3.1. Use of Temporary Files

The MET application and library code uses temporary files in several places. Each specific use of temporary files is described below. The directory in which temporary files are stored is configurable as, described in User's Guide Section 5.2.5.

Whenever a MET application is run, the operating system assigns it a process identification number (PID). All temporary files created by MET include the PID in the file name so that multiple instances can run concurrently without conflict. In addition, when creating a temporary file name, the make_temp_file_name(...) utility function appends _0 to the PID, checks to see if the corresponding file name is already in use, and if so, tries _1, _2 and so on, until an available file name is found.

Note that creating, reading, and deleting temporary files from the local filesystem is much more efficient than performing these operations across a network filesystem. Using the default /tmp directory is recommended, unless prohibited by policies on your system.

In general, MET applications delete any temporary files they create when they are no longer needed. However, if the application exits abnormally, the temporary files may remain.

3.1.1. PB2NC Tool

The PB2NC tool reads input binary files in the BUFR or PrepBUFR format, extracts and/or derives observations from them, filters those observations, and writes the result to a NetCDF output file.

PB2NC creates the following temporary files when running:

  • tmp_pb2nc_blk_{PID}

    PB2NC assumes that each input binary file requires Fortran blocking prior to being read by the BUFRLIB library. It applies Fortran blocking, writes the result to this temporary file, and uses BUFRLIB to read its contents.

  • tmp_pb2nc_blk_{PID}_tbl: PB2NC extracts Bufr table data that is embedded in input files and writes it to this temporary file for later use.

3.1.2. Point2Grid Tool

The Point2Grid tool reads point observations from a variety of inputs and summarizes them on a grid. When processing GOES input files, a temporary NetCDF file is created to store the mapping of input pixel locations to output grid cells unless the MET_GEOSTATIONARY_DATA environment variable defines an existing grid navigation file to be used.

If that temporary geostationary grid mapping file already exists, it is used directly and not recreated. If not, it is created as needed.

Note that this temporary file is not deleted by the Point2Grid tool. Once created, it is intended to be reused in future runs.

3.1.3. Bootstrap Confidence Intervals

Several MET tools support the computation of bootstrap confidence intervals, as described in User's Guide Section 5.2.22 and User's Guide Appendix D, Section 35. When bootstrap confidence intervals are requested, up to two temporary files are created for each CNT, CTS, MCTS, NBRCNT, or NBRCTS line type written to the output.

  • tmp_{LINE_TYPE}_i_{PID}: When the BCA bootstrapping method is requested, jackknife resampling is applied to the input matched pairs. Statistics are computed for each jackknife resample and written to this temporary file.

  • tmp_{LINE_TYPE}_r_{PID}: For each bootstrap replicate computed from the input matched pairs, statistics are computed and written to this temporary file.

Where {LINE_TYPE} is cnt, cts, mcts, nbrcnt, or nbrcts.

Note

Consider whether or not it’s realistic to hold the resampled statistics in memory rather than writing them to temporary files. If so, that would reduce the I/O.

3.1.4. Stat-Analysis Tool

The Stat-Analysis tool reads ASCII output created by the MET statistics tools. A single job can be specified on the command line or one or more jobs can be specified in an optional configuration file. When a configuration file is provided, any filtering options specified are applied to all entries in the jobs array.

Rather than reading all of the input data for each job, Stat-Analysis reads all the input data once, applies any common filtering options, and writes the result to a temporary file.

  • tmp_stat_analysis_{PID}: Stat-Analysis reads all of the input data, applies common filtering logic, and writes the result to this temporary file. All of the specified jobs read data from this temporary file, apply any additional job-specific filtering criteria, and perform the requested operation.

Note

Consider revising the logic to only use a temp file when actually necessary, when multiple jobs are specified along with non-empty common filtering logic.

3.1.5. Python Embedding

As described in User's Guide Appendix F, Section 37, when the MET_PYTHON_EXE environment variable is set, the MET tools run any Python embedding commands using the specified Python executable.

  • tmp_mpr_{PID}: When Python embedding of matched pair data is performed, a Python wrapper is run to execute the user-specified Python script and write the result to this temporary ASCII file.

  • tmp_met_nc_{PID}: When Python embedding of gridded data or point observations is performed, a Python wrapper is run to execute the user-specified Python script and write the result to this temporary NetCDF file.

The compile-time Python instance is run to read data from these temporary files.

3.1.6. TC-Diag Tool

The TC-Diag tool requires the use of Python embedding. It processes one or more ATCF tracks and computes model diagnostics. For each track point, it converts gridded model data to cylindrical coordinates centered at that point, writes it to a temporary NetCDF file, and passes it to Python scripts to compute model diagnostics.

  • tmp_met_nc_{PID}: Cylindrical coordinate model data is written to this temporary NetCDF file for each track point and passed to Python scripts to compute diagnostics. If requested, these temporary NetCDF files for each track point are combined into a single NetCDF cylindrical coordinates output file for each track.