1. Overview of MET

1.1. Purpose and organization of the User’s Guide

The goal of this User’s Guide is to provide basic information for users of the Model Evaluation Tools (MET) to enable them to apply MET to their datasets and evaluation studies. MET was originally designed for application to the post-processed output of the Weather Research and Forecasting (WRF) model. However, MET may also be used for the evaluation of forecasts from other models or applications, including the Unified Forecast System (UFS), and the System for Integrated Modeling of the Atmosphere (SIMA) if certain file format definitions (described in this document) are followed.

The MET User’s Guide is organized as follows. Section 1 provides an overview of MET and its components. Section 2 contains basic information about how to get started with MET - including system requirements, required software (and how to obtain it), how to download MET, and information about compilers, libraries, and how to build the code. Section 3 - Section 9 focuses on the data needed to run MET, including formats for forecasts, observations, and output. These sections also document the reformatting and masking tools available in MET. Section 10 - Section 14 focuses on the main statistics modules contained in MET, including the Point-Stat, Grid-Stat, Ensemble-Stat, Wavelet-Stat and GSI Diagnostic Tools. These sections include an introduction to the statistical verification methodologies utilized by the tools, followed by a section containing practical information, such as how to set up configuration files and the format of the output. Section 15 and Section 16 focus on the analysis modules, Stat-Analysis and Series-Analysis, which aggregate the output statistics from the other tools across multiple cases. Section 18 - Section 20 describes a suite of object-based tools, including MODE, MODE-Analysis, and MODE-TD. Section 21 - Section 27 describes tools focused on tropical cyclones, including MET-TC Overview, TC-Dland, TC-Pairs, TC-Stat, TC-Gen, TC-RMW and RMW-Analysis. Finally, Section 28 includes plotting tools included in the MET release for checking and visualizing data, as well as some additional tools and information for plotting MET results. The appendices provide further useful information, including answers to some typical questions (Appendix A, Section 30) and links and information about map projections, grids, and polylines (Appendix B, Section 31). Appendix C, Section 32 and Appendix D, Section 33 provide more information about the verification measures and confidence intervals that are provided by MET. Sample code that can be used to perform analyses on the output of MET and create particular types of plots of verification results is posted on the MET website). Note that the MET development group also accepts contributed analysis and plotting scripts which may be posted on the MET website for use by the community. It should be noted there are References (Section 29) in this User’s Guide as well.

The remainder of this section includes information about the context for MET development, as well as information on the design principles used in developing MET. In addition, this section includes an overview of the MET package and its specific modules.

1.2. The Developmental Testbed Center (DTC)

MET has been developed, and will be maintained and enhanced, by the Developmental Testbed Center (DTC). The main goal of the DTC is to serve as a bridge between operations and research, to facilitate the activities of these two important components of the numerical weather prediction (NWP) community. The DTC provides an environment that is functionally equivalent to the operational environment in which the research community can test model enhancements; the operational community benefits from DTC testing and evaluation of models before new models are implemented operationally. MET serves both the research and operational communities in this way - offering capabilities for researchers to test their own enhancements to models and providing a capability for the DTC to evaluate the strengths and weaknesses of advances in NWP prior to operational implementation.

The MET package is available to DTC staff, visitors, and collaborators, as well as both the US and International modeling community, for testing and evaluation of new model capabilities, applications in new environments, and so on. It is also the core component of the unified METplus verification framework. METplus details can be found on the METplus webpage.

1.3. MET goals and design philosophy

The primary goal of MET development is to provide a state-of-the-art verification package to the NWP community. By “state-of-the-art” we mean that MET will incorporate newly developed and advanced verification methodologies, including new methods for diagnostic and spatial verification and new techniques provided by the verification and modeling communities. MET also utilizes and replicates the capabilities of existing systems for verification of NWP forecasts. For example, the MET package replicates existing National Center for Environmental Prediction (NCEP) operational verification capabilities (e.g., I/O, methods, statistics, data types). MET development will take into account the needs of the NWP community - including operational centers and the research and development community. Some of the MET capabilities include traditional verification approaches for standard surface and upper air variables (e.g., Equitable Threat Score, Mean Squared Error), confidence intervals for verification measures, and spatial forecast verification methods. In the future, MET will include additional state-of-the-art and new methodologies.

The MET package has been designed to be modular and adaptable. For example, individual modules can be applied without running the entire set of tools. New tools can easily be added to the MET package due to this modular design. In addition, the tools can readily be incorporated into a larger “system” that may include a database as well as more sophisticated input/output and user interfaces. Currently, the MET package is a set of tools that can easily be applied by any user on their own computer platform. A suite of Python scripts for low-level automation of verification workflows and plotting has been developed to assist users with setting up their MET-based verification. It is called METplus and may be obtained on the METplus GitHub repository.

The MET code and documentation is maintained by the DTC in Boulder, Colorado. The MET package is freely available to the modeling, verification, and operational communities, including universities, governments, the private sector, and operational modeling and prediction centers.

1.4. MET components

The major components of the MET package are represented in Figure 1.1. The main stages represented are input, reformatting, plotting, intermediate output, statistical analyses, and output and aggregation/analysis. The MET-TC package functions independently of the other MET modules, as indicated in the Figure. Each of these stages is described further in later sections. For example, the input and output formats are discussed in Section 3 as well as in the sections associated with each of the statistics modules. MET input files are represented on the far left.

The reformatting stage of MET consists of several tools which perform a variety of functions. The ASCII2NC, PB2NC, MADIS2NC, LIDAR2NC, and IODA2NC tools read a variety of point observation input file formats and, optionally, derive time summaries for each observing location. They all write to a common NetCDF point observation file format which can be read by the other MET tools. The Point2Grid tool reads that common NetCDF point observation file format and interpolates the point data onto a user-specified grid. The Regrid-Data-Plane, Shift-Data-Plane, MODIS-Regrid, and WWMCA-Regrid tools read a variety of gridded input file formats and interpolate user-requested input fields to a user-defined output grid. While the MET statistics tools can interpolate many input file formats in-memory and on-the-fly, manually regridding upstream is sometimes useful. The Pcp-Combine tool adds, subtracts, or derives fields across multiple time steps. It is often run to accumulate precipitation amounts into a user-specified time interval - if a user would like to verify over a different time interval than is included in their forecast or observational dataset. The Gen-Vx-Mask tool provides a variety of methods for creating bitmapped masking areas. Those masks can then be used to efficiently limit verification to the interior of a user-specified region in the downstream statistics tools. The Gen-Ens-Prod tool derives basic ensemble products (mean, spread, probabilities) from multiple gridded input ensemble members. The GSI tools reformat binary GSI diagnostic data to be read by the Stat-Analysis tool.


Figure 1.1 Basic representation of current MET structure and modules. Gray areas represent input and output files. Dark green areas represent reformatting and pre-processing tools. Light green areas represent plotting utilities. Blue areas represent statistical tools. Yellow areas represent aggregation and analysis tools.

Several optional plotting utilities are provided to assist users in checking their output from the data preprocessing step. Plot-Point-Obs creates a postscript plot showing the locations of point observations. This can be quite useful for assessing whether the latitude and longitude of observation stations was specified correctly. Plot-Data-Plane produces a similar plot for gridded data. For users of the MODE object based verification methods, the Plot-MODE-Field utility will create graphics of the MODE object output. Finally, WWMCA-Plot produces a plot of the raw WWMCA data file.

The main statistical analysis components of the current version of MET are: Point-Stat, Grid-Stat, Series-Analysis, Ensemble-Stat, MODE, MODE-TD (MTD), Grid-Diag, and Wavelet-Stat. The Point-Stat tool is used for grid-to-point verification, or verification of a gridded forecast field against a point-based observation (i.e., surface observing stations, ACARS, rawinsondes, and other observation types that could be described as a point observation). In addition to providing traditional forecast verification scores for both continuous and categorical variables, confidence intervals are also produced using parametric and non-parametric methods. Confidence intervals take into account the uncertainty associated with verification statistics due to sampling variability and limitations in sample size. These intervals provide more meaningful information about forecast performance. For example, confidence intervals allow credible comparisons of performance between two models when a limited number of model runs is available.

Sometimes it may be useful to verify a forecast against gridded fields (e.g., Stage IV precipitation analyses). The Grid-Stat tool produces traditional verification statistics when a gridded field is used as the observational dataset. Like the Point-Stat tool, the Grid-Stat tool also produces confidence intervals. The Grid-Stat tool also includes “neighborhood” spatial methods, such as the Fractional Skill Score (Roberts and Lean, 2008). These methods are discussed in Ebert (2008). The Grid-Stat tool accumulates statistics over the entire domain.

Users wishing to accumulate statistics over a time, height, or other series separately for each grid location should use the Series-Analysis tool. Series-Analysis can read any gridded matched pair data produced by the other MET tools and accumulate them, keeping each spatial location separate. Maps of these statistics can be useful for diagnosing spatial differences in forecast quality.

Ensemble-Stat is a hybrid tool that provides based post-processing capability of the ensemble members as well as computing measures of ensemble characteristics. Basic post-processing capability includes computing the ensemble mean, min, max, standard deviation, and ensemble relative frequency or probability. These fields can then be used in other MET tools for additional evaluation. Note however that the Gen-Ens-Prod tool also performs ensemble product generation, and this functionality will be removed from Ensemble-Stat in future versions. The ensemble characteristics include computation of rank and probability integral transform (PIT) histograms, the end-points for receiver operator curve (ROC) and reliability diagrams, and ranked probabilities scores (RPS) and the continuous version (CRPS).

The MODE (Method for Object-based Diagnostic Evaluation) tool also uses gridded fields as observational datasets. However, unlike the Grid-Stat tool, which applies traditional forecast verification techniques, MODE applies the object-based spatial verification technique described in Davis et al. (2006a,b) and Brown et al. (2007). This technique was developed in response to the “double penalty” problem in forecast verification. A forecast missed by even a small distance is effectively penalized twice by standard categorical verification scores: once for missing the event and a second time for producing a false alarm of the event elsewhere. As an alternative, MODE defines objects in both the forecast and observation fields. The objects in the forecast and observation fields are then matched and compared to one another. Applying this technique also provides diagnostic verification information that is difficult or even impossible to obtain using traditional verification measures. For example, the MODE tool can provide information about errors in location, size, and intensity.

The MODE-TD tool extends object-based analysis from two-dimensional forecasts and observations to include the time dimension. In addition to the two dimensional information provided by MODE, MODE-TD can be used to examine even more features including displacement in time, and duration and speed of moving areas of interest.

The Grid-Diag tool produces multivariate probability density functions (PDFs) that may be used either for exploring the relationship between two fields, or for the computation of percentiles generated from the sample for use with percentile thresholding. The output from this tool requires post-processing by METplus or user-provided utilities.

The Wavelet-Stat tool decomposes two-dimensional forecasts and observations according to the Intensity-Scale verification technique described by Casati et al. (2004). There are many types of spatial verification approaches and the Intensity-Scale technique belongs to the scale-decomposition (or scale-separation) verification approaches. The spatial scale components are obtained by applying a wavelet transformation to the forecast and observation fields. The resulting scale-decomposition measures error, bias and skill of the forecast on each spatial scale. Information is provided on the scale dependency of the error and skill, on the no-skill to skill transition scale, and on the ability of the forecast to reproduce the observed scale structure. The Wavelet-Stat tool is primarily used for precipitation fields. However, the tool can be applied to other variables, such as cloud fraction.

Results from the statistical analysis stage are output in ASCII, NetCDF and Postscript formats. The Point-Stat, Grid-Stat, and Wavelet-Stat tools create STAT (statistics) files which are tabular ASCII files ending with a “.stat” suffix. In earlier versions of MET, this output format was called VSDB (Verification System DataBase). VSDB, which was developed by the NCEP, is a specialized ASCII format that can be easily read and used by graphics and analysis software. The STAT output format of the Point-Stat, Grid-Stat, and Wavelet-Stat tools is an extension of the VSDB format developed by NCEP. Additional columns of data and output line types have been added to store statistics not produced by the NCEP version.

The Stat-Analysis and MODE-Analysis tools aggregate the output statistics from the previous steps across multiple cases. The Stat-Analysis tool reads the STAT output of Point-Stat, Grid-Stat, Ensemble-Stat, and Wavelet-Stat and can be used to filter the STAT data and produce aggregated continuous and categorical statistics. The MODE-Analysis tool reads the ASCII output of the MODE tool and can be used to produce summary information about object location, size, and intensity (as well as other object characteristics) across one or more cases.

Tropical cyclone forecasts and observations are quite different than numerical model forecasts, and thus they have their own set of tools. The MET-TC package includes several modules: TC-Dland, TC-Pairs, TC-Stat, TC-Gen, TC-RMW, and RMW-Analysis. The TC-Dland module calculates the distance to land from all locations on a specified grid. This information can be used in later modules to eliminate tropical cyclones that are over land from being included in the statistics. TC-Pairs matches up tropical cyclone forecasts and observations and writes all output to a file. In TC-Stat, these forecast / observation pairs are analyzed according to user preference to produce statistics. TC-Gen evaluates the performance of Tropical Cyclone genesis forecast using contingency table counts and statistics. TC-RMW performs a coordinate transformation for gridded model or analysis fields centered on the current storm location. RMW-Analysis filters and aggregates the output of TC-RMW across multiple cases.

The following sections of this MET User’s Guide contain usage statements for each tool, which may be viewed if you type the name of the tool. Alternatively, the user can also type the name of the tool followed by -help to obtain the usage statement. Each tool also has a -version command line option associated with it so that the user can determine what version of the tool they are using.

1.5. MET Release Notes

When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature (MET GitHub issues). Important issues are listed in bold for emphasis.

1.5.1. MET Version 10.1.2 release notes (20220516)

  • Bugfixes:

    • Update static arrays with hard-coded size (METplus-Internal #14).

    • Fix TC-Gen to only count misses from requested initialization hours and lead times (#2148).

1.5.2. MET Version 10.1.1 release notes (20220419)

  • Bugfixes:

    • Fix support for reading rotated lat/lon grids from CF-compliant NetCDF files (#2115).

    • Fix support for reading rotated lat/lon grids from GRIB1 files (grid type 10) (#2118).

    • Fix support for int64 NetCDF variable types (#2123).

1.5.3. MET Version 10.1.0 release notes (20220314)

  • Repository and build:

    • Installation:

      • Enhance the MET compilation script and its documentation (#1395).

    • Static Code Analysis:

      • Automate calls to the SonarQube static code analysis tool in the nightly build (#2020).

      • Fix Fortify High finding for src/libcode/vx_data2d_nccf/nccf_file.cc (#1795).

      • Fix the findings from SonarQube (#1855).

      • Reduce the Security hotspots from SonarQube (#1903).

      • Address findings from the Cppcheck code analysis tool (#1996).

    • Testing:

      • Review and revise the warning messages when running the MET unit tests (#1921).

      • Investigate nightly build output wind direction differences caused by machine precision (#2027).

      • Modify plot_tcmpr.R script to support plotting of extra-tropical cyclone tracks not verified against BEST tracks (#1801).

      • Fix failure in plot_tcmpr.R script when a directory is passed in with -lookin (#1872).

    • Continuous Integration:

      • Implement Continuous Integration with GitHub Actions in MET (#1546).

      • Treat warnings from the documentation as errors to facilitate continuous integration with GHA (#1819).

  • Documentation:

    • Create and publish a PDF of the MET User’s Guide via Read-The-Docs (#1453).

    • Enhance the MET documentation to follow the standard for sections (#1998).

    • Add anchors to link directly to configuration items in the MET User’s Guide (#1811).

    • Update FAQ in User’s Guide with info from webpage FAQ (#1834).

    • Document the statistics from the RPS line type in Appendix C (#1853).

    • Enhance the documentation with meta-data that is expected by MET for netCDF (#1949).

    • Update documentation to reference GitHub Discussions instead of MET Help (#1833).

    • Fix broken URLs in default MET config files (#1864).

  • Library code:

    • Bugfixes:

      • Add check for the start offset and data count are valid before calling NetCDF API (#1852).

      • Fix the MET library code to correclty parse timing information from Grid-Stat NetCDF matched pairs output files (#2040).

      • Fix bug with the incrementing of numbers in temporary file names (#1906).

    • Python embedding enhancements:

      • Enhance Ensemble-Stat, Point-Stat, Plot-Point-Obs, and Point2Grid to support python embedding of point observations (#1844).

      • Fix python embedding when using a named grid with MET_PYTHON_EXE set (#1798).

    • Miscellaneous:

      • Enhance MET to use point observations falling between the first and last columns of a global grid (#1823).

      • Support percentile thresholds for frequency bias not equal to 1 (e.g. ==FBIAS0.9) (#1761).

      • Reimplement the NumArray class based on an STL template (#1899).

      • Modify the interpretation of the message_type_group_map values to support the use of regular expressions (#1974).

      • Sort files read from directories to provide consistent behavior across platforms (#1989).

      • Print warning message for fields that contain no valid data (#1912).

      • Update error messages to redirect users from the MET-Help desk to METplus Discussions (#2054).

      • Update the copyright year of the source code to 2022 (#2013).

    • NetCDF library:

      • Implement a common API for reading and writing the common NetCDF point observation file format (#1402 and #1581).

      • Enhance the MET library code to read Rotated Lat/Lon data from CF-compliant NetCDF files (#1055).

    • Statistics computations:

      • Add Scatter Index to the CNT line type (#1843).

      • Add the HSS_EC statistic to the MCTS line type and a configurable option for its computation (#1749).

  • Application code:

    • ASCII2NC Tool:

      • Fix ASCII2NC to check the return status when reading ASCII input files (#1957).

    • Ensemble-Stat Tool:

      • Enhance Ensemble-Stat to compute probabilistic statistics for user-defined or climatology-based thresholds (#1259).

      • Enhance Ensemble-Stat to apply the HiRA method to ensembles (#1583 and #2045).

      • Enhance Ensemble-Stat and Gen-Ens-Prod to read all ensemble members from a single input file (#1695).

      • Add logic to Ensemble-Stat to handle an ensemble control member (#1905).

      • Enhance Ensemble-Stat and Gen-Ens-Prod to error out if the control member also appears in the list of ensemble members (#1968).

      • Add Point-Stat and Ensemble-Stat obs_quality_exc configuration option to specify which quality flags should be excluded (#1858).

      • Print a warning message about switching from Ensemble-Stat to Gen-Ens-Prod (#1907).

      • Fix failure of Ensemble-Stat when verifying against gridded ECMWF GRIB1 files (#1879).

    • Gen-Ens-Prod Tool (NEW):

      • Create the new Gen-Ens-Prod tool for ensemble product generation (#1904).

      • Enhance Ensemble-Stat and Gen-Ens-Prod to read all ensemble members from a single input file (#1695).

      • Enhance Gen-Ens-Prod to standardize ensemble members relative to climatology (#1918).

    • Gen-Vx-Mask Tool:

      • Refine logic to prevent rounding shapefile points to the nearest grid point (affects GenVxMask -type shape masks) (#1810).

      • Change -type for Gen-Vx-Mask from an optional argument to a required one (#1792).

      • Fix Gen-Vx-Mask to handle named grids and grid specification strings for -type grid (#1993).

      • Fix Gen-Vx-Mask so that the -input_field and -mask_field options are processed independently (#1891).

    • Grid-Diag Tool:

      • Fix integer overflow in Grid-Diag (#1886).

    • Grid-Stat Tool:

      • Enhance Grid-Stat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS (#1926).

      • Add G and G-Beta to the DMAP line type from Grid-Stat (#1673).

      • Fix Point-Stat and Grid-Stat to write VCNT output even if no VL1L2 or VAL1L2 output is requested (#1991).

    • IODA2NC Tool:

      • Fix IODA2NC to handle the same input file being provided multiple times (#1965).

      • Fix IODA2NC bug rejecting all input observations in unit tests (#1922).

    • MADIS2NC Tool:

      • Enhance MADIS2NC to handle the 2016 updates to its format (#1936).

      • Fix MADIS2NC to correctly parse MADIS profiler quality flag values (#2028).

    • MODE Tool:

      • Add support for Multi-Variate MODE (#1184).

    • MTD Tool:

      • Fix MTD to compute the CDIST_TRAVELLED value correctly (#1976).

    • PB2NC Tool:

      • Enhance PB2NC to derive Mixed-Layer CAPE (MLCAPE) (#1824).

      • Enhance the PBL derivation logic in PB2NC (#1913).

      • Update the PB2NC configuration to correct the obs_prefbufr_map name as obs_prepbufr_map (#2044).

      • Add entries to the default obs_prepbufr_map setting (#2070).

      • Fix PB2NC to better inventory BUFR input data when processing all variables (#1894).

      • Fix PB2NC to reduce redundant verbosity level 3 log messages (#2015).

      • Resolve PB2NC string truncation warning messages (#1909).

    • Point2Grid Tool:

      • Enhance Point2Grid to support double type latitude/longitude variables (#1838).

      • Fix the output of Point2Grid which is flipped and rotated with lat/lon to lat/lon conversion (#1817).

    • Point-Stat Tool:

      • Add ORANK line type to the HiRA output from Point-Stat (#1764).

      • Add Point-Stat and Ensemble-Stat obs_quality_exc configuration option to specify which quality flags should be excluded (#1858).

      • Fix Point-Stat and Grid-Stat to write VCNT output even if no VL1L2 or VAL1L2 output is requested (#1991).

    • Series-Analysis Tool:

      • Enhance Series-Analysis to compute the BRIERCL statistic from the PSTD line type (#2003).

    • Stat-Analysis Tool:

      • Enhance Stat-Analysis to compute the CBS Index (#1031).

      • Enhance Stat-Analysis to write the GO Index and CBS Index into a new SSIDX STAT line type (#1788).

      • Modify the STAT-Analysis GO Index configuration file (#1945).

      • Fix Stat-Analysis skill score index job which always writes a dump row output file (#1914).

      • Fix consumption of too much memory by Stat-Analysis (#1875).

    • TC-Gen Tool:

      • Enhance TC-Gen to verify genesis probabilities from ATCF e-deck files (#1809).

      • Enhance TC-Gen to verify NHC tropical weather outlook shapefiles (#1810).

    • TC-Pairs Tool:

      • Enhance TC-Pairs to only write output for a configurable list of valid times (#1870).

    • TC-Stat Tool:

      • Fix TC-Stat event equalization logic to include any model name requested using -amodel (#1932).

    • Wavelet-Stat Tool:

      • Make the specification of a binary threshold in Wavelet-Stat optional (#1746).

1.5.4. MET Version 10.0.0 release notes (20210510)

  • Repository and build:

    • Migrate GitHub respository from the NCAR to DTCenter organization (#1462).

    • Switch to consistent vX.Y.Z version numbering, from v10.0 to v10.0.0 (#1590).

    • Switch from tagging releases as met-X.Y.Z to vX.Y.Z instead (#1541).

    • Add a GitHub pull request template (#1516).

    • Resolve warnings from autoconf (#1498).

    • Restructure nightly builds (#1510).

    • Update the MET unit test logic by unsetting environment variables after each test to provide a clean environment for the next (#1624).

    • Run the nightly build as the shared met_test user (#1116).

    • Correct the time offset for tests in unit_plot_data_plane.xml (#1677).

    • Enhance the sample plotting R-script to read output from different versions of MET (#1653).

    • Update the default configuration options to compile the development code with the debug (-g) option and the production code without it (#1778).

    • Update MET to compile using GCC version 10 (#1552).

    • Update MET to compile using PGI version 20 (#1317).

  • Documentation:

    • Migrate the MET documentation to Read the Docs (#1649).

    • Enhance and update documentation (#1459 and #1460, and #1731).

    • Enhance the python embedding documentation (#1468).

    • Document the supported grid definition templates (#1469).

    • Update comments at the top of each MET config file directing users to the MET User’s Guide (#1598).

    • Migrate content from README and README_TC in data/config to the MET User’s Guide (#1474).

    • Add version selector to the Sphinx documentation page (#1461).

    • Make bolding consistent across the documentation (#1458).

    • Implement hanging indents for references (#1457).

    • Correct typos and spelling errors (#1456).

    • Update the Grid-Diag documentation to clarify the -data command line option (#1611).

    • Documentation updates to correct typos and apply consistent formatting (#1455).

    • Correct the definition of H_RATE and PODY in MET User’s Guide Appendix C (#1631).

  • Library code:

    • Bugfixes:

      • Apply the GRIB ensemble filtering option (GRIB_ens) whenever specified by the user (#1604).

      • Fix the set_attr_accum option to set the accumulation time instead of the lead time (#1646).

      • Fix ASCII file list parsing logic (#1484 and #1508).

      • Fix parsing error for floating point percentile thresholds, like “>SFP33.3” (#1716).

    • Python embedding enhancements:

      • Note that the netCDF4 Python package is now required in place of the pickle package!

      • Replace the pickle format for temporary python files with NetCDF for gridded data (#1319, #1697).

      • Replace the pickle format for temporary python files with ASCII for point observations in ascii2nc and matched pair data in Stat-Analysis (#1319, #1700).

      • Complete support for Python XArray embedding (#1534).

      • Treat gridded fields of entirely missing data as missing files and fix python embedding to call common data processing code (#1494).

      • Clarify error messages for Xarray python embedding (#1472).

      • Add support for Gaussian grids with python embedding (#1477).

      • Correct error messages from python embedding (#1473).

      • Enhance to support the “grid” being defined as a named grid or specification string (#1471).

      • Enhance to parse python longlong variables as integers to make the python embedding scripts less particular (#1747).

      • Fix the read_ascii_mpr.py python embedding script to pass all 37 columns of MPR data to Stat-Analysis (#1620).

      • Fix the read_tmp_dataplane.py python embedding script to handle the fill value correctly (#1753).

    • Miscellaneous:

      • Enhance support for rotated latlon grids and update related documentation (#1574).

      • Parse the -v and -log options prior to application-specific command line options (#1527).

      • Update GRIB1/2 table entries for the MXUPHL, MAXREF, MAXUVV, and MAXDVV variables (#1658).

      • Update the Air Force GRIB tables to reflect current AF usage (#1519).

      • Enhance the DataLine::get_item() error message to include the file name, line number, and column (#1429). * Add support for climatological probabilities for complex CDP thresholds, like >=CDP33&&<=CDP67 (#1705).

      • Update the NCL-derived color tables (#1568).

    • NetCDF library:

      • Enhance to support additional NetCDF data types (#1492 and #1493).

      • Add support for the NetCDF-CF conventions time bounds option (#1657).

      • Extend CF-compliant NetCDF file support when defining the time dimension as a time string (#1755).

      • Error out when reading CF-compliant NetCDF data with incomplete grid definition (#1454).

      • Reformat and simplify the magic_str() printed for NetCDF data files (#1655).

      • Parse the “init_time” and “valid_time” attributes from MET NetCDF input files (#1346).

    • Statistics computations:

      • Modify the climatological Brier Score computation to match the NOAA/EMC VSDB method (#1684).

      • Add support for the Hersbach CRPS algorithm by add new columns to the ECNT line type (#1450).

      • Enhance MET to derive the Hersbach CRPSCL_EMP and CRPSS_EMP statistics from a single deterministic reference model (#1685).

      • Correct the climatological CRPS computation to match the NOAA/EMC VSDB method (#1451).

      • Refine log messages when verifying probabilities (#1502).

  • Application code:

    • ASCII2NC Tool:

      • Fix to handle bad records in little_r format (#1737).

      • Create empty output files for zero input observations instead of erroring out (#1630).

    • GIS Tools:

      • Fix memory corruption bug in the gis_dump_dbf utility which causes it to abort at runtime (#1777).

    • Grid-Diag Tool:

      • Fix bug when reading the same variable name from multiple data sources (#1694).

    • Grid-Stat Tool:

      • Add mpr_column and mpr_thresh configuration options to filter out matched pairs based on large fcst, obs, and climo differences (#1575).

      • Correct the climatological CDF values in the NetCDF matched pairs output files and correct the climatological probability values for climatgological distribution percentile (CDP) threshold types (#1638).

    • IODA2NC Tool (NEW):

      • Add the new ioda2nc tool (#1355).

    • MADIS2NC Tool:

      • Clarify various error messages (#1409).

    • MODE Tool:

      • Update the MODE AREA_RATIO output column to list the forecast area divided by the observation area (#1643).

      • Incremental development toward the Multivariate MODE tool (#1282, #1284, and #1290).

    • PB2NC Tool:

      • Fix intermittent segfault when deriving PBL (#1715).

    • Plot-Point-Obs Tool:

      • Overhaul Plot-Point-Obs to make it highly configurable (#213, #1528, and #1052).

      • Support regridding option in the config file (#1627).

    • Point2Grid Tool:

      • Support additional NetCDF point observation data sources (#1345, #1509, and #1511).

      • Support the 2-dimensional time variable in Himawari data files (#1580).

      • Create empty output files for zero input observations instead of erroring out (#1630).

      • Improve the Point2Grid runtime performance (#1421).

      • Process point observations by variable name instead of GRIB code (#1408).

    • Point-Stat Tool:

      • Add mpr_column and mpr_thresh configuration options to filter out matched pairs based on large fcst, obs, and climo differences (#1575).

      • Print the rejection code reason count log messages at verbosity level 2 for zero matched pairs (#1644).

      • Add detailed log messages when discarding observations (#1588).

      • Update log messages (#1514).

      • Enhance the validation of masking regions to check for non-unique masking region names (#1439).

      • Fix Point-Stat runtime error for some CF-complaint NetCDF files (#1782).

    • Stat-Analysis Tool:

      • Process multiple output thresholds and write multiple output line types in a single aggregate_stat job (#1735).

      • Skip writing job output to the logfile when the -out_stat option is provided (#1736).

      • Add -fcst_init_inc/_exc and -fcst_valid_inc/_exc job command filtering options to Stat-Analysis (#1135).

      • Add -column_exc job command option to exclude lines based on string values (#1733).

      • Fix Stat-Analysis failure when aggregating ECNT lines (#1706).

    • TC-Gen Tool:

      • Overhaul the genesis matching logic, add the development and operational scoring algorithms, and add many config file options (#1448).

      • Add config file options to filter data by initialization time (init_inc and init_exc) and hurricane basin (basin_mask) (#1626).

      • Add the genesis matched pair (GENMPR) output line type (#1597).

      • Add a gridded NetCDF output file with counts for genesis events and track points (#1430).

      • Enhance the matching logic and update several config options to support its S2S application (#1714).

      • Fix lead window filtering option (#1465).

    • TC-Pairs Tool:

      • Fix to report the correct number of lines read from input track data files (#1725).

      • Fix to read supported RI edeck input lines and ignore unsupported edeck probability line types (#1768).

    • TC-Stat Tool:

      • Add -column_exc job command option to exclude lines based on string values (#1733).

1.6. Future development plans

MET is an evolving verification software package. New capabilities are planned in controlled, successive version releases. Bug fixes and user-identified problems will be addressed as they are found and posted to the known issues section of the MET User Support web page. Plans are also in place to incorporate many new capabilities and options in future releases of MET. Please refer to the issues listed in the MET GitHub repository to see our development priorities for upcoming releases.

1.7. Code support

MET support is provided through the METplus GitHub Discussions Forum. We will endeavor to respond to requests for help in a timely fashion. In addition, information about MET and tools that can be used with MET are provided on the MET web page.

We welcome comments and suggestions for improvements to MET, especially information regarding errors. Comments may be submitted using the MET Feedback form available on the MET website. In addition, comments on this document would be greatly appreciated. While we cannot promise to incorporate all suggested changes, we will certainly take all suggestions into consideration.

-help and -version command line options are available for all of the MET tools. Typing the name of the tool with no command line options also produces the usage statement.

The MET package is a “living” set of tools. Our goal is to continually enhance it and add to its capabilities. Because our time, resources, and talents are limited, we welcome contributed code for future versions of MET. These contributions may represent new verification methodologies, new analysis tools, or new plotting functions. For more information on contributing code to MET, please create a post in the METplus GitHub Discussions Forum.

1.8. Fortify and SonarQube

Requirements from various government agencies that use MET have resulted in our code being analyzed by both the Fortify and SonarQube static source code analysis tools. Fortify and SonarQube analyze source code to identify for security risks, memory leaks, uninitialized variables, and other such weaknesses and bad coding practices. They categorize issue as low priority, high priority, or critical, and report these issues back to the developers for them to address. The goal is to drive the counts of both high priority and critical issues down to zero.

The MET developers are pleased to report that Fortify reports zero critical issues in the MET code. Users of the MET tools who work in high security environments can rest assured about the possibility of security risks when using MET, since the quality of the code has now been vetted by unbiased third-party experts. The MET developers continue using Fortify routinely to ensure that the critical counts remain at zero and to further reduce the counts for lower priority issues.