30. References
Aberson, S.D., 1998: Five-day Tropical cyclone track forecasts in the North
Atlantic Basin. Weather and Forecasting, 13, 1005-1015.
Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009: Application of
spatial verification methods to idealized and NWP-gridded precipitation forecasts.
Weather and Forecasting, 24 (6), 1485 - 1497, doi: 10.1175/2009WAF2222298.1.
Barker, T. W., 1991: The relationship between spread and forecast error in
extended-range forecasts. Journal of Climate, 4, 733-742.
Bradley, A.A., S.S. Schwartz, and T. Hashino, 2008: Sampling Uncertainty
and Confidence Intervals for the Brier Score and Brier Skill Score.
Weather and Forecasting, 23, 992-1006.
Brill, K. F., and F. Mesinger, 2009: Applying a general analytic method
for assessing bias sensitivity to bias-adjusted threat and equitable
threat scores. Weather and Forecasting, 24, 1748-1754.
Brown, B.G., R. Bullock, J. Halley Gotway, D. Ahijevych, C. Davis,
E. Gilleland, and L. Holland, 2007: Application of the MODE object-based
verification tool for the evaluation of model precipitation fields.
AMS 22nd Conference on Weather Analysis and Forecasting and 18th
Conference on Numerical Weather Prediction, 25-29 June, Park City, Utah,
American Meteorological Society (Boston), Available at
Buizza, R., 1997: Potential forecast skill of ensemble prediction and spread
and skill distributions of the ECMWF ensemble prediction system. Monthly
Weather Review,125, 99-119.
Bullock, R., T. Fowler, and B. Brown, 2016: Method for Object-Based
Diagnostic Evaluation. NCAR Technical Note NCAR/TN-532+STR, 66 pp.
Candille, G., and O. Talagrand, 2008: Impact of observational error on the
validation of ensemble prediction systems. Quarterly Journal of the Royal
Meteorological Society 134: 959-971.
Casati, B., G. Ross, and D. Stephenson, 2004: A new intensity-scale approach
for the verification of spatial precipitation forecasts. Meteorological
Applications 11, 141-154.
Davis, C.A., B.G. Brown, and R.G. Bullock, 2006a: Object-based verification
of precipitation forecasts, Part I: Methodology and application to
mesoscale rain areas. Monthly Weather Review, 134, 1772-1784.
Davis, C.A., B.G. Brown, and R.G. Bullock, 2006b: Object-based verification
of precipitation forecasts, Part II: Application to convective rain systems.
Monthly Weather Review, 134, 1785-1795.
Dawid, A.P., 1984: Statistical theory: The prequential approach. Journal of
the Royal Statistical Society A147, 278-292.
Ebert, E.E., 2008: Fuzzy verification of high-resolution gridded forecasts:
a review and proposed framework. Meteorological Applications, 15, 51-64.
Eckel, F. A., M.S. Allen, M. C. Sittel, 2012: Estimation of Ambiguity in
Efron, B. 2007: Correlation and large-scale significance testing. Journal
of the American Statistical Association,* 102(477), 93-103.
Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories.
J. Appl. Meteor., 8, 985-987, 10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2.
Gilleland, E., 2010: Confidence intervals for forecast verification.
NCAR Technical Note NCAR/TN-479+STR, 71pp.
Gilleland, E., 2017: A new characterization in the spatial verification
framework for false alarms, misses, and overall patterns.
Weather and Forecasting, 32 (1), 187 - 198, doi: 10.1175/WAF-D-16-0134.1.
Gilleland, E., 2020: Bootstrap methods for statistical inference.
Part I: Comparative forecast verification for continuous variables.
Journal of Atmospheric and Oceanic Technology, 37 (11), 2117 - 2134,
doi: 10.1175/JTECH-D-20-0069.1.
Gilleland, E., 2020: Bootstrap methods for statistical inference.
Part II: Extreme-value analysis. Journal of Atmospheric and Oceanic
Technology, 37 (11), 2135 - 2144, doi: 10.1175/JTECH-D-20-0070.1.
Gilleland, E., 2021: Novel measures for summarizing high-resolution forecast
performance. Advances in Statistical Climatology, Meteorology and Oceanography,
7 (1), 13 - 34, doi: 10.5194/ascmo-7-13-2021.
Gneiting, T., A. Westveld, A. Raferty, and T. Goldman, 2004: Calibrated
Probabilistic Forecasting Using Ensemble Model Output Statistics and
Minimum CRPS Estimation. Technical Report no. 449, Department of
Statistics, University of Washington. Available at
Haiden, T., M.J. Rodwell, D.S. Richardson, A. Okagaki, T. Robinson, T. Hewson, 2012:
Intercomparison of Global Model Precipitation Forecast Skill in 2010/11
Using the SEEPS Score. Monthly Weather Review, 140, 2720-2733.
Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble
forecasts. Monthly Weather Review, 129, 550-560.
Hersbach, H., 2000: Decomposition of the Continuous Ranked Probability Score
for Ensemble Prediction Systems. Weather and Forecasting, 15, 559-570.
Jolliffe, I.T., and D.B. Stephenson, 2012: Forecast verification. A
practitioner’s guide in atmospheric science. Wiley and Sons Ltd, 240 pp.
Knaff, J.A., M. DeMaria, C.R. Sampson, and J.M. Gross, 2003: Statistical,
Five-Day Tropical Cyclone Intensity Forecasts Derived from Climatology
and Persistence. Weather and Forecasting, Vol. 18 Issue 2, p. 80-92.
Mason, S. J., 2004: On Using “Climatology” as a Reference Strategy
in the Brier and Ranked Probability Skill Scores. Monthly Weather Review,
132, 1891-1895.
Mason, S. J., 2008: Understanding forecast verification statistics.
Meteor. Appl., 15, 31-40, doi: 10.1002/met.51.
Mittermaier, M., 2014: A strategy for verifying near-convection-resolving
model forecasts at observing sites. Weather Forecasting, 29, 185-204.
Mood, A. M., F. A. Graybill and D. C. Boes, 1974: Introduction to the
Theory of Statistics, McGraw-Hill, 299-338.
Murphy, A.H., 1969: On the ranked probability score. Journal of Applied
Meteorology and Climatology, 8 (6), 988 - 989,
doi: 10.1175/1520-0450(1969)008<0988:OTPS>2.0.CO;2.
Murphy, A.H., and R.L. Winkler, 1987: A general framework for forecast
verification. Monthly Weather Review, 115, 1330-1338.
North, R.C., M.P. Mittermaier, S.F. Milton, 2022. Using SEEPS with a
TRMM-derived Climatology to Assess Global NWP Precipitation Forecast Skill.
Monthly Weather Review, 150, 135-155.
Ou, M. H., Charles, M., & Collins, D. C. 2016: Sensitivity of calibrated week-2
probabilistic forecast skill to reforecast sampling of the NCEP global
ensemble forecast system. Weather and Forecasting, 31(4), 1093-1107.
Roberts, N.M., and H.W. Lean, 2008: Scale-selective verification of rainfall
accumulations from high-resolution forecasts of convective events.
Monthly Weather Review, 136, 78-97.
Rodwell, M.J., D.S. Richardson, T.D. Hewson and T. Haiden, 2010: A new equitable
score suitable for verifying precipitation in numerical weather prediction.
Quarterly Journal of the Royal Meteorological Society, 136: 1344-1463.
Rodwell, M.J., T. Haiden, D.S. Richardson, 2011: Developments in Precipitation
Saetra O., H. Hersbach, J-R Bidlot, D. Richardson, 2004: Effects of
observation errors on the statistics for ensemble spread and
reliability. Monthly Weather Review 132: 1487-1501.
Santos C. and A. Ghelli, 2012: Observational probability method to assess
ensemble precipitation forecasts. Quarterly Journal of the Royal
Meteorological Society 138: 209-221.
Schwartz C. and Sobash R., 2017: Generating Probabilistic Forecasts from
Convection-Allowing Ensembles Using Neighborhood Approaches: A Review
and Recommendations. Monthly Weather Review, 145, 3397-3418.
Stephenson, D.B., 2000: Use of the “Odds Ratio” for diagnosing
forecast skill. Weather and Forecasting, 15, 221-232.
Stephenson, D.B., B. Casati, C.A.T. Ferro, and C.A. Wilson, 2008: The extreme
dependency score: A non-vanishing measure for forecasts of rare events.
Meteorological Applications 15, 41-50.
Tödter, J. and B. Ahrens, 2012: Generalization of the Ignorance Score:
Continuous ranked version and its decomposition. Monthly Weather Review,
140 (6), 2005 - 2017, doi: 10.1175/MWR-D-11-00266.1.
Weniger, M., F. Kapp, and P. Friederichs, 2016: Spatial Verification Using
Wavelet Transforms: A Review. Quarterly Journal of the Royal
Meteorological Society, 143, 120-136.
Wilks, D.S. 2010: Sampling distributions of the Brier score and Brier skill
score under serial dependence. Quarterly Journal of the Royal
Meteorological Society, 136, 2109-2118. doi:10.1002/qj.709
Wilks, D., 2011: Statistical methods in the atmospheric sciences.
Elsevier, San Diego.