ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2018 are given in the box.
The overall performance of the operational forecasts is summarised using a set of headline scores endorsed by the TAC, which highlight different aspects of forecast skill. Upper-air performance is monitored through the continuous ranked probability score (CRPS) for temperature at 850 hPa for the ensemble forecast (ENS) and the anomaly correlation of 500 hPa geopotential height of the high-resolution forecast (HRES), both over the northern hemisphere extratropics. After a slight decrease in predictability in 2017 (as inferred from comparison with the new ERA5 reanalysis), both ENS and HRES skill increased again in 2018. The most recent upgrade of the Integrated Forecasting System (IFS Cycle 45r1 on 5 June 2018) has brought improvements in the extratropics for some aspects of the forecast, notably an increase in precipitation forecast skill in the HRES, which is very clear when compared with a baseline of ERA5‑based forecasts, and improvements in the tropics for most parameters. In terms of surface skill, in 2018 there was a slight increase for 2‑metre temperature. Forecasts of tropical cyclones have further improved in terms of position, intensity, and speed errors for both the ENS and HRES. With regard to ocean waves, ECMWF has maintained its lead compared to other global wave forecasting systems for forecasts of significant wave height, and its position among the leading systems for peak period. The change from La Niña to El Niño conditions in the first half of 2018 was well captured several months in advance although the positive sea-surface temperature anomalies were weaker than predicted. ECMWF forecasts predicted warm anomalies in Europe in spring and summer 2018 weeks in advance, while the northward extent and intraseasonal variations during the heatwave were reflected in forecasts up to two weeks ahead.
Each summer ECMWF invites Member and Co-operating States to submit updated reports on the application and verification of ECMWF’s forecast products. Many reports focus on comparing HRES with forecasts produced by limited-area models (LAMs), and for this reason usually centre on the shorter ranges (up to about 48 h). A common finding was that biases in IFS forecasts have a diurnal cycle. Annual cycles are also often present. Overall, whilst there are large model-to-model and parameter-to-parameter variations, and some large differences between countries, LAM performance tends to be slightly better than HRES performance.
ENS-related verification was only included in a few reports. Comparisons between ENS and LAM-EPS systems were provided by Finland (2‑metre temperature), Switzerland (precipitation) and Denmark (10‑metre wind), while Hungary (surface parameters), the UK (tropical cyclones and regimes), Israel (ocean waves) and Switzerland (vertically-integrated water transport) provided some evaluation of ENS forecast outputs.
The complete set of annual results is available in two ECMWF Technical Memorandums, No. 831 on ‘Evaluation of ECMWF forecasts, including the 2018 upgrade’ and No. 840 on ‘Use and Verification of ECMWF products in Member and Co-operating States (2018)’. Both are downloadable from http://www.ecmwf.int/en/research/publications.
The following are other sources of information about verification and forecasting system changes.
- Verification pages: http://www.ecmwf.int/en/forecasts/charts
- Inter-comparisons of global model forecast skill: http://apps.ecmwf.int/wmolcdnv/
- A list of ‘Known IFS Forecasting Issues’: https://confluence.ecmwf.int/display/FCST/Known+IFS+forecasting+issues
- All IFS forecasting system cycle changes since 1985: http://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model
Assessment of ECMWF’s Technical Advisory Committee, 11–12 October 2018
With regard to its overall view of the performance level of ECMWF’s operational forecasting system, the Committee:
a) welcomed the new format of the presentations for the forecast performance and feedback from Member and Co-operating States as a webinar in advance of the TAC; appreciated that many questions raised during the webinars had been addressed by ECMWF in time for the TAC meeting;
b) noted that ECMWF headline scores continue to show high skill, and noted that the recent plateauing is due to natural variability in predictability, as revealed by comparison with ERA5;
c) welcomed improvement seen in the new headline score for large 2 m temperature errors and encouraged ECMWF to investigate a similar measure for 10 m wind;
d) took account of Member and Co-operating States’ feedback on precipitation biases and encouraged further investigation;
e) appreciated the very good support ECMWF provided to Member and Co-operating States over the last year, in particular for high-impact weather events such as heat waves, cold spells, windstorms, and tropical cyclones, and through specific products such as the EFI, precipitation type, and improvements in EFAS and fire danger indices;
f) noted that the extended-range forecast was useful for assessing the duration of the summer heatwaves in Europe;
g) welcomed the improvements in snow, but noted the problem of excessive accumulation of wet snow, and slowness in melting accumulated snow; welcomed the changes in the forthcoming cycle to address the first issue and encouraged future work including the multi-level snow model;
h) noted the degradation in spread–error at the longer ranges (T+168 and T+240) from 2013, appreciated the work to understand the changes and encouraged ECMWF to closely monitor this in future;
i) noted that the recent drop in skill for 24‑hour precipitation accumulations in Europe is also seen in the ERA5 verification, which indicates that it is related to year‑to‑year atmospheric variability; encourages Member and Co-operating States to investigate whether this is also seen in high-resolution, convection-resolving LAMs over Europe and report back to ECMWF;
j) welcomed plans to investigate and introduce monitoring of forecast jumpiness and ‘flip-flopping’;
k) acknowledged that the new products relating to CAPE, CAPE-shear are well appreciated, and welcomed ECMWF plans to review the computation of CAPE, and encouraged exchanges with experts in Member and Co-operating States on this;
l) welcomed the new online Forecast User Guide with the flexibility for more frequent updating;
m) welcomed ECMWF’s role in WMO verification, acting as Lead Centre for deterministic NWP verification, and now also as Lead Centre for ocean wave forecast verification;
n) noted the potential impact of data cut-off time and forecast delivery time on the interpretation of forecast scores, and would welcome further work to take this effect into account in forecast comparisons;
o) appreciated the use of high-density observations to improve the verification and the demonstration of their benefits in model evaluation, and encouraged other countries to provide such data;
p) encouraged ECMWF to make use of high-resolution calibrated radar and/or other gridded precipitation datasets in verification;
q) appreciated ECMWF’s invitation to Member and Co-operating States to identify additional products and services for use by forecasters, and appreciated ECMWF’s management of these requests with the URMS to assess priorities;
r) appreciated the value of ECMWF training courses in increasing the benefit of ECMWF forecasts to Member and Co-operating States’ forecasters and end users; welcomed the new blended format combining e-learning with classroom face-to-face interactions; but stressed that such face-to-face interactions remain important and should not be reduced further.