ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2020/21 are given in the box.
In 2021, ECMWF’s headline scores showed consistently high values similar to the previous year, while some aspects of forecast skill improved beyond what has been seen before. Compared to forecasts from other global centres, ECMWF has been able to maintain the overall lead for upper-air parameters in the medium range. For surface parameters, especially in the short range, some of the other centres have drawn closer to ECMWF.
Model Cycle 47r2, implemented on 11 May 2021, brought an increase in the number of levels in the vertical for the ensemble forecast (ENS) to match those of the high-resolution forecast (HRES). The upgrade led to a small but statistically significant improvement in upper-air ENS skill. Model Cycle 47r3, implemented on 12 October 2021, included a major upgrade to the moist physics of the model, resulting in more realistic precipitation characteristics. Together with a number of other changes, this cycle brought small improvements in upper-air skill in the early medium range.
ECMWF has started to routinely monitor the upper-air meteorological forecast performance of the EU-funded Copernicus Atmosphere Monitoring Service (CAMS) run by ECMWF. Its skill is similar to that of other global centres, as seen in the first figure, and the difference between HRES and CAMS at day five amounts to about half a forecast day. The main differences between the two suites, in addition to the treatment of atmospheric constituents, are their initialisation and horizontal resolution.
Position errors for forecasts of tropical cyclones were smaller than in the previous year, but the decrease was paralleled by forecasts based on the ERA5 reanalysis system, which indicates that it was part of natural variability. As in previous years, ECMWF has a modest but consistent lead in ocean wave forecasting over other centres in terms of significant wave height.
There has been a noticeable further increase of the skill of the Extreme Forecast Index (EFI) for 2‑metre temperature, as shown in the second figure. After a decade of little systematic change, the ROC skill at day five has recently increased beyond its previous average value of 0.9. There has also been a further significant improvement for 2-metre temperature in week 3, as monitored by the extended-range headline score based on re‑forecasts. It is not clear yet how much of the increase is due to inter-annual variability.
ECMWF’s seasonal forecast system, SEAS5, predicted the change from La Niña conditions in the equatorial Pacific at the end of 2020 to more neutral conditions in 2021 reasonably well, although the change occurred somewhat earlier than predicted. Due to the absence of strong tropical forcing during most of 2021, the skill of the extratropical seasonal forecast was relatively low.
The complete set of annual verification results is available in ECMWF Technical Memorandum No. 884 on ‘Evaluation of ECMWF forecasts, including the 2021 upgrade’, downloadable from http://www.ecmwf.int/en/publications/technical-memoranda.
The following are other sources of information about verification and forecasting system changes:
- Verification pages: https://apps.ecmwf.int/webapps/opencharts/
- Inter-comparison of global model forecast skill: http://apps.ecmwf.int/wmolcdnv/
- Ocean wave model inter-comparison results: https://confluence.ecmwf.int/display/WLW/WMO+Lead+Centre+for+Wave +Forecast+Verification+LC-WFV
- A list of ‘Known IFS Forecasting Issues’: https://confluence.ecmwf.int/display/FCST/Known+IFS+forecasting+issues
Assessment of ECMWF’s Technical Advisory Committee, 6–7 October 2021
With regard to its overall view of the performance of ECMWF’s operational forecasting system, the Committee:
a) noted that ECMWF’s headline scores continue to show high skill, at least maintaining the level of skill reached in 2019 with some ENS measures reaching their highest ever scores;
b) noted that evaluation of extended-range real-time forecasts for 2 m temperature anomalies shows some increase in week 2 and a little improvement in skill at week 3 and 4 in winter;
c) recognised that ECMWF maintains a lead compared to other centres in terms of upper air medium- and extended-range ensemble spread and error, though there remains some under-dispersion in the summer;
d) recognised that ECMWF has an overall lead in verification scores against buoys for extratropical ocean wave height but not for peak period;
e) recognised that EFI ROC and Diagonal Elementary skill scores over Europe aimed at high-impact weather showed new high points for skill at Day 5 were reached for 2 m temperature, 10 m wind speed and 24 h precipitation; in the case of 2 m temperature this is especially pleasing as there seemed to have been a plateau in skill over the last 10 or so years;
f) noted that skill scores for mean position errors of tropical cyclones on Days 3 and 5 are not as low as 2019, but improved relative to ERA5 and the gap between HRES and ERA5 for intensity and speed has been maintained;
g) noted that ECMWF’s seasonal forecasts joined other models in underestimating magnitude of La Niña and then was too slow in initially returning to a neutral ENSO;
h) noted that the winter 2020/21 SEAS5 forecast missed the cold anomaly over northern Siberia and then missed magnitude of colder conditions late winter/early spring, more so over northern Europe. The summer 2021 SEAS5 forecast did not capture the magnitude of warm anomaly over northern Europe but was better with details over southern Europe;
i) noted that CAMS, for meteorology, ranks third behind other centres, including the HRES, and can be better than the HRES at times in some areas of the Tropics where biomass burning plays a role due to having prognostic, interactive aerosol;
j) noted that some users would like additional Europe-centric verification for ENS scores, perhaps including comparison with other centres;
k) congratulated ECMWF on the successful implementation of IFS Cycle 47r2 in May 2021, which included a move to single-precision for both the ENS and HRES and an increase in ENS vertical resolution to 137 levels which led to an improvement in ENS verification scores;
l) congratulated ECMWF on the successful implementation of Open Charts in October 2020 and welcomed ECMWF’s commitment to responding to user feedback in further developing Open Charts;
m) welcomed the implementation of IFS Cycle 47r3 scheduled for October 2021, recognising that components such as the major upgrade to moist physics will aid the forecasting community in many ways, for example improved precipitation, whilst also providing a stronger foundation to move to higher resolution models in the future. Appreciated the many other improvements 47r3 brings, noting that some of these improvements have been driven by user feedback;
n) appreciated the support ECMWF has given in developing some new products, for example ecPoint, and welcomed further developments with post-processing and machine-learning;
o) welcomed the improvements to CAMS, the ongoing development of the Climate Data Store and ECMWF’s commitment to maintaining the integrity and quality of their contributing elements to C3S;
p) welcomed proposed improvements at 48r1 in late 2022/ early 2023, noting that data volumes will then also change as ENS horizontal resolution improves and the extended range commences running with 100 members daily;
q) appreciated the training, documentation and feedback processes provided by ECMWF and welcomed future training opportunities and webinars introducing new products and developments;
r) appreciated the continued very good support ECMWF provided to Member and Co-operating States over the last year, particularly in the face of COVID‑19 when training and events such as the annual UEF continued.