ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2017 are given in the box.
The overall performance of the operational forecasts is summarised using a set of headline scores endorsed by the TAC, which highlight different aspects of forecast skill. Upper-air performance is monitored through the anomaly correlation of 500 hPa geopotential height of the highresolution deterministic forecast (HRES) and the continuous ranked probability score (CRPS) for temperature at 850 hPa for the ensemble forecast (ENS), both over the northern hemisphere extratropics. The most recent upgrades to the Integrated Forecasting System (IFS) on 22 November 2016 (Cycle 43r1) and 11 July 2017 (Cycle 43r3) have enabled ECMWF to remain the leading centre in terms of overall medium-range forecast skill, for both deterministic and ensemble forecasts. The 12-month running mean value of the ENS skill now consistently exceeds nine days, which is an increase of more than a day over the last decade, as shown in the first figure. Note that ERA5 now provides a useful benchmark and reference for the identification of interannual variations in predictability, replacing ERA-Interim in this regard.
Headline scores for the surface weather focus on precipitation for both HRES and ENS. The ENS skill for precipitation is shown in the second figure – skill is maintained consistently out to 7 days. The resolution upgrade from 32 to 18 km for the ENS in 2016 contributed to a noticeable increase in skill. HRES skill also continues to increase for precipitation, as does the skill for the Extreme Forecast Index (EFI) for precipitation.
Surface skill has increased for 2 m temperature, humidity and 10-m wind speed. However, there are still issues with seasonally and diurnally varying regional biases in 2 m temperature which need to be resolved. Forecasts of tropical cyclones have improved in terms of position errors both for the HRES and ENS, with both showing lowest values so far, while speed errors slightly increased compared to the previous year. Mean absolute intensity errors decreased compared to the previous year, but there are still significant intensity biases, especially for the ENS. Wave forecast skill has further increased both with respect to wave height and peak period, and ECMWF generally maintains a lead compared to other global wave forecasting systems.
The El Niño of 2015–16 ended in early summer 2016, and subsequent conditions have remained relatively close to neutral. With no strong El Niño Southern Oscillation (ENSO) forcing, a general drop in seasonal predictive skill would be expected for 2016–17. Nevertheless, there were some large-scale temperature anomalies that the forecast captured to some extent. The pattern of 2 m temperature in the northern-hemisphere winter (December–January–February 2016–17) was characterized by strong warm anomalies in the Arctic, and over North America and Eurasia. These warm anomalies, which are a combination of the effect of global warming and interannual variability, were captured reasonably well by the seasonal forecast in North America, but not in Eurasia. Parts of Europe experienced a hot summer season in 2017. The forecast for June–August predicted positive anomalies over Southern Europe and the Mediterranean, as well as parts of Siberia (again, part of this pattern is due to global warming). The verifying analysis confirmed the basic pattern, although the magnitude of the observed cold anomaly stretching from Scandinavia eastward was less well captured.
Each summer ECMWF invites Member and Co-operating States to submit updated reports on the application and verification of ECMWF’s forecast products. In this year’s reports, verification corresponds mainly to periods since the introduction of the higher-resolution HRES and ENS (IFS Cycle 41r2 in March 2016), and several countries noted recent improvements in the verification results for HRES. A number of countries also noted improved scores for winter temperature since last year, consistent with the expected improvements from IFS Cycle 41r2 (resolution increase) and 43r1 (e.g. modified surface coupling for 2 m temperature).
These reports tend to focus on HRES, and there is less verification reported for ENS. However, Denmark reported that verification of the ENS is used as a reference (benchmark) for the development of their COMEPS limited-area ensemble system. They note especially that for 10 m wind COMEPS outperforms the ENS because of its much higher horizontal resolution (2.5 km HARMONIEAROME grid). Switzerland compares probabilistic scores for the ENS and the COSMO-E ensemble. Some of the IFS performance issues raised by the users are known, and most of these are also listed on the ECMWF ‘Known IFS Forecasting Issues’ web page, which is regularly updated.
The complete set of annual results is available in two ECMWF Technical Memorandums, No. 817 on the ‘Evaluation of ECMWF forecasts, including 2016–2017 model upgrades’, and No. 818 on the ‘Use and Verification of ECMWF products in Member and Co-operating States (2017)’. Both are downloadable from https://www.ecmwf.int/en/research/publications. These documents present recent verification statistics and evaluations of ECMWF forecasts (including weather, waves and severe weather events), whilst No. 817 also includes information about changes to the data assimilation/forecasting and post-processing systems. The performance of the monthly and seasonal forecasting systems is also assessed.
Other sources of information:
- Verification pages on the ECMWF web server are regularly updated. They are accessible at https://www.ecmwf.int/en/forecasts/charts
- Interactive plots showing intercomparisons of global model forecast skill can be found on the WMO Lead Centre for Deterministic Forecast Verification (WMO-LCDNV) web page at http://apps.ecmwf.int/wmolcdnv/
- All IFS forecasting system cycle changes since 1985 are described at https://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model
Assessment of ECMWF’s Technical Advisory Committee, 12–13 October 2017
With regard to its overall view of the ECMWF operational forecasting system, the Committee:
- acknowledged the very good level of information provided by ECMWF in its evaluation of performance, including some of the information that the TAC Subgroup on verification measures is proposing to add to the current set;
- appreciated the very good summary of feedback received from Member and Co-operating States and supported the new deadline for providing this feedback, which gives ECMWF more time for extracting the most important information;
- was pleased to see continued improvement of important headline scores, for example ECMWF maintaining and sometimes increasing its lead with respect to other global NWP providers;
- noted that the use of ERA5 as a benchmark provides very useful assistance in interpreting the impact of model improvements vs interannual variability;
- noted the improvement of precipitation forecasts with special interest; HRES, ENS and the EFI verification scores have reached their best ever levels this year and several Member and Co-operating States have also reported positive feedback on this improved quality;
- noted some remaining problems such as too much light precipitation and insufficient inland advection of precipitation. ECMWF has reported preliminary results showing improvements to these aspects, which will be included in the coming model cycle;
- welcomed the prospect of improvements in the handling of lying snow in future model cycles;
- noted that the improvement for surface parameters, relative to ERA5, is smaller than for upper level parameters. Random errors have been further reduced although biases persist;
- noted that higher model resolution has been reported by several Member States as significantly improving the quality of wind forecasts. However, little improvement for extreme wind events was found in the EFI headline score over the last few years;
- noted that temperature forecasts still have some biases, notably in the representation of the diurnal cycle; appreciated the creation of a dedicated project to investigate this and is looking forward to their findings;
- appreciated the inclusion, or future inclusion, of new useroriented parameters such as visibility and lightning density. Member and Co-operating States have started using and evaluating some of these, and improvements for some of the identified biases were included in the most recent model cycle;
- noted that for ocean waves, ECMWF’s lead in verification scores compared to others has been reduced but that improvements are expected with 45r2;
- welcomed the smallest ever tropical cyclone track errors for HRES and ENS. Improvements are expected in forecasting tropical cyclone intensity in future model cycles, including improvements in tropical cyclone analysis;
- noted that the monthly forecast has shown improvement over time for week 2 and no clear trend for weeks 3 and 4;
- welcomed promising results from SEAS5 and noted successful modelling of summer 2017 by the seasonal forecasting system;
- welcomed the earlier delivery of ENS forecasts that has been achieved without compromising quality. This has been appreciated by users, especially forecasters.