Dr Peter Dueben, ECMWF Scientist working on numerical methods
How can single precision help?
There are two standard precision options on high performance computing hardware. In double precision, 64 bits are used to represent each real number and the numbers are represented with at least 15 decimal digits precision. In single precision, 32 bits are used and at least 6 decimal digits are available.
Today, almost all weather and climate models use double precision as the standard. However, if single precision could be used instead, this would allow significant savings for simulations, since less bits need to be processed and moved. Such savings could be reinvested to increase resolution or complexity of the models that are used, or to increase the number of ensemble members to improve predictions.
Such investigation of single precision fits very well into the scalability programme at ECMWF that aims to improve computational efficiency to make the most of current and future high performance computing hardware.
How did the research develop?
There is an ongoing effort to investigate the use of reduced numerical precision in Earth System modelling within the research group of Prof Tim Palmer at the University of Oxford. As a PostDoc of Tim, I was involved in this effort and showed in 2014 that single precision is sufficient to perform simulations with the OpenIFS model (the portable version of IFS; Dueben et al. MWR 2014). This proof-of-concept was the motivation for scientists at ECMWF to join the investigation.
A single precision version of the IFS (cycle 41R2) was developed by Filip Vana and the use of single precision was studied in more detail (Vana et al. MWR 2017). I am now working on further developments of the single precision version of IFS as part of the EU ESiWACE project.
What changes to the ECMWF Integrated Forecast System (IFS) are needed?
In principle, the change from double to single precision is as easy as a change of the default precision level of the model code. However, many technical difficulties had to be addressed such as adjustments of input/output and parallelisation routines, fixes of links to external libraries and system functions, replacements of hard-coded model thresholds, and additional safeguards were added to avoid divisions by zero. Some parts of the code, such as the setup of the Legendre transformation and the vertical integration scheme, need to be kept at high precision. It is a challenge to find all model components for which a reduction in precision generates differences in model simulations due to the sheer size of the IFS.
What’s the status of the research?
Within the latest IFS cycle (43R3), single precision simulations with the forecast model achieve almost equivalent results for both short and long term simulations. This includes simulations at the resolution of operational forecasts (9km and 18km). However, mass conservation error is significantly larger in single precision simulations and differences in temperature at 100 hPa as well as in surface temperature are still measurable. These issues are being addressed at the moment.
So far, all efforts to run in single precision have focussed on the weather forecast model. However, it is likely that single precision could also be used to calculate forecasts and trajectories in the ECMWF 4DVAR data assimilation system.
How much faster can our weather forecasts run with single precision?
The magnitude of savings with single precision depends on various factors. When computing speed is limited by memory bandwidth, single precision can typically increase performance by a factor of two since data volume is halved. If floating point operations are the performance bottleneck, the speed-up factor will depend on the level of code vectorization. Single precision is twice as efficient when the code is fully vectorised.
For forecast simulations with the ECMWF IFS on our Cray supercomputer, single precision runs are typically ~40% faster compared to double precision references. However, those numbers depend on the model resolution and the number and configuration of processors that are used in parallel.
What’s the future for single precision at ECMWF?
The use of single precision will potentially allow significant savings of computing cost in the near-term future and an increased throughput on supercomputers. Single precision will also help us to perform tests with the IFS at very high resolution. For these simulations, a very large number of computing nodes is needed to fit the model into the memory that is available. Single precision is reducing memory requirements significantly and less nodes can be used. Tests with single precision in simulations at 2.5 and 1 km global resolution are being performed at the moment.
Driven by the need for high performance and low precision in the community of deep learning and artificial intelligence, more and more high performance computing hardware is available that allows the use of half precision (16 bits), for example the Pascal GPU architecture by NVIDIA. This may open the door for further trade-offs between precision and performance in weather and climate predictions.
Figure: Surface temperature in degree Celsius for five day forecasts for 8th January 2017 0:00 UTC. This date is during the European cold wave that caused very low temperature in Eastern and Central Europe. Results are shown for single precision and double precision simulations at 9km (TCo1279) resolution (left and middle) and the analysis as a reference (right). Differences between single and double precision are very small.
Many thanks to Michail Diamantakis and Filip Vana for feedback on the text and the many people who are involved in the single precision project for their constant support.