Fact sheet: Supercomputing at ECMWF

10 September 2021

Atos BullSequana supercomputer

The Centre’s next supercomputer will be made up of four Atos BullSequana XH2000 clusters and is planned to become operational in 2022. (Image: Atos)

Why does ECMWF need a supercomputer?

Supercomputers are made up of many thousands of processers, which can efficiently exchange information and work as one powerful machine. Only a supercomputer can handle the enormous amounts of mathematical calculations and data processing involved in modern-day weather forecasting.

Every day at ECMWF, hundreds of millions of observations must be filtered and combined with recent short-range forecasts. This process is known as data assimilation, and it provides the input to a sophisticated numerical model of the Earth system that simulates weather across the globe. The numerical model is based on physical laws that govern changes in the atmosphere, land, and oceans. To make a forecast, mathematical equations that describe these laws are solved, using the analysed state of the Earth system as a starting point. This type of forecasting procedure is known as numerical weather prediction (NWP).

ECMWF’s computing facilities also store and disseminate forecasts to our Member and Co-operating States and a range of other users – all within exacting operational time constraints.

Why does ECMWF need a global Earth system model?

Weather over Europe can be influenced by disturbances that occurred on the other side of the world a few days earlier, and by longer-term conditions over land and in the ocean. Hence, a global Earth system model is essential. It allows ECMWF to provide forecasts for anywhere in the world, for days, weeks and months ahead.

How does ECMWF use its supercomputer?

Time on the supercomputer is carefully managed.

Half of the available time is allocated to day-to-day operational forecasting. The core operational forecast runs twice a day (midday and midnight). There are also two further runs of the model in-between, which are primarily used to provide boundary conditions to local area models run by our Member and Co-operating States. Member States also have a resource allocation on the supercomputer for activities such as running some of their own operational forecasts, or for specific project work. 

The other half of the available time on the supercomputer is allocated to ECMWF research, vital for continually improving our forecasts. It also includes testing the next upgrade of the forecasting system. This is run in parallel with the operational system for an extended period of testing, before operational implementation.

How do developments in supercomputing help to provide more skilful weather forecasts?

Improving the representation of Earth system processes in models, running finer-resolution ensemble forecasts, and improving the assimilation of weather observations can all bring improvements in forecast skill. However, such developments demand ever greater computer power, hence ECMWF’s supercomputer is upgraded on average every four or five years. Increasing computer power while improving energy-efficiency, and maintaining affordability, are all key for the future.

ECMWF’s supercomputer comprises two identical Cray XC systems, which provide two self-sufficient clusters of processors and storage. An Atos BullSequana XH2000 supercomputer will come into operation in 2022, to be housed in ECMWF’s data centre in Bologna, Italy. The upgrade will achieve around a five-fold increase in computing power. While an average desktop PC has 2 or 4 processors, ECMWF’s Atos supercomputer will have over one million processors spread across 4 clusters and a performance of about 30 petaflops (1015 or a million billion calculations per second).

What does the future hold?

A new generation of computing systems with exascale capabilities (1018 or a billion billion calculations per second) is emerging, which promise improved energy efficiency. However, it is not just raw computing power that matters, radical changes are needed to optimise our forecasting system so we can exploit the full potential of exascale machines.

ECMWF’s major Scalability Programme is tackling the challenge with meteorological modellers, computer scientists and hardware providers from around the world, in a collaborative approach to software and hardware development. Key challenges include: adapting computer codes to work efficiently on a range of novel computer architectures, improving data handling and exploiting machine learning.

Working at the interface between computational science and weather prediction will allow ECMWF to continue to push the boundaries of weather forecasting skill.

Atos BullSequana XH2000 system  
Clusters 4
Each cluster has  
Compute nodes 1,872
General purpose nodes 112
Racks 20 water-cooled, 2 air-cooled
Weight (kg) 42,000
Each node has  
Processor type AMD Epyc Rome
Cores 64 cores/socket, 128 cores/node
Memory/node (GiB) 256 (compute nodes) / 512 (general purpose)
Memory (PiB) 2.05
Nodes 7,488 compute, 488 general purpose
Cores 1,015,808

Atos BullSequana XH2000 supercomputer – facts and figures.