Emanuele Danovaro, Analyst in ECMWF's Forecast Department
The news of the Bonn and Bologna openings brought great enthusiasm among EuroHPC ACROSS project partners, looking forward to cooperating even more closely with ECMWF, to develop the high performance, low power and scalable computer architectures that are vital for the future of numerical weather prediction (NWP).
Figure 1: The opening of ECMWF's new data centre in Bologna, Italy, housing the Atos BullSequana supercomputer system, will offer even greater opportunities for collaboration.
The ACROSS project started in March 2021 and will run for three years. It aims to develop an HPC, big data, and artificial intelligence convergent platform and will provide effective mechanisms to easily describe and manage complex workflows. Energy efficiency will play a key role, with massive use of specialised hardware accelerators, high-frequency systems monitoring and applying smart mechanisms of job scheduling. The platform will be co-designed by Links foundation (IT), Atos (FR), IT4I (CZ), Inria (FR) and Cini (IT) and will exploit EuroHPC computing resources, namely CINECA Leonardo and Lumi pre-exascale supercomputers and IT4I Karolina petascale machine.
The platform will be validated by three use cases:
- The first use case aims to increase the propulsive efficiency of aeronautics turbines by designing a machine-learning based system for the exploration of their parametric design space.
- The second use case will face the problem of data management in large-scale hydro-meteorological and climatological workflows addressing cloud-resolving global-scale NWP and efficient post-processing techniques.
- The third use case will focus on the development of a system for the simulation of large-scale CO2 storage scenarios.
ECMWF, in cooperation with domain experts including Max-Planck Institute (MPI) (Hamburg, DE), Deltares (NL) and Neuropublic (GR), is leading the ACROSS partners in the development of the hydro-meteorological and climatological use case.
Figure 2: A schematic of the hydro-meteorological and climatological use case being developed under the EuroHPC ACROSS project. Image credit: EuroHPC ACROSS project.
Our goal is to demonstrate global simulations at 5 km resolution (or better) with multiple concurrent in-place data post-processing applications (i.e. product generation, machine-learning inference) and low-latency support for downstream applications (e.g. regional downscaling, hydrological simulations). In-place post-processing is crucial in reducing the pressure on the input/output (IO) system and scaling workflows to very high resolutions. In ECMWF’s current operational system, roughly 70% of the data generated by the Integrated Forecasting System (IFS) and stored in the Fields Database (FDB) is read back during the time-critical windows for product generation. This results in file-system contention and a resultant slowdown of the numerical model.
We plan to achieve in-place post-processing by adopting the MultIO software stack, developed as part of the MAESTRO project, and we will work on efficient exploitation of heterogeneous computing resources by experimenting with graphics processing unit (GPU)-accelerated post-processing. This will improve end-to-end run times by reducing unnecessary data movements.
Data generated by global-scale NWP models will be stored and made available to the downstream applications in the workflow. ECMWF will use the FDB to approach this problem. The FDB is a domain-specific object store which efficiently stores and indexes NWP output according to semantic metadata.
While versions of the FDB have been used operationally for decades, increases of data volume and velocity generated by high-resolution forecasts (5 km or better resolution), and increased heterogeneity in the hardware landscape require further improvements. ECMWF is committed to improving the FDB to fully exploit the multi-layer data stores available on the EuroHPC computing resources, that may offer a combination of non-volatile memory express (NVMe) solid-state storage devices (SSDs) and traditional parallel file systems.
ACROSS infrastructure will provide an invaluable opportunity for testing such new technologies at scale and will support the ECMWF Strategy. Moreover, ECMWF's new data centre in Bologna and the CINECA Leonardo pre-exascale supercomputer will be side-by-side in the new Bologna Technopole. We are really looking forward to benefiting from the synergies that such a close relationship will bring to ACROSS and Destination Earth, that will exploit Leonardo as one of the main computing resources.
Innovation within the ACROSS project is not limited to the meteorological domain; in cooperation with MPI and DKRZ, we are contributing to the use of the FDB in the ICON numerical model, adopted by MPI for climatological simulations. This joint effort will pave the way to closer collaboration between meteorological and climatological communities and simpler data interchange among different models in the complex ACROSS workflows. While we have always had good cooperation with MPI and DKRZ, we expect that the new Bonn site will provide additional opportunities for closer cooperation and collaboration.
The ACROSS project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955648. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Italy, France, Czech Republic, United Kingdom, Greece, Netherlands, Germany, Norway.