Pedro Maciel is the main developer of ECMWF’s new interpolation and regridding software package.
It has taken nearly four years to develop and test ECMWF’s new Meteorological Interpolation and Regridding (MIR) software package. Now its main developer, Pedro Maciel, is happy to certify that it is ready for operations.
Pedro joined ECMWF’s Development Section in January 2015 and has been working on MIR ever since. At the same time, he has also improved the interpolation software used at ECMWF until now, which is part of the EMOSLIB package.
Pedro's main area of expertise is fundamental to numerical weather prediction: computational fluid dynamics.
His fascination with space prompted him to study aerospace engineering. This led to a PhD in multiphysics, using computational fluid dynamics to study complex flows.
Postdoctoral work and a spell of working as a fluid dynamics research engineer gave him the skills he needed to take on a challenge in software development for numerical weather prediction.
“The task was to replace ECMWF's ageing interpolation package, which had become difficult to maintain and to develop further, by a state-of-the-art, more flexible package,” Pedro explains.
“Once it has been made operational, MIR will be widely used not just at ECMWF but also by external users of our forecasts and of the MARS meteorological archive.”
Interpolation estimates values at intermediate points between a given set of points, as shown here for a simple ‘nearest neighbour’ interpolation (left), linear interpolation (middle) and cubic interpolation (right), with estimates of values at intermediate points shown as red dots and the values at the given set of points as blue dots. (Image: Cmglee – CC BY-SA 4.0)
Wide range of applications
Starting from a set of values associated with a given set of points, interpolation estimates the values at intermediate points.
This is required for many weather forecasting applications. Users may, for example, want to map weather data from ECMWF’s weather model grid to a different grid adapted to their needs.
MIR can do this, and more. It can also transform data from spectral space (a mathematical space used in ECMWF’s model) to physical space (a grid describing the Earth’s surface and atmosphere).
The data could be any type of weather variable, such as wind, temperature or precipitation.
Handling requests submitted to MIR typically involves multiple steps.
Pedro gives the example of a user making a request to retrieve a wind field in east–west (u) and north–south (v) components from ECMWF’s MARS meteorological archive.
“MIR converts the vorticity and divergence fields in spectral space to u and v components, transforms them to the gridded space and finally interpolates them to the grid the user requested,” he says.
“The principle is that users should just have to tell the software package what they want to do. The software acts like a ‘black box’ which will do all the work for them in the background.”
Surface geopotential height field (in m2 s-2) interpolated from the octahedral reduced Gaussian grid TCo1279 used operationally for high-resolution forecasts at ECMWF (back) to a lower-resolution grid (TCo31 – front) using MIR. The colour palette is qualitative, for illustrative purposes only.
In addition to MARS requests, MIR also handles interpolation and regridding in product generation and for the meteorological workstation application Metview.
“MIR is the post-processing engine for MARS, product generation and Metview,” Pedro says.
Rigorous testing
Pedro worked with colleagues in the Research and Forecast Departments to get MIR to the point where it can do all the tasks which are currently performed by EMOSLIB.
The last 10 months have been devoted to putting MIR through a rigorous validation programme.
Nearly 200,000 MARS requests have repeatedly been submitted to MIR to make sure the package handles them successfully.
The proportion of problematic cases has gone down from about 1.5% three months ago to just 0.001% in the latest test run.
“This is far less than the failure rate of the EMOSLIB package, showing that MIR is now ready for operations,” Pedro says.
“As a research scientist, I used to work with software that was not necessarily very robust, and I fixed issues as I went along. Here I’ve had to put the bar much higher to get this package ready for operations based on systematic testing.”
MIR can interpolate from a point cloud and associated unstructured mesh (front – the mesh is shown in green) to the octahedral reduced Gaussian grid TCo1279 used at ECMWF (back), shown here for an upper-atmosphere temperature field.
Greater flexibility
The previous package, EMOSLIB, was written in the 1980s. It has provided a robust service over many years, but it has become difficult to maintain and update.
MIR has been designed to provide greater flexibility.
“It is more scalable and brings efficiency gains, a high degree of user configurability, and support for a wider range of grids than EMOSLIB,” Pedro says.
“It uses the data infrastructure provided by the recently developed software library Atlas to support unstructured grids.”
MIR is implemented in the C++ programming language, with a clean architecture. The code is concise, readable and extensible, making it easy to maintain.
MIR streamlines processing pipelines by merging operations that match. This minimises the number of total operations and thus runtime and I/O.
Once MIR is operational, Pedro will work on adding additional features that go beyond the functionality of EMOSLIB.
These will concern aspects such as conservativity, higher-order interpolation and improved interpolation methods.
It looks like MIR will keep him busy for some time yet.