ECMWF is developing Python packages and interfaces to help people work with vast weather and climate datasets faster and more efficiently. The Centre also provides training to help the scientific community and technical users to interact with services, make use of ECMWF’s open source software and create maximum value for their applications.
As a numerical weather prediction (NWP) centre, ECMWF generates and holds vast amounts of weather forecast data. Offering ensembles of forecasts, increasing model resolutions and an increasing number of physical parameters has meant the data volumes generated are constantly growing.
The Copernicus services operated by ECMWF on behalf of the European Commission also offer various datasets that are growing in size and popularity.
These datasets have great value for decision makers and scientists, but too often people cannot take full advantage of the datasets’ potential because they are too large for time-critical applications or training for machine learning algorithms.
Python has become the programming language of choice for many users processing large datasets in the Earth system sciences, and ECMWF is investing in this area to help users interact with its data.
Simplified access
For many years now ECMWF has provided Python interfaces to its key software packages to access, process and visualise meteorological data.
In recent years Metview, ECMWF’s meteorological workstation and batch system, has also adopted a Python interface that encapsulates all the functionality of the various Python packages provided by ECMWF. By working with well-known efficient Python data structures, this allows scientists and analysts to work with all their data as usual whilst also making use of the possibilities offered by the established Python scientific ecosystem.
While very popular in the meteorological domain, ECMWF software packages are still very domain specific. ECMWF is developing a new Python package called CliMetLab, aimed at data scientists using machine learning on weather and climate data.
From the beginning this package has been developed to be used in JupyterLab environments. It integrates closely with the PyData ecosystem to ease its adoption. Popular datasets can be loaded with a single command and are immediately ready for analysis with common Python data processing tools.
The new CliMetLab development features example-based documentation through Jupyter notebooks.
Training with Jupyter notebooks
ECMWF provides training for the scientific community and technical users to learn how best to interact with services, make use of its open source software and create the biggest value for their applications. Jupyter notebooks have been used in all these areas.
One example is European research projects such as HiDALGO, where notebooks have been used successfully to train users on the benefits of weather and climate data and provided examples on how to access data from across various centres.
A large set of notebooks has been developed to provide examples on how to use the various Python open source software packages that ECMWF provides for the community. ECMWF participated with two posters in the recent JupyterCon conference and learned about the latest developments in both using and providing Jupyter-based services.
Example of how JupyterLab allows scientists and analysts to easily work with weather data.
ECMWF is investigating the use of server-side data processing using Jupyter notebooks for its forecast and Copernicus datasets. The work will build on the experiences of the Copernicus Climate Data Store (CDS) Toolbox, which already allows users to access all Copernicus climate datasets without any data download. This would open the possibility for data scientists to interact with the full datasets without technical barriers on their side.
ECMWF will hold a virtual workshop in February 2021, entitled Weather and Climate in the Cloud, to engage with the community on this topic.