ECMWF is stepping up its efforts to provide processing and visualisation options in the Python programming language. To engage with the developer community, the Centre hosted a Python frameworks workshop on 28 and 29 November 2017.
High-profile uses at ECMWF include the Python-based toolbox for the Climate Data Store, which will be at the heart of the Copernicus Climate Change Service (C3S) from next year, and the new Python interface of the post-processing and visualisation package Metview.
But building a Python framework is challenging.
The aim of the November workshop was to bring together key actors from the Python community who develop packages for Earth system sciences.
“The workshop was a great success as many participants had not met before and could for the first time exchange experiences,” says workshop organiser Stephan Siemen.
Presentations showed the functionalities provided by different packages and offered insights into the challenges involved in their development and distribution.
Three working groups were set up to discuss the various challenges and how the community can work together better. Each group tackled a different aspect:
- Deploying and packaging Python frameworks
- Handling Big Data in Python
- (Code) Interoperability and common data structures
The discussions focused on the challenge of enabling improved interoperability between the various frameworks represented at the workshop.
Outcomes
Some of the main points made in the working groups were that:
- there needs to be a good mapping of metadata between data formats because metadata plays a crucial role in giving meaning to the data
- it is essential to make as much use as possible of core Python packages, such as NumPy, Pandas, xarray and Dask, since this will automatically reduce incompatibilities between packages
- ECMWF needs to support efforts to engage with the Python community and explain its community’s needs, for example by participating in a Python-for-Earth-system-sciences session at the annual European Conference on Python in Science.
All participants were encouraged to contribute to the core packages to improve them instead of implementing new solutions. Participants stressed that it is also important to pick the right tool for the job, ideally one with good community support.
There was a strong message on community-led developments and distribution: all packages presented were Open Source, but it was pointed out that Open Development is the first step towards building a community.
The Python frameworks workshop was attended by 55 developers, who explained the functionalities of different packages and made recommendations for future collaborative work.