Media hub‎ > ‎

ESCAPE News

Atlas software library released

posted 5 Jul 2018, 04:21 by Daniel Thiemert   [ updated 5 Jul 2018, 04:21 ]

ECMWF has publicly released Atlas, a software library supporting the development of Earth system model components and data processing, with an open source Apache 2 licence - partially developed under the ESCAPE project. The source code can be downloaded freely at www.github.com/ecmwf/atlas and contains installation instructions.
Emerging diverse and complex hardware solutions have a large impact on the programming models traditionally used in software for numerical weather prediction (NWP). Furthermore, NWP models are becoming increasingly complex as more Earth system components are introduced. Each model component may use different numerical algorithms and may be based on structured or unstructured grids.
In response, ECMWF is developing Atlas, an object-oriented (OO) programming library for developing flexible next-generation NWP models on existing and emerging hardware, such as GPUs. Atlas addresses the complexities associated with managing parallel distributed data structures on Earth grids. It provides mesh-generation capabilities from a wide catalogue of grids. Through its OO design, the memory layout and parallelisation of fields defined on meshes can be abstracted to accommodate specific numerical methods and hardware implementations. Atlas not only supports structured global grids such as the ones used operationally at ECMWF, but also unstructured meshes and structured regional grids as used in limited-area models. Although Atlas’s main features are implemented using C++, a modern OO Fortran interface allows Atlas features to be used in most existing NWP software packages.

Distributed mesh generation
Distributed mesh generation. Atlas is here used to generate a distributed mesh from a very coarse reduced octahedral Gaussian grid. Each distributed mesh partition can be constructed with parallel halos to allow stencil operations. Distributed fields can then be created based on a specific ‘function space’, which is tightly coupled to numerical schemes of how discrete data is interpreted in a continuous space

At ECMWF, Atlas is being used in the development of a finite-volume module (FVM) for the Integrated Forecasting System (IFS) (see ECMWF Newsletter No. 145, pp. 24–29). Its parallel mesh-generation capabilities build a distributed unstructured mesh about the grid points of the IFS’s reduced Gaussian grid. The resulting triangular or quadrilateral elements are then used to evaluate finite-volume differential operators. Atlas’s mesh generator is also used in ECMWF’s new Meteorological Interpolation and Regridding (MIR) software (see ECMWF Newsletter No. 152, pp. 36–39), where the resulting elements are used to construct linear interpolation operators.

The ESCAPE project: towards exascale computing in NWP

posted 21 Feb 2018, 02:43 by Daniel Thiemert   [ updated 21 Feb 2018, 02:47 ]

ESCAPE project logo

Future developments in high-performance computing will place even more emphasis on parallel processing. They promise greater energy efficiency but rely on parallelisation at levels to which current numerical weather prediction (NWP) codes are not adapted. Making codes more scalable to run on more processors is among the top priorities in NWP.

Through its Scalability Programme, ECMWF is involved in a range of projects designed to cover the entire NWP processing chain, from processing and assimilating observational data to delivering forecasts to Member and Co-operating States and other licensed users.

One of these projects, ESCAPE, is about providing better insight into the computing performance and energy-efficiency of those model components that drive overall cost. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. Funded by the EU and coordinated by ECMWF, the three-year project started in October 2015 and combines the expertise from 12 world-leading global and regional numerical weather prediction centres, academia, high-performance computing centres and hardware vendors.

Weather and climate dwarfs

The project’s innovative approach is to extract elements called ‘dwarfs’ from a weather prediction model for adaptation to different hardware options. Dwarfs are key functional components that also impose specific computational challenges on the system. Working at dwarf level allows problem-specific code adaptation, optimisation and benchmarking, and also facilitates knowledge dissemination and training. The experience gained will provide guidance for model and hardware configurations that provide the best cost–benefit relationship across all applications considered in ESCAPE.

Early in the project, eight dwarfs were extracted from ECMWF’s Integrated Forecasting System (IFS). Of these, four have been ported and assessed on different processor types.

ECMWF’s flexible and parallel data framework Atlas, which serves as the underlying data structure for many of the dwarfs, has been used to adapt the dwarf code. This makes it possible to perform model field operations on various grid/mesh options in a unified and parallelised way. Atlas also links to the well-established GridTools library developed by MeteoSwiss, ETH Zurich and the Swiss National Supercomputing Centre (CSCS). GridTools is a software layer that employs a domain-specific language concept to separate science code from hardware-dependent libraries. This concept is being further extended in the second phase of the project and promises great potential for future Earth system model code development.

Efficiency gains

The performance metrics follow the so-called Roofline model, which uses the ratio of the number of performed calculations and the amount of data being moved across the memory hierarchy to assess performance. Since energy-efficiency is at the heart of the ESCAPE project, this ratio is essential for estimating energy cost because moving data is so much more expensive than computing itself. Measurements and simulations of energy cost make it possible to identify sources of efficiency gains and to extrapolate energy cost estimates to scale.

For the Spherical Harmonics dwarf, for example, the project team investigated different memory management approaches using the Intel compiler suite on the Xeon Broadwell and Xeon Phi architecture. The impact of clock boosting and vectorisation was also considered. This dwarf is particularly relevant for the IFS as it relies on a spectral discretisation in space, which implies performing costly spectral transforms between spectral and grid-point space at every time step. In regional, limited-area model configurations of the IFS, spectral transforms consist of Fourier methods in two spatial dimensions (BiFourier Transform dwarf); in the global model, Fourier and Legendre transforms are combined (Spherical Harmonics dwarf).

Other code modifications focused on vectorisation, which aims to maximise the amount of variables to which the same operations are applied. The performance impact of these optimisations is shown in the chart below. While some of the initial transformations did not lead to a significant improvement in performance, optimisation no. 3 led to a performance boost of between 4% and close to 12% on a single core. This option combined vectorisation and memory layout optimisations.

ESCAPE optimisation results on Xeon for Spectral Transform–Spherical Harmonics dwarf

First optimisation results on Xeon for the Spectral Transform–Spherical Harmonics dwarf: Optimisations 1 and 2 represent vectorisation improvements and optimisation 3 data layout optimisation (data courtesy of P. Messmer, NVIDIA).

For the BiFourier Transform dwarf, the CPU investigations were also focused on vectorisations, and a 17.6% speed-up was achieved on an Intel Xeon Haswell-based node.

On the GPU architecture, investigations targeted an OpenACC implementation of the Spherical Harmonics dwarf. Profiles of the original implementation initially focused on the Fourier transform part of the algorithm, revealing two areas for optimisation: minimising data layout transformations, especially transpositions, as well as increasing the amount of calculations for the Fourier transforms. By changing the data layout and grouping the fast Fourier transforms (FFTs) into batches of the same size, an acceleration close to a factor of 2 was achieved.


ESCAPE GPU optimisation of Spherical Harmonics dwarf

GPU optimisation of the Spherical Harmonics dwarf with double-precision floating point (FP64) format (data courtesy of P. Messmer, NVIDIA).

To return the dwarf experience back into entire models, limited-area prediction models have been installed at ECMWF that will serve as reference standards for performance evaluations once selected dwarfs are reintegrated into the models. These reference installations also include the use of the Atlas library. This capability will eventually make it possible to gauge the impact of running optimised dwarfs on novel hardware in full-sized forecast systems.

Experience and results from the first half of the project have been shared through workshops and training. In August 2107, the ESCAPE Young Scientist Summer School in Copenhagen, Denmark, was attended by 28 students from 12 countries. Lecture materials are available on the ESCAPE website. ESCAPE also held its second dissemination workshop in September 2017 in Poznan, Poland, attended by some 40 researchers from Europe and the United States. Close collaborations have been established with two centres of excellence (ESiWACE and POP) and the EuroEXA FET project.

Looking ahead

The design and implementation of the dwarfs through a common, well-defined library is a major step towards re-structuring existing computational kernels for weather forecasting. The mid-term review report describes the dwarf kernels as building blocks for exascale-ready solutions, which will impact the whole weather forecasting community.

Since the review, the work of the ESCAPE consortium has focused on the upcoming release of the second batch of dwarfs, including two MPDATA dwarfs that perform advection calculations in the future IFS finite-volume model option, an elliptic solver and a radiation dwarf. Further, selected dwarfs have been ported to different architectures (GPU, KNL).

The Atlas library will be released to the public as open source early in 2018. In total, 12 deliverables have been produced at the end of 2017, of which 7 will be accessible publicly.

License for ESCAPE dwarfs available

posted 23 Nov 2017, 01:57 by Daniel Thiemert   [ updated 23 Nov 2017, 01:57 ]



ESCAPE is making part of its developments available to the scientific community by providing a free-of-charge license for the dwarfs developed within ESCAPE.

A numerical weather & climate prediction dwarf is a self-contained set of algorithms representing a key functional block of a forecast model. More specifically, a weather & climate prediction dwarf should encapsulate a relevant characteristic or required functionality of a weather and climate prediction model and it is meant to be a runnable and verifiable mini-application. The specific implementation of a dwarf in ESCAPE is referred to as prototype, where the prototype should represent a new concept or implementation of a weather and climate prediction model. Examples from this perspective are the specialization of a particular dwarf to different hardware (e.g. accelerators) or a new concept allowing the solution of a required functional block in a different manner (e.g. different advection schemes for the dynamical core).

More information on the license is available here.

2nd Dissemination Workshop - Agenda published

posted 16 Aug 2017, 02:38 by Daniel Thiemert   [ updated 16 Aug 2017, 02:38 ]




The second dissemination workshop organised by the ESCAPE project promises to share plenty of experience and results from the first 18 months of the project, including topics such as Weather and Climate Dwarfs, Diagnostics, Code Adaptation, and Hybrid Computing. The final agenda has now been published on the workshop page, and registration is still open until the 28th of August.

Deliverables, Presentations and Publications

posted 19 May 2017, 04:17 by Daniel Thiemert   [ updated 19 May 2017, 04:17 ]




ESCAPE has produced a number of documents in its first 18 months. All (public) deliverables, publications and presentations have now been updated on the website and are available for download. 

2nd Dissemination and Training Workshop

posted 19 May 2017, 04:06 by Daniel Thiemert   [ updated 19 May 2017, 04:07 ]

After the successful first 18 months of the project there is plenty of experience and results we would like to share with the community. The sessions of the 2nd ESCAPE Dissemination and Training Workshop, organised by PSNC in Poznan, Poland, will be organised according to the following topics, including oral presentations, keynote speakers and panel discussions:
- Weather & Climate Dwarfs – aspects and fundamental building blocks underlying weather and climate services;
- Diagnostics methods and tools, co-design and algorithms development for use in extreme-scale, HPC applications, minimising time- and cost-to-solution;
- Code adaptation and hybrid computing to exploit higher-level data structures, language abstractions, specialised accelerators and alternative technologies;
- Relevant EU projects involving HPC for weather, climate, ocean and sea, Earth System Modelling communities;
- Panel discussions on key issues from the workshop.
Details on the workshop will be published on the workshop page.

Summer School - Extended Deadline and Full Programme

posted 6 Apr 2017, 00:11 by Daniel Thiemert   [ updated 6 Apr 2017, 00:11 ]



The ESCAPE-organised Young Scientist Summer School 2017 has extended the deadline for applications to attend the Copenhagen event until the 1st of May 2017. In addition, the full programme is now available. You can find the details on the application process and the programme on the event page.

Training Material and Grant Schemes

posted 14 Mar 2017, 06:21 by Daniel Thiemert   [ updated 14 Mar 2017, 06:29 ]




Following the successful Undergraduate Training organised by ESCAPE partner PSNC in Poznan, Poland, a new section has been added to the ESCAPE website, focusing on Young Scientists. Here you can find the training material provided by ESCAPE partners related to High Performance Computing and Numerical Weather Prediction, as well as information on available grant schemes.

Undergraduate Training - Final Programme Online

posted 23 Jan 2017, 01:15 by Daniel Thiemert   [ updated 24 Jan 2017, 01:10 ]


The Undergraduate Training organised by ESCAPE will take place between the 27th and 28th February in Poznan, and can also be attended through online streaming for those unable to go to Poznan. It will cover aspects of Numerical Weather Prediction and High Performance Computing. The final programme has been published here and registration is still open.

Young Scientist Summer School 2017

posted 16 Jan 2017, 06:39 by Daniel Thiemert   [ updated 16 Jan 2017, 06:40 ]


ECMWF is organising a Young Scientist Summer School (YSSS) in August 2017. The YSSS will introduce the young generation of researchers in numerical weather prediction, ocean, climate relevant topics as well as modern computational physics & energy efficient computing. During the school, the participants will get familiar with ESCAPE dwarfs – fundamental algorithmic building block – and recognize why HPC and energy‐efficient computations are important. The programme will include theoretical lecture material as well as practical exercises (to be realized by students as small‐scale research projects started from the 1st day of the school with defence of project results) on selected dwarfs. For more information, and registration details, please go to the workshop page.

1-10 of 26