ETP4HPC SRA, Multi-dimensional HPC vision: Since October 2014, ECMWF has assumed full membership status of the ETP4HPC as the first NWP centre as an effort to generate a concerted European approach to produce sustainable excellence in HPC for NWP. ESCAPE is the first project aiming to address specific aspects of the ETP4HPC research agenda and their realisation in numerical fine-scale modelling of the atmosphere. Regarding the ETP4HPC’s multi-dimensional HPC-vision these are:          

• HPC stack elements: ESCAPE aims at filling the software gap between complex hardware and complex applications through its focus on advancing energy efficient algorithmic building blocks optimized for data flow, data locality and communication patterns across processors. These blocks will form the basis for designing new spectral transform kernels, solvers, meshes, grids, time stepping, and multi-scale selective algorithms and data layouts required by future NWP. While portability for top-level kernels is of high priority, the potential for offloading selected components to specialized hardware will be established.  
• Extreme scale requirements: Enhancing energy efficiency is a prime target for ESCAPE since operational NWP imposes strict runtime requirements on product delivery with direct implications on FLOP/second rates. As energy efficiency and resilience are related, the algorithmic methods developed in ESCAPE aim at enhancing numerical accuracy and stability, and by assessing the reliability of heterogeneous hardware options employed for specific computing tasks.
• New HPC developments: Developments for facilitating front-end access to HPC computing are not targeted by ESCAPE.      
• HPC usage expansion: ECMWF as a European member state organisation, a centralized HPC facility and through its tight link to regional consortia and downstream services assumes a central responsibility in (i) enhancing the effectiveness of access and migration to future hardware and (ii) preparing the next generation of weather research scientists facing future HPC challenges. The development of shared and portable code libraries is at the heart of ESCAPE and thus all associated organisations will benefit. ESCAPE will also allow the extension of the existing numerical aspects training agenda towards exascale computing science, facilitated by the already existing openly accessible OpenIFS code[3].

ETP4HPC SRA, Technical research priorities: At the system architecture level, ESCAPE investigates selected options for specialized compute units but does not contribute directly to hardware development. Based on existing technology, the offloading of compute-intensive tasks, e.g. microphysics parameterizations and radiation, to low-energy-cost NVIDIA GPUs (GPU ensembles) and Intel MICs will be established in partnership with HPC centres and vendors. Different intrusion levels of code adaptation and their effect on performance will be benchmarked. An innovative and entirely new approach to perform spectral transforms and large matrix operations, which are major contributors to computational cost in NWP, will be exploited through low-energy optical processing of such operations at the speed of light in partnership with partner OSYS. ESCAPE will perform the interfacing of such processors with existing hardware and the programming environment, not hardware design. This ESCAPE model for employing specialized hardware is applicable to the majority of codes used in weather and climate prediction and thus has community wide impact.             

ESCAPE only contributes indirectly to the programming environment by assessing the effectiveness of compiler directives and domain-specific languages for enhancing computing and energy efficiency in key algorithmic components. The collaboration with vendors (and NVIDIA in particular) introduces an interface to compiler design in support of hardware abstraction towards a higher management level of data locality and concurrency.

Enhancing energy efficiency in weather and climate prediction is essential when approaching kilometre-scale simulations at global scale and under operational time constraints. ESCAPE combines a paradigm change for NWP relevant algorithms (Weather & Climate Dwarfs) with a concept for employing specialized hardware in a heterogeneous environment for dedicated tasks dealing with the resolved flow (model dynamics) and unresolved processes (physical parameterizations). New approaches to enhancing energy efficiency represent the core of ESCAPE. ESCAPE proposes the development of novel techniques for minimizing communication, enhancing data locality and for overlapping of computation and communication. Key is to accommodate the diverse set of numerical operations performed, the large dimension of prognostic variables integrated per model time step and the complex dependencies in the internal workflow of NWP models. At the same time, code portability and readability requirements are addressed. In ESCAPE, performance models that simulate the most cost-intensive components in terms of memory usage, computing and communication will be developed for both global as well as regional modelling. These performance models will achieve community-wide applicability by providing a detailed quantification of efficiency gains obtained from algorithmic changes as well as compiler directive and domain-specific language implementations.

Accelerator usage will target the identified compute-intensive tasks, with optical processors specifically targeting spectral transforms and matrix-matrix multiplications, thus offloading the most compute-intensive and costly elements to low-energy hardware. The optimal integration of these units into the overall architecture and workflow will be assessed based on the above described performance models.

ETP4HPC SRA, Completing the value chain: ECMWF combines advanced research and operational application which benefits both the application and service layers spanned by ECMWF (including Copernicus services), its member states and ESCAPE project partners as they represent a significant portion of the European weather forecasting community. The push-through of the envisaged ESCAPE developments follows the same impact route. While the ETP4HPC SRA focuses its recommendations on the industrial impact, a similar value-chain template applies to environmental application and service provision.