4-9 September 2022, Bonn, Germany
Enter zoom meeting

OSA1.10

From HPC to Cloud and Edge computing in Meteorology

Cloud computing has emerged as the dominant paradigm, supporting practically all industrial applications and a significant number of academic and research projects. Since its introduction in early 2010s and its widespread adoption thereafter, migration to the cloud computing has been a considerable task for many organizations and companies. Meteorology is no exception, there is a great diversity in the adoption of cloud computing and related technologies like fog/edge computing, microservices etc. The processing of big meteorological data close to their physical location is a perfect use case for cloud technologies and cloud storage infrastructures which offer all the necessary infrastructure and tools, especially if cloud infrastructures are offered together with HPC resources. European Weather cloud, coordinated by ECMWF and EUMETSAT, is one of the notable cloud computing infrastructures following this paradigm and is provided to the meteorological services of these Organization’s Member and Co-operating States. Moreover, in DestinE, the new EC coordinated initiative implemented by ESA, ECMWF and EUMETSAT is envisaged that most of the processing workloads will be across different cloud/HPC facilities and distributed cloud-based storage infrastructures will be hosting the produced and stored data.
This session focuses on Cloud/Fog/Edge computing use cases in meteorology (in combination with HPC) and aims to identify the current status and the steps ahead towards a wider cloud computing adoption in Meteorology.
We encourage contributions describing all kinds of Cloud/Fog/Edge computing efforts in the Meteorological domain, such as (but not limited to):
• Cloud Applications, Infrastructure and Platforms (IaaS, PaaS SaaS and XaaS).
• Cloud federations and cross domain integrations.
• Service-Oriented Architecture in Cloud Computing
• Cloud Storage, File Systems, Big Data storage and Management.
• Networks within Cloud systems, the Storage Area, and to the outside
• Virtualization in the Context of Cloud Computing Platforms
• Fog and Edge Computing
• Operational systems on the cloud.
• Big Data processing use cases, techniques, models, and algorithms on the cloud.
• Big Data Infrastructures and platforms
• Data lakes and warehouses on the cloud.
• Machine Learning Techniques for Edge, Fog, and Cloud.
• Cloud computing and HPC convergence in Meteorology
• Edge/Fog/Cloud computing and HPC workload unification.

Convener: Vasileios Baousis | Co-conveners: Umberto Modigliani, Mihai Alexe, Charalampos Kominos, Xavier Abellan, Roberto Cuccu
Orals
| Thu, 08 Sep, 11:00–13:00 (CEST)|Room HS 3-4

Thu, 8 Sep, 11:00–13:00

Chairpersons: Vasileios Baousis, Roberto Cuccu, Umberto Modigliani

11:00–11:15
|
EMS2022-16
|
Onsite presentation
Roope Tervo et al.

The European Weather Cloud (EWC) is set to be the cloud-based collaboration platform for meteorological application development and operations in Europe and enables the digital transformation of the European Meteorological Infrastructure.

It consists of cloud infrastructure established by the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) and the European Centre for Medium-Range Weather Forecasts (ECMWF) but is also open to federation partners with relevant data or infrastructure assets. All resources in EWC are specifically designed to have fast access to the EUMETSAT and ECMWF data holdings.

Major data accessible by the EWC users is the sum of all online data products available at ECMWF and EUMETSAT, accessible in a seamless manner across the boundaries of their respective cloud infrastructures. Services to access the data include the basic data access services supported by related functions, such as display, reformat, etc., as per the applicable data and service policies and already available at different levels at EUMETSAT and ECMWF. The data offering will be augmented over time based on user needs in line with the aspiration of the EWC driven by meteorological applications development. Where relevant, data federation agreements will be sought with federation partners for third-party data access in line with the basic architecture of the system.

From a technological viewpoint, EUMETSAT and ECMWF seek to offer services that carry the highest benefits from cloud technology by exploiting the full potential of hosting a given processing function, a project, or a service close to a large variety of readily available data. The hosted services are augmented with the Software Marketplace, which allows EWC users to easily share and use meteorological applications and algorithms.

The EWC is available for EUMETSAT and ECWMF Member States activities. Resources will also be allocated to research initiatives via specific EUMETSAT Research and Development calls and ECMWF Special Projects, experiments, or investigations of a scientific or technical nature, undertaken by one or more EUMETSAT or ECMWF Member States, likely to be of interest to the general scientific community.

EWC has reached the end of the pilot phase at the end of 2021 and is currently in 'operational ramp-up'. During the pilot phase, the EWC hosted over 40 different types of use cases containing, for example, data processing, application development, training, and experimenting with cloud technologies. The target is to reach a fully operative state by the end of September 2022. After launching the operational service, EWC looks forward to developing the service towards Platform as a Service (PaaS) based on user feedback.

How to cite: Tervo, R., Schulz, J., Saalmueller, J., Abellan, X., Modigliani, U., and Baousis, V.: The European Weather Cloud (EWC) – Collaboration Platform for Meteorological Application Development and Data Processing, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-16, https://doi.org/10.5194/ems2022-16, 2022.

11:15–11:30
|
EMS2022-26
|
Onsite presentation
Nils Wedi et al.

The talk describes ongoing efforts to create digital replicas of the Earth as part of the the European Commission's Destination Earth programme.

Global, coupled storm-resolving simulations are feasible and can contribute to building such information systems and are no longer a dream thanks to recent advances in Earth system modelling, supercomputing and the adaptation of weather and climate codes for novel computing architectures. Such simulations for example explicitly represent essential climate processes, such as deep convection and mesoscale ocean eddies, that today need to be parametrised even at the highest resolution used in global weather and climate information production. These simulations, combined with novel data-driven deep learning advances, thus offer a window into the future, with a promise to significantly increase the realism of Earth system information. Despite the significant compute and data challenges, there is a real prospect to better support global to local climate change mitigation and adaptation efforts, and complement the existing information derived with today's operational simulations in the range of 10-100 km.

Digital Twins of Earth thus encapsulate both the latest science as well as technology advances to provide near-real time information on Extremes and climate change in a wider digital environment. Here users can interact, modify and ultimately create their own tailored information. This is facilitated through complex workflows managed by ECMWF's digital twin engine that closely connects EuroHPC resources for the production of digital twin data, manages diverse data access patterns through cloud-based ancilliary systems such as Eumetsat's Data Lake, and provides a diverse range of tools to faciliate user interaction and data-driven applications creating new information through ESA's user service platform. The underlying system architecture and design choices will be described and justified.

How to cite: Wedi, N., Quintino, T., Modigliani, U., Baousis, V., Geenen, T., Sandu, I., Bauer, P., Hoffmann, J., and Thiemert, D.: Destination Earth: Digital Twins of the Earth System, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-26, https://doi.org/10.5194/ems2022-26, 2022.

11:30–11:45
|
EMS2022-28
|
Online presentation
Paraskevi Vourlioti et al.

To promote cloud and HPC computing, GRAPEVINE* project objectives include using these tools along with open data sources to provide a reusable IT service. In this service a predictive model based on Machine learning (ML) techniques is created with the aim of preventing and controlling grape vine diseases in the wine cultivation sector. Aside from the predictive ML, meteorological forecasts are crucial input to train the ML models and on a second step to be used as input for the operational prediction of grapevine diseases. To this end, the Weather and Research Forecasting model (WRF) has been deployed in CESGA’s HPC infrastructure to produce medium-range and sub-seasonal forecasts for the targeted pilot areas (Greece and Spain). The data assimilation component of WRF – WRFDA-   has been also introduced for improving the initial conditions of the WRF model by blending in observations from weather stations and satellite precipitation products (Integrated Multi-satellitE Retrieval for GPM – [IMERG]). The operational production of the aforementioned forecasts is achieved by the cloudify orchestrator on a Kubernetes cluster. The connection between the Kubernetes cluster and the HPC infrastructure, where the model resides, is achieved with the croupier plugin of cloudify. Blueprints that encapsule the workflows of the meteorological model and its dependencies were created. The instances of the blueprints (deployments) were created automatically to produce operationally weather forecasts and they were made available to the ML models via a thredds server.  Valuable lessons were learned with regards the automation of the process and the coupling  with the HPC in terms of reservations and operational production.  

*hiGh performAnce comPuting seRvices for preVention and coNtrol of pEsts in fruit crops

How to cite: Vourlioti, P., Kotsopoulos, S., Mamouka, T., Agrafiotis, A., Sánchez, C. F., Llerena, C. G., Nieto, F. J., and García, S.: GRAPEVINE project: Operational weather forecasting in HPC managed by an orchestrator in Kubernetes cluster. , EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-28, https://doi.org/10.5194/ems2022-28, 2022.

11:45–12:00
|
EMS2022-98
|
CC
|
Onsite presentation
Florian Prill and Christian Eser

The ICON model, which is developed by the Deutscher Wetterdienst (DWD) and its partners, represents an example for modern NWP models with their intricate structure of numerous subcomponents. Besides, parallel processing is massively needed, where workloads often change in the short term and the dynamic requirements exceed the available local hardware. This increasingly complex setup and software stack is contrasted with the limited staff resources of small NMHSs or university groups. Even large developer communities, such as that of the ICON model, do not have the necessary support capacity for this situation. During the past decade two key answers to these problems have been identified: container deployment and cloud service models.
The ICONIC project, developed at DWD and financially supported by the World Bank, addresses the use case of the Central Asian region. Its technical concept employs user-friendly,  auto-scaling Slurm clusters and Singularity containers for the ICON model. The project results confirm that flexible short-term NWP workload demands can be handled by IaaS service models. We show by example that commercial and community cloud infrastructure is sufficiently capable for small and medium-size applications in a cost-effective scenario. The ICONIC setup is integrated into the data supply of the DWD weather service and takes into account a separation of the roles of administrators, operators, and forecasters. Portability between on premise software deployment community clouds and several commercial cloud providers is shown under the condition of minimal hardware requirements. The outlook discusses the impact of this deployment model on current operational centers.

How to cite: Prill, F. and Eser, C.: Deploying the ICON model on cloud architectures (ICONIC for Central Asia), EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-98, https://doi.org/10.5194/ems2022-98, 2022.

12:00–12:15
|
EMS2022-183
|
Onsite presentation
Jacob Weismann Poulsen et al.

Cloud computing provides on-demand availability of all computer system resources and is, consequently, of  
general interest to National Meteorological and Hydrological Services. However, how does one get started and is true HPC a real option in the cloud?"
In connection with the work on a forecast benchmark for United Weather Centre-West project and the
upgrade of the operational forecast system at the Danish Meteorological Institute (DMI), both of
which are based on the Harmonie-arome 43h2.1 released by the Hirlam-C consortium, we ported and
tested the Harmonie forecast system on the Amazon Web Services (AWS), a public cloud. In these explorations, we
tested both a stand-alone benchmark configuration and a full system configuration. The latter includes
boundary preparation and observation pre-processing, data assimilation and forecast cycling, and the
subsequent postprocessing; i.e. execution of a full NWP cycling dataflow with the required typical
suite of applications using the Harmonie scripting system.
We will share our experiences of porting the Harmonie-Arome weather forecasting system to AWS.There are both differences and the similarities between running
NWP on-premise versus running it on AWS and we will first focus on describing these. Then we will cross-compare
the performance obtained on the on-premise systems versus the ones obtained on AWS. Finally we will describe
our approach to integrate the on-premise clusters with AWS clusters. The approach will be using layered builds
of AMIs (Amazon Machine Images). Moreover, these builds will be launched in a fully automatic fashion using Git
CI/CD pipelines. This approach has the potential to have 24/7 NWP production systems running on our on-premise clusters that
will have AWS sitting in the back 24/7 both as a fallback option and as a bursting option to our on-premise
production. 

How to cite: Weismann Poulsen, J., Yang, X., Whelan, E., and Raman, K.: Harmonie-Arome on AWS, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-183, https://doi.org/10.5194/ems2022-183, 2022.

12:15–12:30
|
EMS2022-323
|
CC
|
Onsite presentation
Open Innovation and Open Development of the Unified Forecast System - An EPIC Approach
(withdrawn)
Maoyi Huang
12:30–12:45
|
EMS2022-466
|
Onsite presentation
Tristan Carion et al.

For both military and civilian purposes, the assessment of the impact of a CBRN agent release is crucial. To assess the area of contamination of an agent, atmospheric dispersion models can be used. A web application for assessing the impact of CBRN-type incidents is currently being developed in the frame of a joint project of the Royal Military Academy of Belgium, the Royal Meteorological Institute of Belgium and ECMWF. It is hosted on the European Weather Cloud system, benefiting from the closeness to the ECMWF meteorological data and from the computing power. The application is designed to provide both deterministic and ensemble dispersion modelling capabilities for any worldwide hazardous release. 

The dispersion model programs are wrapped in Julia, a modern technical computing language that runs in the backend of the web application. The Julia package manager offers an efficient and easy way to port third-party libraries, avoiding the sometimes painful installation of scientific softwares. For example, the Lagrangian dispersion model FLEXPART can be installed with a one-liner, which makes it very easy to use on the cloud once Julia is installed.

The main objective of the application is to provide a quick and reliable decision-making input for non-experts in dispersion modelling in case of CBRN-incidents. Response models will also be implemented using event driven simulation to account for external data (population density, topography etc.). Besides, the application can also be used as a user-friendly interface to multiple dispersion models, making it easier to run, visualize and execute simulations for more experienced users.

How to cite: Carion, T., Jassens, B., and Delcloo, A.: Atmospheric dispersion modelling in the cloud , EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-466, https://doi.org/10.5194/ems2022-466, 2022.

12:45–13:00
|
EMS2022-694
|
Online presentation
Mikko Partio
Finnish Meteorological Institute (FMI) provides weather forecasts for a variety of users, including aviation, agriculture, media and the general public. FMI spends significant resources to prepare meteorological data and products to suit our specific needs, dictacted by our geography and culture. Our data-oriented production system forces us to constantly develop novel methods to keep up with the increasing data volume of NWP. Like many other organisations FMI started to move towards cloud computing in the first half of 2010s, a process which is still continuing today. Cloud computing has given us new tools to handle the increasing data volume. 
 
While we initially started with a “virtual machines in the Cloud” approach, a more holistic adaption to modern, cloud-native technologies such as S3, Kubernetes and AWS Lambda has enabled us to expand our operations efficiently and flexibly. While we are not a cloud-first organization, our on-premise computing infrastructure has also been upgraded to match the common cloud technologies. This streamlines the operating environment as same software stacks can be used on-prem and in the cloud. Using containers is increasingly important as it makes easier to move code between systems. Containers offer natural isolation from operating system upgrades, which is important when pace of development is high and software depends on specific library versions.
 
A big challenge still facing us is the support of software that is not easily translated to cloud workflows. In the meteorological domain this is rather common as software has long lifetime and code refactoring is in many cases not feasible. We must also ensure that our users and developers maintain their skills and knowledge as new technologies come and go.

How to cite: Partio, M.: From virtual machines to serverless: NWP Postprocessing at Finnish Meteorological Institute, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-694, https://doi.org/10.5194/ems2022-694, 2022.

Posters

Supporters & sponsors