Hybrid Automation of Numerical Weather Prediction Across DSRC and IL5 Clouds
Improving automation processes for complex numerical weather prediction (NWP) workloads is critical for advancing next generation forecasting capabilities and mission critical decision-making within Defense environments. While current weather forecasting systems already incorporate significant automation, ongoing efforts seek further enhancements in flexibility, robustness, and efficiency. In this presentation we will detail recent advancements and practical experiences in automating the development workflow for the NEPTUNE atmospheric model from the Naval Research Laboratory (NRL).

The presented automation approach leverages the HPCMP High Security Platform (HSP) for securely orchestrating tasks across Defense Supercomputing Resource Centers (DSRCs) and approved IL5 cloud environments. Employing the HSP REST API with secure and compliant short-term tokens, the team established automated workflows for data transmissions between the DSRCs and the IL5 cloud platforms, and has begun exploring comprehensive continuous integration topologies capable of dynamically allocating and managing computational tasks across hybrid infrastructures.

The automation improvements aim to ensure computational fluidity and operational continuity, taking strides to proactively redistribute workloads as necessary. Automated data management further enhances the NWP research, development, and transitions to operations. Comprehensive continuous integration testing capabilities, from unit testing to end-to-end regression testing, on the cloud improves the robustness of and the confidence in the numerical weather prediction suites developed at NRL, key factors for efficient and timely transitions to operations. 

Observability of resource allocation and usage is another crucial feature highlighted, providing precise accountability and detailed project-level chargeback. Such capabilities are essential for efficiently managing resource-intensive, project-driven research environments typical within organizations like NRL.

This presentation will finally draw practical parallels from similar successful ensemble automation efforts in the weather domains. Attendees will gain insights into overcoming challenges related to hybrid computational orchestration, secure data handling, and workflow robustness. The discussion will help inform and guide DoD researchers and practitioners in enhancing their own computational ensemble operations.
IMPACT
Accomplishment: Automated continuous integration (CI) testing and secure data management for NRL’s NEPTUNE weather model using HPCMP’s High Security Platform (HSP); Result: Increased forecasting model robustness and accelerated transition-to-operation timelines, enhancing weather situational awareness for defense missions.
PRESENTER
Shaxted, Matthew
shaxted@parallelworks.com
847-254-0230
Parallel Works
CO-AUTHOR(S)
Long, Matt
mlong@parallelworks.com
Gary, Dr. Stefan
sfgary@parallelworks.com
Heinzeller, Dr. Dominikus
heinzell@ucar.edu
Reinecke, Dr. Alex
patrick.a.reinecke.civ@us.navy.mil
CATEGORY
Weather &  Ocean Modeling & Sim
SECONDARY CATEGORY
Mod, Sim & Analysis for Decision Making
SYSTEM(S) USED
Nautilus, IL5 AWS on HSP