Scaling Scientific Analysis: Application of Workflow Management Tools on HPC's

Delivering solutions to address the complex problems encountered in science and engineering requires significant development and consideration of methods to automate and scale end-to-end analysis pipelines and workflows. Traditional methods of managing scientific workflows for High Performance Computing (HPC) environments have become increasingly difficult to implement in response to the growing number of heterogenous compute resources, availability of data, and ongoing advancements of simulation and data analysis tools. Workflow managers provide a high-level interface between system resource management and available programming models to abstract the process of defining computational tasks and how they should be executed. These interfaces and tools have become important components of recent scientific discoveries, demonstrating the ability to accelerate the time-to-solution while preserving data provenance and ensuring reproducible results. In this presentation, we will discuss three such interfaces: Galaxy Simulation Builder, Maestro, and Parsl, and how each can be used to accelerate the development, management, and execution of scientific workflows. To demonstrate the applicability of these tools to the broader HPCMP community and to provide a comparative evaluation between them, the development of three workflows related to modeling/simulation, optimization and machine learning will be discussed.

PRESENTER

Simin, Andrew
andrew.simin@gdit.com
201-888-1797

HPCMP PET / General Dynamics Information Technology

CATEGORY

Other: HPC Workflow Management

SYSTEMS USED

Narwhal, Onyx, Koehr

SECRET

No