Automated HPC Batch Post-Processing with the Visulization Pipeline (Vispipe)

One of the advantages of using High Performance Computers is the ability to simultaneously run tens, hundreds, or even thousands of numerical model simulations at one time. This allows for researchers to test many combinations of variables and starting conditions in a relatively short amount of time. This produces a large amount of data which is often not in an easily readable or understandable format. Often, the data requires additional processing to visualize the results for analysis. This becomes tedious if the model does not have built-in visualization capabilities, and even models with those capabilities often provide limited options in terms of image customization.

The newly-developed Visualization Pipeline (Vispipe) offers a solution to this problem. Vispipe is a Python 3-based batch visualization program that automates the visualization of simulation data. Developed for use with the Coastal Storm Modeling System (CSTORM-MS) utilized by the U.S. Army Corps of Engineers’ Coastal Hydraulics Laboratory to model coastal surge and wave responses under extreme events, Vispipe allows for significant post-processing image customization but is simple to set up and execute. The run configuration files are capable of passing options directly to the plotting functions, making it easy to customize individual plots. They can also be adapted as templates for reuse across multiple simulations. Additionally, it takes advantage of the HPC environment by parallelizing all plotting jobs. It uses open-source packages and is structured into swapable modules. This allows the user to add in their own plotting functions, which allows for Vispipe to be easily implemented into existing workflows. Vispipe provides a powerful asset to researchers by reducing the time and effort required to post-process and create imagery for large datasets. In this presentation, a basic overview of the functionality of Vispipe will be provided, along with examples of how it is currently being applied within the CSTORM modeling production system.

PRESENTER

Gramm, Joshua
joshua.d.gramm@usace.army.mil
817-851-8765

US Army Corp of Engineers

CATEGORY

Big Data / Visualization

SYSTEMS USED

Carpenter, Jim, Mike

SECRET

No