Efficient lossy compressor design for unsteady CFD data on distributed meshes
High fidelity unsteady computational fluid dynamics (CFD) simulations generate extremely large datasets would ideally be preserved completely for machine learning model development, visualization, and data assimilation. However, limited disk space and filesystem bandwidth constraints typically force users save only small data subsets, resulting in significant loss of valuable flow physics information. To address this challenge, a novel lossy compression approach is presented that substantially reduces dataset size while increasing disk write frequency. This method employs a delta-key frame technique that leverages temporal coherence between snapshots to minimize errors. Key frames are stored in reduced precision with lossless compression, while delta frames utilize state-of-the-art lossy compression algorithms. Preliminary results demonstrate that this approach achieves both high compression ratios and significant error reduction while maintaining exceptional encoding and decoding performance.
IMPACT
Accomplishment: Developed a new algorithm for efficiently compressing block structured CFD data on DoD HPC systems while incurring minimal error (~0.1%). Result: Significant (~100x) reduction in data storage on HPC systems allowing for more efficient usage of DoD disk resources
PRESENTER
Hoy, Jonathan
jonathan.hoy.5@us.af.mil
626-826-6474
Air Force Research Laboratory
CATEGORY
Big Data
SECONDARY CATEGORY
Comp Fluid Dynamics
SYSTEM(S) USED
Carpenter