Abstract

Current trends in high performance computing include the use of graphics processing units (GPUs) as massively parallel co-processors to CPUs that can accelerate numerical operations common to computational fluid dynamics (CFD) solvers. GPU-parallel CFD achieves speedups with CPUs from additional fine grain, or second-level parallelism under existing CPU-based distributed memory, or first-level scalable parallelism. For most CFD implementations, the GPU focus is on implicit sparse iterative solvers whereby linear algebra matrix operations that would be processed on the CPU are offloaded to the GPU for numerical acceleration, resulting in an overall simulation speedup. During 2019, the OpenCFD OpenFOAM HPC Technical Committee introduced the PETSc4FOAM library that permits the plug-in of external solvers that conform to PETSc formats. A collaboration among members of the HPC TC developed an external solver for GPU offload of OpenFOAM pressure solves based on the Open Source AmgX solver library that was first introduced in 2012. In the initial implementation, the OpenFOAM system matrix is copied from the CPU to GPU, and the AmgX library applies an AMG preconditioner to a preconditioned conjugate gradient (PCG) linear solve for acceleration of the pressure solve. Results are copied back to the CPU which completes the OpenFOAM simulation on the CPU as the end-user normally observes. This work will preview details of the AmgX development and its integration with OpenFOAM for multi-GPU and multi-node computations. Initial experiments with the standard benchmarks of (i) 3d lid-driven cavity and (ii) motorbike cases demonstrate that AmgX can achieve as much as a 9x speedup of the pressure solve on a the GPU vs. an OpenFOAM GAMG-PCG solve on a dual-CPU server node. With this OpenCFD community-supported solution, frequent software updates made to community-based libraries like PETSc, AmgX, and PETSc4FOAM will ensure that OpenFOAM users naturally benefit from the latest system software, compilers, system and processor hardware architectures, and OpenFOAM future releases. Future work will be described which will include investigations of more complex geometries and turbulence treatment such as LES. In addition the TC plans parallel scaling optimizations for strong scaling across computational nodes of multiple GPUs, and to explore the potential for more OpenFOAM code applied to GPU acceleration such as matrix assembly procedures. The overall objective of the TC collaboration and current and future contributions is towards a community supported GPU-enabled OpenFOAM for CFD practice at industry scale.