Abstract

The processing of big data from large-scale CFD simulations is a challenges task. On current HPC systems, one observes an increasing gap between the amount of data that is generated during simulations and the amount that can be stored to disk, in particular due to limitations of the data I/O system. For this reason, there is a recent trend for in-situ analyses where the data is processed during the run-time of the simulation and only the analysis results are stored. Furthermore, the extraction of physical and engineering knowledge from highly-resolved simulations leads to a need for new algorithms. While traditional post-processing techniques based on scalar or integral quantities can capture only specific aspects of the flow, the simulation usually contains much more information that can be exploited. Data-driven techniques, e.g. based on Machine Learning (ML), have the potential to extract global features, construct surrogates or find patterns in the data that can be employed in the subsequent scientific and engineering tasks. In this work, we introduce an in-situ data analysis framework with the goal to support both, a flexible application and efficient parallel execution of new algorithms during the development and evaluation process. As most existing Machine Learning libraries provide a Python API and allow an interactive usage through ipython servers, this is also a fundamental design strategy for our software. Therefore, the core API of our framework follows the definition of well-known ML libraries and enhances a simple integration of our code with external methods into a single processing pipeline. By connecting to established in-situ interfaces known from the visualization community, e.g. Catalyst, our code becomes non-intrusive and independent of particular CFD solvers. To meet the goal of an efficient execution, we keep the python layer as thin as possible on the one hand and build on existing math libraries, such as Elemental, for heavy computations on the other hand. To demonstrate the functionality of our framework, a Principal Component Analysis (PCA) of the turbulent velocity field of an industrial relevant use-case, the HVAC duct from an automotive set up, is computed during the simulation based on a batch processing approach. As a result, a significant reduction of data in- and output during and after the simulation can be achieved. A further benefit of the demonstrated in-situ approach lies in earlier availability of the results. This can be exploited by investigating intermediate results during the simulation. A monitoring of the simulation is demonstrated to find early trends in the results by comparing them to previous simulations runs. Based on this, the overall computing time can be reduced by aborting simulations that show non-physical or unwanted behavior.