In Situ processing lets researchers preview results of long run exascale simulations as they are running

by

The SENSEI in situ processing project takes aim at a set of research challenges for enabling scientific knowledge discovery within the context of in situ processing at extreme-scale concurrency. This work is motivated by a widening gap between FLOPs and I/O capacity which will make full-resolution, I/O-intensive post hoc analysis prohibitively expensive, if not impossible.

They focus on new algorithms for analysis, and visualization – topological, geometric, statistical analysis, flow field analysis, pattern detection and matching – suitable for use in an in situ context aimed specifically at enabling scientific knowledge discovery in several exemplar application areas of importance to DOE.

Complementary to the in situ algorithmic work, they focus on several leading in situ infrastructures, and tackle research questions germane to enabling new algorithms to run at scale across a diversity of existing in situ implementations.

The intent is to move the field of in situ processing in a direction where it may ultimately be possible to write an algorithm once, then have it execute in one of several different in situ software implementations. The combination of algorithmic and infrastructure work is grounded in direct interactions with specific application code teams, all of which are engaged in their own R and D aimed at evolving to the exascale.

In situ processing ability is to analyze simulation output while it is still resident in memory. This is appealing and sometimes necessary for researchers running large-scale simulations.

The team recently completed a study, to be presented at Supercomputing later this month, evaluating the performance impacts of running SENSEI alongside simulations. The largest scale simulation was a 1,048,576 MPI rank fluid dynamics simulation using PHASTA. This is believed to be the largest in situ MPI simulation ever performed.

“This is part of the exascale initiative,” project member Andrew Bauer of Kitware tells The Next Platform, “trying to get more accurate analysis and visualization, trying to get more scientific insight or more engineering insight into the types of problems that we’re trying to solve.”