Technical Area: Unstructured Mesh
Unstructured meshes can yield required levels of accuracy using many fewer degrees of freedom at the cost of more complex data structures and algorithms to achieve parallel scalability and performant execution. Application developers often lack the time and/or expertise to develop the tools necessary to take full advantage of unstructured meshes in addressing their simulation needs. The goal of the FASTMath unstructured meshing team is the development of interoperable tools that application developers can employ within their software to create simulation workflows that take full advantage of unstructured mesh technologies. A key emphasis is performant execution on the latest heterogeneous supercomputers. Current FASTMath unstructured mesh development areas include:
High-order unstructured mesh PDE discretization methods
Unstructured mesh adaptation tools
Architecture aware task placement and load balancing
Unstructured meshes particle-in-cell technologies
Machine learning for unstructured mesh applications
The SciDAC Center for Integrated Simulation of Fusion Relevant RF Actuators is focused on the accurate FR simulation of fusion devices, such as the tokamak fitted with RF antenna as shown above. To effectively support these simulations FASTMath unstructured mesh technologies, including high-order finite elements methods (MFEM), needed to provide the high rates of convergence required, and curved mesh adaptation (MeshAdapt), to support mesh discretization control, are being extended and applied
Unstructured mesh PDE discretization methods:
The MFEM unstructured mesh code has been implementing high-order methods that are well suited for performant execution on up-coming GPU based exascale systems. Building on efforts to date for performant GPU execution, developments will address GPU based matrix-free solvers, preconditioners and error estimation. Discretization methods developments will include: discretization of electromagnetics in MFEM for the fusion SciDACs, advancement of the PETSc centric Landau form Fokker-Planck collision operator code, and continued advancement of PHASTA for complex flow problems.
Unstructured mesh adaptation tools:
FASTMath has developed tools for conforming and non-conforming mesh adaptation that are now applied in multiple SciDAC fusion and other DOE applications. Ongoing FASTMath developments are full functionality mesh adaptation on GPUs, GPU assembly, and high-order curved mesh adaptation. Efforts will be carried out to address these areas for both conforming and non-conforming mesh adaptation.
Architecture aware task placement and load balancing:
The architectures of DOE leadership-class systems, containing multi-GPU/CPU nodes with complex communication networks, necessitate novel partitioning and task placement for performant acceleration of DOE application codes. We are developing general purpose hierarchical approaches for architecture-aware partitioning, load balancing, and task placement, and appling them to a wide range of applications. Examples applications under development include task placement and load balancing in earth systems simulations in E3SM, dynamic load balancing adaptive analysis workflows for fusion applications, and mesh and particle load balancing in the mesh-based PIC applications.
Unstructured mesh particle-in-cell (PIC) technologies:
FASTMath has defined a new approach to unstructured mesh PIC calculations targeted to support fully parallel (both particle and mesh) PIC applications. Currently targeted SciDAC fusion applications include the XGC edge plasma and GITR impurity transport codes. We will continue to advance the core infrastructure, PUMIPic [24], to be performant on the upcoming exascale systems, support dynamic particle and mesh load balancing and additional mesh/particle operations, and to couple with mesh adaptation.
Machine learning for unstructured mesh applications:
ML/AI methods have the potential to accelerate simulations, support smart simulation data management and enable in situ simulation steering to provide increased insights. Physics-informed graph-based neural network methods to support application’s machine learning needs are being investigated. ML methods will be extended to support smart in situ data compression and feature extraction. The resulting procedures will be applied to CFD applications and PIC based fusion applications.
In loose coupling of earth system components, geometric partitions like these Recursive Coordinate Bisection partitions from the Zoltan library enable efficient search for mesh intersections. Coupling the atmosphere (right) and ocean (left) partitions (e.g., by inferring the atmosphere’s partition from the ocean’s) further reduces mesh intersection time by assigning both atmosphere and ocean elements from a single region of the globe to a processor.