MOAB + Applications:
Connecting, Data Sharing, etc.
Tim Tautges
Mathematics & Computer Science Division
Argonne National Laboratory
Outline
●
MOAB primer
●
Connecting to applications: Nek5000, UNIC
●Data sharing
Mesh-Oriented datABase (MOAB)
●
Library for representing, manipulating structured, unstructured
mesh models
●
Supported mesh types:
– FE zoo (vertices, edges, tri, quad, tet, pyramid, wedge, knife, hex) – Polygons/polyhedra
– Structured mesh
●
Optimized for memory usage first, speed second
●
Implemented in C++, but uses array-based storage model
– Avoids C++ object-based allocation/deallocation– Allows access in contiguous arrays of data ●
Mostly an ITAPS-like data model
– Entity, entity set, tag, interface ●
Mesh I/O from/to various formats
– HDF5 (custom), vtk, CCMIO (Star CD/CCM+), Abaqus, CGM, Exodus ●
Main parts:
– Core representation
4
MOAB Parallel Model
•
Each proc has one MOAB instance, which appears locally as a serial
mesh
• All serial MOAB function calls available locally
•
Parallel model based on element-based partition
• Each element contained in exactly one part (entity set), with vertices (& possibly elements) shared between processes but owned by one
• Arbitrary number of layers of ghost elements
•
Supported parallel mesh constructs:
• For each shared entity, every sharing proc knows all other sharing procs & handles on those
• Sharing data stored as either (proc,handle) or (procs[n]/handles[n])
• Shared/multishared/interface/owned status also stored (bitmask)
•
Parallel model usually initialized by loading from some decomposition
in file
• Can be any subset structure that’s a “covering” (each entity in exactly 1 subset)
• Material set, geometric volume, or Zoltan-generated partitioning
•
Single-file parallel read/write using parallel HDF5
MOAB Entity Storage
Entity Handle:
• Unsigned long type • Bitmask
• Sorts by dimension,
type 4-bit type 28-or 60-bit ID
EntitySequences:
Represent used portions of handle space Have pointer to SequenceData
Have start and end handle values Arranged in binary tree by start handle
Typically one
EntitySequence for an entire SequenceData
SequenceData:
Represent allocated
portions of handle space
Have start and end handle Coordinates or Connectivity Dense Tag Data
Connectivity array
6
Two Example Applications
Nek5000
UNIC
Physics
Incompressible NS
Neutron transport
Discretization
SEM
(NxNxN GLL basis)
FEM
(linear, quadratic)
Solver
Native semi-implicit
with AMG
3-level hierarchy
(power, energy, space/angle),
with PETSc for space/angle
Materials, BCs
User-defined
functions
ExodusII-like element blocks,
sidesets
Mesh type
Ucd hex
Ucd hex, tet, prism
Implementation
F77 + C
F90
Mesh, data storage
Common blocks
'include TOTAL'
F90 modules
●
Basic steps for MOAB-based execution:
– Import verts & elements, material types, boundary conditions – [Nek: Compute GLL points]
– Copy state vars to MOAB @ each timestep
Ordering Issues
● partition generated w/o regard to
material types
● element, set order from file, ghost
order from ghost exchange
● fluid elements first, then solid
elements
● one contiguous index space
● distinguish between “modeled” (has
non-ghost elems) and “unmodeled” (no non-ghost elems) ghost materials
elements material_set 2 1 3 1 vertices P0 ghosts
Serial/file:
P1 P2 …MOAB/Parallel:
P0 P1 P2 …Nek:
fluid solid P0 modeled P1 P2 …UNIC:
unmodeledApplication-MOAB Solution Memory Sharing
●
Recent enhancement in MOAB allows applications to get pointer to
(dense & sparse) tag memory, connectivity, vertex coordinates
Application-MOAB Solution Memory Sharing
●
Recent enhancement in MOAB allows applications to get pointer to
(dense & sparse) tag memory, connectivity, vertex coordinates
●
Implemented as tag/connect/coord iterator over contiguous chunks:
Range::iterator iter = myrange.begin(); int count; double *data;
while (iter != myrange.end()) {
tag_iterate(tag_handle, iter, myrange.end(), count, (void*&)data_ptr ); iter += count;
}
●
In most cases, including parallel with ghosts, will have 1 chunk
– Each unique combination of (entity type, #verts/entity) is a separate chunk ●
Exposed in iMesh (as extension) too
pointer (rpxm1, xm1(lx1, ly1, lz1))
call iMesh_initEntArrIter(%VAL(imeshh), %VAL(rootset), %VAL(iBase_REGION), %VAL(iMesh_HEXAHEDRON),%VAL(num_hexes), iter, ierr)
10
Application-MOAB Solution Memory Sharing: Pros/Cons
●
Pros:
– Any mesh library-based services can operate with no data copying (solution transfer, parallel I/O, partitioning)
– Future advances possible & free once application is hooked up: in-situ viz (w/ help from LANL CESAR participants), many-core memory movement ●
Cons/questions
– Lower cache efficiency than globally-numbered DOFs ordered for locality (e.g. vertex-edge-face-element variables)
● How many apps order vertex-based and element-based variables together?
– May not work for dynamic mesh applications?
● But, reordering/compression of handle/field space always necessary in
Going Forward
●
Working with LANL on vtk copy-less data arrays as part of CESAR
– Should allow in-situ viz out of the box● Complication: visualizing higher-order fields stored in GLL basis...
●
