Large-scale ocean modelling with adapting unstructured mesh methods

(1)

Large-scale ocean modelling with

adapting unstructured mesh

methods

Matthew Piggott

(Imperial College London)

Annual Science Meeting Liverpool 1-3 June 2009

(2)

Acknowledgments

A large number of people have contributed to this work including groups at:

• Imperial College London

– Applied Modelling and Computation Group – Grantham Institute for Climate Change

– Department of Earth Science and Engineering – Department of Aeronautics

– Department of Computing

– High Performance Computing Service

• University of Oxford

– Department of Physics - AOPP

• National Oceanography Centre

– Ocean Modelling and Forecasting – Ocean Observing and Climate

• Proudman Oceanographic Laboratory

• Daresbury Laboratory (HECToR)

• Fujitsu Laboratories of Europe

• and others

(3)

Some aims of this work

To work towards a computational modelling framework with attributes:

• Open source community model development

• Flexible application to multi-scale ocean problems from laboratory to global scale • Non-hydrostatic, Boussinesq/non-Boussinesq dynamics

• Choices between incompressible/compressible and multi-material/phase • Optimised representation of coastlines and bathymetry

• Cartesian space – no pole singularities

• Finite element/control volume discretisation options

• Element pairs including continuous/discontinuous and higher order • Respect large scale geophysical balances

• h, r and p type mesh adaptivity algorithms

• Solvers/preconditioners which perform for poorly conditioned matrices • Make use of massively parallel computing systems

(4)

Resolution is critical, both to define adequately

the domain of interest and also to resolve the

relevant dynamics within it

1 minute The future 6 minute Eddy resolving 20 minute Eddy permitting 60 minute Climate

An unstructured (and adaptive) mesh is able to give an optimal representation of coasts, bathymetry and the dynamics,

without the need for high resolution everywhere

Inc_rea sed rea lism , bu t at _the ex pen se of _ma ssiv_e com put_atio nal _cos ts

Models used today use uniform

(5)

• Hence the interest in the flexibility for

variable resolution that unstructured

meshes provide

• However this flexibility does comes at

a cost of higher CPU times

• The aim is to mitigate (and more!) this

through the use of static or dynamic

mesh adaptivity to focus

computational resource where it is

most required

• This can be due to numerical errors,

or simply what is of relevance to the

simulation being conducted, or the

region you are interested in

Mesh optimised for coastlines and bathymetry, using the adaptive methods presented here

(6)

GFD example: The lock exchange problem

(Hiester 2009)

Compare speed of head at free and no slip boundaries

against DNS results (Hartel et al, 2000): 0.01509 & -0.01284.

(7)

Validation against Laboratory

experiments

(Maddison; Power)

The differentially heated rotating annulus at two rotation rates. The bounds of the normalised temperature have been limited to aid visualisation at the end of the movie

R. Hide and P.J. Mason 1975: Sloping convection in a rotating fluid. Adv. Phys. 24, 47-100

(8)

Two snapshots in time at the faster rotation rate

Here the maximum computational cost corresponds to the time of maximum kinetic energy

(9)

Internal wave breaking at the

continental shelf break – important

for mixing in the vertical,

biogeochemistry on the shelf

(Martin)

Open ocean deep convection –

an important process linking the

surface and deep parts of the

thermohaline circulation

(Bricheno)

(10)

Here the adaptive mesh is focusing in and preserving the integrity of the

thermocline in a lab scale simulation of internal wave generation, breaking

and mixing over bathymetry

(Martin)

(11)

Westward intensified

boundary currents: Effect of

varying the maximum allowed

aspect ratio in the adapted

mesh

Uniform mesh

Approximately a two orders of magnitude improvement in the error/cost relationship between uniform and anisotropic adaptive refinement

(12)

Advecting a tracer field through the boundary layer (cf. Hecht et al, Hanert et al)

40k 10k 160k 10k 40k 160k Fixed Adaptive Adaptive

(13)

Wind driven gyre, western boundary current and eddies resolved

with an adapting anisotropic mesh

Wind stress

Anisotropy reflecting that although the flow is

intense North-South, it is in the East-West direction that the important variation (shear) occurs

(14)

High aspect ratio

large-scale baroclinic circulation

Need for stable discretisations respecting geophysical balances, fast linear solvers for ill-conditioned pressure equation, minimally dissipative advection methods Also shown here the use of 2+1D

adaptivity to preserve some structural difference between horizontal and vertical

(15)

Evolution of the upper mixed

layer with an adapting mesh

over a column

Here atmospheric data from Station Papa in the Pacific is used to force the model. A k-epsilon turbulence model is being validated against measurements of the mixed layer. The adaptive mesh is reflecting the seasonal nature of the stratification.

Summer _Summer

Winter

(16)

Biology module with four species

Aim to use adaptivity to simulate important small scale physics and seasonal biological activity.

Currently developing

methods in single column domains.

(17)

Mesh movement

Solve a system of equations for node locations of the form

where w is a monitor function which is large in regions

requiring finer resolution, e.g. Ceniceros & Hou 2000. Also working in 3D

Approach being developed to track isopycnals and eddies for example

Have shown improved

diapycnal mixing properties for simple problems

(18)

Modelling geohazards: landslide generated tsunami waves

(Wilson)

Lituya Bay, Alaska, 1958: Largest recorded landslide induced tsunami runup.

Experimental models (Fritz et al., 2001) are now being used to validate our 2D wave generation models.

Stromboli, 2002: Landslide induced tsunami destroyed coastal infrastructure. Current 3D modelling of wave generation and propagation is building on previous laboratory scale models (DiRisio et al., 2009) of the event.

(19)

Indian Ocean Tsunami, Boxing Day 2004

Arrival times (minutes) Location 198 195 197 Gan, Maldives 208 212 211 Hanimadhoo, Maldives 220 217 226 Diego-Garcia, UK 193 191 195 Male, Maldives 161 165 170 Colombo, India 156 136 156 Viakhap-atam, India 204 207 205 Tuticorin, India 122 82 107 Sibolga, Indonesia ICOM model Delfin model, 2007 Tide Gauge

Validation against arrival times from tide gauge data and another tsunami model. Also possible to look at free surface height profiles from satellite tracks (Jason-1).

An idealised tsunami on ‘aqua-planet’ – no problem with polar singularities with this approach

(20)

Dynamic balancing of the

computational load between

processors as the mesh

adapts

Cost of adaptivity approx 10% of simulation, cost of data migration minimal

Recent scaling analysis shows good results on up to 1024 processors of a Cray XT4 (HECToR): 0.8

efficiency going from 128-1024 with a 5M node adapting annulus

(21)

Conservative Interpolation: crucial for many

applications, with adaptivity, coupled models

‘donor’ mesh ‘target’ mesh

‘super mesh’ and mapping from donor mesh

Following construction of the supermesh we are able to perform projection operations to achieve

conservative, bounded (etc…) interpolation schemes. Farrell et al. Comp Meth Appl Mech Eng, in press. Also now in 3D (Farrell+Maddison)

(22)

• If a failure is detected in a test problem, the developers are notified details via email.

• Statistical information about code quality is automatically collected from the newly validated code. This allows for the monitoring of performance. Results available via a web interface. • This modern approach to software engineering has yielded dramatic improvements in code

quality and programmer efficiency. • All models require rigorous

validation/verification. A continuous automated approach is required as the codebase changes.

• A central copy of the source is kept in a

subversion repository. When developers commit changes this must result in a pass of the code test suite.

• Buildbot checks out and builds on various platforms with several different compilers. Any errors are relayed back to developers.

Buildbot: automated

code verification system

(Farrell et al.)

(23)

A nice thing about having a general Navier-Stokes code at the core is that a

larger number of validation & verification cases is available

E.g. Comparisons between drag calculation in flow past a sphere at a range of Reynolds numbers

Computed drag coefficient compared against correlation (from Brown and Lawler, 2003) with lab data

(24)

Diamond: new automatic pre-processor tool to improve usability

(Ham et al. Geoscientific Model Development, 2009, Vol: 2, Pages: 33 - 42)

• An xml schema file describes the rules that govern ICOM options • Diamond uses this to

automatically generate a GUI based on the

schema

• Options are entered and output as another xml file containing the options values

• This is written to an

options library accessible from anywhere in ICOM • Includes many features,

including the ability to define python functions executed at run time by ICOM

(25)

Current status:

• Parallelisation of latest discretisation methods which respect balance and are needed for large-scale high aspect ratio applications – complexity (higher order and discontinuous solution fields) means a complete revamp of our entire parallel framework needed which is nearing completion • Non-hydrostatic model means that we have costly elliptic equations to solve, high aspect ratios

leads to poor matrix conditioning. New algebraic multi-grid methods have been developed which perform better than other solvers available. These have now been parallelised

• For tracer advection the twin requirements of resolving weak stratification as well as sharp

fronts/interfaces makes this a very challenging problem for standard advection methods. Control volume limited and discontinuous based methods are being investigated

• Above are being combined on the large scale baroclinic double gyre problem as a stepping stone to basin/global scale applications

• Further development of subgrid-scale parameterisations

• Verification and validation on idealised problems and laboratory data; range of process studies • Interfacing with NetCDF and bulk formulae for realistic forcing; open boundary conditions

• Development of a biology module

• Coupling with ice (sea ice and ice shelves), atmosphere, structures, …

• Work towards a full adjoint and also adjoint produced from reduced order models (POD) • Pre and post processing tools; documentation etc…

• Important cross-fertilisation and re-application of technology and code with other areas: coastal engineering, atmospheric pollution, geohazards, geodynamics, oil reservoir engineering,

(26)