Multi-scale studies, such as those described by Gulrajani [1998], Clayton and Holden [2002], Sundnes et al. [2006] have sought to estimate electrocardiogram (ECG) measurements by placing a fine-grained model of the heart into a coarse-grained model of the torso. Studies of this kind
offer a link between the sub-cellular detail provided by in silico modelling, and the primary
clinical tool for study of the heart.
1.9
Computational Methods
To be solved on a computer, the continuum tissue equations (1.7–1.9), or (1.10) for monodomain, and boundary conditions (1.11) must be discretised in both space and time. The sense in which the word ‘discrete’ is used here is distinct from the discrete tissue models and discrete states discussed above. For partial differential equations, discretisation is required only to provide a numerical solution. Two methods are commonly used for space discretisation: finite differences and finite elements.
1.9.1
Space Discretisation
Finite Elements
Finite Element Methods (FEM), well developed in computational fluid dynamics, has been in the mainstream of cardiac modelling in the last 3 decades [e.g. Buist et al., 2003, Franzone and Pavarino, 2004, Franzone et al., 2006, Hooks et al., 2002, Sundnes et al., 2006, LeGrice, 2001, Rogers and McCulloch, 1994]. Using FEM, the solution domain is divided into a number of polygonal — commonly triangular (2D) or tetrahedral (3D) — elements. One of the major attractions of FEM is its treatment of irregular domains such as the heart. Because the size of the elements may differ, it is possible for meshes to have more, smaller elements near regions of interest, such as rapidly changing surfaces.
The variability of finite element meshes requires considerable care in the discretisation of any given geometry. A broad selection of software tools now exist to generate finite element meshes. In finite elements, the solution to the continuous equations is approximated by a set of piecewise polynomial basis functions on the whole domain, thus implicitly incorporating the discretisation of the boundary conditions (1.11).
The result of this approximation will be a system of linear equations for the values of the
basis functions at the points, or nodes of the discretised mesh, which has to be subsequently
solved by linear algebra tools.
The solution matrix of the discretised linear system is typically sparse and asymmetric. It is difficult to construct iterative methods for general classes of non-symmetric problems.
1.9. COMPUTATIONAL METHODS 17
Finite Differences
In Finite Differences, the entire domain is divided into cells, or voxels by a structured, Cartesian grid. Continuum equations are written in terms of finite difference equations. At each inte-
rior point, derivatives are approximated by difference quotients over a small interval, i.e. ∂u
∂x is
replaced by ∆u
∆x where ∆xis small. For example,
∂2u
∂x2 ≈
(u−−2u·+u+)
∆x2 (1.12)
where u· is the central point, u− and u+ are x-axis neighbours in the negative and positive
directions, respectively, and ∆xis the space step.
The solution matrix of the discretised system is a trilinear, banded matrix. The banded matrix relates to the limited neighbourhood of points in the FD mesh. Matrices of this kind are inherently simpler to solve than the sparse asymmetric matrices created by FEM. Regular meshes also lend themselves to implicit methods such as the Crank-Nickelson method. Another attraction of Finite Differences is the relative simplicity of coding finite difference equations.
When simulating the anatomically realistic geometry of the heart, integration of data from MRI is more easily used in Finite Differences schemes. Clayton and Panfilov [2008] state that “[s]ince MRI images are typically obtained in tissue slices and assembled into a 3D volume representation, this approach lends itself to a description based on a Cartesian grid.”
Neumann boundary conditions in finite differences can be maintained by the introduction of ‘fictitious’ points at points immediately outside the domain. By assigning these points values
from their immediate interior neighbour. In irregular domains, wherendoes not lie parallel to a
coordinate axis, several methods exist for computing boundary conditions. Some methods, such as that described by Noye and Arnold [1990] and Dumett and Keener [2003] require additional information about the precise location of the boundary closest to the irregular point, which are problematic for cardiac meshes, where the discretisation ordinarily matches the resolution of the measurement equipment used. Buist et al. [2003] describe the use of a three point one-sided difference scheme to be used in the case where one point is missing in the direction of interest. The ‘phase-field’ method introduced by Fenton et al. [2005] approximates additional resolution at the boundary, by introducing weighted points in a manner similar to image anti-aliasing.
1.9.2
Time Discretisation
As discussed by Keener and Bogar [1998], there are four common timestep methods: 1. Fully explicit (forward Euler):
ut+1 =ut+ ∆t[Dut+f(ut)]. (1.13)
2. Fully implicit (backward Euler):
ut+1−∆t[Dut+1+f(ut+1)] =ut. (1.14)
3. Mixed explicit-implicit (semi-implicit):
ut+1−∆tDut+1=ut+ ∆tf(ut). (1.15)
4. Crank and Nicolson [1996]:
The fully explicit method is broadly understood to be easy to implement. It also lends itself well to parallelism. To ensure numerical stability in Finite Differences, step size is critical. Numerical stability requirements of the fully implicit method insist that, for finite differences,
D∆∆xt2 ≤
1
2d, where d is the dimensionality of the medium. Highly non-linear systems may
require thatD∆∆xt2 ≤≤
1
2d. The numerical stability of a discretisation can be tested by running
a simple simulation with varying step sizes and examining the results for numerical artefacts. For Finite Elements, a similar numerical stability condition relates the timestep to element size. Consequently, explicit schemes are expensive to compute in fine-grained meshes. While the fully explicit method computes the future state of the system from its current state, the fully implicit method computes the future state of the system based on both the current and future states. The fully implicit method is not restricted by timestep, but does require additional computation, and effort to implement, due to the large non-linear system of equations to be solved at each step. Semi-implicit methods offer improved numerical stability over the fully explicit method, allowing longer timesteps to be used, but require only the solution of a linear system of equations at each step.
1.10
Software Implementation
Computer simulations of cardiac tissue involve vast scale ranges, with activity atµm andµs scales
being observed in several cm3 of tissue over tens of seconds. Moreover, modern biophysically
realistic models have brought about an order of magnitude increase in both space and time complexity. The complexity is further increased as simulations expand to include far greater detail of heterogeneous tissue structure and cell types, on increasingly large meshes. In this context, it is unsurprising that the timely completion of simulations relies on modern High Performance Computing (HPC) hardware.
Use of HPC facilities, although essential, is limited by the ad-hoc manner of software de- velopment seen across the research community. This can be compounded in situations where funding is more easily obtained for hardware purchase than for the accompanying skilled staff. The result is that many small software development projects are undertaken in isolation, each supporting a particular short-to-medium-term aim. Code developed in this environment is often poorly structured and documented — hampering reuse within the lab — and is unlikely to be portable, preventing use on different platforms elsewhere. Results produced from such software are difficult to compare or repeat, limiting the scope of peer review. This rather insular approach to development excludes a great many members of the community for whom code-level involve- ment with simulations is neither desirable nor practical. The disparity of cardiac simulation software at present is, arguably, impeding scientific progress.
1.10.1
Shared Memory Parallelism
Shared memory parallelism (SMP) involves multiple threads of execution, each of which have access to a single, shared memory resource. Each thread may read and write the same regions of memory, unless explicitly prohibited from doing so. This approach avoids the need for explicit communication between processes, but requires that developers take great care to avoid race
conditions. By default, shared memory programs run in a single,master, thread, until a parallel
region is entered, at which point additional threads of execution are spawned. The developer must explicitly define parallel regions in which work may be shared between threads. The execution of a multithreaded program is illustrated in Figure 1.8.
1.10. SOFTWARE IMPLEMENTATION 19
Slave Threads Master Thread
Parallel Region
Figure 1.8: Execution of a multithreaded program. A master thread persists throughout execu- tion, while new threads are created as needed.
POSIX Threads. Also, many modern programming languages such as Java and Python implic- itly support threads. For HPC applications, the OpenMP library for C/C++ and FORTRAN is popular.
By definition, shared memory applications require that all threads have access to a local block of main memory. In hardware terms, this requires that threads are assigned to multiple processors or processor cores of a single machine. Despite the advent of multicore processors and the increasing number of cores per processor, there exist physical limits with regard to the number of processor cores that can practically surround a single block of main memory and still maintain reasonable access times. This limits the number of threads that may be run concurrently, and ultimately the scalability of parallel codes run on such machines.
For very data-intensive applications, shared memory machines may also lack sufficient RAM to hold all of the necessary program data in memory at once.
1.10.2
Distributed Memory Parallelism
In Distributed Memory Parallelism (DMP), the program is executed on multiple processes. A
process is a thread of execution with its own allocation of memory and no direct access to that of other processes. Where processes need to exchange data, communication is required via explicit send and receive actions. Unless otherwise specified, all program code runs on every process. This requires DMP processes to be restricted to operating on specific tasks or sets of data to prevent redundant processing. The execution of a parallel program is illustrated in Figure 1.10. In DMP, an executable runs in one or more processes, where each process has its own, ring-fenced area of memory in which to operate. In a distributed memory machine, several conventional servers, referred to as nodes, are connected to a network. Traditionally, a process maps to a single node. With multi-processor and multicore machines, however, it is likely that each processor or processor core (hereafter referred to as cores) will handle a single process. Fig- ure 1.11 illustrates the hardware architecture of a distributed memory machine with multicore processors. Utilising all of the available cores in this way is likely to yield improved performance,
M a i n M e m o r y
CPU CPU CPU CPU CPU CPU
Figure 1.9: Schematic of a typical Shared Memory machine’s architecture. Each CPU is has equal access to a single, shared block of main memory.
Send/Receive
Gather
File I/O
Figure 1.10: Execution of a DMP program. All processes run throughout execution. Work is divided between processes to create speedup. Where coordination is required, processes commu- nicate explicitly. All processes have access to shared I/O resources.
1.11. STATE OF THE ART SOFTWARE PACKAGES 21