Computational Parameters

Along with improvements of the code, increased computational resources allowed the usage of an improved model representation in terms of discretization. Differences in computational parameters for setup I and II are discussed briefly in the following.

Horizontal resolution Horizontal resolution of the three-dimensional velocity model

increased from 200 m to 100 m. This results in a better and smoother representation of the sedimentary basin edges. As discussed below the lowest shear wave velocity used in the model was reduced from 1400 m/s to 1000 m/s. It is noteworthy that the lowest velocity was not reduced to half of the original value as it would have been possible from the denser grid spacing. At the same time the target frequency was kept at 1 Hertz. Assuming a surface wave speed of 90 % the shear wave velocity the new setup samples surface waves with a minimum of about 9 points per wavelength (at target frequency) whereas the old simulations were carried out with a sampling of about 6 points. In most finite difference studies concerning wave propagation at local to regional scale, involving wave propagation distances of tens to several hundreds of kilometers, a sam- pling of 5 points per wavelength is concerned to be sufficient in order to avoid numerical dispersion and amplitude errors. It can be found from verification that a number of 5 points per wave length at the dominant period is just at boundary between the steep part of a error versus grid spacing relation and the flat part where no significant improve- ment in terms of accuracy can be expected at reasonable increased computational cost. From denser sampling up to about 10 points a distinct improvement can be expected with reasonable effort.

Vertical resolution The most fundamental improvement between the two setups as

for model representation concerns vertical resolution. Whereas with setup I this pa- rameter was 200 m alike the sampling in horizontal directions, setup II has a vertical grid sampling of 50 m and therefore a four times denser resolution of sedimentary basin

Figure 5.1: Grid representation for the new simulations. Vertical spacing increases from 50 m at the top of the model where low velocities are present, to 120 m at the bottom. For clarity reasons only every 5th grid line in horizontal direction is shown.

depths. A better representation of basin topography is achieved and more details of the wavefield effects should be revealed. To make this goal achievable within the limits of computational resources a varying grid spacing is used, ranging from 120 m at the bottom covering the whole range of about 20 km bedrock layers to the mentioned 50 m within the sedimentary part of the model and its surroundings. Technical aspects of this method are discussed in chapter 3. An example of the possible savings in terms of both memory and computation time can also be found in that section. Figure 5.1 illustrates the resulting grid as it was used in this study. Due to the limited graphical resolution only every twelfth grid line is shown in this plot.

It is noteworthy that the grid stretching algorithm can be applied in the other two di- rections, the horizontal directions x and y, in a straightforward way. Adapted to special requirements for basin simulation it is carried out only in the vertical direction where the potential of savings overwhelms the additional computation cost significantly.

Temporal discretization According to the stability criterion in finite difference sim-

ulations temporal discretization by means of time step is determined by the choice of spatial discretization by means of grid sampling and the velocities applied to the model. It is notable that the use of a fine mesh in the top layers does not necessarily lead to a proportionally smaller time step. In the setup used in this study vertically increasing velocities are applied to the bedrock representation. Therefore the highest velocities are not present at the same location as the smallest grid cells.

Absorbing boundaries The reduction of artificial reflections from the model bound- aries is a general problem in numerical simulations. The Cerjan type boundaries used in the old simulations were replaced by absorbing boundaries following the approach of perfectly matched layers (PML) as described in section 3.7 in order to circumvent their main drawbacks. One drawback of the Cerjan type absorbing boundaries is the frequency dependent degree of efficiency of this method. Practically this leads to re- maining reflections of the low frequency part of the wavefield. Whereas this could be tolerated for the point source approximations with very limited long period content, for finite source simulations the use of frequency independent boundary conditions be- comes inevitable. The PML approach which has become popular in the seismic com- munity in recent years provides this capability. The width of absorbing layers necessary for efficient suppression of artificial reflections is another important issue. Cerjan type boundaries work well when a width of more than 40 grid points is used. In the setup I simulations a value of 50 grid points was chosen. PML boundaries provide a better ab- sorption for all frequencies at a width of only 20 points. Consequently, this leads to an improved ratio of computational (including the absorbing margins) to physical (without boundaries) model space. Comparing setup I and setup II with respect to this quantity demonstrates a significant improvement achieved with PML boundaries. The usage of Cerjan type absorbing boundaries inflated the model space in the old simulations from 84,000,000 grid points (physical model) to a computational model with 144,000,000 grid points. Thus, only 58.3 percent of the model were actually used for the physical model, whereas nearly half of the model was wasted in absorbing margins. With the PML boundaries a computational model space consisting of 417,000,000 grid points is necessary in order to discretize the physical model into 352,800,000 cells. The ratio of physical to computational points is 84.6 percent. Usage of resources on the reduction of artificial reflections has been reduced to about 15 percent.

Memory Despite the above discussed improvements on the code simulations using

setup II require significantly larger amount of core memory than it was the case with the setup I. In fact, memory consumption is increased by a factor of about 4.5. Addi- tional memory is necessary for three reasons. First, the number of grid points has been increased nearly threefold. Second, the memory variables necessary for the implemen- tation of viscoelasticity are costly. The third additional field variables that must be held are the ones containing the derivative operator weights for the nonuniform grid layout. In total about 110 Gigabyte of core memory were necessary to describe the model and wavefield in the simulations with setup II. 32 nodes of the Hitachi SR8000 were used to carry out the simulations.

Computation time Using the above mentioned 32 nodes the calculation time was

kept at 12 h. That means roughly only 4 times the CPU time of the setup I simulations was necessary. It is noteworthy that floating point performance was increased by a factor of 3.5 between different versions of the code. The increase in performance is mainly due to optimization of computation loops. The setup II simulations were carried out on the Hitachi SR8000 with a performance of roughly 17 percent of the theoretical peak performance.

Table 5.1: Comparison of computational parameters for the different simulation setups. Parameter Setup I Setup II

Horizontal Discretization 200 m 100 m Vertical Discretization 200 m 50 - 120 m

Time Step 0.0198 s 0.0035 s Lowest S-wave velocity 1400 1000 Grid Size (physical model) 700×800×150 1404×1404×180

Grid Size (computational model) 800×900×200 1444×1444×200 Number of Time Steps 3034 16960

Simulation Time 60 s 60 s

Required Core Memory 24 GB 110 GB Computation Time 12 h×8 nodes 12 hours×32 nodes

Table 5.2: Comparison of model parameters used in the different simulation setups. Parameter Setup I Setup II

Sedimentary structure

Model type homogeneous velocity gradient S-wave velocity 1400 m/s 1000 - 1950 m/s P-wave velocity 2425 m/s 1732 - 3377 m/s

Density 2200 kg/m3 _{2200 kg/m}3

Bedrock

Model type homogeneous layered P-wave velocity 5000 m/s 5500 - 8000 m/s S-wave velocity 2890 m/s 3175 - 4620 m/s

Density 3000 kg/m3 _{3000 kg/m}3

Summary Above discussion gives an impression of the additional effort put into the

new simulation setup. It is noteworthy that this effort was not put into higher frequency content, but into a better model representation and resolution. Simulation results using setup II should be comparable to the ones achieved with setup I and reveal how im- provements on the model side affect the results. Table 5.1 gives an overview of old and new computational parameters.