The Benchmark Operations

The nal step in dening a benchmark is the denition of the operations. It follows the guidelines discussed for operation denition introduced on Page 42. We, therefore, include op- erations for creation of all entities that are dened in the benchmark schema, various traversals through the syntax graph in order to measure performance of unparsing operations, changes to attribute values and operations that delete objects. The main point in the detailed oper- ation denition is to dene parameter values for the operations in a way that they respect the structure of the initial database. Thus, the denition of those values is also based on the above mentioned analysis results.

A problem that we have to address is the variance in execution times of complex benchmark operations. The execution times vary because they depend on the states of external devices, such as the position of disk heads, the hardware bus, caches of disk controllers, of the operat- ing system and of the database system itself. As these states are beyond our control we must execute benchmark operations repeatedly and compute mean values of the operation execu- tion times to obtain reliable results. Repeated operation executions, of course, must be done from exactly the same logical database state3. For two reasons it is not appropriate to create

the database anew before measuring each operation. First, creating a database of a consider- able size is rather time-consuming (in the order of magnitude of hours) whereas benchmark operations will execute within a few hundred milliseconds. Secondly, the physical database organisation will vary and this might even increase the variance of operation execution times. Instead, we suggest to organise time measurement of benchmark operations within benchmark runs. A benchmark run is a sequence of benchmark operations that leaves the database in exactly the same logical state that it had before the run began. This means that we rst have to perform create operations, then perform traversals and attribute changes (or vice versa) and then have to delete all created objects again. We can then repeat the execution of benchmark runs and have achieved that a benchmark operation is in each run performed in the same logical database state.

According to the above considerations, the operations of the Merlin Benchmark are clustered into four groups. Operations of the rst group create increments of the two document types like modules, types, operations, parameters and comments. The second group is dedicated to measuring the impact of changing attribute values. The third group simulates tool operations that perform traversals through the database. The last group deletes all previously created objects. A Merlin Benchmark run, therefore, consists of the following operation sequence.

OpenOMS:

Open the access to the OMS, i.e. perform all necessary operations such as authorisation or reservation of buers to access data in the OMS.

3We will not be able to inuence the physical database state, i.e. the way how objects are stored on disk

pages, since it is controlled by the database system and this is in fact one of the reasons why multiple operation executions are required.

CreModul:

Create

n

new modules in the architecture. Set their names to ModuleNo j+2i,

(

j

2f0

;:::;n

,1g

;i

= levels in the initial database). Set the graphical coordinates to

(50

j;

100).

CrModTyp:

For each created module set the export type identier toType No j+2i, (

j

2

f0

;:::;n

,1g

;i

= levels in the initial database).

CrModCom:

For each module created by

CreModul

expand the module comment incre- ment to an arbitrary string which is 256 bytes long.

CrModImp:

For each created module create four import relationships to arbitrarily given modules of the bottom-most level of the initial database.

CrImpObj:

For each created import relationship of each created module increment expand the list of imported objects to the exported type and the 1st

_;

₃rd

_;

₇th

_;

₁₁th

_;

_{and 14}th

operation of the respective module.

CrExpOpe:

Create a list of 17 operation increments in each created module, update their name to Mod

j

Func

k

with

j

being the number of the module as determined by the

CreModul

operation and

k

2 f1

;:::;

17g and expand their type to the type which is

exported by the module.

CrOpePar:

Expand each operation parameter increment in each created module to a pa- rameter list with three call by reference parameters. Expand the parameter names to arbitrary unique identiers (12 bytes) and the parameter types to the types which are imported by the rst three import lists of the module.

CrOpeCom:

For each operation expand the operation comment increment to an arbitrary string of length 256 bytes.

UnparMod:

Create textual representations of the created modules in a string of the host programming language.

UnparArc:

Create a textual representation of the architecture in a string of the host pro- gramming language.

AnaUsage:

For each module of the bottom most level of the initial database, return the names of modules to which it exports objects. Return the names as a list of the host programming language.

ClosTrav:

Return the names of those modules that are reachable from the module at level 1. Return the names as a list of the host programming language.

ChModNam:

Change the module name of each module to another unique name of the same length and update the name in the import parts of those modules which use the module.

ChModTyp:

Change the exported type of each module to another name of the same length and update the name in the import parts and parameter lists of those modules which use the module.

ChModCom:

Change the contents of the module comment of each created module to another arbitrary string of the same (256 bytes) length.

ChOpeNam:

Change the names of the exported operations of each module to another name of the same length and update the name in each import list which imports the operation.

ChParNam:

Change the names of each operation parameter to a new value of the same length.

DlOpeCom:

Delete every operation comment in all modules that has been created during the benchmark.

DlOpePar:

Delete each operation parameter in each module that has been created during the benchmark.

DlOperat:

Delete all exported operations in all modules that have been created during the benchmark.

DlImpObj:

Delete all imported objects in all modules that have been created during the benchmark.

DlImpRel:

Delete all import relations in all modules that have been created during the benchmark.

DlModule:

Delete all modules that have been created during the benchmark.

CloseOMS:

Perform all operations that are necessary to start the benchmark again.

4.1.4 Summary

In this section we have discussed database performance evaluation using abstract database benchmarks. We have argued, why the three general purpose benchmarks Simple, Hypermodel and OO7 Benchmark are inappropriate for our as well as many other database performance evaluation tasks. We, therefore, consider database performance evaluation as a problem that cannot be solved in general. To solve it, it is required to exploit knowledge of a particu- lar application domain. We have suggested a method to acquire this knowledge in terms of application-specic load characteristics and use it for the denition of application-specic benchmarks. The load characteristics are elicited based on existing schemas and data. A conceptual benchmark schema is dened based on a representative schema of the application domain. Then existing data is analysed in order to determine realistic cardinalities for rela- tionships in an initial database. In a third step, benchmark operations are dened that create objects of the entities contained in the benchmark schema, change attributes, traverse along relationships and nally delete the objects again. Following the suggested method, the Merlin Benchmark has been dened in order to measure database performance for syntax-directed tools. We have implemented the benchmark for representative database systems of dierent classes. In the next sections, we discuss the Merlin Benchmark implementations and the re- sults of its execution along with the assessment of the respective classes using the functional requirements presented in the previous chapter.

In document Tool specification with GTSL (Page 58-60)