Obtaining benchmark results

Determine execution time We used the default PostgreSQL timing mechanism to retrieve query execution times. This mechanism is enabled with the command ‘\timing’. We measured the time it takes to execute the result of an SQL query mapped from an XPath query. The time it takes to map an XPath query into an SQL query is not taken into account, because we are only interested in the performance of the URDBMS. For query execution benchmarking, we use the average execution time over the last 100 runs. We executed 101 runs to ensure the database was hot.

The time to create indices is included in the execution time for the data mapping. Indices created for element tables are included in the execution time to map the flesh; indices created for phase table are included in the execution time of a DO glue process. In particular, BB glue processes create intermediate indices.

Normalized execution times In order to give a fair comparison between the performance of different approaches that evaluate a set of varying expressions, we normalize execution results. We explain normalization by example. Leta1,a2,a3 be three algorithms to evaluate a certain queryq

and lett1(q),t2(q),t3(q) be the query execution times of each of the three methods. We normalize

these query execution times with a functionn as follows:

n(t1(q)) =t1(q)/P{t1(q),t2(q),t3(q)}

n(t2(q)) =t2(q)/P{t1(q),t2(q),t3(q)}

CHAPTER 12. EXPERIMENTS 117 0.25 1 0.65 0.11 NT(a3) 1 0.06 0.44 NT(a3) 0.50 0.44 NT(a1) 1 0.25 0.31 30 sum 49 74 31 q1 T(a3) Queries T(a2) 3 12 6 T(a1) 12 6 6 65 12 q3 q2

Default speedup of a1 compared to a2: 0.61 Default speedup of a1 compared to a3: 1.51

Figure 12.3: Normalized speedup compared to default speedup

In more general, for query execution timesa, . . . ,z derived from the same query, we obtain their normalized exection times as follows:

n(a,b, . . . ,z) = (a/s,b/s, . . . ,z/s) wheres =a+b+· · ·+z

An advantage of normalizing execution times is that the overall performance of an algorithm is not ruined by one outlier. For example, if two algorithmsa1 anda2 have the same performance

for three queries and a third algorithma3 evaluates the first query in twice the execution time as a1anda2 but evaluates the other two queries four times as fast, all three algorithms would have a

total normalized execution time of 1 regardless of the actual execution time.

In general, algorithms that manage to execute a certain query far more efficient than the average execution time of that query are rewarded. The algorithm with the lowest total normalized execution time has the best performance for queries that are considered to be hard to evaluate for the majority of the algorithms with which the winning algorithm is compared.

Speedup The conceptspeedup refers to how much a certain algorithma is faster than a corres- ponding equivalent algorithma0. Speedup is defined as the execution time of the former algorithm

T(a) divided by the execution time of the latter algorithm T(a0).

Normalized speedup We define the concept normalized speedupas the normalized execution time of an algorithm a divided by the normalized execution time of another algorithma0. Like the default definition of speedup, normalized speedup is a measure of how much an algorithma

is faster than a corresponding algorithm a0. Normalized speedup and default speedup give the same result for execution times of individual queries, because normalization divides the execution times by a common denominator. For a set of query execution results, default speedup disfavors algorithms that have outliers. We illustrate this with Figure 12.3. Observe that the normalized speedup of all combinations of algorithmsa1,a2 anda3 is 1. This indicates that a random picked

query in {q1,q2,q3} is evaluated by all three algorithms even efficient. However, the absolute

difference in execution time for query 3 result in a favoring of algorithma2 while this algorithm is

the slowest for queries 1 and 2.

Speedup should be used if a query workload is representative for default user behaviour with the restriction that each query should be assigned with a certain weight that captures how often that query is used compared to others. In contrast, normalized speedup should be used if a query workload contains queries that are likely to be evaluated by the system, however, it is unknown if the query workload gives a proper reflection of user behaviour.

Materialized vs. unmaterialized Materialized views and unmaterialized views are discussed in Section 2.6. In our experiments, we make a distinction between queries for which we evaluate the flesh result with a materialized view –denoted with postfix (mv)– and queries for which we evaluate the flesh result with an unmaterialized view –denoted with postfix (v).

For queries with the postfix (mv), we force the PostgreSQL query planner to calculate the flesh result prior to the QRO glue process. This restriction does not hold for queries with the postfix (v). As a consequence, a wider set of query plans apply to unmaterialized views, including the