2
First of all, I would like to talk about the experiences we have made with several proof-‐of-‐concepts when comparing different Oracle platform architectures like the Exadata versus conventional platforms.
There are several reasons why WK͛Ɛ may become very time consuming, complicated and expensive:
Owing to security requirements databases with sensitive data have to be masked (Oracle provides an EM pack for data masking)
It is often difficult to simulate OLTP Systems with a large user community (Oracle provides also a tool for this requirement: real application testing)
The embedding in a productive environment like SOA or the interface to messaging Systems for data transfer is complex
Several customers use engineered system, like the Exadata, for database consolidation. How can we simulate this without migrating many databases possibly from different platforms (different ͙ operating Systems, byte order, database releases) to the target system?
The experiences of WK͛Ɛ show, that you have to invest at least 50+ labor days. And the whole POC will probably last some months.
In order to execute the POC one have to travel to a benchmark center, or to the benchmark center of selected Oracle partners. In both cases one will get only a limited time window for tests.
If a POC is run with a single application, this application must be highly scalable to challenge the performance capabilities of a new platform architecture.
What we also have seen is, that application change their behavior when moving to a system with complete new architecture like the Exadata. If an application was former I/O-‐bound, it may now becoming more CPU-‐bound, because the I/O subsystem delivers much better I/O performance. Or the system can not be utilized, because the application is not designed to do so.
And the results are ambiguous, because an application changes over time and therefore the results would be different with the next release of the application software. The POC may still not deliver any performance data suitable for capacity planning and database consolidation.
And, it is important to keep in mind: a POC is NOT reality, but some level of modeling reality. Like each benchmark.
>Ğƚ͛Ɛ take a look at other industries and how they work with performance numbers. For example the automobile industry.
Have you ever seen a car driver running a proof-‐of-‐concept to validate the performance data of his car?
The automobile industry publishes key performance metrics for cars, like speed, acceleration, horsepower and torque.
If I ask you, how fast is your car, you will probably be able to answer my question immediately. But if I ask you about the key performance metrics of your database platform, it will probably be more difficult.
Why?
There are two reasons:
First, there are no useful standards for database platform key performance metrics
Second, most people do not measure the performance of their database platforms unless users complain about bad performance or management complains about high cost
6
Why is it so difficult to obtain key performance metrics of our database platforms? ItDue to the huge amount of combinations
Components
Configurations
Architectures (e.g. for HA and DR)
there are probably no two identical Oracle platforms -‐ even in the same company. The performance of such an complex Oracle platform is not predictable. Benchmarking is the only way to predict performance of an Oracle database platform.
Lets summarize so far:
1. The performance of a database platform is not predictable due to its complexity. 2. Benchmarking is the only way to determine the key performance metrics.
9
The Transaction Processing Council has published standards for database platformbenchmarks. Unfortunately these benchmarks are only used by vendors. I have never seen a customer running a TPC benchmark to certify the performance or to evaluate components of his database platform.
Industry benchmarks like TPC are only used by IT vendors, but not by IT customers because they are unpractical.
Certain rules have to be followed to set up the database in terms of sizing and usable features. But a customer would like to use ALL database features which help to increase overall efficiency and to get the most performance for each license dollar. A benchmark has to follow customer needs, not the other way around.
The benchmark must be operational on each customer system with any database size, because the customer wants to know the key performance metrics of his platform.
If you take a detailed look at the full disclosure report of some of these benchmarks, you will find out, that for a 10 TByte database about 3͛000 disks are used. Completely unrealistic. Even the largest banks in Switzerland use by far less disks for a single database. Take a look at the TPC-‐H performance metric. Its just one number. Assume the value is 17. What do you know about the power of the Wh͛Ɛ from this value? How much data can be loaded in a given time period? Or how much data can be scanned or backed up in a given time window? Does this value of 17 help anybody in the project team to size the system? We have a very similar situation with TPC-‐C benchmarks. TPC-‐C measures the transaction rate at highest system load (> 95% CPU utilization). But this is a situation every system administrator wants to avoid in OLTP Systems. This high load is an exception in OLTP Systems, not the rule.
Other topics concerning TPC benchmarks:
No values about best-‐case behavior (cache) and worst-‐case behavior (disk)
No values about system behavior with increasing load (from 1 process to n processes/system saturation)
Because there is no useful benchmark method around, during the past few years we developed our own benchmark method and benchmark tool, the benchware Loader. We are measuring Oracle platform performance right above the database layer. At this interface we identify the performance of the complete Oracle platform. The key performance metrics of an Oracle platform can be used as a foundation for service level agreements between IT operations and the business applications.
The Benchware Loader evaluates the key performance metrics of CPU and server system, storage system and database system. The benchmark tests are representative for OLTP and DWH systems. They analyze system behavior for all major database operations in different load situations from a single process up to complete system saturation.
The Benchware Loader examines both best and worst case performance. All components of the platform (server, storage and database system) are checked for their performance characteristics and limitations. Bottlenecks will be detected immediately.
The Benchware Loader delivers simple and understandable key performance metrics, this is an important difference to TPC benchmarks. The key performance metrics are the foundation for validating the chosen system architecture and for capacity planning when upgrading systems or consolidating servers and databases. Performance metrics are also utilized, taking into consideration Oracle license costs, to conduct price-‐ performance platform comparisons.
The Benchware Loader, written in SQL, PL/SQL and Java, runs on all Oracle platforms and scales from small server to very large high-‐end systems. It uses synthetic data and allows benchmarks to be repeated and compared at any time to verify the impact of system changes on overall application performance.
The Benchware Loader evaluates the key performance metrics of CPU and server system, storage system and database system. The benchmark tests are representative for OLTP and DWH systems. They analyze system behavior for all major database operations in different load situations from a single process up to complete system saturation.
The Benchware Loader examines both best and worst case performance. All components of the platform (server, storage and database system) are checked for their performance characteristics and limitations. Bottlenecks will be detected immediately.
The Benchware Loader delivers simple and understandable key performance metrics, this is an important difference to TPC benchmarks. The key performance metrics are the foundation for validating the chosen system architecture and for capacity planning when upgrading systems or consolidating servers and databases. Performance metrics are also utilized, taking into consideration Oracle license costs, to conduct price-‐ performance platform comparisons.
The Benchware Loader, written in SQL, PL/SQL and Java, runs on all Oracle platforms and scales from small server to very large high-‐end systems. It uses synthetic data and allows benchmarks to be repeated and compared at any time to verify the impact of system changes on overall application performance.
Customers are using Benchware Loader in different situations, here some examples:
Testing existing architectures (health check) to identify current performance bottlenecks or to certify platform performance capabilities (usable for service level agreements between IT and Business)
Optimizing storage performance by comparing different product, different configurations, different architectures, etc.
Evaluating different CPU types to optimize Oracle license cost
Comparing the impact of different disaster recovery technologies to application performance
Based on benchmark results, Benchware provides a sound foundation for decisions: components, configurations, architectures and complete systems.
In an evaluation of a new system you have always the old system (blue) with relative cost of 1.0 and relative performance of 1.0. The new system (red) needs an investment, in this case factor 0.6 of the old one (blue), and provides a performance improvement of factor 2.5, based on key performance metrics.
The Benchware Loader has several value propositions, it is
Comprehensive ʹ benchmarks all basic database operations
Simple to install and to use
Fast to run, within a few hours
Scalable -‐ from laptop to high-‐end system
Reproducible ʹ uses its own data
Provides understandable performance metrics
Uses and supports important Oracle features like RAC, Data Guard, Flash Cache, Exadata, ͙
