used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser’s personal use without the written permission of Quest Software, Inc.
The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR
INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document.
If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters
LEGAL Dept 5 Polaris Way
Aliso Viejo, CA 92656 email:[email protected]
Refer to our Web site (www.quest.com) for regional and international office information. Quest, Quest Software, the Quest Software logo, AccessManager, ActiveRoles, Aelita, Akonix, Benchmark Factory, Big Brother, BridgeAccess, BridgeAutoEscalate, BridgeSearch, BridgeTrak, BusinessInsight, ChangeAuditor, CI Discovery, Defender, DeployDirector, Desktop Authority, Directory Analyzer, Directory Troubleshooter, DS Analyzer, DS Expert, Foglight, GPOADmin, Help Desk Authority, Imceda, IntelliProfile, InTrust, Invirtus, iToken, JClass, JProbe, LeccoTech, LiteSpeed, LiveReorg, LogADmin, MessageStats, Monosphere, NBSpool, NetBase, NetControl, Npulse, NetPro, PassGo, PerformaSure, Point, Click, Done!, Quest vToolkit, Quest vWorkSpace, ReportADmin, RestoreADmin, ScriptLogic, SelfServiceADmin, SharePlex, Sitraka, SmartAlarm,
Spotlight, SQL Navigator, SQL Watch, SQLab, Stat, StealthCollect, Storage Horizon, Tag and Follow, Toad, T.O.A.D., Toad World, vAutomator, vConverter, vEcoShell, VESI,vFoglight, vPackager, vRanger, vSpotlight, vStream, vToad, Vintela, Virtual DBA, VizionCore, Vizioncore vAutomation Suite, Vizioncore vEssentials, Vizioncore vWorkflow, WebDefender, Webthority, Xaffire, and XRT are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Software’s trademarks, please seehttp://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered trademarks are property of their respective owners.
Benchmark Factory for Databases User Guide
New For This Release
18
New for Benchmark Factory 6.5 18
New for Benchmark Factory 6.1.1 18
New for Benchmark Factory 6.1 18
New for Benchmark Factory 6.0 18
Understanding Benchmark Factory
19
About Benchmark Factory
19
The Benchmark Factory Environment
19
Integration with Other Quest Software Products 20
Understanding the Benchmark Factory Environment
20
Understanding Benchmark Factory Terminology
21
How Benchmark Factory Works
24
Create a Script With Benchmark Factory
25
Benchmark Factory Components
32
Benchmark Factory Console
32
Agent(s)
33
Run Reports Node
34
Cutting and Pasting of Run Reports 34
Testing Server Throughput 34
Testing SQL Scalability 35
Testing Real Application Clusters 35
Views
38
Script View 38
Jobs Queue View 39
Properties View 40 Agents View 40 Output View 40 Properties 41
How Do I...
71
Overview
71
How do I...
71
User Scenario Wizard
72
Profile Creation Wizard
72
Overview 72
Edit a Profile
74
Submit Job Dialog
74
Run Reports Node
76
Cutting and Pasting of Run Reports 77
Testing Server Throughput 77
Testing SQL Scalability 78
Testing Real Application Clusters 78
Advanced Creation Objects (Object Mapping)
79
Benchmark Objects Wizard
79
Changing Scaling Factors
81
To Change Scale Factors 82
Execute File Wizard
83
Overview 83
Script Assist 85
SQL Scalability Wizard
86
Overview 86 OverviewTab 87 Transactions 88 Summary 90 Messages 90Copy a Load Scenario to a Replay, Mix, Goal, or Scalability Test
91
Running SQL Preview
92
Replace Child Latencies
93
Goal Testing Quick Start
94
Purpose of this Quick Start 95
Actions Performed During This Quick Start 95
Use Global Search and Replace
104
Creating an ODBC Trace File
105
Import Profiles from the Repository
105
Real Application Clustering (RAC) Quick Start
106
Overview 106
Purpose of this Quick Start 106
Actions Performed During This Quick Start 106
Save the Selected Job to the Script View
116
Schedule A Job
116
About Latency
118
Set the Number of Users on an Agent
119
Stagger User Startup
120
Benchmark Factory Console 121
Benchmark Factory Agent 121
Running a Job With Multiple Agents 121
Creating Load Scenarios
123
Load Scenario Wizard
123
Overview 123
Settings
125
About Settings
125
General Settings
125
Benchmarks Settings
127
Timing Settings
128
User Load (Measurement Intervals)
129
Latency Settings
131
Repository Settings
133
Agent Settings
134
Drivers
136
Driver Selection
136
Adding BFScript to the Profile Page 136
IBM DB2CLI 137
Microsoft SQL Server (Native) 138
Microsoft SQL Server (ODBC) 138
MySQL (ODBC) 139 MySQL ODBC 139 Oracle 139 Sybase 140
Profiles
142
Overview
142
Import Profiles
143
Profile Properties
143
Overview 143
General Tab 144
Options Tab 145
Profile Properties Real-Time Counters Tab 146
Clustering Tab 149
Wizards
150
Overview
150
Benchmark Objects Wizard
150
Benchmark Objects Wizard 150
Data Repository Migration Wizard
152
Data Repository Migration Wizard 152
Execute File Wizard
153
Execute File Wizard 153
Load Scenario Wizard
154
Load Scenario Wizard 154
Industry Standard Benchmark (TPC-C, TPC-H...) 155
Validate Oracle RAC Cluster Scalability 156
Test Database Scalability 157
Test SQL For Scalability 159
Replay Load from an Oracle Trace File 160
Test Stored Procedures for Scalability 162
Replay Load from SQL Server 2000/2005 Trace Table 163
Import from Foglight Performance Analysis Repository 164
Import from a Delimited Text File 165
Import from Oracle Dynamic Performance View 168
Import from SQL Files 169
Create a Custom Load Scenario (Scalability, Goal...) 170
Mix Test 170
Replay Test 171
Goal Test 172
Scalability Test 174
Profile Creation Wizard
175
Profile Creation Wizard 175
BFScript
176
BFScript Wizard 176
SQL Scalability Wizard
179
SQL Scalability Wizard 179
Submit Job Dialog
184
Submit Job Dialog 184
Transaction Wizard
186
Transaction Wizard 186
Test Stored Procedures for Scalability 187
Create SQL Statement Transaction 188
Create Prepared SQL Statement Transaction(s) 189
Create Execute Prepared SQL Statement Transaction 193
Import Transactions 193
User Scenario Wizard
194
User Scenario Wizard 194
Import and Replay Load Scenarios From an Oracle Trace File 194
Replay Load from a SQL Server Trace Table 195
Import from a Delimited Text File 196
Testing Results (Run Reports)
198
Benchmark Factory Testing Results
198
Benchmark Factory Testing Results Terminology
198
Exporting Test Results to Excel Using the Command Line
199
Real-Time Load Testing Results
200
Overview-Viewing Real-time Statistics Workloads 200
Change Graph Views 200
Run Status Node 203
Real-Time Statistics Node 203
Summary Node 205
Transaction Node 206
Benchmark Factory Run Reports
207
Run Reports Node 207
Change Graph Views 210
Run Reports Viewer 212
View Load Testing Results 213
Using Benchmark Factory Run Reports 214
View Multiple Run Results 220
Export Test Results to Excel 221
Benchmarks
222
Overview
222
Realistic Expectations When Using Benchmarks 222
What Benchmarks Measure 222
Provided Benchmarks 222
AS3AP Benchmark
223
Best Practices 223
How the Scalable Hardware Benchmark Works 224
Scaling Factor 224
Best Practices 224
TPC-B Benchmark
225
Overview 225
Certification of Transaction Processing Council (TPC) Testing Results 225
Best Practices 225
Scaling Factor 226
TPC-C Benchmark
226
Overview 227
TPC-C Tables 227
Certification of Transaction Processing Council (TPC) Testing Results 228
Best Practices 228
TPC-D Benchmark
229
Overview 229
Certification of Transaction Processing Council (TPC) Testing Results 229
Best Practices 229
Scaling Factor 230
TPC-E Benchmark
230
Overview 230
Certification of Transaction Processing Council (TPC) Testing Results 230
Best Practices 231
Scaling Factor 231
TPC-H Benchmark
231
Overview 231
Certification of Transaction Processing Council Testing Results 232
Best Practices 232
About Scripts
234
Using Scripts in Username and Password Fields
236
Dates and Numbers
236
$BFCreditCardExp 236 $BFCurrentDate 236 BFCurrentDateTime 237 $BFDate 238
File Access
239
$BFFileArray 239Global Variable
240
$BFGetGlobalVar 240 $BFSetGlobalVar 240 $BFSetGlobalVarRtn 241Name and Address
241
$BFAddress 241 $BFAddress2 241 $BFCity 242 $BFCompany 242 $BFCountry 243 $BFEmail 243 $BFFirstName 243 $BFFullName 244 $BFLastName 244 $BFMiddleInitial 244 $BFPhone 245 $BFState 245 $BFZipCode 246
Numerical Manipulation
246
$BFFormat 246 $BFProd 247 $BFSum 247Random Numbers
248
$BFRand 248 $BFRandRange 248 $BFURandRange 249 $BFURand 249Random String
250
$BFCreditCard 250 $BFRandList 250 $BFRandMultiList 251 $BFRandStr 252 $BFURandList 253 $BFList 254String Manipulation
255
$BFAsc 255 $BFChr 255 $BFConcat 255 $BFLeft 256 $BFLen 256 $BFLower 257 $BFMid 257 $BFRight 258 $BFTrim 258 $BFTrimLeft 258 $BFTrimRight 259Test Info
260
$BFGetVar 260 $BFMaxNode 260 $BFNode 261 $BFNumberOfIterations 261 $BFProfile 262 $BFRunID 262 $BFSetVar 263 $BFSetVarRtn 263 $BFUserCounter 264 $BFUserID 264 $BFUserLoad 265 $BFNextUserload 265 $BFPrevUserload 265Quick Starts
267
About the Quickstarts
267
Real Application Clustering (RAC) Quick Start
267
Overview 267
Purpose of this Quick Start 268
Actions Performed During This Quick Start 268
Upgrading Your Database Server Quick Start
278
Overview 278
Purpose of this Quick Start 278
Actions Performed During this Quick Start 279
Oracle Trace Import-Database Replay Quick Start
286
Purpose of this Quick Start 286
Standard Benchmark Quick Start
293
Purpose of this Quick Start 293
Actions Performed During this Quick Start 293
Goal Testing Quick Start
300
Purpose of this Quick Start 300
Actions Performed During This Quick Start 301
Troubleshooting
310
Agent Connection
310
Use Benchmark Factory with SQL Server 2005 Client
310
MySQL Initialization Settings
310
Appendix
311
Change Graph Views
311
Show Data/Show Graph 311
Graph Legend 312
Toolbar 312
Print 313
Copy Data to Clipboard 313
Copy Graph to Clipboard 313
Load Configuration 313
Save Configuration 313
Set as Default 313
Clear Configuration 313
Customize List Controls
313
Create a SQL 7 Trace Table Using the SQL Server Profiler
314
Create a SQL 2000 Trace Table Using the SQL Server Profiler
315
Create a SQL 2005 Trace Table Using the SQL Server Profiler
316
Oracle Instant Client Installation
317
Store Procedure Examples
318
Oracle Trace File Activation
320
Support Bundle
321
System Requirements/Upgrade Requirements/Supported Databases
322
System Requirements 322
Shortcut Keys
324
Copyright, Licensing, and Licensing Agreement
327
Licensing and Virtual User Packs
328
Modify the Benchmark Factory License 329
Modify the Benchmark Factory Virtual User Pack 330
330
330
Appendix:Contact Quest
342
Contact Quest/About Quest Software
342
About Quest 342
Contacting Quest Software 342
Contacting Quest Support 342
Getting Started
Getting Started
Benchmark Factory is an intuitive, easy to use database load testing tool. The following links provide you with information to quickly start the testing process and gather results.
l Video: Take a Tour of Benchmark Factory l About Benchmark Factory (page 19)
l Understanding the Benchmark Factory Environment (page 20) l Understanding Benchmark Factory Terminology (page 21) l How Benchmark Factory Works (page 24)
l Benchmark Factory Components
l Create a Script With Benchmark Factory (page 25) l Benchmark Factory Testing Results (page 198) l Quick Starts
l Real Application Clustering (RAC) Quick Start (page 267) l Upgrading Your Database Server Quick Start (page 278) l Oracle Trace Import-Database Replay Quick Start (page 286) l Standard Benchmark Quick Start (page 293)
l Goal Testing Quick Start (page 300)
l Quest Benchmark Factory Web Site (contains additional videos and white papers)
Benchmark Factory Community
Get the latest product information, find helpful resources, and join a discussion with the Benchmark Factory for Databases team and other community members. Join the Benchmark Factory for Databases community athttp://www.benchmarkfactory.inside.quest.com.
New For This Release
New for Benchmark Factory 6.5
l New Oracle Trace Record and Playback Wizard-A new wizard has been added that walks users through the entire process of capturing the Oracle workload, including the database backup.
l Enhanced Connection Profile Page-The connection profile pages have been enhanced to be more specific to the particular database vendor (Oracle, Microsoft SQL Server, etc.). l Oracle Profile Improvements-The Oracle profile now supports multiple Oracle Homes,
Direct Connections, as well as being able to specify "Connect As". The Oracle Profile also has a new Statistics page to allow the user to automatically capture Statspack or AWR snapshots during test execution.
l New Activation Window-The new Activation window provides trial users with more information such as expiration date, days remaining, licensing information, etc. l New Licensing Window-A new licensing window is now available to provide better
information about the user's licensing for the different Benchmark Factory licensing keys.
New for Benchmark Factory 6.1.1
This release of Benchmark Factory for Databases provides critical bug fixes.
New for Benchmark Factory 6.1
l Support for 11gR2.
l New global variable support for BFScript. l Enhanced bind types.
New for Benchmark Factory 6.0
l The TPC-E benchmark is now available in this release for both Oracle and Microsoft SQL Server.
Understanding Benchmark Factory
About Benchmark Factory
Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. This enables developers, DBAs, and QA teams to validate that their databases will scale as user load increases, application changes are made, and platform changes are
implemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL, and other databases via ODBC connectivity. The following topics provides an overview of the Benchmark Factory Environment:
l The Benchmark Factory Environment (page 19)
l Understanding the Benchmark Factory Environment (page 20) l Understanding Benchmark Factory Terminology (page 21) l How Benchmark Factory Works (page 24)
l Create a Script With Benchmark Factory (page 25)
The Benchmark Factory Environment
Benchmark Factory places enormous stress on a database system, which is typically hard to achieve in a standard testing environment. A system typically breaks under extreme load. By identifying system capacity and performance bottlenecks before they occur, Benchmark Factory facilitates proactive testing, which in turn reduces downtime, development costs, and potential loss of revenue. Benchmark Factory allows you to:
l Determine system throughput and capacity for database systems
l Simulate thousands of concurrent users with a minimal amount of hardware l Find applications that do not scale well with an increase in the number of users l Find breaking points, weak links, or bottlenecks of a system
l Quantify application or server performance
All test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a vast amount of statistics, including overall server throughput (measured in transactions per second, bytes transferred, etc.) and detailed transaction statistics by
Integration with Other Quest Software Products
Benchmark Factory integrates with otherQuest Softwareproducts to allow IT departments to quickly measure the capacity and performance of their systems to ensure that users will experience fast response times. For instance, Benchmark Factory can run multiple workloads in conjunction with Spotlight™ products. This provides the ability to detect and diagnose issues, allowing users to resolve bottlenecks, slow performance, and application flaws before an application is entered into production.
Understanding the Benchmark Factory Environment
Note: The Benchmark Factory Quick Starts provide procedures that introduce you to the
Benchmark Factory environment. See "About the Quickstarts" (page 267) for more information. The following provides a narrative on the terminology, application, and mechanics of using Benchmark Factory.
Automated Testing, Virtual Users, and Agents
When you load test with Benchmark Factory, you are performing automated testing. Automated testing is the process of using software to test computer hardware or software implying that the software is used instead of actual users. The software acts as a "virtual" user. For example, say you would like to test your database with two hundred virtual users over a given period of time. Benchmark Factory lets you select two hundred virtual users and the length of time you wish to perform the test.
Benchmark Factory usesAgentsto deploy the virtual users. An Agent is a software routine that waits in the background and performs an action when a specified event occurs. One Agent can simulate thousands of virtual users (limited by hardware and workload characteristics) at a time. Each virtual user has their own connection to the system under test.
Understanding Benchmarks
A benchmark is a performance test of hardware and/or software on a system-under-test.
Benchmark Factory provides the option of using industry standard benchmarks during the load testing process. Benchmarks measure system peak performance when performing typical operations.
Benchmark Factory comes equipped with the following industry standard benchmarks: l AS3AP Benchmark (page 223)
l TPC-C Benchmark (page 226) l TPC-C Benchmark (page 226) l TPC-D Benchmark (page 229) l TPC-E Benchmark (page 230) l TPC-H Benchmark (page 231)
Understanding Benchmark Factory Terminology
The following provides a list of terminology required to understand the Benchmark Factory load testing process.
l A script is the user's workspace that allows you to create and save jobs.
l Scripts are created in the Script View. See "Script View" (page 38) for more information.
l When ready to run, jobs are submitted to the Jobs Queue View. See "Jobs Queue View" (page 39) for more information.
l The Load Scenario Wizard is the starting point for the creation of scripts. See "Load Scenario Wizard" (page 154) for more information.
l A job contains load scenario, execution of a file, creation/deletion of benchmark objects, or a combination of each.
l A load scenario is an assembled test comprised of user scenarios and/or transactions. These tests can be run with multiple virtual users. A load scenario can be one of the following:
l Mixed Workload: A Mixed Workload test runs for a specified time at each predetermined user load level. Each user will run a transaction mix based upon the weights defined on the transactions tab. For example, if a test has two transactions, A and B, with A having a weight of one and B having a weight of four, on average B will run four times for every time A is run once. The run order will be randomly generated for each user so they are not all running the same transaction simultaneously. That run order is used for that user each time the test is performed to ensure reproducible results.
l Goal Test: A goal test is used to find maximum throughput or response time goals. A transaction mix is executed at user load levels, determined by setting a beginning, ending, and interval value. When run, the specified goal criterion is evaluated at the end of each iteration and the test ends once the goal or maximum user load has been reached.
l Replay Test: A Replay Test runs multiple transactions with each one running independently on a specified number of users. The test will run until the defined number of executions for each transaction or a specified time limit is reached.
test could have two transactions, A and B, and two userloads of 10 and 20, with an iteration length of one minute. Transaction A would execute continually for one minute at userload 10, then B would do the same. Next A will run at userload 20, followed again by test B, for a total time of 4 minutes.
Click here for load scenario creation procedures.
l A user scenario is a series of one or more transactions to be executed in order to make a single transaction, normally associated with user behavior simulated against the system-under-test. An Execute SQL Statement from a file User Scenario will execute SQL statements as defined in an XML file. See "User Scenario Wizard " (page 194) for more information.
l A transaction is a single unit of work in the testing process, used in load scenarios and user scenarios. For example, a SQL statement. Click here for procedures to create a transaction.
l Execute a file allows you to execute a file during the running of a job. See "Execute File Wizard" (page 153) for more information.
l Create or delete benchmark objects configures a system-under-test for industry standard benchmark tests to measure system performance. Each standard benchmark has been developed with specific system configuration requirements, that include tables, indexes, data, etc. See "Benchmark Objects Wizard" (page 150) for more information.
How Benchmark Factory Works
The following steps provide an overview of how Benchmark Factory components interact during the load testing process.
1. The Benchmark Factory Console (page 32) implements the workload testing process and controls one or more distributed agent machines. Each agent machine can simulate thousands of users. Each simulated user executes transactions and records statistics. 2. Benchmark Factory Agent(s) (page 33)machines simulate virtual users. The Agents send
transactions to the system-under-test. The Agents record statistics that includes how much data the resulting transaction contained and how long it took to get the results. At the end of an iteration, each agent machine reports its findings back to Benchmark Factory. 3. A server (system-under-test) is the database Benchmark Factory connects to. Benchmark
Factory is server neutral and network protocol independent. Benchmark Factory uses vendor client libraries of the system-under-test. Any system that the client software can support, Benchmark Factory can support. Its only requirement is that Agent machines must be able to connect to a server through an appropriate manner and it supports the test system.
4. The Repository Manager (page 36) stores all testing data. 5. Results are viewed from Run Reports Node (page 207).
Create a Script With Benchmark Factory
The following provides a narrative discussion on how to create a script using Benchmark Factory. This process consists of following steps:
1. Set up Benchmark Factory 2. Create a Script
3. Create a Profile 4. Create a Job 5. Run the Script 6. Analyze Test Results
Note: The Benchmark Factory Quick Starts provide procedures that introduce you to the
Benchmark Factory environment. See "About the Quickstarts" (page 267) for more information.
1. Set up Benchmark Factory
The Settings window displays default values used when creating new scripts. Changes to these settings affect only new load scenarios, not existing load scenarios.
To Display Benchmark Factory Settings
l Select Edit | Settings. The settings dialog displays. The Settings window provides the following tabs:
l General Settings (page 125) l Benchmarks Settings (page 127) l Timing Settings (page 128) l Latency Settings (page 131) l Repository Settings (page 133) l Agent Settings
2. Create a Script
Using the Benchmark Factory graphical user interface, you can create scripts in the Script View. You open one script a time when using Benchmark Factory. See "Benchmark Factory Console" (page 32) for more information.
All tasks required to complete a Benchmark Factory script can be accomplished using wizards. The first wizard you will use is the Load Scenario Wizard. The Load Scenario Wizard takes you through the steps required to create a script and complete a database load test. See "Load Scenario Wizard" (page 154) for more information.
The first step the Load Scenario wizard asks you to perform is to select a load scenario type from the load scenario dialog. Load Scenarios are an assembled load test comprised of:
l User scenarios: A series of one or more transactions to be executed in order normally associated with a behavior to be tested on the system-under-test. These are the components used to build a load scenario.
l Transaction: A single unit of work in the testing process, such as retrieving a Web page, executing a SQL statement, writing a file, or sending an email.
After selecting the desired task type, the Load Scenario Wizard will guide you through the steps for script creation, depending on which type of task you select.
3. Create a Profile
Profiles are sets of information that Benchmark Factory uses to communicate with the system-under-test . This information includes, server name, IP address, and user name. You create profiles using the:
4. Create a Job
A job holds a load scenario, execution of a file, creation/deletion of benchmark objects, or a combination of each. Jobs are created in Benchmark Factory when a load scenario is added to a script. A script can hold as many jobs as you chose to create.
Change Script Properties
Benchmark Factory script properties can be changed or adjusted to meet load testing requirement. You change properties by clicking on the node of the property you wish to change. For example, if you wish to change the user load test you click on the Load Scenario node. SeeAbout the Script Nodefor more information.
To change the User Load properties, select the User Load tab and select the desired number of users.
5. Run the Script
After completing the wizards and setting in the desired properties, you run the job by selecting the Job Node and clicking the Submit Job icon , or right-clicking and selecting Submit Job. See "Submit Job Dialog " (page 184) for more information.
6. Analyze Test Results
Benchmark Factory provides a number of metrics to analyze a system-under-test. These include: l Response, retrieval, and transactions times
l Transaction throughput in transactions per second (TPS) l Amounts of data transferred in bites per second (BPS) l Errors incurred
l Any user defined counters (Windows NT/2000 Performance Monitor) added using the Benchmark Factory Console.
Benchmark Factory provides the following testing results:
l Real time statistics (See "Real-Time Statistics Node " (page 203) for more information.) l Run Reports (See "Run Reports Node " (page 207) for more information.)
l Testing results exported to Excel (See "Export Test Results to Excel" (page 221) for more information.)
Benchmark Factory Console
The Benchmark Factory console implements the database workload testing process. This interface is where:
l Scripts are created and initiated. A script is the user workspace that allows you to design and save jobs.
l Jobs are created and submitted to the Jobs Queue View. See "Jobs Queue View" (page 39) for more information.
l All scriptpropertiesare set. SeeAbout the Script Nodefor more information. l Settings can be made. See "About Settings" (page 125) for more information.About
Scripts (page 234)
l Users can access test results. See "Benchmark Factory Testing Results" (page 198) for more information.
l Agents are managed. See "Agent(s)" (page 33) for more information. The Benchmark Factory Console contains the following views:
l Script View (page 38) l Jobs Queue View (page 39) l Properties View (page 40) l Agents View (page 40) l Output View (page 40)
Agent(s)
Benchmark Factory Agent(s) reside on each physical client machine and spawn multiple virtual-user sessions. Each Agent generates an information screen with tabs that contain a variety of Agent information. Benchmark Factory can control hundreds of Agent machines. Each Agent can simulate up to 1500 virtual users, depending upon hardware and test configuration. Virtual users simulate the load against the system-under-test. Each virtual user is a separate thread, acting independently of the other virtual users, with its own connection to the system-under-test. Each virtual user tracks its own statistics that includes transaction times and the number of times a transaction executes.
The Agent provides the follow tabs: l Agent Test Status Tab l Agent Virtual Users Tab l Agent Transaction Tab l Agent Output Tab
Run Reports Node
Benchmark Factory Run Reports is a separate executable that opens outside of Benchmark Factory and provides a comprehensive and detailed collection of database load testing results. With Benchmark Factory you can drill down into a database to view a wide array of information and statistics that gives you accurate insight into database performance. Run Reports Viewer allows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page 212) for more information.
Note: Three instances of Run Reports can be viewed at one time.
Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. See Cut and Pasting Reportsfor more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you to view a variety of server throughput statistics that include:
l Transaction/Second (TPS) l Transaction Time l Bytes/Second (BPS) l Rows/Second (RPS) l Total Bytes l Total Errors l Total Rows l Response Time
Testing SQL Scalability
The Benchmark Factory transaction comparison graph provides users with a comparison of the individual transaction performance as they compare to each other. This is most useful when analyzing a SQL Scalability test.
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determine system capacity. The cluster performance graph allows users to view the performance of
Repository Manager
Note: If you create a new Benchmark Factory 5.5 repository, earlier versions of Benchmark
Factory will not work against this repository.
The Repository is a database where all of the test results are stored. Benchmark Factory inserts test results into the repository and provides an easy way to access the data. By default, the Repository is a MySQL database that resides on the same machine as Benchmark Factory for Databases. The Repository can reside on another database server if required. Tto change the database select the Data Source Name of the ODBC connection for the new database. Note: If the database structure does not exist on the selected database, a prompt to create the
structure will appear when OK is clicked.
The supported repository database types are: l Sybase Adaptive Server Anywhere l Microsoft SQL Server
l MySQL l Oracle
To launch the Repository Manager
1. Select Start | All Programs | Quest Software | Benchmark Factory for Databases | Repository | Repository Manager. The Repository Manager dialog displays.
Section Field/Icon Description Connection
Parameters
Data Source Name
Data Source name of the ODBC connection used to connect to the repository database. ODBC Driver Current ODBC driver
User Name The User Name used to log into the selected database.
Password The Password associated with the user name used to log into the database.
Displays the ODBC connection information dialog for the selected data source.
Displays the ODBC Data Source
Administrator dialog, used to add and edit ODBC connections.
Tests the connection of the currently selected ODBC Data Source.
Maintenance Creates the repository objects on the selected database.
Deletes the repository objects on the selected database.
Warning: This will delete all test results
stored in the Repository.
Launches the Data Migration Wizard. See "Data Repository Migration Wizard" (page 152) for more information.
Views
Script View
The Script View displays the currently open script and provides user workspace that allows you to design and save jobs. When ready to run, jobs are submitted to the Jobs Queue. The Script View Toolbar provides user functionality when creating or editing load testing scripts. Note: You must be in the script view to create jobs and edit jobs.
Script View Toolbar
The Script View toolbar provides user functionality when creating or editing load testing scripts.
Creates a blank job. You must click on the Script Node to active the icon.
Displays the Load Scenario Wizard (page 154)
Displays the Transaction Wizard (page 186)
Displays the Benchmark Objects Wizard (page 150)
Displays the Execute File Wizard (page 153)
Displays the Submit Job Dialog (page 184)
Moves an item up in the Script View
Moves an item down in the Script View
Jobs Queue View
The jobs view displays the currently open jobs. From the Jobs Queue, jobs are run, status can be viewed, and real-time testing results can be viewed. The Jobs toolbar provides user functionality when running a job.
Jobs Queue icons provides user functionality when running jobs.
Deletes the selected job from the Jobs Queue.
Saves the selected job to the Script View. See "Save the Selected Job to the Script View" (page 116) for more information.
Expands all jobs when the root is selected. Collapses all jobs when the root is selected.
Properties View
The Properties View allows you to monitor and managed scripts. Properties can be viewed for the:
l Script Node Properties (page 41) l Jobs Queue Node Properties (page 55)
Agents View
The Agent view displays the status of connected agents.
Output View
The Output window displays errors and messages that occur outside of running a job. For example, a .dll error or re-registering a benchmark.
Properties
About the Properties View
Benchmark Factory properties allow you to: l Adjust script parameters
l View script properties
l Access Benchmark Factory Run Reports See "Run Reports Node " (page 207) for more information.
l Viewrealtime statistics (See "Real-Time Statistics Node " (page 203) for more information.
l Set and view latency See "About Latency " (page 118) for more information. The following provides Benchmark Factory properties definitions:
l Script (SeeAbout the Script Nodefor more information.)
l Job (See "Jobs Queue Node Properties" (page 55) for more information.) l Latency (See "About Latency " (page 118) for more information.)
l Benchmark Properties See "Benchmark Properties" (page 67) for more information.)
Script Node Properties
Script Node
Jobs Node
Note: In the Jobs Queue view, not all properties are editable. The Jobs node provides the following tabs:
l General l Job Schedule l Jobs Alerts
l Real-Time Counters
l Global Variables Node (page 56)
Load Scenario Node
Note: In the Jobs Queue view, no properties are editable. The Load Scenario node provides the following tabs:
l Transaction Tab l Goal Test (page 172)
l Timing Tab
l Options Tab-Mixed Workload, Goal, and Scalability Test (page 48)
User Scenario Node
The User Scenario node properties allows you to: l Delete transactions
l Adjust transaction order
l Adjust latency (See "About Latency " (page 118) for more information.)
User Scenario From File
The User Scenario From File node properties allows you to: l Executing a SQL Statements from a File (page 44)
When large Oracle trace files are imported into Benchmark Factory, they are converted to editable XML files.
To Edit SQL Statements from a file
1. Click Edit.
2. Click File | Open.
The Open dialog displays.
3. Navigate to and select the required file XML file. 4. Click Open.
5. Under Session Details, click on the SQL statement you wish to edit. That statement displays in the SQL Statement view.
6. From the Prepare SQL tab, you can click in the upper right-hand corner to: l Run SQL Preview See "Running SQL Preview" (page 92) for more
information.
l Launch the BFScript Wizard See "BFScript Wizard" (page 176) for more information.
Transaction Node
Note: In the Jobs Queue view, no properties are editable.
The transaction node displays properties based on transaction type.
To view the Transaction node
Options Tab-Replay Test
The Repository Properties tab controls what information is saved in the Repository during the running of a test, and allows you to set error handling properties. The database where all Benchmark Factory information is saved. It contains information about the transactions stored in each benchmark DLL and is where all of the statistical information is stored about benchmark executions.
To change Repository properties
1. Click the Load Scenario node. 2. Click the Options tab.
Setting Description
Repository Settings
Save Real-Time Counter Information
Saves Benchmark Factory real time counters and performance monitor counters to the repository during test.
Save User Statistics Saves user level statistics.
Save Distribution Saves transactions per second and response time distribution.
Options Tab-Mixed Workload, Goal, and Scalability Test
The Options tab specifies Repository, error handling, file execution, and database checkpoint settings.
Section Field/Icon Description
Repository Settings SaveReal-Time Counter Information
Note: To change real-time counters please refer to Profile Properties Real-Time Counter Tab.
Saves counters and performance monitor counters to the Repository during test. Sample rate is determined in job settings.
Save User Statistics Saves user level statistics
Section Field/Icon Description
second and response time per user distribution. Log Transaction Results to disk Logs the transaction results
of a job to disk.
Note: Checking this option imposes performance penalties.
Error Handling Stop test after first error When checked, the test is stopped when an error is reported.
Execute the following program at the beginning of each iteration
File Name
Note: This field accepts BFScripts.
Name of the program to execute at the beginning of an iteration.
Enforce Timeout Enforces a timeout on the file being executed.
Execute the following program at the end of each iteration
File Name
Note: This field accepts BFScripts.
Name of the program to execute at the end of an iteration.
Enforce Timeout Enforces a timeout on the file being executed so that if the program does not complete in the specified time, it will be stopped, and the load scenario will continue.
Database Checkpoints Perform checkpoint at start of each test iteration
Initiates a database
checkpoint at the beginning of a test iteration.
Perform checkpoints during each iteration
Initiates a database checkpoint during a test iteration.
Number of checkpoints Specifies the number of checkpoints to initiate.
Run Reports Node
Benchmark Factory Run Reports is a separate executable that opens outside of Benchmark Factory and provides a comprehensive and detailed collection of database load testing results. With Benchmark Factory you can drill down into a database to view a wide array of information and statistics that gives you accurate insight into database performance. Run Reports Viewer allows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page 212) for more information.
Note: Three instances of Run Reports can be viewed at one time. Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. See Cut and Pasting Reportsfor more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you to view a variety of server throughput statistics that include:
l Transaction/Second (TPS) l Transaction Time l Bytes/Second (BPS) l Rows/Second (RPS) l Total Bytes l Total Errors l Total Rows l Response Time
Testing SQL Scalability
The Benchmark Factory transaction comparison graph provides users with a comparison of the individual transaction performance as they compare to each other. This is most useful when analyzing a SQL Scalability test.
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determine system capacity. The cluster performance graph allows users to view the performance of
SQL Statement Node
The SQL statement node displays individual SQL statements and allows you edit a SQL statement or use the BFScript Wizard. See "BFScript Wizard" (page 176) for more information.
Create/Delete Benchmark Objects Node Note: In the Jobs Queue view, no properties are editable.
The Create/Delete Benchmark node allows you to either create the benchmark objects or delete the benchmark objects. Benchmark objects are used by standard benchmark load scenarios. Benchmarks can be added to a job using the Benchmark Objects Wizard. See "Benchmark Objects Wizard" (page 150) for more information.
Field Description
Benchmark Name of the benchmark whose benchmark properties are being edited.
Scale The factor used to increase the amount of data added to the database.
Valid benchmark scales are dependent upon the benchmark selected. For example, the AS3AP scales by a factor of 10 with valid scales of "1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1, with valid scales of "1, 2, 3, 4..."
Displays the Benchmark Properties page and allows you to edit the benchmark scale and map database tables and indexes. See
"Benchmark Properties" (page 67) for more information.
Always delete objects before create
Checking this box deletes benchmark objects before creating new ones.
When unchecked, Benchmark Factory checks for the benchmark tables. If they exist, then the creation will be skipped.
Note: The scale factor is not checked.
Execute File Node
Note: In the Jobs Queue view, no properties are editable.
The Execute File node allows you to place a file execution in a job as a sibling to a load scenario. The file executes during the running of a job.
Field Description
Execute File Type in or Select the File Name to execute:
Name of the file you wish to execute.
Note: This field accepts BFScripts. See "About Scripts" (page
234) for more information.
Click to open a file.
Enforce Timeout Places a time constraint on the number of seconds a file has to execute.
Field Description between 30 and 10000
[n] Secs.
Jobs Queue Node Properties About the Jobs Node
Some of the nodes in the Jobs Queue are the same as the Script view.
Note: Not all nodes in the Jobs Queue are editable and provide property viewing only. The following nodes display:
l Jobs Queue Node
l Load Scenario Node (page 56) l User Scenario Node (page 56) l Transaction Node (page 57)
l Create/Delete Benchmark Objects Node (page 57) l Run Reports Node (page 207)
l Run Status Node (page 203)
l Real-Time Statistics Node (page 203) l Summary Node (page 205)
l Messages Node (page 64)
Jobs Node
Note: In the Jobs Queue view, not all properties are editable. The Jobs node provides the following tabs:
l General l Job Schedule l Jobs Alerts
Global Variables Node
From the global variable node, you can add or delete the required variables.
To add a global variable:
1. Right-click to display the global variable dialog. 2. Enter the name and value.
3. Click Add.
Load Scenario Node
Note: In the Jobs Queue view, no properties are editable. The Load Scenario node provides the following tabs:
l Transaction Tab l Goal Test (page 172)
l User Load Tab (Measurement Intervals) l Timing Tab
l Options Tab-Mixed Workload, Goal, and Scalability Test (page 48)
User Scenario Node
The User Scenario node properties allows you to: l Delete transactions
l Adjust transaction order
Transaction Node
Note: In the Jobs Queue view, no properties are editable.
The transaction node displays properties based on transaction type.
To view the Transaction node
» Click the Transactions node in the Jobs Queue or Script View.
Create/Delete Benchmark Objects Node Note: In the Jobs Queue view, no properties are editable.
The Create/Delete Benchmark node allows you to either create the benchmark objects or delete the benchmark objects. Benchmark objects are used by standard benchmark load scenarios.
Field Description
Benchmark Name of the benchmark whose benchmark properties are being edited.
Scale The factor used to increase the amount of data added to the database.
Valid benchmark scales are dependent upon the benchmark selected. For example, the AS3AP scales by a factor of 10 with valid scales of "1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1, with valid scales of "1, 2, 3, 4..."
Displays the Benchmark Properties page and allows you to edit the benchmark scale and map database tables and indexes. See
"Benchmark Properties" (page 67) for more information.
Always delete objects before create
Checking this box deletes benchmark objects before creating new ones.
Field Description
When unchecked, Benchmark Factory checks for the benchmark tables. If they exist, then the creation will be skipped.
Note: The scale factor is not checked.
Run Reports Node
Benchmark Factory Run Reports is a separate executable that opens outside of Benchmark Factory and provides a comprehensive and detailed collection of database load testing results. With Benchmark Factory you can drill down into a database to view a wide array of information and statistics that gives you accurate insight into database performance. Run Reports Viewer allows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page 212) for more information.
Note: Three instances of Run Reports can be viewed at one time. Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. See Cut and Pasting Reportsfor more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you to view a variety of server throughput statistics that include:
l Transaction/Second (TPS) l Transaction Time l Bytes/Second (BPS) l Rows/Second (RPS) l Total Bytes l Total Errors l Total Rows l Response Time
The Benchmark Factory transaction comparison graph provides users with a comparison of the individual transaction performance as they compare to each other. This is most useful when analyzing a SQL Scalability test.
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determine system capacity. The cluster performance graph allows users to view the performance of
Run Status Node
Run Status Node properties displays information on the job when it is running. You can monitor the status of the test, which userload its running, transactions per second (TPS), and a variety of other datapoints that give you insight into the actual performance your database provides under the userload being tested.
Iteration Overruns
Iteration overruns occur at the end of an iteration to allow time for all transactions submitted within the test iteration cycle to complete, so that all transaction statistics can be collected. For example, an agent may execute a transaction during the last five seconds of test iteration, if this transaction takes 15 seconds to complete, an iteration overrun of 10 seconds will occur.
To view the Run Status node
1. Click the Run Status node in the Jobs Queue.
Real-Time Statistics Node
The Real Time Statistics node provides you with real-time graphs and raw data. This data allows you to spot system-under-test issues that may be affecting server performance. The Real Time Statistics node displays real-time counter data. Right-clicking inside the graph displays a drop-down that allows you to change graph settings and view.
To view the Real-Time Statistic node
2. Select Click here to Add Datapoints. The Add Datapoints dialog displays.
3. Select the desired datapoints to view. 4. Click OK.
5. To change graph views, right-click. See "Change Graph Views" (page 311) for more information.
Summary Node
The Summary node displays user selected statistics for the job timing period following completion of an iteration.
To view the Summary node
1. Click the Summary node in the Jobs Queue.
information.
5. Select the desired option.
Messages Node
The Messages node displays: l Job status
l Any errors or warnings
Latency
About Latency
Latency is the speed in which transactions are sent to a server or the period between receiving results of one transaction and sending the next transaction to a serve. Benchmark Factory allows you to select the following latency models and distribution models:
Latency Types Distribution Models
No Delay (page 65) Absolute Distribution (page 66)
Interarrival Time (page 65)
Uniform Distribution (page 66)
Keying Time (page 65) Negative Exponential Distribution (page 66)
Think Time (page 65) Normal Distribution (page 67)
No Delay
No Delay means that transactions execute as fast as possible. As soon as one transaction is processed, the next transaction is issued against the server. In the case of a mixed workload test, each virtual user issues transactions as fast as possible. The No Delay option is used when the goal of the test is to stress the system to its limits, without concern for accurately simulating users. With No Delay specified, a relatively low number of users can stress the system to its limits. As such, there is no easy way to correlate N virtual users running with no delay to some number of real users.
Interarrival Time
Interarrival Time derives its name from achieving a specific transaction rate as seen by a server (the rate at which transactions are arriving at a server.) When specifying Interarrival Time, Benchmark Factory is instructed, that regardless of how long a transaction actually takes to execute, to ensure that the transactions arrive at a server at a specific interval.
Keying Time
Keying Time is used to simulate an amount of time performing data entry, before executing a transaction (entering information). In many cases, Keying Time is used with Think Time to provide a delay before and after a transaction execute. Selecting Keying Time inserts a fixed or variable delay before executing a transaction.
Think Time
Think Time is used to simulate an amount of time "to think" about the results of a previous transaction. This could be time spent performing analysis on the results of a database query. Selecting Think Time inserts a fixed or variable delay after executing a transaction.
About
Benchmark Factory provides the following distribution models l Absolute Distribution (page 66)
l Uniform Distribution (page 66)
l Negative Exponential Distribution (page 66) l Normal Distribution (page 67)
l Poisson Distribution (page 67)
Absolute Distribution
Absolute is a fixed length delay. If a 2000 ms delay is specified for Keying Time (page 65) and a 3000 ms delay is specified forThink Time (page 65),each time the transaction should execute, Benchmark Factory waits 2000 ms, then fires the transaction, then waits an additional 3000 ms before deciding which transaction to execute next. If an Inter-Arrival model is chosen with a delay type of Absolute set to 2000 ms, Benchmark Factory marks the time, executes the transaction, and assuming the transaction finishes in less than two seconds, waits until two seconds from the marked time has elapsed before determining which transaction to execute next.
Uniform Distribution
Selecting a Uniform delay instructs that random delay should be chosen, with an equal
probability of being the minimum value, the maximum value, or any value in between. Uniform delays are chosen when it is suspected that the delay is highly random within a range or a minimal amount of statistical analysis has been performed to determine how the actual users react. Suppose a uniform Keying Time (page 65) is selected with a minimum value of 1000 ms and a maximum value of 1500 ms. If the transaction is executed more than 500 times, there is a high probability that each possible delay has been selected at least once. With the other delay types, this is not the case. If the same 2000 ms to 2500 ms uniform delay is set for Interarrival time, the tester essentially is setting the test so that a server sees the transaction every 2 to 2.5 seconds, instead of exactly two seconds as in the Absolute delay time.
Negative Exponential Distribution
Similar to the Normal distribution, Negative Exponential inserts a random delay based on a mathematical model. A Negative Exponential distribution should be chosen when most users
have latency of the mean latency or less. Negative Exponential is weighted heavily toward smaller latencies being selected relative to the mean and maximum latency.
Normal Distribution
Normal distributions differ from Uniform delays in that most of the delays chosen by Benchmark Factory will be close to the average, but can vary by as much as ±10% of the mean. While a Uniform delay is used when users have latencies within equal likelihood of being anywhere between two values, Normal distributions are chosen when all users fall within a range, but most of the modeled users have a latency close to the average latency.
Poisson Distribution
A Poisson distribution is very similar to the Normal distribution and can be used most places where a Normal distribution delay could be used. The biggest difference between a Normal distribution and a Poisson distribution is that Poisson selects discreet values.
Benchmark Properties Benchmark Properties
Benchmark scale factors are used to increase the size of the database testing tables during the creation of benchmark objects, and to correctly execute the benchmark transactions as needed to fully utilize the database objects created.
The Benchmark Properties page displays information on: l Scales (page 68)Scales (page 68)
l Advanced (page 69) l History Tables (page 69)
The scale dialog specifies the benchmark scale used to create table data.
Field Description
Benchmark Scale The factor used to increase the amount of data added to the database. Valid benchmark scales are dependent upon the benchmark selected. For example, the AS3AP benchmark scales by a factor of 10 with valid scales of "1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1, with valid scales of "1, 2, 3, 4..."
Show Empty Tables
Displays any tables that will be created but not populated with data.
Estimated Size The approximate size that the tables will use in the database without indexes. Additional space is needed for indexes.
Advanced
Advancer mapping provides views of the tables and indexes that allows you to determine where there tables and indexes will be stored in the database. This allows your database to run more efficiently. See "Advanced Creation Objects (Object Mapping)" (page 79) for more information.
History Tables
History tables allows you to set the number of history tables to create. The TPC-B benchmark is made up of only one transaction that updates three tables and inserts a record into a history table. Inserting one record into one history table limits testing performance. The Benchmark Factory properties page allows the user to set the number of history tables to create during a test. The best ratio of history tables to virtual users is based on database configuration and hardware. The number of history tables to use is determined by the tester.
Field Description
Number of History tables
How Do I...
Overview
The How Do I section points you to the procedures required to perform load testing.
How do I...
l Create a Load Scenario- This wizard walks you through the steps tocreate and run scriptsand load scenarios.
l Create a User Scenario - This wizard walks you through the steps to create a user scenario.
l Create a Profile- This wizard walks you through the steps to hook up to the data base you are testing.
l Edit a Profile
l Submit a Job- This wizard walks you through the steps to submit a job to theJobs Queuefor testing.
l Use Run Reports
l Use Advanced Creation Objects
l Create Benchmark Objects for Load TestingThis wizard walks you through the steps to create benchmark objects for load testing.
l Execute a File- This wizard walks you through the steps to execute a file outside of Benchmark Factory.
l Change Scale Factors
l Use Benchmark Factory Scripting Capabilities- This wizard walks you through the steps to use Benchmark Factory built-in functions for use in scripts and the load testing process. l Use SQL Scalability Wizard
l Copy a Load Scenario to a Replay Test, Mix Test, or Goal Test l Run SQL Preview
l Replace Child Latencies l Perform a Goal Test
l Use Global Search and Replace l Create a Customized ODBC Trace File l Import Profiles from the Repository
l Set the Number of Users on an Agent l Stagger User Startup
l Run Benchmark Factory with Multiple Agents
User Scenario Wizard
The User Scenario Wizard allows you to build user scenarios. A user scenario is a series of one or more transactions to be executed in order, normally associated with user behavior simulated against the system-under-test. From the User Scenario Wizard you can:
l Import and Replay Load Scenarios From an Oracle Trace File l Replay load from SQL Server trace table
l Import from a delimited text file l Replay from an ODBC trace File l Create a custom user scenario
Profile Creation Wizard
Overview
Profiles are used by Benchmark Factory to connect to the system-under-test. The profile contains information required to connect to the system-under-test, such as server name, IP address, and user name. The Profile Creation Wizard walks you through steps required to create a Benchmark Factory profile.
To create a profile
1. Select Wizards | Profile Creation. 2. Click Next.
3. From the Driver drop-down list, select the required driver type. 4. Click Next.
6. Enter the Password.
7. From the TNS Driver drop-down, select the required database. or
Click the Direct tab and enter the required fields.
8. From the Connect as drop, select the required connection, Normal, SYSDBA, or SYSOPER.
9. Select the installed client if applicable.
10. Click the Verify connection before proceeding check box to verify the connection to the specified database.
11. Click Next.
12. Enter a profile name. 13. Click Finish.
Edit a Profile
To edit a profile
1.
Click the Edit Profile icon .
2. Double-click on a profile to edit or click View Properties in the upper right-hand corner of the dialog.
3. The Edit Profiles dialog displays. The following options are provided. l Adds a profile l Imports a profile l Deletes a profile l
Displays profile properties allowing you to edit the profile
Submit Job Dialog
A job contains load scenario, execute file, creation/deletion of benchmark objects, or a
combination of each. The Submit Job Wizard walks you through the process of adding a job to the jobs queue. See "Jobs Queue View" (page 39) for more information.
To submit a job to the Jobs Queue
1. Select the Job node. 2. Right-click.
The Submit Job dialog allows you to tailor your job needs and provides the following options:
Section Description
Profile Allows you to select profiles already created.
Displays the Profile Creation Wizard. See "Profile Creation Wizard" (page 175) for more information.
Displays the Profile setting dialog. See "General Tab" (page 144) for more information.
Actions Allows you to:
Run Job: Submits the job to the queue and runs the job.
Run Job when queue becomes idle: Submits the job to the jobs queue and puts it in a ready state so that it will run when the currently ready or running job completes.
Run Job per schedule: Submits the next job scheduled.
Start Agent and Run Job: Starts the agent and runs the job.
Start Agent and Run Job when queue becomes idle: Starts the agent and runs the job when all previous jobs have finished.
Start Agent and Run Job per schedule: Starts the agent and runs the next job scheduled.
No action: Closes the dialog after you select Submit.
Comments Allows you to add a comment to the job.
Alerts Allows you to send an email, pager, or net-send operator alerts when the job completes.
Job Counters Allows you to addJob Countersto the job.
Stop job after first error
When checked, the job will stop if an error is encountered.
4. After filling in the desired parameters, click Submit.
Run Reports Node
Benchmark Factory Run Reports is a separate executable that opens outside of Benchmark Factory and provides a comprehensive and detailed collection of database load testing results. With Benchmark Factory you can drill down into a database to view a wide array of information and statistics that gives you accurate insight into database performance. Run Reports Viewer allows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page 212) for more information.