Q: What is the difference between the other load testing tools which enables the wan emulation, location based load testing and Gomez load testing?

(1)

Q:

Gomez

is

standlone

web

application

testing

tool?

Gomez provides an on‐demand platform that you can use for both testing and monitoring your Web

applications from the “outside‐in” — across your users, browsers, mobile devices and geographies — using a

global network of 100,000+ locations, 168+ countries and 2,500+ local ISPs. Reality Load is Gomez’s web load

and performance testing solution and you can reuse the same scripts for both load testing and production

monitoring

Q:

What

is

the

difference

between

the

other

load

testing

tools

which

enables

the

wan

emulation,

location

based

load

testing

and

Gomez

load

testing?

A few key differences are the scale of the Gomez offering, the number of locations that you can test from and

the fact that it is not ‘emulating’ a WAN. In terms of scale, the Gomez Last Mile network includes 100,000+

peers located in countries on every continent. (Reality Load runs on real user’s machines on the local ISPs and

connection speeds available to them). The Gomez network is already deployed and requires no configuration

other than to choose which computers (we called them peers) you want to use for load testing or monitoring

purposes. . This gives you an unprecedented view into user experience that cannot be gained by simply

emulating different configurations. If you are interested in how your real users would experience your site

under load you need to realistically test under the same conditions, and the Gomez Last Mile is the only way to

do that. For example with Gomez Last Mile you can find geographical response time discrepancies that may

surface only under load What will be the impact on the business if average response time stays under 4

seconds (your target goals), but users from key markets are seeing response time of 10+ seconds?

Q:

How

would

we

use

Gomez

for

the

internal

applications...not

the

external

app?

Reality Load, and the rest of the Gomez platform, is designed to test and monitor your applications across the

internet, from an outside‐in customer perspective. But testing internal applications is a common request. At

present there are a couple of options. One would be to allow the Gomez test machines access to the internal

system, either by allowing requests from our IPs or by indentifying our agent string. Another option currently

available for our Monitoring offerings, though not our load testing, is to deploy a Gomez private peer behind

the firewall from which the performance measures could be taken.

Q:

Also

what

will

be

the

approach

of

Gomez

while

testing

any

web

based

application

running

on

a

mobile

phone?

For testing mobile applications Gomez has 2 complimentary offerings, Reality Load (discussed in our webinar)

and Active Mobile for monitoring mobile web applications. Users could run high volume load test using Reality

Load, including device specific user agent strings and HTTP headers, while utilizing Active Mobile for the most

accurate measure of end user experience. Active Mobile emulates devices across major mobile carriers in the

US (Chicago, Boston, Seattle) , the UK, Germany and China, giving an accurate view of the impact of the

increasing load on the mobile user experience.

Q:

So

the

agent

actually

mimics

browser

behavior,

is

not

actually

running

the

scripts

inside

browsers

?

The browsers used by Gomez’s Reality Load are real Web browsers. That means that they are actually running

scripts and processing the pages, not just playing back HTTP type requests.

Q:

does

it

support

Virtual

private

network

usage?

(2)

Q:

Also

does

this

tool

monitors

servers

during

the

test

run

and

co

relate

the

bottle

necks?

Today most of our RL customers (like Jim from Dollar Thrifty mentioned during the Webinar) are using existing

in‐house system mgmt tools to correlate performance inside and outside the firewall during a load test. The

Gomez platform has an open integration philosophy, and we will continue to integrate with any EMS tools that

you may be using. Now that Gomez is a Compuware Division we are looking into offering an integrated view of

application performance under load by connecting the Gomez platform and Compuware Vantage “behind‐

the‐firewall” infrastructure monitoring capabilities, so you can quickly correlate bottlenecks and performance

issues across both the enterprise and the Internet.

Q:

Do

you

use

a

3rd

party

load

test

tool

behind

the

GOMEZ

Reality

load

tool?

Something

like

HP

LR,

RPT

or

a

combination

of

it?

The Gomez load testing tool uses our own tool set. It uses the same script recorder and test agent as the rest

of the Gomez Platform. This makes it very easy to move between load testing and 24/7 monitoring and vice

versa.

Q:

Can

the

raw

data

be

used

for

the

CompuWare

Load

testing

toolsets?

The raw data is exported in CSV files, so it accessible to a wide range of toolsets.

Q:

Gomez

software

is

opensource?

Is

there

a

trial

version?

it

runs

in

any

OS

or

web

explorer?

The Gomez platform is a SaaS platform accessible on‐demand through a web browser –that means that you

don’t need to allocate hardware or install any software products, and relies on. The Gomez network— 100k+

testing locations, 168+ countries, 2,500+ ISPs. It is not opensource, but if you are interested in a trial just let

us know and we can set you up.

Q:

On

your

interface,

when

and

why

would

you

check

the

checkboxes,

Exclude

JS,

CSS

or

Images?

This was a feature requested by a number of our customers. Basically, it can serve two purposes, one is

around test methodology and the other around cost. Methodology wise, we have customers who prefer to

start their testing with simple steps, focusing very specifically on pieces of their applications. This focused

approach starts with hitting just the basic pages and not any of the ancillary content like images, JS, CSS that

may not be needed. Here they are focusing just on the impact of the requests for their application pages on

the server under load.

The second reason is cost. Many of our customers use CDNs that charge by the amount of content served.

When running a lot of load tests the costs can get pretty high. So they will exclude the objects from their early

tests and only include them for a final test or two. Also, just wanted to highlight that we see a lot of issues with

third‐party providers and external components performing poorly, especially under load, and regional

discrepancies as well, so don’t forget to include test cases that cover external components and geographical

testing into your plans

We also allow users the option to exclude requests by host.

Q:

Are

you

taking

care

of

that

pages

may

be

cached

in

a

transaction?

If

not,

can

you

be

sure

that

the

front

end

is

not

getting

too

much

load?

Our load test tool performs as a real user’s browser would perform for each transaction. So if images are

cached for instance, each iteration of each user run would request the object once, but not again within a

(3)

Q:

If

you

are

testing

in

the

PRODUCTION

environment

during

off

peak

hours,

you

still

cannot

know

how

much

USER

traffic

is

hitting

your

site.

So

you

cannot

isolate

your

results

to

your

test.

How

do

you

account

for

UNKNOWN

load

during

your

test?

This is definitely a concern for testing of production systems. Here it is important to monitor the utilization of

the production system prior to the load test, during the load test and after the load test. This can give you a

profile of the live traffic and assist in understanding the utilization on the servers.

Also, by monitoring things like firewall bandwidth (at the network level), and comparing the overall total with

the load reported for the test users (from the test tool) the load from live users can be extrapolated. For

instance, if Reality Load was showing that the test was consuming 80 Megabits per second of bandwidth, and

the firewall statistics were showing 90 Megabits per second of overall traffic, then you can get a sense of how

much live traffic was on the site (in this case, 10 mbps worth of traffic). The same could be said for page

request rates or other measures with the difference between the load tool measures and the basic system

measures providing the sense of live traffic.

The best way to ensure quality of experiences and that an application scales properly under load is to

realistically test your application –all the elements of the web application delivery chain—from the outside in,

using real users and desktops worldwide to access your production environment

Q:

How

do

you

calculate

transactions

per

second?

One of the Reality Load metrics is transactions per minute, I assume this is what the question refers to. This is

calculated as the sum of the transactions (complete runs of the test scripts) complete in each minute.

Q:

Do

you

help

customers

identifying

workload

profile,

if

yes

what

is

your

approach?

Most of our customers operate in a self‐service model and utilize their own test methodologies. However we

do offer collateral around best practices for things like workload profiling as well as offering professional

services assistance where needed or requested..

In general though, the approach is to evaluate the traffic the web application is currently seeing with focus on

three key areas: the most heavily hit pages/functionality, the most business critical pages/ functionality and

any pages/functions known to cause problems.

Q:

Can

you

terminate

a

test

run

when

things

start

to

break?

Does

the

initiator

of

the

test

have

access

to

the

test

admin

console?

Can

they

terminate

a

test

in

a

timely

way

to

avoid

crashing

the

application?

The Gomez tool is a self‐service platform, so users have access to the controls to schedule a test, stop it, view

real time results, etc. Tests can be stopped within a few seconds just by clicking a stop button. In addition,

Reality has a Max Failure Rate settings (Reality Load checks that the % of users encountering errors at any

given minute remains below the Max Failure Rate value) and it can automatically reduce the load level looking

for essentially a ‘last‐known good’ state if needed.

Q:

We've

got

a

staging

environment

that

matches

production

and

it

has

a

web

server

in

the

DMZ

and

is

accessible

from

the

Internet.

We

use

Loadrunner

to

test

the

system.

What

is

the

advantage

of

using

Gomez?

Traditional inside‐the‐firewall testing tools, like LoadRunner, are built on the philosophy of generating load and

measuring performance behind the firewall. This approach is valuable, but will only tell you part of the story,

since these tools are focused on testing internal systems. However, your end‐users don't live in data centers ‐‐

they are located at the end of a long and complex web application delivery chain. When you generate load and

(4)

components (ISPs, CDNS, etc) will scale under load. And all components and services need to perform

optimally in order to deliver quality web experiences across all users and regions. Some examples of problems

that you could be missing are: Inconsistent geo performance especially under load, CDN configuration issues or

oversubscribed POPs, major ISPs outages, SMS routing, latency issues, etc

Q:

Did

you

support

flex

applications?

Yes, Gomez supports flex applications.

Q:

Who

does

the

scripting

?

Do

customers

write

own

scripts

or

do

you

write

the

scripts/scenarios

for

the

customer?

Specifically

scripting

web

services

The Gomez Platform, including Reality Load is a self‐service platform, and so most of our customers are

building their own scripts for load testing and monitoring. However, there is a service desk option for free

assistance with script creation if our customers run into any difficulties.

Q:

What

language

are

scripts

written

in?

The scripts for the Gomez Platform are recorder through a visual interface. Users click through their

application and the Gomez recorders recorder the users interactions with the web application. When the

scripts are played back, the agent builds out the page (document object model, javascript and CSS processing

etc) and then replays the recorded actions.

Behind the scenes, the scripts are saved as XML, and can be extended using javascript, but there is no scripting

language per se.

Q:

How

do

you

create

the

scripts

(clicking

through

the

site)?

Yes, scripts are recorder by clicking through the site.

Q:

How

easy

is

to

script

Q:

How

difficult

is

it

to

create

testing

scripts

and

what

tools

are

used

for

that

task?

Scripting is relatively simple and uses the Gomez Script recording tools for script creation. It involves just

clicking through the web site while the Gomez recorder captures your actions. When the scripts are played

back the agent starts at the first URL and then executes the recorded actions.

Q:

Can

we

capture

screens

of

application

at

predefined

points

in

the

script...?

Reality Load allows users to collect screen captures on error (SCoE) in load tests.

Q:

Do

you

have

any

facility

for

testing

client

side

rendering

using

javascript

or

ajax?

The test users are executing the javascript, ajax calls, etc during the loadtesting, so those metrics are already

included in the data.

Q:

Raw

data

exported

is

going

to

be

cumulative

of

the

ser

responses

how

often

it

has

been

collected

(or)

for

every

user

for

every

request

until

the

end

of

the

test?

Reality Load allows users to export with the aggregated (minute by minute) data, or the full raw data set. The

full raw data set includes data on every hit of every page request of every transaction run in a load test.

Q:

Is

the

summary

report

export

format

in

XML?

The summary report for Reality Load is exported as a PDF, though all of the data behind the reports can be

(5)

Q:

How

did

you

insure

that

you

correctly

backed

out

the

scripted

transactions

without

losing

any

real

customer

traffic

data.

Gomez transactions are easily identifiable by text in the request header. This text can be used to ensure that

traffic monitoring tools do not count the load test requests in their reports.

Q:

what

was

the

size

of

Page

0 (the

megabits/s

seem

to

grow

really

quickly

is

this

representative

of

real

life

use?)

Page 0 in the demo was a 250KB page. This is pretty common for a homepage, and in fact smaller than many I

have seen recently.

Q:

Can

the

results

point

to

database

transaction

bottlenecks?

Interpreting load test results is a combination of effective test planning, collecting the right metrics and proper

analysis. Looking at the user experience metrics, and combining that with data collected from the back end

systems can point to a the location and likely cause of a bottleneck. For example, if only certain pages in a

transaction are seeing performance degradation the first step would be to ask what they have in common. If

they all make database hits then check the CPU of the database server. Is it within tolerable parameters or

was it under duress. This correlating of functions exercised, user experience changes and server metrics can

be highly effective in isolating performance bottlenecks.

Q:

Our

test

environment

is

significantly

slower

than

our

production

environment.

How

do

you

load

test

in

that

scenario

and

insure

that

your

production

environment

can

actually

handle

the

load?

In scenarios where a test system and production system are not identical (and this is often the case) it is

important to understand the 'factor of scalability', or the degree of difference between the two systems. This

may take a little testing, or some estimation. In this way you could execute your test on your staging system

and project them out to the production capacity. However, unless you actually run a test against your

production system you'll never know for sure. I've seen many situations over the years where tests on staging

go well, and then production fails under live load because of a missed configuration, unexpected change, etc.

So despite the difficulties of testing against a production environment, I believe it’s critical to run at least some

tests against it before the customers do...

Q:

General

load

test

question:

So

would

starting

with

50 users

for

15 minutes

and

then

running

100 for

15 minutes

and

so

forth

to

see

where

the

throughput

breaks

down

as

far

as

number

of

users...would

this

be

a

viable

test

My preference in load testing is to bring users on slowly over time, so that when a problem is encountered,

you'll know exactly the load level that can be supported. In the example presented here, if the site could

handle 75 users you wouldn't know that for sure. You'd have a 'yes' for 50 users, but only a no for '100'. I

would start at 1 user and ramp to 50 over 10 minutes, then hold at 50 for 5 minutes (or until some transactions

can complete at this load level), then ramp to 100 over another 10 minutes and hold again at 100. That way

you would see the impact of increasing load as well as the flat load.

Q:

Is

throughput

test

is

the

same

name

as

"Stress

test"

for

you?

There is some overlap in the terms, but the goal of a throughput test isn’t necessarily to push a system to the

breaking point. The goal of a throughput test is to measure the system’s ability to deliver pages, hits,

bandwidth, orders etc (things that are typically measured in terms of metric per second/minute/hour),

independently of how many users are running. So typically in a throughput test a small number of users run

(6)

with fewer users and less time than larger tests (what we would call concurrency tests), which are focused on

measuring performance in terms of the number of users the system can support.

Q:

Does

this

product

only

test

pure

http

traffic

or

can

it

handle

new

technologies

like

flex,

java

scripts,

ajax

etc

and

be

able

to

emulate

the

client

side

processing

as

well?

Reality Load uses the same transaction engine as the rest of the Gomez platform. This agent can handle flex,

java script, ajax, etc and client side processing

Q:

Do

your

browsers

include

mobile

devices

such

as

the

iPhone

and

Blackberrys

(on

simulated

3G

connections)

?

Presently Gomez supports mobile simulation for Monitoring, but not for load testing. The monitoring is

simulating connections through modems on a all major carriers in the US (Chicago, Boston, Seattle), and in UK,