Secrets of Successful Java Testing

P R E S E N T A T I O N

International Conference On

Software Testing, Analysis & Review

Presentation

Bio

Return to Main Menu

Presentation

Paper

Bio

Return to Main Menu

W7

Wednesday, Nov 10, 1999

Secrets of Successful

Gary Guillon

Java Testing

Secrets of successful Java testing

Gary Guillon

JVM JVM JVM JVM

The Java Virtual Machine levels the playing field...

Different operating systems have different facilities for running programs

OS 1 OS 2! OS 3 OS 4

APP

An application written for one OS, will not run on another...

JAVA APP JAVA APP JAVA APP JAVA APP

The same identical Java application can run on any OS, without modification!

Refresher : What is Java?

Cost of Software Defects

Defect

Cost

Design Implementation Release Deployment

• The cost of software defects increases rapidly.

Test Earlier and

More Often

The Testing Process

•Test Earlier and More Often

Design

Implement

Tra

d

iti

o

na

l

App

roa

ch

Release

Deploy

Design

Recomm

ended

App

roa

ch

Unit testing

Regression

Release

Implement

_Deploy

Load

Static Analysis

When and what to check

Design

Unit testing

Code coverage analysis

Regression

Load testing

Static Check

Dynamic Check

Implement

Release

Deployment

Load testing

Load testing

Why is it different to test?

Java Apps

JVM

Java OS

Java Apps

Java Chip

Native

Apps

Java Apps

JVM

Native OS

Platform Specific Problems

Java Apps

JVM

Java OS

Java Apps

Java Chip

Native

Apps

Java Apps

JVM

Platform-Specific Test Tool

Unix test script

Win 32 tool

Win 32 script

Mac Test tool

Mac test script

OS/2 Test tool

OS/2 test script

Win3.x Test tool

Win3.x test script

AS/400 test tool

AS/400 test script

JavaOS tool

? ? ?

Unix Test tool

Java Chip tool

? ? ?

Is your OS supported?

?

?

Platform-specific tools require

one

test for

each

platform

Platform-specific test tools are “blind” to most Java events

Java Test Tools Run Anywhere

Java Apps

Java Tool

JVM

Java OS

Java Apps

Java Tool

Java Chip

Java Apps

Java Tool

JVM

Native

Apps

Native OS

Pure Java testing tools work on all Java platforms

The Java test tool runs in the same process as the application

100% portable tests, “write tests once, run tests anywhere”

Can you spot the difference?

A G U I testing tool that cannot

resolve objects and is forced

to resolve bitm ap im ages..

W ill

FA IL

a test for the w rong

reason.

There is no difference, these

G U Is are functionally identical!

A test

PA SS!

The Advantages

Java-based

Platform-Specific

Return Value

Color.Grey

“Replace All”

100,120

Boolean.TRUE

Boolean.TRUE

(Bitmap image)

D ata, m ethod

getBackground ( )

getLabel ( )

getLocation ( )

isV isible ( )

isEnabled ( )

Java Apps

Java Tool

JVM

Native OS

PlatformTool

Peerless components are fully visible to Java-based tools

Peerless components are invisible to platform specific tools

Testing Applets

Applets can be tested using

Suns appletviewer technology,

or in the browser itself. (This

may require the installation of

the java plug-in)

Write tests once, run tests anywhere!

Script is reviewed once for

thoroughness and coverage

Application

Test

Script

Application is installed on

target configurations

C

O

M

P

A

R

E

What is the Highest Priority?

What gets

effected , when

you shift

priorities?

Time to Market ? 3

Quality of Product ? 6

Cost Saving =

1 !!

Total Available Marks -

10

-6 = 4

-3 = 1

Are you dealing with the real problem?

Ship #1: Please divert your course 15 degrees to the

North to avoid a collision.

Unknown contact#2

: Recommend you divert your course

15 degrees to the south to avoid a collision.

Ship #1: This is the Captain of a US Navy ship. I say

again, divert YOUR course.

Contact #2

: This is the watch ensign. I say again, you

divert YOUR course.

Ship #1:

THIS IS THE AIRCRAFT CARRIER

ENTERPRISE! WE ARE A LARGE WARSHIP OF

THE US NAVY! DIVERT YOUR COURSE NOW!

Contact #2

: This is a lighthouse. It's your call!!!!

Load Performance

Time (sec)

0

20

40

60

80

100

120

System

Crash!

In the past Clients

have estimated

performance based

on an existing

system or

available hardware

User Count

W hat is the response tim e at 100 users?

Time by ‘Finger in Air’ method

Time by Load testing!

Asynchronous Load testing…. Ouch.

Sam e System , sam e end result, but different internal transaction organisation!

Synchronous arrangement. Client always waits for server.

Query and response are always in the same order. GUI client waits whilst server is busy.

Multiple request can be sent, data can arrive in any order! GUI can do useful work whilst server busy.

New Era ORB based or distributed application provides asynchronous communication for greater efficiency!

Asynchronous Load testing….cont.

Script captured from Scenario 1

Tries to play against Scenario 2… but the system under test

responds differently...

“Record, capture and

encapsulate” load testing

technologies are all

synchronous!

FORCES SYSTEM TO BE SYNCHRONOUS!

NOT A TRUE REPRESENTATION OF THE SYSTEM !!!

Input 1 Waiting for response 1… waiting… waiting… waiting...

Script doesn’t send

input 2 until response

1 is received, forcing

system to behave

differently,

IN V A LID A TIN G

TH E TEST!

Network Java Load Testing

App Server

.

.

.

Distributed 3-tier

SAP / Baan / Oracle

Mainframe / Legacy

Internet

Internet

Defect Tracking

Defect Tracking

CM Systems

CM Systems

JavaScope

Coverage Tools

Coverage Tools

The Testing Infrastructure

Software

Under Test

Software

Under Test

System

Scripts &

Results

Unit Testing

Unit Testing

Manual

GUI Testing

GUI Testing

JavaStar

JavaSpec

GUI

API

!

Test M anagem ent is essential!!!

?

Test

Management!

Test

Need to know more?

W eb

: http://w w w .sun.com /suntest

Em ail

: suntest_saleseurope@ sun.com

Phone

: 0800 962761 (U K )

That was…

Secrets of successful Java testing

By...

Gary Guillon

Sun Microsystems

Gary Guillon

European Manager - Java Testing Tools for Sun Microsystems Gary was a Engineer for GEC Avionics working on Hardware Data Conditioning for the Tornado Aircraft and moved into the software group to test the Nimrod - Early Warning Submarine Detection equipment, which is still being used today. He has a number of humorous stories of 'Testing Disasters' that appear during his testing presentation.

Java is rapidly becoming the language/environment of choice in the Times Top 1000 companies world-wide. With it comes a whole host of Testing concerns. Asynchronous Testing being one of the frequently overlooked elements of testing software running on parallel processors. This talk addresses the uniqueness of Java and how Asynchronous

Testing can be done effectively with technology that was invented by Sun Microsystems, the author of Java itself.