TRUE PERFORMANCE ENGINEERING

(1)

TRUE PERFORMANCE

ENGINEERING

Quality with Testing, Testing with Quality

(2)

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

INTRODUCTION

Complex applications leveraging real-time interactive push services are becoming the norm. The rate at which these services present streamed dynamic push data, presents a new scale of opportunity for providers to offer differentiated services where clients subscribe to applications with constantly changing price screens, performance indicators, 3d graphics and analytics.

User interaction and ongoing loyalty depends on the accuracy, speed and reliability of the data presented, and as applications wait for the user to commit to bet, buy or sell or place an order massive volumes of data are being simultaneously transported across the web.

As a consequence of the requirement to push data such as prices, gaming odds and other real time data, accurately and identically to all parties, true performance engineering, quality performance and stress testing are becoming more and more critical.

Traditional testing tools, which remain effective for static data functions in mainframe and “pull” data applica-tions as seen in many retail websites, are proving to be inadequate for today’s complex rich internet applicaapplica-tions where complexity of functionality within the client is increasing, this is particularly the case in systems where the volumes of pushed data to the client is driving markets and profitability.

This white paper explores the traditional versus new methodology of performance testing, and explains how The Test People’s solutions mitigate risks by enabling high volume, true performance engineering and load testing, with full visibility of speed, flows and access to simultaneous pricing.

This paper is intended for CIOs, IT directors and managers, in enterprises of all sizes, especially those who have the need to test evolving complex client applications with true performance engineering in the Cloud.

THE CHALLENGES

Dependable Applications Testing

Credibility and dependency on real time data is becoming paramount across a range of industries, as custom-ers exercise choice of providcustom-ers, it is essential that providcustom-ers are able to demonstrate the highest levels of true performance to retain clients. Users will change provider if screens do not appear to be updated in real-time, or if confirmations are requested for prices or goods, which, due to market or stock movement are then unavailable. The evolving nature of business critical systems has lead to demands for more tests to cover functionality and operational detail, new information distribution methods and interoperability of applications with browsers and platforms. Testing is now much more complex as there are now many different browsers, delivery platforms and protocols that must be taken into account.

Consequently, the time, cost and effort to test these inter-dependencies, has become much greater, and in some cases unmanageable.

(4)

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

TRADITIONAL LOAD TESTING

Testing of application loading typically involved repli-cating the traffic between client and server and check-ing the time for messages to be returned.

Replicating the user activity in this way proves that the server is performing. While this is very important, it does not give true visibility as to how fast data is being returned to users by the client, and whether this data is distributed consistently, securely and accurately.

LARGE CAPITAL EXPENDITURE

Building the necessary hardware for internal injector farms (racks of servers and communications in-frastructure), the costs of an internal QA lab, test soft-ware, test maintenance and expert consultants may be cost-prohibitive for many organizations who need to scale to potentially millions of users. Also, these proj-ects tend to be inefficient, as servers need to be provi-sioned and scaled for the worst-case load tests and are redundant when testing projects are not taking place. Additionally, budget constraints may mean that insuf-ficient hardware can be purchased to replicate maxi-mum loads.

INCOMPLETE APPLICATIONS TESTING

Capturing and maintaining a large user base on the Internet is dependent on fast, reliable, low la-tency accurate content and delivery. The com-plexity of modern applications means that content comes from multiple sources, mixing external static or dynamic data with their own internal information. Organisations may use their own injector farms, how-ever the challenge is compounded in that they may not fully replicate an environment with its rapidly chang-ing mix of internal and external sources and connectiv-ity inter-dependencies.

ONE-SHOT PERFORMANCE TEST SCRIPTS

The dynamic nature of data and integrated applications means that every time content, volume or versions change, new test scenarios need to be created to test environments and the datasets within them.

COMPLEX BROWSER TESTING

It is possible to fool a server into accepting scripts rep-licating the client server traffic from a user on an

inter-As more application functionality is executed at the client, the use of traditional performance testing methodologies to simulate message-level interaction with the server for Rich Internet Applications (RIAs) is becoming more complex and potentially less representative of the behaviour of the application it-self.

Many Modern web applications now depend on the use of application frameworks such as SilverLight, Flash and JavaFX or ever increasingly complex javas-cript, and their compatibility with browsers such as IE, Firefox or Chrome.

RIGHT INFORMATION AT THE SAME TIME

Typically, large volume systems deliver information to tens of thousands of subscribers simultaneously, each with their own preferences and licenses, and the ven-dors internal systems must gather the prices, and send subscribed data to each client.

However as prices change, there is often no way to know the true fan-out of this information, how long updates have taken to reach the user, or how long the browser may take to parse the information, potential-ly leaving users blind to latest prices and the vendor incorrectly believing they have fulfilled their obligations in supplying on-time service.

For example, this is true in Foreign Exchange mar-kets, where internet-based individuals are empow-ered to trade in real-time, waiting to exercise their buy or sell strategies through a plethora of browsers. In fast moving markets, users with complex trading applications need to be assured that they are receiving prices and trading opportunities accurate-ly and quickaccurate-ly as they scan the constant stream of activity.

(5)

Browser processing

A

User

B

Client

C

Datacentre

Click Button

Enter Text

Read Back

Values

Protocol variants:

AMF,RTMP,XML,STOMP

Or Bespoke Binary Formats

HTTP

TCP / IP

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

DIAGRAM 1 – APPLICATION OVERVIEW

IN PUSHED DATA APPLICATIONS:

Interaction between users and the GUI (A to B) is relatively simple, the user subscribes to a number of data channels, data is returned automatically when changes occur and the user then acts on the data when they wish, for example, placing bets and placing/modifying/cancelling order and trades.

Interaction between the GUI and the server (B to C) is complex. The GUI may use differing protocols to communicate with the server negotiated at connection time depending on network capability. When updates are received from the GUI it executes business logic on the data updating its own internal data structures, it may auton-omously query the server or another 3rd party for additional information and then render the updates on the screen to the user. All of this happens automatically within the GUI without any interaction between the user and the GUI.

THE VISION

To remain competitive companies must engage optimal true performance engineering methodologies to ensure they are fully aware of and are able to manage any latency and potential bottleneck issues with their system. Competitive advantage may also be gained as a consequence of being able to demonstrate to clients that optimal levels of performance are consistently achieved.

The vision therefore is to provide service providers with cost effective and easy to deploy performance tools, envi-ronments and analytics such that are able to prove consistently superior levels of performance and service delivery.

THE SOLUTION

As evidenced by industry experts and client experience, performance testing of pushed data systems is not achiev-able using traditional performance engineering methods.

Consequently, a new approach is required.

(6)

• Uses the REAL GUI

• Controls the GUI through User Interaction • Observable responses are when the user sees a

response

• GUI Complexity is abstracted from the load tester – Regardless of complexity or type of protocol used, interaction is performed at user level (key presses, mouse clicks etc.)

• Does NOT use the real GUI, but mimics the GUI protocol level messaging to SIMULATE the GUI behaviour.

• Rework test scripts if GUI protocol activity changes

• Monitor protocol level response times – if the GUI is adding to response times this will not be picked up

• GUI business logic has to be replicated through performance test tool scripts

• Tool vendors must provide support for increasing numbers of dynamic, differing protocols

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

DIAGRAM 2 – APPROACHES TO PERFORMANCE ENGINEERING

Real GUI Real GUI Real GUI

Controller

User

Actions

Protocol

Level

Comms

Controller

GUI Approach

Traditional Approach

Simulated

Client

Simulated

Client

Simulated

Client

Server

User

Actions

Protocol

Level

Comms

Protocol

Level

Comms

Protocol

Level

Comms

MINIMAL CAPITAL EXPENDITURE

(7)

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

COMPLEX BROWSER TESTING

Using frameworks and applications within the Cloud, based on virtual servers with wire capture capability, there is full understanding of the protocols involved, and how data is moved and accessed specifically for the tests. The Test People have deployed injector models enabling running of many thousands of clients in truly representative scenarios driving the real web browser (or windows based client), with an in-depth appreciation of the technology and cost implications.

TRUE PICTURE OF REAL-USER EXPERIENCE

In protocol level performance testing the transaction time clock stops when the simulated client receives a response back from the server. From an end user perspective however the transaction is only completed after the client has received the message, processed the message and rendered the update on the screen. The Test Peo-ple have used their true performance engineering and smart client application testing experience to performance test the client itself and include client processing time in reported transaction times, accurately capturing the elu-sive end-to-end response time as observed by real users. This measure is becoming increasingly important in an environment where bottlenecks or performance issues in a complex browser application can severely impact end-user transaction times and go unnoticed in protocol level testing.

RAPID AND LOW COST AVAILABILITY

Cloud computing offers “pay for what you use” operations, enabling ‘on demand’ expansion of load infrastruc-tures without the need for scheduled load management, often months in advance. In addition to the technical advantages, this model offers end-users true business benefits in terms of flexible project prioritisation and reductions in hardware budgets, which directly affect bottom line return on investment.

The TTP solution enables clients to gain significantly more confidence in their production systems. With experience of providing true performance engineering, testing skills and architectural advice, industrial strength is proven faster, with improved application quality and speed of release cycles. By harnessing the cloud, the ability to cost-effectively manage and monitor exponential load tests has finally arrived.

DIAGRAM 3 – LOAD MANAGEMENT

GUI GUI GUI GUI

Load O

rchestr

ation

Load Injector 2

Load Injector N

W

ire - C

aptur

e/Dec

oder

Load Injector 1

System Under Test

Load Management Server

Deserialised

Updates

Notes

Load Management server orchestrates all data to/from ALL load injectors

Load injector comprises a Load Orchestration Module (applies the user journeys to the GUIs)

(8)

KEY BENEFITS

• Significantly faster scripting allows delivery of performance testing results more quickly •

• Fast performance testing of browser/applications combinations •

• I/O of messages can be monitored and managed through their life-cycle •

• Testing scenarios are scalable, less brittle and not reliant on underlying protocols. •

• Cost efficiency - only using the cloud resources as required for limited periods •

• No interference with live operations •

• Promotes rapid deployment of new solutions •

• Still gives visibility of underlying messaging between client and server.

With proven industrial strength load injector tools enabling providers to test their applications in the process of broadcasting push prices accurately and speedily, the Cloud based true performance engineering solutions of The Test People offer flexible on demand virtual unlimited load capability at low cost whilst also bringing other benefits such as more robust and easier to maintain performance test scripts.

DIAGRAM 4 - LOAD INJECTOR PLATFORM

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

(9)

THE TEST PEOPLE - TRUE PERFORMANCE ENGINEERING

CONCLUSION

True Performance Engineering enabling high-volume, data-intensive push applications to obtain meaningful infor-mation on stability, interoperability and scalability is vital.

As such, True Performance Engineering in the cloud has emerged as a cost-effective, secure and resilient solution to testers and developers by delivering bandwidth and computing power to run real-time exact replica tests at user levels never before possible, enabling industry strength applications to be delivered with reliability, certainty and time critical accuracy.

The problems that arise from untested and incorrectly tested complex real-time push client applications can result in loss of customers, revenue and credibility, but powerful, cost-effective testing solutions are available to protect your business.

With its experience in multiple methods and markets, The Test People are leaders in this specialist field, where un-derstanding the issues from a content delivery, RIAs and end user perspective has enabled it to identify and monitor fan-out at the client - the all important time between the first and last user update for a single piece of information, the implications of which may affect customer retention, loyalty and trust.

The Test People’s suite of solutions give users peace of mind that testing truly reflects the scale and variety of actual usage, delivering a framework for push platform providers and ultimately the customers to interact with what they need, reliably, with low latency, simultaneously, at the time they need it.

For corporations seeking to seamlessly market and move their data across multiple digital channels and empower push applications, with advanced delivery capabilities, comprehensive monitoring, analysis, powerful management and true performance engineering tools, The Test People provides a compelling combination.

To learn more about true performance engineering in the cloud, visit www.thetestpeople.com for more details.

CONTACT THE TEST PEOPLE

London Office: +44 (0) 207 100 7794 Phone: Leeds Office +44 (0) 113 834 5975

Email: [email protected]

ABOUT THE TEST PEOPLE:

Founded in 2007 and with offices in Leeds and London, The Test People Ltd. Is an innovative testing consulting firm specializing in True Performance Engineering, Test Automation and Test Management.

TRUE PERFORMANCE ENGINEERING