Test Preparation
Phase I:
Test Definition
The core theme of the previous chapters is that a methodical and structured approach to PT is rather necessary right from the early phases of the development cycle. This will ensure predictability and controllability of application performance in practice. The preparatory activities associated with PT (Figure 4.1) are of great importance and are distributed over the life cycle phases of requirement elicitation and analysis, High Level Design (HLD), Detail Level Design (DLD), and several sets of system builds. These activities help define and provide continuity between the high level requirements for application performance, strategies for testing, a framework for designing the tests (see Barber, 2004), and artifacts used to plan and carry out tests. This chapter contains a detailed consideration of the definition phase while Chapters 5 and 6 highlight issues related to the design and build phases associated with the preparatory activities as shown in Figure 4.1.
Need for Test Definition Phase
Customarily, designers address performance issues close to the end of the project life cycle, when the system is available for testing in its entirety or in significantly large modular chunks. This, however, poses a difficult problem, since it exposes the project to a potentially large risk related to the effort involved in both identifying as well as rectifying possible problems in the system at a very late stage in the life cycle. (Refer to
Chapter 1 and Chapter 2 on the problems of debugging.) A more balanced approach would tend to distribute such risks by addressing these issues at different levels of abstraction (intended to result in increased clarity with time), multiple times (leading to greater effectiveness and comprehensiveness in testing application performance), and at differ- ent stages during the life cycle. The very first component of activities related to preparation for such testing is in collecting and analyzing requirements related to the performance of the system alongside those related to its features and functions. Hence, this activity would have to take place during the requirement study of business functions and is termed the test definition phase. The main objectives of this phase are to:
•
Define goals of PT;•
Remove ambiguities in performance goals;•
Determine the complexity involved in PT;•
Define performance metrics to measure the performance;•
List risk factors involved in PT;•
Define the strategy for PT.Although most of these activities are carried out during the requirement study phase, some of these have to be revisited for further tuning and clarification during the design phase.
While requirements for modeling business functions and those for performance comple- ment each other, their gathering, as well as analysis, need to be treated as different activities. The performance requirements must be well defined before the design phase, as shown in Figure 4.2, so as to facilitate a more precise design of tests of critical performance parameters.
Collection of performance requirements is a difficult and complex task due to a variety of reasons. It is often easier to elicit requirements associated with business functionality
Performance Test Preparation
Definition Phase
Performance Requirement Document Test Strategy Document
Design Phase Build Phase
Benchmark Design Operation Profile Workload Design Tools Selection Test Plan Test Environment Test Scripts Test Schedule Testing Process
than those related to performance of a system developed to cater to the functionality. In addition, performance parameters are influenced by multiple factors such as:
•
The business background and objectives of the organization;•
Information related to current operations of the organization;•
Business growth strategies of the organization;•
User or customer base associated with the organization;•
Operational and deployment logistics as envisaged by the organization;•
Existing and prospective technical infrastructure for the automated system;•
Network environment, hardware and software systems required to run the busi- ness.Typically, most of these factors are not directly available for analysis during the initial stages and/or are implicit and therefore need to be derived from other information. In the following sections, a detailed description and approach is provided to understand the various activities related to performance requirement elicitation, analysis, structuring, and documentation. Figure 4.2 provides the process flow to generate a performance requirement document wherein business requirements, customer interaction, and infor- mation on infrastructure are the main inputs.
Performance Requirements and
Their Importance
Performance is one of the nonfunctional requirements and has a severe impact on the success of the project. Since designers are not bothered about the performance during the requirements gathering, the real impact will be known only when it fails to meet the users’ expectations. If the system fails during its live operation, rectification is a very expensive activity, which may lead to loss of customers. To avoid such implications, treat the performance elicitation activity much in the same way as functional requirements. Broadly, the activities in this phase can be divided into three categories, as related to:
Performance Requirement Analysis Performance Requirement Document
Business Req Doc Customer
Interaction Infrastructure
•
Business functions related performance requirement;•
Infrastructure and network environment;•
Explicitly specified requirements for performance.In each category, intensive performance requirement elicitation must be carried out as discussed in the following sections.
Business Functions Related
Performance Requirement
Functional requirements gathering is the first step in SDLC. This process is well understood. However, nonfunctional requirements like performance may not be col- lected during the requirements study due to uncertainties of the expectations on performance. However, it is possible to collect performance requirements based on many business related functions like:
•
Business context;•
Business events;•
Business operations;•
Business objectives of the system;•
Logistics of the operations;•
User base.These functions are explained in the following sections:
Business Context
Many of the detailed plans and processes employed by organizations to achieve business goals can have a significant impact on performance. The business context diagram normally available with the organization provides the in-depth details. The following questionnaire will help to collect information related to a business context.
• What is the uniqueness in the business as relevant to the performance?
The idea is to find out how the business uniqueness impacts the performance of the system. To illustrate, providing mobile commerce (m-commerce) for the custom-
ers of a bank is unique compared to other banks but requires the system to be performance conscious.
• What are the business functions that are critical for a company’s survival?
Some of the business functions are critical and must be given top priority while handling the performance. These functions are important to the company’s survival in order to stay ahead of their competitors. Identify those functions critical to performance with no scope for compromise.
• What is the organizational structure?
Organizational structure always helps to determine the stakeholders of the perfor- mance results. Different groups at different levels in the organization will demand different types of performance results.
• What are the business goals? List the business goals per the organizational
structure.
Knowing the organizational structure helps in formulating the performance goals to suit the different levels. For instance, in any organization, senior managers may have different goals compared to marketing or information system (IS) groups. Senior managers may be looking at the growth projection, in terms of increase in number of users, for the next five years whereas the marketing and information system (IS) groups may be working on how efficiently services can be offered to existing customers. In the former, the user base increases whereas the latter addresses the response time of various activities. However, both of them will have an impact on the performance of the system.
• What is the overall structure of the system?
Knowing the structure of the overall system helps in planning the performance. Part of the system may be used internally, and performance may not be the criterion for its function. For instance, executing batch processing could be done during nonprime time so that the impact on the performance is reduced.
Business Events
Many business operations are triggered by various events. These events may originate within the system or driven by the external system. The dependency of operation on an event definitely affects the performance. Events may be triggered manually or automati- cally. These events are categorized as follows:
External Events
These events are triggered by external entities, which depend on many factors like communications, infrastructure, and people. For instance, fund transfer in a banking system is allowed through a customer’s mobile phone. To complete the transaction, the event depends on the health of the mobile phone, features enabled on the mobile phone, and the service provider. The banking system does not have any control over these
external events. The performance of the complete transaction depends on the perfor- mance of these external entities.
Temporal Events
On occasion, some of the business events are triggered in real time. To illustrate, the purchase order for a specific material is triggered only during the first week of every month. In such cases, there will be an additional load on the system during the specified period, which must be accounted for by measuring the performance of the system.
Business Operations
Different groups within the organization manage business operations, and typically, the objectives of each group are different from one group to another. In some cases, the organizational structure and business operation group may be the same. The intention of making groups according to the business operations is to ensure that the stakeholders, involved in appreciating performance results, will define the performance requirements correctly. To illustrate, in the banking industry, the groups may be categorized broadly into:
•
Senior management to plan the business growth of the bank;•
Customer division to manage the customers efficiently and build customer confi- dence;•
Middle management to strategize the business of the bank for efficient operation and building more profit;•
Information system groups to maintain and monitor the various automated sys- tems.The objectives and operations of each group must be known so that the performance analyst can assess the impact on performance of the system due to the groups operations. This helps to define the performance objectives at the group level.
Another significant impact on the performance is altering the existing system by:
Introducing the New Business Operation
Based on the business growth and plan, demand may arise to start a new business operation. Depending on the complexity of the new operation, the performance of the system may vary. In-depth study and investigation on new operations must be carried out during this phase. For instance, the service of fund transfer through a mobile phone, if introduced to bank customers at a later stage, will impact the performance of the system.
Enhancing the Existing Operation
The existing business operation may not always be effective and requires modifications that include enhancements. Such modifications may impact the performance of the system.
To illustrate, a money withdrawal facility is available through ATM in a banking system. The authentication procedure used to withdraw money is the ATM card PIN. However, due to business compulsion, management decided to enhance the authentication procedure by introducing the voice recognition of the card holder. This requires upgrading the system using emerging technology and implementing the speech synthe- sis technique within the system.
Table 4.0 provides the proposed checklist, which highlights various activities involved to carry the business operation and the corresponding outputs for each activity. Each operation helps in capturing performance requirements.
Business Objectives of the System
Though the overall business objectives of the system are listed as part of the business context, it is necessary to define operation-wise business goals. Many of the business functions are known during the systems requirements study phase. However, those functional requirements (see Robertson, 2005) that impact performance must be revisited and defined properly. Business plans must be explored properly from the angles of scalability testing. Proposed guidelines that may help in obtaining information on business objectives would be:
•
List the objectives of the system and categorize them according to business operations.•
What are the business plans for next Y years? Quantify, if possible.•
What are the growth projections for the coming years? Quantify, if possible.•
Identify the parameters that impact growth projections of the organization.Activity Output of the activity
Identify different working groups wit h in the
organization A list of stakeholders
List the operations of each working group. Determine the complexity of each operation.
A list of operations Determine how each operations impacts
their business growth Scalability information Identify any business process in place. Nil
Determine the critical operations, if any, of the group.
Identify possible bottlenecks for performance.
•
Are there any specific performance based goals defined by management? If so, list all the goals.Logistics of Operations
If the business activities are distributed at many places, modeling the logistics of operation is of paramount importance. This is because performance of the system depends on the bandwidth available at the place of operation and vicinity of ISP providers. Volume and type of transactions also vary from place to place which impacts performance. The following guidelines provide an overview on logistics of operations.
•
List different places of operations.•
List the complete set of business operations at different places.•
Identify type and volume of transactions at each place.•
Identify the topology of networking and ISP providers place-wise.•
Determine the expectations at each place of operation.•
If you think the priority of objectives changes at different places, list them properly.User Base
User-wise information on the user base at different locations of operation and type of operations are required. The user base may be broadly categorized into user groups, and each user group can be identified with a set of operations. There may be different groups such as internal and external users. Internal users may be employees. If their base is
U s er
Internal
S enior M anagem ent
M iddle M anagem ent
E x ternal
A d-H oc U s er
(W eb S urfer) C us tom er
IS G roup C us tom er S upport G roup
C ritic al
C us tom er C us tom erR are
Mo st Fre q u e n t V isito r
considerably high, create subgroups based on their activities. Similarly, external users may be customers who undertake different activities with the Web site. They may just surf the Web site as a Web surfer, or some customers may transact through the Web site. Activities vary with customers’ requirement, and the number of customer visiting the Web site varies with time.
Proposed guidelines to be used to collect information on user bases would be:
•
Identify the user base and its growth potential;•
Differentiate internal and external users;•
Determine different groups based on operations;•
Determine different groups based on logistics;•
Prioritize the user group based on their requirements; • Frequency of accessing the Web site by each user group.Figure 4.3 provides a possible user base for a typical industry. Observe that internal users are also contributing to performance requirements like external users are. Subsequent categorization depends on the specific industry. User groups will help in planning reload on the system during PT.
Infrastructure and
Network Environment
Infrastructure available for the proposed system must be studied exhaustively. Infra- structure mainly involves the computer hardware and software systems. If the proposed system is to be developed from the ground up, the infrastructure recommended for the system must be accounted for in the study. If the proposed system is a reengineered system, existing and new infrastructure must be accounted for in the study. Infrastruc- ture requirements are divided into (1) network environment and (2) hardware/software system.
Network Environment
Evaluate the network environment in terms of Local Area Network (LAN) and Wide Area Network (WAN). If the organization is using LAN, it may be composed of several servers working in a C/S environment connected by fast Ethernet using the 100 BaseT standard (see Guide to using fast enthernet, 2005) and rated at 100 Mbps. If the performance result identifies LAN as the bottleneck, the speed of the LAN could be increased to gigabit per second using gigabit Ethernet. Also, T1 lines may be shared between the Web and telephone traffic.
Domain 6 Domain3 Domain 1 Domain 2 Domain 5 Domain 4 Workstation 5 Workstation 3 Workstation 2 Workstation2 Workstation1 Workstation3 Workstation 1 Workstation 3 Workstation 2 Workstation 2 Workstation 1 Workstation 4 Remote Application /Web Server Local Server 2 Local Server 3 Workstation 4 Workstation 1 Workstation 4 Workstation 3 Local Server Local Server 1
Remote Data Server
INTERNET
Figure 4.4. Typical computer network (WAN)
If the organization is using WAN, different offices might be connected through dedicated T1 lines. To illustrate, a typical networking environment for the banking project could be as shown in Figure 4.4.
The system will operate in a client-server network as well as be supported by Web. There will be a cluster of LAN situated at different places. Some of the clusters may have their own Web server and remote data server. All clusters are linked to the headquarters by a WAN. The client-server network will include approximately 2,000 users, with one client workstation per user. A complete topology of the network environment is available with the IS department of the bank.
Hardware/Software System
The proposed/existing application system requires proper hardware/software infra- structures to work efficiently. The organization may procure the relevant ones or may also be in excess. These resources may be utilized exclusively for the proposed system or in addition to the system; they may be used for other activities. In such cases, careful examination is required, as other systems will be considered noises during PT. Table 4.1 provides the checklist for the basic infrastructure required for a typical Web system.
Though the hardware/software is used for the proposed system, there may be other applications running on the system. These applications impact performance by unnec- essarily loading the system and network. If these applications are required to run along with the proposed system, the implication of these applications on the performance must be studied.
Other important issues to be noted are the people issues. Management may not be knowledgeable enough to understand the performance issues. The IS department may be very stringent on certain issues related to performance results requirements. All users may not have full knowledge about the expectations on performance of the Web site they are going to use. All these issues must be carefully handled during the performance requirement study.
Sl.
No. Hardware/Software Types Descriptions
1 Web Servers Web servers (see Brain, 2004) are connected to routers, which in turn connected to the Internet through multiple T1 lines. If there are host of web servers, load balancers are used to balance. The load on these load balancers impacts the performance. Web servers may have its own databases but not all data are available in the local database.
2 Application Servers Application servers may be installed on a single machine
along with the web servers or may be in different