February, 2010
Testing and Quality Management — Best
Practices
Any enterprise must accurately design, plan, and test core business processes to optimize operational capabilities and revenue-producing opportunities. Rather than make ad-hoc investments and business changes to meet temporary needs, organizations must test and configure key applications so that they are ready to work from the moment that they are deployed in production. By testing and optimizing their IT environments to align with business processes, Best-in-Class organizations were able to reduce operating costs by 26% and improve process efficiencies by 41%. However, these improvements required a wide range of modeling, testing, and optimization procedures. By learning what top companies did in comparison to their counterparts to reduce costs and simplify their work environments, end users can adopt these superior practices and create a more stable IT environment for their business.
Key Goals for Testing and Quality Management
Companies must understand all steps of their process architecture to optimize business processes throughout the organization, ranging from business process design to enterprise application testing across all modules and applications. These organizations must obtain the 99.5% uptime and ease of end-user support associated with premier application performance to enable the business and accomplish the business gains associated with Best-in-Class application performance. When these goals are measured in a production environment, they are most easily linked with the challenges of supporting peak load as the application is stressed at specific bottlenecks. However, focusing strictly on these bottlenecks is a short-sighted and reactive approach to application testing.Organizations that are able to proactively fix problems, optimize application uptime and increase transaction conversions focus on the quality and process alignment of applications early in the implementation process. In isolation, stress-testing does not ensure that businesses have optimized application performance. IT must provide solutions that align formally designed business processes on an end-to-end basis with specific applications and IT infrastructure elements. By doing so, IT operations, application tests, and metrics are directly related to business metrics, thereby making IT a provider of business productivity and revenue. As companies struggle to do more with less, they must stop making ad-hoc and temporary technology investments and move to a mature and strategic model of IT investment and alignment to ensure that mission-critical business processes are productive, flexible, durable, and scalable.
Analyst Insight
Aberdeen’s Insights provide the analyst perspective of the research as drawn from an aggregated view of the research surveys, interviews, and data analysis
This focus on the business processes associated with enterprise applications is accentuated by the fact that Best-in-Class organizations are 27% more likely than all other companies to have formal executive-level support for a business process management effort. Because these changes go far beyond the IT organization to all areas of the organization, companies that have identified and demonstrated clear executive support have been more successful in deploying these business process management efforts.
Companies seeking to improve the efficacy of their business must optimize their business applications to match the demands of the company and the particular needs of enterprise constituents. Although enterprise applications are typically chosen for their scalability and flexibility, they must still be properly configured to meet the demands associated with established network challenges, varied server and cloud infrastructure demands, and even the specific nuances of supporting specific endpoints, browsers, and operating systems.
Testing Enterprise Applications
There are a number of tests that must be performed before applications are placed in a production environment. Although this document does not cover the entirety of these topics, organizations must be aware of each of these testing categories as they implement enterprise applications:
Unit Testing - Although unit testing is often used to describe granular tests of individual programming functions, unit testing from a pre-packaged enterprise application perspective should be seen somewhat differently. Instead of focusing on a specific programming function, enterprise
application unit testing should focus on supporting individual transactions or business rules.
Requirements Testing - This series of tests is intended to identify the use cases, scenarios, transactions, external applications, and risks associated with the application.
Integration Testing - Often scenario-based, these tests are intended to show that a number of requirements can be fulfilled in the context of a specific business process. This testing can either be within the scope of operations covered in a single application or include the execution of business processes associated with multiple applications. This process can also be accelerated through automated regression testing that validates business processes on a cross-module basis.
System Testing - Performs all manual test cases and automated test scripts in context to provide a baseline for defect analysis and testing. This testing step allows the installation team to make sure that the application is working according to spec after all business requirements have been added. There is typically a round of defects testing associated with unit, integration, and system testing to fix problems that are found at each stage.
Acceptance Testing - This round of tests is end-user driven to make sure that all modules and applications are working together in a cross-functional
Definition of Best-in-Class In the August 2009 report Benchmarking Network and Application Visibility: Reduce Downtime to Keep the Enterprise Running, the top 20% of respondents had the following performance:
√ 83% of application issues resolved before end-users were affected.
√ 90% of network issues resolved before end-users were affected.
manner. This level of testing must be performed by end-users in an
environment that can fully simulate the production environment, functions, and features that the application must support. Any environmental details that cannot be replicated in the test environment must be treated as known risks when the application is released to a production environment.
Load Testing - One of the top challenges for enterprise applications is the challenge of meeting peak loads. A consistent practice for Aberdeen's Best-in-Class organizations is to load and stress-test applications in a pre-production stage to simulate the challenges of a live environment. Usability Testing - Organizations must optimize the user interface associated with enterprise applications to gain the greatest value from their investments. As the enterprise application becomes the primary portal for all work activities, end-users must be able to access key business processes quickly and efficiently. These tests involve a combination of business process alignment, interface testing, and application flexibility that allows for
potential customization that drives business differentiation and competitive advantage.
Performance Testing - Although the enterprise application may be working from a general perspective, performance testing provides information about the application based on specific environmental requirements or challenges that are not obvious in a generalized testing environment.
Security Testing - When employees are using and accessing enterprise applications, the data and functions involved must be managed under enterprise standards. To secure the enterprise environment, applications must meet governance, risk management, and compliance (GRC) issues defined by the company, industry standards, and national/provincial standards. This testing must be done in parallel with the pertinent legal, regulatory, and IT security personnel responsible for enforcing and documenting compliance.
Standardizing Business Processes
To support mature application testing processes, businesses must optimize the mapping and planning of business processes. By doing so, top
enterprises can improve operating costs, increase customer responsiveness, and achieve greater process efficiencies compared to all other respondents by aligning their testing and quality processes with the business processes involved.
Best-in-Class companies understood the connection between visibility to business operations and improved business performance. These companies were twice as likely as all other organizations to measure end-to-end process cycles in their organization (Figure 1).
Figure 1: Best-in-Class Performance Management Capabilities 47% 40% 28% 35% 15% 23% 10% 20% 30% 40% 50%
Ability to measure performance against defined SLAs
Ability to measure end-to-end process cycle times
% of R e s pondent s
Best-in-Class Industry Average Laggard
Source: Aberdeen Group, November 2008
To gain this capability, these respondents first needed to test each link of their value chain from a load and quality perspective. Afterward, they needed to define each process and workflow associated with the business process that needed to be analyzed from end-to-end. Although this can be a difficult process, by gaining real-time visibility of their operations, these companies are able to conduct analytical efforts that can reduce process cycle times and improve performance throughout the organization. Top companies also leveraged dashboards and alerts to monitor business processes. In BPM and Beyond: The Human Factor of Process Management, 75% of Best-in-Class companies had business intelligence (BI) dashboards and scorecards. In addition, Best-in-Class companies were three times more likely than all others to have BI integrated with their Business Process Management (BPM) solution. By having these reporting capabilities associated with their BPM solution, these top performing organizations were able to gain real-time monitoring and forecasting capabilities based on their business processes. By deeply contextualizing business intelligence within the relevant product and service-based processes, the company is better able to proactively solve problems impacting end-users and avert issues by aligning operational changes with business value.
Application Quality Drives Efficient Businesses
To meet the challenges of application management and provide business value to the enterprise, top-ranked organizations focused on functional application testing solutions to improve the quality and response of their applications. Organizations must have a mature application testing process that starts long before the technical challenges of understanding response times and conducting appropriate load testing.
Functional testing consists of a testing regiment designed to ensure that a newly-installed application will provide the expected end-user and
business-Fast Facts
√ On average, Best-in-Class companies report that 45% of their workforce uses the BPM solution
Compared with
√ 35% of the workforce for all other companies
based functionalities that are expected. These tests are designed to emulate end-user feeds and interactions to determine that the application will be able to support and drive the business processes that it is intended to support.
Best-in-Class performers also focus on testing performance and quality before applications are released to a live enterprise environment. In data collected for March 2009's Application Performance Management: Getting IT on the C-Level's Agenda, 34% of respondents stated that the inability to
effectively test application performance in development and pre-production stages was a top challenge for launching enterprise applications.
To develop performance and quality testing capabilities, application requirements and potential defects must be defined from a business perspective. Companies must initiate a testing release process that creates test scripts to match work scenarios that the production application will face. In addition, these firms can improve application releases through mature script-driven and data-driven testing capabilities compatible with test automation and end-user testing. By pursuing this standardized approach for requirements testing and interface testing, organizations can test a number of business-based scenarios without having to fully create and develop each test. In addition, this approach allows companies to take a more agile approach to application testing by adding elements of user acceptance and performance testing to the initial requirements and integration testing. This approach should not fully replace acceptance and performance testing steps, but is intended to create a more integrated testing process that allows for the alignment of business and information technology to occur throughout the installation of enterprise applications.
Once this environment has been simulated, testing can occur in the context of the workflows and service level metrics associated with each business process. By testing these workflows with automated efforts, code
optimization and debugging, and end-user testing efforts, these applications can be optimized by recreating production challenges as fully as possible. This release process allows organizations to better understand when they are able to launch a product based on all stages of their pre-production testing efforts. Through this approach, companies can gain the benefits of understanding application response times, maintaining application availability, and improving application-based revenue and transaction-based goals. By thoroughly testing these solutions and understanding the obstacles to obtaining optimal response times prior to initial production deployment, these top organizations kept vital business processes up and running without impacting employees and customers. Even though they were challenged with the same demands that all other companies faced, 42% of Aberdeen's Best-in-Class respondents in the August 2009 report
Benchmarking Network and Application Visibility: Reduce Downtime to Keep the Enterprise Running, were able to predict the response times of new
applications prior to deployment. This ability to predict response times was 2.7 times greater than all other respondents (Figure 2).
"Originally we thought that we didn't have enough bandwidth to run an ERP application and were thinking about adding an additional T1 line. However, a technology solution that we had in place helped us to realize that the root-cause of the problem was on the application side - not the network. This allowed us to avoid adding more bandwidth, but it also allowed us to be able to effectively manage
application performance on an ongoing basis and address potential problems before they could impact end-users."
~ IT Director, Manufacturing Company
Figure 2: Best-in-Class Understand Application Responses 10% 19% 42% 0% 10% 20% 30% 40% 50% Ability to predict application response times before deployment Percentage of Respondents (n=156) Best-in-Class Industry Average Laggards 10% 19% 42% 0% 10% 20% 30% 40% 50% Ability to predict application response times before deployment Percentage of Respondents (n=156) Best-in-Class Industry Average Laggards
Source: Aberdeen Group, August 2009
Meeting the Demands of Peak Application Loads
A mature and thorough load testing capability also differentiated our Best-in-Class companies from all others in reaching these goals. For major infrastructure enhancements, the use of load testing ensures that any server, data center, or enterprise application deployment will not serve as a
bottleneck to throttle the enterprise environment. By taking this step to perform holistic load testing for new and updated applications, Best-in-Class companies mitigate the disruption of business processes and insure that their applications are able to handle the strain of peak traffic (Figure 3). Figure 3: Enterprise Usage of Load Testing
24% 33% 33% 41% 46% 49% 0% 10% 20% 30% 40% 50% 60% Major infrastructure enhancement New web application rollout Enhancement to existing web application Percentage of Respondents (n=94) Best-in-Class All Others 24% 33% 33% 41% 46% 49% 0% 10% 20% 30% 40% 50% 60% Major infrastructure enhancement New web application rollout Enhancement to existing web application Percentage of Respondents (n=94) Best-in-Class All Others
Aberdeen Group, October 2009
Fast Fact
√ 50% of respondents reduced the number of calls to their help-desk related to network or application performance as a result of testing and improving their network and application environment.
One of the surprising aspects that Aberdeen has noticed is the low adoption of load testing capabilities among all respondents. Even as Best-in-Class organizations have shown a correlation between their adoption of load testing and their ability to proactively reduce application and network issues, a majority of all respondents are still not using load testing for a variety of tasks ranging from infrastructure enhancements to new web application rollouts.
Effective load testing also allows companies to prepare for peak traffic challenges by establishing strategies for link load balancing. This allows any excess ingoing or outgoing traffic to route to a server with excess capacity. By optimizing the usage of all traffic-processing servers, companies can handle higher levels of peak traffic without adversely affecting the network and application traffic to the general population accessing corporate resources.
The Specific Challenges of Web Applications
Best-in-Class companies were also consistently more likely than their counterparts to support load testing efforts to enhance current web applications and sustain new applications. These enhancement and support roles each represent a different perspective regarding the importance of load testing.
For current web applications with traffic load that has already been defined and supported, the importance of load testing is to understand how new application versions will affect traffic handled by existing applications. Companies can find that the addition of even one new application to a webpage may increase traffic for specific applications by an order of magnitude. For new web applications, companies may face the additional issue of driving unexpected traffic that brings the entire corporate network to a standstill. Even the most functional and useful web applications are useless if they drive network latency to the point where employees and customers avoid using the application.
Best-in-Class companies focus on the difficulties of testing web applications on an end-to-end basis. In the October 2009 benchmark report, Optimizing Performance of Web Applications, Aberdeen found that the top 20% of respondents (called the Best-in-Class) had seen the following changes over the past 12 months: a 20% increase in successful transaction conversions, a 13% increase in web application transactions, and a 25% increase in the profitability of their web applications. As these transactions affect a number of different servers and applications, it is vital to insure that each part of the web application chain is fully able to support corporate end users.
To support the entire value chain of web applications, Best-in-Class companies extended beyond load testing and bandwidth management to end-user based testing capabilities. Previous Aberdeen research has shown that extending the time needed to access a new web page from four
seconds to five seconds can lead to a 16% decrease in customer service and an 11% decrease in page views. Companies seeking a root cause analysis for
these delays did not simply look at the internal performance of web servers. Instead, they also looked outward to external networks and even the end user's endpoint to see how their web applications interacted with the full ecosystem involved in delivering the applications to relevant contacts. Since applications such as ERP, CRM, and e-commerce-driven programs are increasingly web-based and play a vital role in strategic and operational initiatives, the delivery of enterprise software is increasingly dependent on non-traditional software delivery models. The web experiences of these software platforms on laptop browsers and mobile devices must be tested to the same extent as the traditional network portals. From an internal perspective, this ensures that employees are able to deliver enterprise projects and deliverables based on one consistent and centralized application channel.
By integrating the IT view of load testing with the business perspective of holistic end-user monitoring, Best-in-Class companies are able to use web applications to improve metrics that affect the entire organization.
Case in Point - Hubbell Incorporated
Hubbell Incorporated, based out of Orange, Connecticut, creates electronic and electrical products on a global scale. They currently have 13,000
employees and 2.7 billion dollars in revenue. Because of their business need to quickly launch global software rollouts, it is vital to fully test all
applications before they are released in a production environment. To meet these needs, Hubbell standardized on a comprehensive solution that provided a foundation for:
• Unit testing, which allowed the company to test each function of the source code at the most granular level.
• Scenario testing, which simulated the software's ability to conduct multi-step and business-relevant roles
• Integration testing to ensure that all modules within a software system work together in a coherent manner
• User acceptance testing, which makes sure that the software passes the ultimate test in the business: supporting corporate subject matter experts and employees in their day-to-day operations • Defect management to accelerate the corporate response to
defects and minimize downtime associated with software problems. Based on this initiative, Hubbell was able to lower costs in several ways. First, by having a coordinated and centralized test platform, test assets could be leveraged and re-used to reduce the total cost of ownership related to application testing. Second, the organization was able to gain real-time reporting and analytical capabilities to provide all stakeholders with any alerts, rather than rely on manual processes and reports. But the most important aspect to this global testing platform was that Hubbell was able to increase test coverage and lower the risk of application deployment by
reducing the time and effort needed to maintain test assets. By reducing the risk component associated with enterprise software application upgrades, patches, and plug-ins, Hubbell was able to reduce downtime and react more quickly to the business needs of the organization.
Case in Point - Linde AG
Linde Gas AG is one of the largest industrial gas producers in Europe. These gases are used to support chemical, food, environmental, and medical organizations as products, tools, and business enablers. To sell these
products, Linde works with third-party distributors known as "depots." Working with these depots, Linde Gas currently conducts business in nearly 100 countries around the world and achieved revenues of 12.7 billion Euros in 2008.
In 2005, Linde Gas sought to augment its sales channel through a web-based portal designed to allow depots direct access to Linde Gas' own sales and distribution system. This application was considered to be mission-critical to sustain the operations and growth of the company. Thomas Steinich,
customer competence center manager, explained, "If our resellers cannot sell our product, we will lose customers and damage our business. And if something harms the sales channel … then we would have big, big problems."
As this portal was developed from proof of concept to a production phase, Linde Gas found that the initial application had not been fully validated to handle a full production load. Although the company originally estimated that this portal needed to support over 1,200 users, the initial portal reached application failure after only 27 users.
To find why this was happening, Linde Gas developed a detailed analysis of the complete application structure needed to establish this portal, including the web client, the internet browsers used, networking and virtualization servers, portal platform software, a web application Server, and the ERP software suite associated with Linde Gas' sales and distribution operations. To fully document this complex and heterogeneous web application environment, Linde turned to a consulting project team lead by Ralph Killenberger, Project Manager at Hewlett-Packard.
Killenberger's team was able to validate the performance of this portal by establishing a complete testing process, starting with test scripts aligning to individual transactions, then simulating business scenarios with multiple concurrent transactions. To enable break-fixes, the team also developed a monitoring process to establish root-cause analysis when the application is stressed by overwhelming load. By taking this approach, the team was able to find multiple bottlenecks throughout the web portal ranging from the configuration and scripting to the need to optimize the portal application to the infrastructure being supported.
After identifying and fixing these documents, the sales and distribution processes supported by this portal improved by an order of magnitude.
Killenberger noted from an overall perspective, "Ultimately we delivered a performance improvement on average of about 1,700%."
To specify, he provided an example from sales operations. "The average transaction response time - for example, the sales process where you sell a bottle of oxygen - started at 29 seconds. After everybody was finished with the tuning engagement, it was 2.5 seconds."
As a result of this strenuous load testing and performance validation, Linde Gas was able to provide an enterprise-grade portal to all of its depots with real-time access to the sales and distribution functions that were necessary. The portal is now available to over 800 portals, which have full access to conduct and manage business on Linde Gas' behalf.
Steinich summarizes his night-and-day experience with and without a load testing solution by stating "It was a real benefit to have [a load testing solution] and the results we achieved when we used it were really fantastic. In the end, I can definitely say it was a total success, because we really had no problems with performance with performance when we went live."
Key Takeaways
To improve optimization of business processes throughout the organization, companies must have the ability to understand all steps of their process architecture. Fundamentally, these organizations must have reliable, high-quality application performance to create the foundation needed to support optimal business processes. Organizations that take the time to test their applications from a load bearing, quality, and process-based perspective see improvements in their ability to proactively fix problems, optimize
application uptime and increase transaction conversions. These goals should be top-of-mind for the CIO as business processes are increasingly
automated, application-enabled, and online.
However, stress-testing applications in a pre-production environment does not ensure that businesses have optimized their operational and tactical capabilities. It is also vital for the enterprise to formally design business processes on an end-to-end basis and align them to specific applications and IT infrastructure elements. By doing so, IT becomes intimately connected to the lifeblood of the business so that IT metrics are directly related to business metrics. Rather than simply working as a cost center, IT now becomes a provider of business productivity and revenue. As companies deal with the struggle of doing more for less, they must stop making ad-hoc and temporary technology investments and move to a mature and strategic model of IT investment and alignment to ensure that mission-critical business processes are productive, flexible, durable, and scalable. The role of the executive cannot be taken for granted in achieving these results. Best-in-Class organizations have CEO, CFO, or CIO commitment to deliver performance management from both a business and technical perspective. Because enterprise applications can affect the entire
that are needed to fully test and optimize these process-driven
implementations. As these testing programs take place, implementations that have won clear executive support have resulted in more successful business process management efforts.
By aligning load testing, quality testing, and business process management efforts, companies can transform their IT environments into true business support environments where the goal is no longer simply creating or launching applications based on company demand. Rather, organizations can progress to a point where IT and line-of-business concerns become
intertwined throughout both application and business process development efforts. This collaboration creates opportunities to eliminate the mistakes that can occur when technical and functional groups work in isolation from the complete enterprise application installation team. By combining the tools, the people, the processes, the reporting capabilities, and key
performance metrics into an integrated quality program, companies can use their testing processes and scenarios to strengthen the business as a whole. For more information on this or other research topics, please visit
www.aberdeen.com
Related Research BPM and Beyond: The Human Factor of
Process Management; November, 2008 Application Performance Management: Getting IT on the C-Level's Agenda; March 2009
Monitoring the End-User Experience: Improving Business Performance through Application Management; June 2009
Benchmarking Network and Application Visibility: Reduce Downtime to Keep the Enterprise Running; August 2009 Optimizing Performance of Web Applications; October 2009
BPM Accelerated: Slashing Cost and Time with Agile Business Processes; October, 2009
Authors: Hyoun Park, Research Analyst, Technology Markets Group
([email protected]); Jeffrey DeBarros, Senior Research Analyst,
Network and Application Performance Management
Since 1988, Aberdeen's research has been helping corporations worldwide become Best-in-Class. Having benchmarked the performance of more than 644,000 companies, Aberdeen is uniquely positioned to provide organizations with the facts that matter — the facts that enable companies to get ahead and drive results. That's why our research is relied on by more than 2.2 million readers in over 40 countries, 90% of the Fortune 1,000, and 93% of the Technology 500.
As a Harte-Hanks Company, Aberdeen plays a key role of putting content in context for the global direct and targeted marketing company. Aberdeen's analytical and independent view of the "customer optimization" process of Harte-Hanks (Information – Opportunity – Insight – Engagement – Interaction) extends the client value and accentuates the strategic role Harte-Hanks brings to the market. For additional information, visit Aberdeen http://www.aberdeen.com or call (617) 723-7890, or to learn more about Harte-Hanks, call (800) 456-9748 or go to http://www.harte-hanks.com This document is the result of primary research performed by Aberdeen Group. Aberdeen Group's methodologies provide for objective fact-based research and represent the best analysis available at the time of publication. Unless otherwise noted, the entire contents of this publication are copyrighted by Aberdeen Group, Inc. and may not be reproduced, distributed, archived, or transmitted in any form or by any means without prior written consent by
Appendix A:
Research Methodology
Between October 2008 and October 2009, Aberdeen examined the use, the experiences, and the intentions of 889 enterprises in business process management, application testing and end-user management in a diverse set of enterprises.
Aberdeen supplemented this online survey effort with telephone interviews with select survey respondents, gathering additional information on IT operational strategies, experiences, and results.
Responding enterprises included the following:
• Job title: The research sample included respondents with the
following job titles: Manager (26%); Director (14%); Consultant (11%); EVP/SVP/VP (7%); CEO/President/Chairman (6%); CIO/VP IT (5%), and other (31%).
• Department / function: The research sample included respondents
from the following departments or functions: Information
Technology manager or staff (41%); Business Development manager or staff (11%); Operations manager or staff (7%); Business Process Management manager or staff (7%); Logistics/Supply Chain manager or staff (7%) and other (27%).
• Industry: The research sample included respondents from 37
different industries including the following: Software/hardware supplier (9%); Insurance/Real Estate (8%); IT Consulting/Services (7%); Finance/Banking/Accounting (6%); Industrial equipment manufacturing (5%); Government (4%); Health/medical/dental (4%); and Other (57%).
• Geography: The majority of respondents (52%) were from North
America. Remaining respondents were from Europe (23%); the Asia-Pacific region (15%); the Middle East and Africa (6%); and Latin America (4%).
• Company size: Thirty-one percent (31%) of respondents were from
large enterprises (annual revenues above US $1 billion); 34% were from midsize enterprises (annual revenues between $50 million and $1 billion); and 35% of respondents were from small businesses (annual revenues of $50 million or less).
• Headcount: Forty-eight percent (48%) of respondents were from
large enterprises (headcount greater than 1,000 employees); 34% were from midsize enterprises (headcount between 101 and 1,000 employees); and 18% of respondents were from small businesses (headcount between 1 and 100 employees).
