NTS Testing Labs
“E-Services” Test Lab
White Paper
Revision 1.0NATIONAL TECHNICAL SYSTEMS
The NTS Mission:
Assisting our Clients in Navigating a Short Course to Global Markets
Written by:
Troy Sukert, Technical Manager
Tracking No. WP201001 Rev. 1.0 Approved by: Osman Sakr Date 2/22/02 Quality Assurance Approval
Derek Fakehany Date 2/22/02
FileName
NTS White Paper - EServices - WP201001.doc
Copyright © 2002, National Technical Systems, Inc. This document is provided to clients and potential clients of National Technical Systems (NTS) for informational purposes only. It may not be reproduced or stored, in whole or
Table of Contents
1. Introduction...1
2. Web Site Testing...1
2.1 Browser Cross Compatibility Testing...1
2.2 Web Site Functional Verification Testing...2
2.2.1 Functional Verification Test Areas...2
2.3 Web Site Localization Testing...4
2.4 Web Site Testing Deliverables...4
3. Web Site & Server Testing...5
3.1 Web Load / Stress Testing...5
3.1.1 Overview...5
3.1.2 Virtual Users & TPS vs. Hits per Day...6
3.1.3 Test Tools...7
3.1.3.1 RadView’s WebLoad...8
3.1.3.2 Mercury Interactive’s LoadRunner...8
3.2 Example Web Server Load / Stress Test Procedures...9
National Technical Systems E-Services White Paper
1. Introduction
This document presents a summary of NTS Testing Laboratories Internet-centric test capabilities and associated service offerings (“E-Services”). The following matrix provides an at-a-glance summary of the topics discussed herein. Additional (more detailed) coverage of specific areas may be found in the associated document sections.
Technology / Test Area Service Offering(s)
Document Section & Starting Page Web Sites Browser Cross Compatibility Testing Section 2.1, Page 1 (content & applications) Site Review & Functional Testing Section 2.2, Page 2 Localization Testing Section 2.3, Page 4 Web Sites & Server(s) Load & Stress Testing Section 3.1, Page 5 (web server & back-end) Tool Term Licensing Program Fees Section 3.1.3, Page 7
2. Web Site Testing
2.1 Browser Cross Compatibility Testing
PC Workstation Image Server PC Workstation PC Workstation Ethernet Hub Image Files (OS + Browser)
Although an estimated eighty percent (80%) or more of the people touring the web use Netscape Communicator / Navigator or Microsoft Internet Explorer, a significant number of people are using a variety of other browsing tools. Add to this the incredible variation between versions of browser software. Each browser interprets web documents differently and in many cases incorrectly or not at all, very often requiring customization on the part of the web site developer to account for such variations. This proliferation of web browser software products and associated versions is both confusing and difficult to test & support. NTS offers a solution. Our Browser Cross-Compatibility Lab (BCCL) specializes in maintaining a full set of easily configured browser software configurations. Fully configured browser versions and associated operating systems are maintained on image servers and may be quickly restored to test platforms representing a wide variety of popular configurations with which to exercise a web site. NTS Test Engineers can quickly cover large portions of a web site with a complete matrix of browsers, insuring that the pages display correctly and reporting client-side script execution errors. Often, it is beneficial to perform a cross-browser compatibility test just to decide which browsers and versions not to support! The following matrix is an example of a typical selection of tests routinely performed in the BCCL (this matrix indicates only a subset of selected configurations; actual tests may include or exclude any number of supported configurations, and may include or exclude additional browsers and/or browser versions):
National Technical Systems E-Services White Paper Browser Wi n 3.1 Wi n 95 Wi n 98 Wi n98SE
WinME WinNT4 Win2K MacOS 8.0 MacOS 8.01 MacOS 8.5 MacOS 9 Web TV
Netscape Navigator v3.04 -X- -X- -X- Netscape Navigator v4.06 -X- -X- -X- -X- Netscape Navigator v4.07 -X- -X- -X- Netscape Navigator v4.08 -X- -X- Netscape Communicator v4.06 -X- -X- Netscape Communicator v4.5 -X- -X- Netscape Communicator v4.51 -X- -X- Netscape Communicator v4.6 -X- -X- Netscape Communicator v4.7 -X- -X- Netscape Communicator v4.7.6 -X- -X- -X- Netscape Communicator v6.0 -X- -X- -X- -X- -X- -X-
Microsoft Internet Explorer v4.01 -X- -X- -X-
Microsoft Internet Explorer v4.01 SP1 -X-
Microsoft Internet Explorer v4.5 -X- -X- -X- -X- -X-
Microsoft Internet Explorer v5.0 -X- -X- -X- -X- -X- -X- -X-
Microsoft Internet Explorer v5.5 -X- -X-
AOL v3.0 -X- -X- -X-
AOL v4.0 -X- -X- -X- -X- -X-
AOL v5.0 -X- -X- -X- -X- -X- -X-
AOL v6.0 -X- -X- -X- -X-
Web TV -X-
2.2 Web Site Functional Verification Testing
A modern web site can contain hundreds of individual pages, with many pages generated programmatically and designed to extract and display constantly changing content. In addition, even the smallest of sites tend to include many internal and external hyperlinks and elements such as tables, style sheets, client-side scripting, Java applets, dynamic HTML and links to server-side scripts (i.e. ASP, Java Servlets, Perl, PHP, XML parsers, etc.).
Problems with script execution, dead links, missing or incorrectly scaled images, browser incompatibility and other such problems and inconsistencies can quickly sour a user’s experience of the site. In addition, a web site that is difficult to navigate can make it troublesome to locate important information which can also make for a very unpleasant user experience. NTS web experts and extensive site testing experience can help. NTS offers a complete suite of web site functionality and design review services, any of which can be individually selected and combined with related services, such as server stress testing and browser cross compatibility testing.
2.2.1 Functional Verification Test Areas
◊ Functionality: Do the pages load and display correctly at both 800x600 and 1024x768 (and above)
resolutions, with the latest versions of MSIE and NetScape? Do both client-side and server-side scripted elements function correctly in these configurations? Are there any dead or incorrect links? NTS will exhaustively exercise the site to insure that users will not run into unexpected problems later on.
◊ Navigation: How easy is it to navigate the site? NTS will examine site organization and verify intra-site
navigation as well as external link functionality. If requested, NTS will provide recommendations for site navigation / organization changes based upon testers reactions, including adding, deleting or altering elements such as overall site indexes, additional links back to start / home pages, frames, pop-ups, etc.
National Technical Systems E-Services White Paper
◊ Graphics: Effective and correct use of visual elements is the focus of this test area. Initially, objective criteria are applied, such as insuring that images display correctly (in relation to layout) at various resolutions and color depths (including 8-bit if desired). Additionally, NTS will verify and report load speed for complex graphical pages in relation to dial-up and ISDN users. NTS, if requested, will then proceed to examine and comment upon subjective areas, such as: Are graphics over used on the page? Are graphics used to draw attention to certain places on the page, to make a particular point, or to enhance the pages visual appeal? Are large graphics overused, causing unacceptable download speeds for dialup users? For sites where the business’ products or services depend on graphical presentation, do the graphics show enough detail so that users make an effective purchasing decision? Are thumbnails used effectively (or at all)?
◊ Readability: Are the Web Pages easy to read? For example, are white spaces and headings used correctly
to separate elements and to direct the readers’ eyes? Are there obvious spelling and/or grammar errors on many pages? NTS will report inconsistencies and offer suggestions for improving the readability of Client’s Web Site.
◊ Background: Is the background distracting or make the pages impossible to read when using 8-bit color depth? Do the backgrounds add interest and integration to Client’s Web Pages? Are the backgrounds too large (and therefore too slow to load)?
◊ Content: The following are common to well designed web sites. These will be used as a base-line criteria
in evaluating the Web Site under test:
• Introductions: Is there an introduction to the company and/or the products? Descriptions of the company, background information, etc. helps present an image to Web Site visitors.
• Contact Information: Does the Web Site contain easy to find information for contacting the company for more information or assistance? Are phone numbers, fax numbers, e-mail, and snail mail addresses provided to allow people to contact the Client? If submission forms are used, do they include assurances that information is kept secure and not used for email lists, etc.?
• Information: Is the information on Client’s Web Site up to date, informative, and useful?
◊ Interaction: The ability to link between information using HTML is the Web’s biggest advantage. This
testing will focus on information such as:
• Links: Are the links well thought out (i.e. not overwhelming but useful)? Are external links also used to utilize the wealth of information on the Web? Are external links at a place where you don’t mind people leaving your site (i.e. after they’ve seen the essential information)?
• E-mail / Information Forms / Surveys: Can users communicate their interest effectively?
◊ Bandwidth Testing: Testing will be performed using both dial-up (28.8, 56k) and higher speed
connections (i.e. ISDN, cable modem / DSL, T1). Sites which deploy unusually large numbers of multimedia elements (i.e. Shockwave Flash presentations, streaming media, large images and image maps, etc.) may preclude the use of dial-up (i.e. NTS will note in test reporting that the site is ‘broadband only’).
◊ Secure Area Testing: Testing will be performed to verify that secure pages function correctly, and to test
National Technical Systems E-Services White Paper
◊ Data Validation: Data validation will be performed to check for pages with fields, such as a search tool on
a site. Does the web server remain stable when a large number of hits are returned or invalid data is entered into a field? Can the server determine and refuse invalid data, such as shipping of an order to Los Angeles, TX?
2.3 Web Site Localization Testing
As an adjunct to either site functionality or browser cross compatibility testing, NTS can review portions or all of a site for correct language localization. NTS, with both in-house language specialists and our relationship with SoftTrans•Berlitz, can support testing in up to thirty-five (35) different languages:
1. Arabic 19. Italian
2. Basque 20. Japanese
3. Catalan 21. Korean
4. Chinese (Simplified) 22. Norwegian
5. Chinese (Traditional) 23. Pan European
6. Czech 24. Persian
7. Danish 25. Polish
8. Dutch 26. Portuguese (Brazil)
9. English (British) 27. Portuguese (Iberian)
10. English (International) 28. Russian
11. Finnish 29. Slovak
12. French 30. Slovenian
13. French (Canadian) 31. Spanish
14. German 32. Swedish
15. Greek 33. Thai
16. Hebrew 34. Turkish
17. Hungarian 35. Vietnamese
18. Indian
NTS’s localization web site localization testing range’s from basic side-by-side verification against existing English language content (no detailed review) to deploying native language speakers while taking into account local customs, culture and business protocols for a detailed word-by-word analysis of translated content.
2.4 Web Site Testing Deliverables
A NTS Web Site Summary Report is the deliverable following the conclusion of NTS web site testing. Test results will be in the form of NTS Test / Problem Reports, Summary Matrices, and (if applicable) a Web Site Design Review.
NTS Test/Problem Reports record system configuration (i.e. browsers, system configurations, operating system versions, RAM, video resolutions, etc.), test procedures (outline) and associated test results for specific tests performed (i.e. browser cross-compatibility, localization review, functionality, etc.). Problem reports will indicate procedures to reproduce test cases which cause or expose anomalies. Test / Problem reports will are developed as developed on a priority basis (pending internal review).
The NTS Summary Matrix provides a quick overview of the entire project. The Summary Matrix is a master grid outlining the individual performed during the project. The matrix, updated daily, provides a quick overview of milestones performed to date, with columns specifying the product(s) tested, pass/fail/warn status, the date performed, Test Engineer identification, and related Problem Report numbers.
The NTS Web Site Design Review will provide a summary of findings, and will offer comments and suggestions related to the presentation and content of the web site under test. Portions of this report may include subjective ‘real world’ analysis by the NTS Test Engineers of the web site under test. Areas such as browser appeal, readability, navigation, graphics, background, content, and interaction will be reviewed and summarized.
National Technical Systems E-Services White Paper 3. Web Site & Server Testing
3.1 Web Load / Stress Testing 3.1.1 Overview
Web servers today range from single CPU PC systems running nearly free versions of Linux & Apache to multi-server Sun enterprise platform ‘multi-server farms’ running Fusion web multi-server software. In addition, back-end database servers and front-end load balancing equipment or software affect overall system response times. Increasingly, scalability and performance under load are critical factors which must be explored and tested to insure future success.
NTS is prepared to exercise web servers and associated two- and three-tier configurations using sophisticated test tools which can drive from hundreds up to tens of thousands of nearly simultaneous users, representing hundreds of millions of hits per day.
To support stress testing for both large and small web server configurations, NTS offers two options. The first option (scenario 1) is to test the servers at NTS’s location; this configuration requires that the complete set of server configurations be replicated at NTS labs (equipment may either be supplied by NTS or by the Client). A direct connection using 100BaseT switches and/or hubs is made between the simulated clients and the server(s) under test (for smaller tests, multiple T1 connections may also be deployed). The second configuration involves NTS’s Satellite High Bandwidth Internet Lab (SHBIL). NTS’s SHBIL can support scalable levels of internet connectivity with dual OC-3 connectivity. NTS’s SHBIL provides real world internet simulation of thousands and tens-of-thousands of users, from an internet backbone as large or larger than nearly any co-locator or ISP could possibly provide. NTS believes that no other company or service provider in the market today can provide NTS’s SHBIL capabilities.
EXAMPLE TEST CONFIGURATIONS
Internet 100M bit Link
W eb Server Under Test
Sim ulator Sim ulator Sim ulator Sim ulator Sim ulator
Switch / Router
W eb Server Under Test Sim ulator Sim ulator Sim ulator Sim ulator Sim ulator
Load Balancing Router / Software Back-End Server Oracle db W eb Server Under Test
Sim ulator Sim ulator Sim ulator Sim ulator Sim ulator
Switch / Router
W eb Server Under Test Sim ulator Sim ulator Sim ulator Sim ulator Sim ulator
Load Balancing Router / Software Back-End Server Oracle db Direct Connection Scenario 1 - Direct Connection
Scenario 2 - SHBIL Internet Connections
National Technical Systems E-Services White Paper 3.1.2 Virtual Users & TPS vs. Hits per Day
TPS and Hits Per Day
Sophisticated internet test tools can simulate thousands and even tens of thousands of ‘nearly simultaneous’ web site transactions. Simulated load testing is performed via “virtual clients”, in which one or more computer platforms generate multiple HTTP/HTTPS requests (transactions). Each platform deploys threaded multitasking, acting as a “load generator”, representing dozens and even hundreds of clients. The load generation is measured in hits per second, or transactions per second (TPS). The most common relevant statistics gathered include response time (for the requests) and overall throughput.
Web server manufacturers tend to quote their server performance statistics in terms of how many millions of hits per day they can sustain. As an example, Sun Microsystems’ claims support for 18 million hits per day with their 200 MHz, dual processor Ultra 2 servers. A typical relatively high-end PC (Pentium II or III) platform running Microsoft Internet Information Server (IIS) is touted to support anywhere from one to five million hits per day. So, how does hits per day relate to transactions per second (TPS)? The following chart presents a basic correlation:
TPS Per Hour (TPS x 3600) Projected Hits Per Day
10 36,000 864,000
100 360,000 8.6 Million
400 1,440,000 34 Million
1000 3.6 Million 86 Million
Keep in mind that a ‘hit’ is a single transaction on a web server (a single request). A typical web page may contain many elements, including several HTML documents (in the case of frames) and perhaps dozens of individual images. All of these elements amount to many requests from the client, which when aggregated, represent a complete single web page. During load testing, it is critical that a “use case” be designed which reflects actual application usage, rather than single hits. A real world use case often includes the loading of one or more pages, execution of scripts and/or submission of a form. The target use case will be deployed via script by the load generation tools. Using a few examples from the above chart, and assuming a web page which requires 25 hits, the following revised projections result:
TPS Transactions Per Hour (TPS x 3600)
Projected Hits Per Day
Projected Web Page Views Per Day
Projected Web Page Views Per Week
400 1,440,000 34 Million 1.36 Million 9.52 Million
1000 3.6 Million 86 Million 3.44 Million 24 Million
Of course, many factors play a role in actual test configurations and server performance, including the defined size of transaction (commonly used “average” transactions can range between 10kb - 60kb), and most importantly, the nature of the application being tested. Some servers may spend 80% or more of available processing time executing server side scripts, waiting for another tiered (database) server, or performing SSL encryption / decryption.
Virtual Users
The use of software tools to generate large numbers of effective virtual users requires high-end PC platforms (typically Intel Pentium III or better with a minimum of 512MB of RAM). Using platforms in this category, the following table indicates a realistic TPS generation capability (using Radview’s WebLoad):
# of Load Generating Platforms Total Virtual Users Estimated TPS
1 300 200
5 1500 1000
National Technical Systems E-Services White Paper
Keep in mind that 5,000 TPS represents could represent an estimated 430 million hits per day, and using the 25 hits per page figure, a staggering 17.2 Million page loads per day. All excepting the top fifty or so web sites in the U.S. could expect to see only a small fraction of this projected traffic.
Bandwidth
A server’s bandwidth on the internet is of tantamount importance when considering, planning and testing web server load. Using a relatively low average figure of just 12k per page, a T1/DS1 line would actually be bandwidth saturated when only ten different broadband users were trying to download the same 12k page simultaneously (a
T1/DS1 supports a maximum of about 125Kb per second). On the other hand, if ten such clients continuously
downloaded the 12k page over and over (as fast as they could), it would represent approximately 6 million pages downloaded a week!
Using this 12k per page figure, however, also shows that the best case bandwidth figure for the same T1 is thus limited to that same 6 million pages per week, regardless of the number of actual users - there is only enough bandwidth for 6 million 12k chunks to pass through a T1/DS1 line in a one week timeframe, whether requested by a few dozen broadband users or hundreds of dial-up users. Any ‘real world’ test configuration thus has to factor in the available bandwidth to ensure that bandwidth is not constantly being over saturated during testing (i.e. the total number of virtual users and the aggregate data being requested must be in-line with the available bandwidth, or the LAN / Internet becomes the bottleneck, affecting overall test statistics). Simulating large numbers of virtual users requires equivalent large amounts of bandwidth. This is the primary reason why many stress tests are performed using switched local 100Mbit and 1GB LAN backbone configurations. NTS can also drive ‘big-pipe’ web server stress tests from our SHBIL, which supports up to 100Mbits of internet access.
3.1.3 Test Tools
NTS maintains key relationships with web test tool vendors, allowing NTS to exercise the industries’ leading high-end web stress and load testing tools on a term licensing basis (TLP). The products are licensed for use on each project based upon the term of the project and the number of users emulated. The TLP program allows smaller companies the opportunity to test on a one-off basis using highly respected (and very expensive) test tools and affords larger companies an opportunity to evaluate whether or not the tool meets their needs before purchase. There is also, in some cases, a discount afforded to NTS clients who later purchase software licenses. Of course, all test scripts developed by NTS for the project become the property of the client following completion of testing. The following sub sections provide a quick overview of two widely used tools which NTS deploys for web site stress and load testing. The table below outlines the TLP pricing for the two tools which NTS deploys. The TLP cost component is separate from other project execution time & materials costs and is non-negotiable. It is also tied to the project duration; projects which extend beyond planned duration due to client delays may incur additional TLP costs.
Radview WebLoad
Total Virtual WebLoad WebLoad
Users 10-Day 30-Day
0-500 (next tier price) $3,750.00 0-1000 $3,350.00 $6,310.00 1001-2000 $5,000.00 $9,550.00 2001-5000 $8,900.00 $15,850.00 5001-10000 $11,900.00 $22,650.00
National Technical Systems E-Services White Paper Mercury Interactive LoadRunner
Total Virtual Up to Users 45 Days 0-1500 $19,300.00 1501-5000 $23,650.00 5000-15000 $32,900.00 15001-25000 $39,000.00 25001-50000 $54,000.00
Note: Mercury offers up to a 10% discount off the purchase price to NTS clients who later decide to license LoadRunner software.
3.1.3.1 RadView’s WebLoad
RadView’s WebLoad software is a premier scalability testing solution for web applications – Internet, Intranet or e-business. RadView’s WebLoad delivers open interoperability with web server applications, comprehensive HTTP support, automated operation, single-point management and real-time graphical and statistical analysis for maximum testing performance, reliability and usability. RadView’s WebLoad is globally recognized as a leader in web scalability testing and is a testing-tool of choice for demanding application environments.
RadView’s WebLoad verifies the scalability of web applications by generating a load composed of Virtual Clients that simulate real-world traffic. JavaScript-based test scripts are created that define the behavior of the Virtual Clients. WebLoad executes these test scripts and monitors the performance response of the web application including Transactions Per Second (TPS), Response Time, and Throughput.
WebLoad is engineered specifically for testing web applications using an optimized and distributed architecture. WebLoad’s ability to generate maximum Virtual Clients per Load Generator Machine combined with the ability to freely incorporate additional Load Generator Machines into test scenarios means unmatched efficiency and load generation capabilities for maximum testing power.
Website: http://www.radview.com/
3.1.3.2 Mercury Interactive’s LoadRunner
Mercury Interactive’s LoadRunner is a load testing tool that predicts system behavior and performance. It exercises an entire enterprise infrastructure by emulating thousands of users to identify and isolate problems. LoadRunner’s integrated real-time monitors enable organizations to minimize test cycles, optimize performance and accelerate deployment.
As e-business applications are becoming an integral part of IT systems, companies are often faced with a high risk due to the combination of multi-tier architectures, secure environments, multiple protocols and components, unpredictable user loads and the uncertainty of Internet bandwidth. Poor application performance can result in lost opportunities, revenue losses or even business failure. LoadRunner helps ensure flawless performance of these mission-critical systems upon deployment and throughout the lifecycle.
At the end of the testing process, you can use LoadRunner to create a series of graphs and reports that summarize your test results and present them in a clear and visual format. For added versatility, these graphs and reports can be easily exported to Microsoft Word, Microsoft Excel, Lotus 1-2-3, e-mail and more. Using LoadRunner’s expanded testing capabilities, you can now not only exercise an entire enterprise infrastructure, but you can also isolate problems in order to optimize performance and accelerate deployment.
National Technical Systems E-Services White Paper
3.2 Example Web Server Load / Stress Test Procedures
Stress test procedures will vary widely, since various web sites and related technologies have different performance requirements. In addition, in order to determine a simulated load profile, a simulated user profile must first be decided upon. The profile is basically a description of what a ‘typical’ site user might do. Most tests involve multiple user profiles, each of which represents a certain type of user. The stress test model then simulates various numbers of each type of user, either all at once, or through a cycle where more and more users are added over time. All web stress test projects involve a ‘discovery’ phase, during which the user profile scripts are developed. During this phase, a smaller number of simulated users is also tested (typically 100 to 300) to exercise any bugs out of the scripts and to prepare for follow-on larger scalability testing.
The following sample test procedure outlines a typical web stress project flow (test procedures have been generalized):
1. Discovery Phase / Overview: Prior to the start of testing, NTS Programmers will develop an automated test script. This automated test script will be designed to exercise Client’s web site with approximately two-thousand (2000) simultaneous virtual users. During development, NTS will also determine the correct tools to be utilized on the servers to determine (1) processor utilization (2) RAM usage and (3) loads for each thread (of the multithreaded processes). After the successful development, NTS will perform “rehearsal” testing with a few hundred users to insure correct script functionality.
2. Setup Workstations: NTS Test Engineers will setup the virtual client “test bed” for load/stress testing. NTS Test Engineers will setup each of the twenty (20) PIII 700MHz workstation platforms with Microsoft Windows NT Server v4.0 SP6, at least 128MB of RAM, and a 100Mbps Ethernet adapter. Each platform will represent up to 100 virtual users during testing. One of the platforms will act as a controller, periodically gathering the statistics and results obtained by the full set of virtual clients.
3. Web Server: NTS Test Engineers will utilized Client provided server test platforms (i.e. Sun Solaris v2.6 and Netscape Enterprise Server v3.61). Client is expected to provide fully configured servers or to be available on-site to aid with configuration issues if NTS is performing the installation. Servers will be connected to a network switch (100Mbps Ethernet network) for testing. Prior to any testing, NTS will verify that the server is functioning correctly;
4. Tier 2/3 Platforms: Client’s web application and Oracle v8 are also required to perform testing. Client will be responsible for providing Oracle and associated licensing for use with the test servers. Additionally, Client will also be responsible for configuring their web application on the test server and additional (Oracle) server prior to testing. NTS test engineers will verify that the complete system is functioning correctly prior to proceeding;
5. Test Tool Installation: The virtual client simulation test tool will be installed onto the full set of load generation test platforms per documentation. The NTS developed test scripts will be installed. Verify that the software is functioning correctly on each platform prior to proceeding;
6. Load/Stress Testing: The stress test will evaluate the performance of Client’s server and site while under stress. Testing will be performed over an extended period of time (approximately two [2] working days) to evaluate performance. Per Client’s request, NTS will measure response times and loads for the processor and RAM usage on the servers. NTS will setup the virtual clients to simulate large number of users based on the test profile and the platform / network bandwidth. NTS will evaluate the functionality of Client’s product with up to two thousand [2,000] virtual users, starting at 100, then scaling up in increments covering 200, 400, 800, 1,000, 1,500, and finally 2,000 virtual users, representing millions upon millions of simulated ‘hits’ per day.
National Technical Systems E-Services White Paper 3.3 Web Server Load / Stress Test Deliverables
Upon the completion of testing, NTS will provide Client with the raw data and summary analysis of stress testing performed (i.e. Transactions Per Second [TPS], Response Time, Throughput, Processor Utilization, and RAM Usage). Depending upon the test tool used, large numbers of additional statistics and source data may also be available. NTS will provide the raw data files (typically Microsoft Excel format) as well as a summary report specific to the test performed (which will include graphs and charts as relevant).
NATIONAL TECHNICAL SYSTEMS
National Technical Systems (NTS) is one of the largest consulting and certification laboratories in the United States. NTS operates extensive test facilities, throughout the United States and Japan, covering operational environment, stress environment, compatibility, interoperability, broadband network testing, EMI/EMC, FCC, NEBS, USB, WHQL, Product Safety testing and ISO Audit/Certification (we certify the National Space Administration for ISO compliance). NTS is a Nationally Recognized Testing Laboratory (NRTL) - as certified by the United States Government. We are accredited to ISO 17025 by the American Association for Laboratory Accreditation. NTS holds many certifications from foreign governments as a Conformity Assessment Body (CAB).
We are the only “one stop” certification and compliance facility offering a broad range of testing capabilities. These capabilities include significant financial resources, numerous certification programs and a high level of lab security. We support extensive test procedures, operating systems and hardware.
We are fully equipped for broadband testing and offer such services as DOCSIS “third-party” pre-certification and DSLAM compatibility and functionality testing. A variety of PSTN and Internet access methods are available ranging in speed from dial-up to 1Gb/sec. Several independent T1 Internet access points and frame relay circuits are available. An OC-12 link permits access ranging from standard T1 to OC-12 (644 Mb/sec) in a variety of formats: ATM, Frame Relay, SuperTrunk, HiCap and various other architectures.
Our particular testing laboratory is located in Culver City, California. This facility is centrally located and is just 15 minutes from Los Angeles International Airport.
IT IS EASY TO START!
Send us your “wish list” covering the areas that you would like addressed in your test, and we will design a test proposal specific to your needs that covers only what you want tested. To turn your request into a formal (no cost to you) proposal, we need the following:
1. Product description and or specification.
2. Target Market – What market your product will sell into. 3. Start and end date – When you need to start and end this test.
4. Company information – Name, contact, address, phone, fax and e-mail. 5. Any concerns about your product that you have or you know about.
Let us know if you want a test proposal for your particular project. Typically, our Proposal Department can prepare one for you within five (5) working days, although more complex products may require additional time. The proposal will contain our test methodology, project schedule, defect classifications, regression information, and the project cost. This proposal should be considered a “working document” that you can change or modify to fit your exact test requirement and budget – Our goal is to work together so that you can achieve your objectives.
CONTACT US
National Technical Systems Computer Testing Division 5730 Buckingham Parkway
Culver City, CA 90230
Telephone: (310) 641-7700
Fax: (310) 641-5827
