Systems Load Testing Appendix
1 Overview
As usage of the Blackboard Academic Suite grows and its availability requirements increase, many customers look to understand the capability of its infrastructure. As part of this analysis, sound business continuity practices are required to maintain uptime commitments; however an understanding of the infrastructure’s performance thresholds provides tremendous insight into long-term system stability. Many customers face challenges in understanding the infrastructure needs their Blackboard usage requires. Blackboard Consulting’s Systems Load Testing service provides institutions with the means to maximize the performance of their Blackboard environment. This service can address all aspects of performance engineering, from specific performance issues to long-term capacity planning. This service is designed to provide a thorough exercise in performance engineering, the end-to-end process of considering
performance throughout the software development lifecycle, from requirements analysis to development and deployment. This type of engagement will help organizations who are looking to achieve higher levels of performance maturity, as shown in the Performance Process Maturity Model below.
Performance Process Maturity Model1
1.1 Objectives
The primary objective of the Systems Load Testing service is to perform the necessary activities to understand, resolve and optimize the Blackboard environment for peak performance. Through this service, customers are able to provide their users with a Blackboard Academic Suite that performs at the highest level possible. Blackboard consultants evaluate the current infrastructure, perform a series of load tests and evaluate the environment holistically. Thereafter, consultants design a plan for improving all aspects of system performance and capacity.
Blackboard’s performance investigation attempts to determine the cause of any actual or potential performance issues; and provide customers with appropriate guidance and talent for systematic resolution. The goals of this performance investigation are:
• Evaluate the current traffic and usage patterns • Define client expectations and goals
• Measure current system capacity and scalability
• Evaluate current architecture Identify and correct any potential problem areas • Ensure operational stability
• Provide short-term recommendations for tuning and optimization
• Develop long term recommendations for architectural changes to improve performance, quality of service and reliability as well as best practices
• Plan for future upgrades
1.2 Performance Audit and Tuning Approach
Through its years of experience, Blackboard Consulting has developed a performance engineering methodology focused on efficiency and quality. Its approach is broken into several, interrelated stages, designed to develop a robust solution that can be managed easily. The approach is highlighted below.
Architecture Review Traffic Analysis and Modeling Performance Measurement Analysis and Redesign Interactive Workshops
1.3 Benefits
Blackboard expects its approach to provide several critical benefits. Through the application of expert knowledge, using a proven approach, customers will gain a thorough understanding of how their Blackboard environment system performs under various load scenarios. Beyond identification,
Blackboard consultants investigate, resolve and plan for optimum performance of the environment as a whole, including the network, hardware and software design. Some of the planned improvements can be implemented rapidly and yield considerable improvement, however the long-term, detailed performance analysis will enable customers to maximize their current infrastructure capability and design for
tomorrow’s needs.
Examples of performance engineering benefits include: • Identification and resolution of performance issues
• Validation of hardware configuration’s ability to handle anticipated load • Guidance on architecture growth
• Provision of insight on future capacity planning, including quantification of current capacity and alignment of growth goals with recommendations for achieving desired capacity.
1.4 Qualifications
Blackboard Consulting maintains a global team of over 120 professional consultants. In fact, Blackboard Consulting has the ability to truly work around the clock to support your eLearning rollout if time is of the essence. As Blackboard Consulting has been implementing and evaluating client environments around the world since 1997, several key trends have emerged:
1) Blackboard customers that utilized this service saw a direct impact on the stability and scalability of their production system.
2) Institutions received the necessary documentation to address performance issues before they became a problem
3) Customers who have utilized this service run some of the highest performing Blackboard applications in the world
4) Institutions have acknowledged performance during peak load times was less than the testing process and, therefore did not introduce system outages or performance problems
The implementation plan below describes the suite of activities that the Blackboard Consulting team and client project team will participate in to complete the proposed solution. Each section of activities has specific outcomes that track towards completion of the project for production purposes while also providing
knowledge transfer for a level client self sufficiency.
3 Implementation
Plan
The following activities describe the services engagement in which your organization and the assigned Blackboard team will participate.
3.1 Project Preparation and Initiation
• Blackboard Consulting team members assigned to your organization’s engagement
o Familiarization with your project requirements, including the final Statement of Work and knowledge transfer from the Blackboard Sales team
o Setup of client centralized communication plan for tracking project activities, progress, issues, and risk
o Preparation of materials for the project kick off conference call
• Conference call kick off with the client team conducted by Blackboard Consulting o Introduction of Blackboard and your team members
o Review scope of the engagement as defined in the Statement of Work o Review the project plan at a high level
o Describe the various client participants required for each stage of the project engagement o Determine the project schedule, including scheduling of the onsite sessions
• After the kick off call, the Blackboard Consulting team prepares and delivers to the client a detailed agenda and supporting documentation for the project’s onsite activities
3.2 Project Management
• Throughout the engagement, the Blackboard Project Manager will provide expert management of all Blackboard performed activities. He/she will:
o Act as the primary point of contact between the client project manager and the Blackboard team for all project-related topics
o Define, document and update comprehensive project plans and project schedules o Track and report on progress against the project schedule
o Coordinate team meetings and provide regular updates via conference calls for both Blackboard and client constituents
o Provide regular status reports and bi-monthly budget tracking and project plan updates. o Track and report on any project issues
o Summarize and coordinate project planning across all activities and Blackboard groups – Consulting, Training, ASP, Quality Assurance, and Product Development
3.3 Load
Testing
Workshop Activities
Prior to the performance evaluation process, it is critical to design a plan that addresses the known issues as well as specific customer needs. Therefore, Blackboard Consultants invest significant energies in this phase of the project. It begins with a series of working sessions to understand the public perceptions regarding system performance. Further, Blackboard consultants meet with IT personnel to understand efforts currently underway to manage system performance and recent capital expenditures or system upgrades. Next, the system as a whole is documented, including the hardware and network design as well as any applicable third-party applications and Blackboard configurations. This information allows for the necessary baselining, prior to testing.
More specifically, consultants lead a series of workshops at the client’s facility. Those include the following:
• Various information-gathering exercises, including interviews with key stakeholders, surveys of the user community, and other activities designed to elicit local expectations and the current perception of the system performance
• Identify and document the current Blackboard architecture at all levels including, but not limited to: software components, customizations, third-party integrations, hardware, network
environment, and database configuration
• Activities structured to identify existing business practices, map them to applicable Blackboard processes, and share best practices around common system issues
• Identify the objectives of the engagement
3.4 Architecture
Review
With the baseline information assembled and an understanding of the current support infrastructure, consultants are able to review the current architecture. This process is expected to review the hardware design for capacity and long-term sustainability. Typically, customers choose among several design standards (scaling up or scaling out for example) when designing the Blackboard environment and while the preferred standard is not critical, its implementation and long-term capacity is. Next, the Blackboard application is evaluated for the customer’s configuration decisions. Each of the critical configuration and tuning parameters are documented for use in the next phase. Lastly, consultants review the network, both to understand how the Blackboard Academic Suite is laid out and to assess its capacity and potential bottlenecks.
With an understanding of the holistic environment, Blackboard consultants review usage patterns in an attempt to model typical usage. Information is gathered from system and application logs and used to determine areas of the application most commonly used and response times during varying times and under varying loads.
During the architecture review phase, Blackboard Consultants will provide the following services: • Perform a full audit of current system architecture (hardware, software, network, configuration) • Audit current and expected usage patterns to create data and behavioral models
• Scan for potential problem areas (signs of abandonment, resource saturation, faulty architecture, errors within logs, etc.)
3.5 Performance
Measurement
Using the results of the client interview and architecture reviews, Blackboard consultants define, document and perform the necessary load testing activities. This phase begins with a detailed
measurement plan. This plan outlines the specific elements of the system to be measured, the expected outcomes as well as a baseline from which comparisons can be made.
Specialized Blackboard tools will place load on the client systems to simulate the behavior models developed during the previous phase. The tests begin with a thorough evaluation of the system as a whole, understanding specific bottlenecks. These issues are typically resolved in short-order and more detailed tests begin. Detailed tests are indented to use the customer’s specific behavior models and evaluate the environment’s ability to respond to user demand. Responses are measured in terms of system capacity (i.e. limitations) and reliability (i.e. scalability) in order to determine weaknesses. More specifically, consultants will perform the following:
• Specialized Blackboard tools will place load on the client systems to simulate various levels of real-life usage conditions
• The test campaign will measure capacity, reliability, scalability, and user experience (response time) as well as identify potential bottlenecks and hotspots
3.6 Performance
Analysis
The load testing process generates a great deal of insight into capacity and system limitations.
Blackboard Consultants lead the customer team through a thorough analysis of the resulting data sets. The specific areas of focus during this analysis are:
• Network response • Hardware response • Software response • Blackboard configurations
As part of the testing process, many of the smaller issues are identified and resolved. During this phase of the project, the broader, long-term issues are discussed. Blackboard consultants lead the discussions and, based on both the data and results of the technical discussions, provide the following
recommendations:
• Short-term optimization and tuning opportunities
• Long-term architectural recommendations to achieve the performance, reliability and quality of service goals
• Guidelines for growth and capacity planning
This phase of the project concludes with a review of the recommendation and a series of discussions regarding the future of the system. These discussions focus largely on scalability options, based on the customer’s chosen strategy and a review of the long term capacity planning suggestions. Consultants then research the various options available to the customer team and model several scenarios. This information is used to understand the next steps regarding system performance and dictates the long-term strategy for maintaining optimum performance.
3.7 Project Closure
At the conclusion of the project, Blackboard will furnish final documentation to the customer in the form of a project control binder. It includes:
• Interview Summaries • Survey Results • Architecture Summary
• Results of the System Audit • Results of Load Testing
• Documentation of any Tuning/Optimization Changes made • Recommendations
• Project Final Report
The documentation will include all materials developed throughout the project as well as suggested next steps for your organization.
The schedule below reflects the necessary timelines required to fulfill the appropriate project milestones. Key to staying on schedule is making sure we have the correct client participants. Our objective is to follow the project schedule as closely as possible providing there are no significant scheduling challenges or changes in project scope.
4 Schedule
During the term of the project the Blackboard project team will guide the client team through a series of exercise focused on gaining a deep understanding of current system performance and identifying any known or suspected problem areas. The engagement begins with a series of client interviews and architecture reviews that outline the process and capture applicable business rules for a performance audit. Next, Blackboard consultants develop the necessary load testing strategy. Finally, they provide recommendations to bring the system up to the optimum level of performance.
4.1 Project
Timeline
The average project is estimated to be four to five weeks from project initiation. A rough timeline is shown above with activities before and after the onsite.
4.2 Scheduling
Requirements
Blackboard Consulting’s methodology assumes a certain level of effort from the customer team. For example, those individuals with direct responsibility for the system’s performance and maintenance must be committed during the three project phases. Blackboard has found that this immersion is necessary to prepare customers for the ongoing demands of the improved system. The following table is intended to aid you in planning for this project:
Please note: Blackboard consulting resources are available on a first come first served basis and will be
4.3 Client
Participants
Project Task Roles Required
Client Interview Executive Sponsor Project Manager Senior IT manager Business Analyst / Educational Strategist Application Administrator System Administrator Database Administrator Network Administrator Educator panel Architecture Review Application Administrator
System Administrator
Database Administrator
Network Administrator Load Testing Application Administrator
System Administrator Database Administrator Network Administrator Recommendations Project Manager Senior IT manager Application Administrator System Administrator Database Administrator Network Administrator
5 Conclusion
Blackboard Inc. and Blackboard Consulting appreciate your interest in working with our skilled team of technology experts. We believe the aforementioned proposal and estimate will you’re your needs by providing the necessary insight and talent to improve the performance and capacity of your Blackboard systems.
