Intel® Cloud Builders Reference
Implementation
Virtualized Enterprise-Class Production Platform for ERP Systems
AUDIENCE AND PURPOSE
This document describes the implementation of a multi-tenant virtualized hosting platform for SAP environments that is easy to manage and utilizes state of the art virtualization management solutions from VMware. It is intended as a proof-of-concept blueprint for IT system architects with a starting knowledge of SAP platform architecture and VMware vSphere* infrastructure setups.
Executive Summary
Capgemini is a global leader in consulting, technology,
outsourcing, and local professional services. Its customer base has traditionally implemented SAP applications for their business needs in various sizes, some requiring high amounts of availability while others demand a certain flexibility with regards to capacity. Such environments can usually be split up in development, quality assurance, training, and production installations. As an IT service provider designing and operating these environments, Capgemini is not only responsible for regular maintenance tasks such as application changes and package patching, including software and hardware updates, but also for driving cost effective continuous innovation. Growth requirements due to data expansion and additional business throughput require Capgemini to manage capacity proactively while maintaining a lowest possible total cost of ownership (TCO). To that end, this paper will describe Capgemini’s reference implementation of a multi-tenant virtualized hosting platform for SAP environments that is able to fulfill the above mentioned requirements regarding sizing, elasticity, availability, and ease of management. It can therefore serve as the basis of a multi-tenant, fully automated private cloud for mission-critical workloads.
Capgemini has designed this reference implementation in an open manner, with the ability to function on multiple virtual and non-virtual infrastructure platforms, and based on industry standards such as Intel® architecture, VMware, and SAP based technologies. It is intended to guide implementers through the early stages of similar deployments and share those best practices which have ultimately proved successful for Capgemini.
Introduction
A typical characteristic of SAP applications is a comparatively high CPU and memory consumption and fairly high I/O
bandwidth requirements. It can be concluded that the virtualized infrastructure needs:
• Easy systems management
• Cloud based deployment of environments
• Ability to re-use current utility based pricing model
• 3-tier setup of SAP environments (Presentation – Application – Database)
• Easy growing and shrinking of CPU/memory capacity • CPU bandwidth requirements for peak load
• Service level availability between 97.8 and 99.9 percent • Disaster recovery of 100 percent capacity for production in
case of major failures of infrastructure and/or data center
• Easy integration with existing traditional infrastructure for SAP environments based on physical installations on HP Superdome*
• Industry standard components
The following challenges had to be overcome for these requirements to be met.
Integration of Virtualized and Traditional Solution Architecture
The current traditional solution is based on an HP Superdome infrastructure in a dual data center setup with clustering in which single instance SAP environments are the standard and High Availability (HA) is realized through storage clustering. Moving from a 2-tier to a 3-tier setup for SAP environments in which, at minimum, the SAP application server tier is deployed on the new virtualized infrastructure gives added flexibility in moving functionality and/or user workload to guarantee performance. However, this also makes HA scenarios somewhat more complex due to the increased number of components in the stack that can principally fail.
The two HA mechanisms differ significantly in the new solution versus the traditional setup. For example, the previous clustering-based approach will be replaced by VMware’s HA features including Site Recovery Manager. This needs to happen transparently, so as to not impact end customer Service Level Agreements (SLA).
Shifting the Architectural Balance Moving from Traditional to Virtualized Deployment
Previous work on deploying SAP in virtualized environments on Intel architecture and with market-leading VMware software solutions has given Capgemini the confidence to follow this path1. Concrete experience, however, still had to be gained in terms of controlling virtualization overhead and production-ready sizing guidelines.
In addition, the change from a traditional 2-tier to a 3-tier architecture for SAP applications also means that communication between the application and database layer will need to go through network connections instead of shared memory. Still, network bandwidth and network latency have to remain in a certain window while the application and database servers need to run in the same data center.
Platform Overview
The architecture is based on a dual data center site setup, in which the capacity of both data centers is utilized (see Figure 1). Production systems are installed at site number one while the
less business critical development, test, and training systems are installed at site number two. In case of failures or disasters, a failover with 100 percent capacity of a production system to the other data center can be performed if the specific service level requires.
In this case, capacity for related development and test systems is reused for the failover of the production system. Failovers are established on the basis of the HA and Site Recovery Manager capabilities of VMware’s vSphere* product. For these systems, data replication with SAN storage is required and is implemented in this architecture as described below. While high speed network connections between the data centers are usually available for Capgemini-hosted installations, it was decided that all production applications of one dedicated customer should run in the same data center, providing advantages for application integration and lowering latencies in the network connections between the data centers.
Based on the previous principles and trade-offs, the architecture in Figure 1 was implemented. Both the dedicated infrastructure based on HP Superdomes hosting the RDBMS and the virtualized environment of the utility-based architecture share a common network and storage section.
In this architecture, both data centers contain the following same components:
• HP Blade Center, LAN pass through, containing five blades each with two 8-core Intel® Xeon® processor E7-2870 and 256 GB memory per blade
• HP Superdome with 64 Intel® Itanium® processor cores • IBM XIV storage (fiber connected), 64 TB
• SAN switches
• LAN switches (Cisco N5000 and N2000)
• VMware management server (HP ProLiant series)
Failover for disaster recovery is set up with HP MC Serviceguard for the Superdome infrastructure and VMware Site Recovery Manager for the VMware Blade Center. Failover within a data center is only available for the VMware infrastructure based on the HA option of VMware. Data center failover is only available for environments that use storage replication. The data centers are connected through a redundant high speed WAN connection. The VMware management stations use a redundant setup for data center disasters.
Due to the workload’s business-critical nature, the processor technology supports the highest levels of availability at a reasonable cost. The Intel® Xeon® processor E7 family provides the greatest amount of reliability, availability, and serviceability (RAS) features supporting the highest required service levels at commercial off-the-shelf pricing. VMware vSphere was chosen as the virtualization platform of choice due to a number of flexibility and availability requirements. This software also benefits from the capabilities of the Intel processor through integrated virtualization support (Intel® Virtualization Technology) and availability features such as Machine Check Architecture Recovery (MCA Recovery)2.
RAS Features of the Intel® Xeon® Processor E7 Family
Reliability, availability, and serviceability (RAS) features of this service infrastructure depend critically on the platforms hosting the applications, the operating systems, and the virtual machine monitors. To support this from the ground up, the Intel Xeon processor E7 family implements a collection of RAS features built around the philosophy of continuous self-monitoring and self-healing. Self-monitoring enables a system to actively and passively monitor all its key interconnects, data stores, data paths, and subsystems for errors. Self-healing features enable the server to proactively and reactively repair known errors and minimize future ones by acting automatically based on configurable thresholds.The Intel Xeon processor E7 family's RAS feature set targets three main goals to secure the foundation of this platform: it protects data, increases system availability, and minimizes planned downtime.
Protect Data
The Intel Xeon processor E7 family provides advanced support for error detection, correction, and containment across all major components and communication pathways. It does this by reducing circuit-level errors, detecting and correcting data errors, and containing uncorrectable errors, thus limiting their impact across the system. Ensuring data integrity consequently helps avoid computation and database corruption.
Increase Availability
For typical enterprise resource planning systems, the need for uninterrupted 24/7 operation is critical. As companies strive to become more responsive to their customers, they move toward real-time business processes, which make high availability more important than ever. The Intel Xeon processor E7 family supports high levels of system availability through the use of multiple levels of redundancy and Operating System (OS)-assisted
recovery from certain uncorrectable errors on the memory, processor, and server system levels.
• Memory: increasing fault-tolerance, the Intel Xeon processor E7 family includes mechanisms for memory entity sparing, mirroring, and failover. Sparing and mirroring are two RAS features that allow on-the-fly failover from a failing component to another component. Sparing allows failover to a physical spare in the same memory controller, and mirroring preserves data in the case of DRAM component failure. Soft errors in the internal communication fabric can be handled by retries while hard errors are handled by having extra “failover” lanes in the data paths, which are activated upon the detection of such errors. For a good overview of the probability of such errors, see reference three at the end of the document.
• Processor/ Socket: at the CPU and socket level, the Intel Xeon processor E7 family provides internal on-die error protection to protect processor registers from transient faults, and enables dynamic case of a failing processor. This set of features also handles errors at the Intel® QuickPath Interconnect (Intel® QPI) and the PCI Express* I/O channels.
• Server: at the full server level, the Intel Xeon processor E7 family supports interactions with the operating system, VMM, and application software running on the server to enable recovery from errors that cannot be corrected by hardware. This technology is called Machine Check Architecture for Recovery (MCA-R) and is also described in detail in4 reference four at the end of the document.
Minimize Planned Downtime
As server density increases in the data center, it becomes increasingly important to reduce operating costs, particularly in the areas of routine service and maintenance. The Intel Xeon processor E7 family provides enhanced serviceability through support of predictive failure analysis and electronically isolated hardware partitioning. By reporting hardware-corrected errors up the software stack to the OS and management layers, the Intel Xeon processor E7 family allows analysis of error patterns that can be used to predict that a component will fail before the actual failure occurs and thus enables preventative maintenance during planned downtime.
These extensive RAS features providing a leading amount of foundational robustness and resilience have motivated Capgemini to build the SAP virtualization concept on top of Intel Xeon processor E7 family. For an in-depth technical description and a lot more details, please seereference four at the end of the document4.
Test-bed Logical System Architecture and Sizing
In accordance with the two deployment modes described above, two different types of SAP environments are going to be hosted on the platform:• System Type 1, fully virtualized: SAP ABAP only, SAP Java* only and SAP Java+ABAP systems, all with their database (Oracle) in a 2-tier Central Instance setup
• System Type 2, RDBMS non-virtualized, SAP Distributed Instance virtualized: SAP ABAP only application servers in a 3-tier setup (in combination with Central Instance and database on Superdome)
The infrastructure will serve a multi-client environment with different VLANs.
Figure 2 shows the possible Virtual Machines (VMs) with SAP environments and their type of functionality (HR, ECC, SRM, PI, EP=Enterprise Portal, BI Java, BI, etc.), whether it serves PRD (production) or DEV/QAS (Development, Quality Assurance) and their assigned capacity in SAPS (SAP Application Performance Specification) and Memory.
The memory and CPU throughput of a host for an SAP environment is based on the SAP-suggested sizing procedure with the SAP Quicksizer tool5. Capgemini’s experience in customer-relevant workloads leads to the use of a best practice of a constant ratio of 8 GB per 1000 SAPS.
The physical server chosen by us – HP’s BL620c G7 blade based on the Intel Xeon processor E7-2870 (2.4 GHz, 10 cores/20 threads) – is represented in SAP’s public benchmark database with a 2-tier SD benchmark result of 34,020 SAPS for SAP enhancement package 4 for SAP ERP 6.0 on SUSE Linux* Enterprise Server 11 SP1 in a physical installation6. This allows us to develop a consistent “rule of thumb” sizing between the physical and virtual installations. Based on this benchmark and an assumed amount of 10 percent virtualization overhead accounted for, a throughput of roughly 30,000 assignable SAPS per virtualization host was employed.
As vSphere is not able to directly handle this metric, vCenter’s reporting based on the artificial metric of “GHz” throughput will have to be used and converted into the SAPS metric from above. A maximum of 40 virtual CPUs (vCPU) are assigned in vSphere to each virtualization host whose two CPUs can handle up to 40 instruction sets simultaneously in hardware (two CPUs with 10 cores each, supporting two threads per core). As each of these vCPUs is based on an exact commit of processor resources (no
overcommitment has occurred), each of them is calculated to have an application throughput of 750 SAPS. Putting this into a linear relationship for the vCenter-reported apparent vCPU clock frequency allows us to size virtual machines according to business needs. See below for that calculation.
VMware Setup
As depicted in Figure 1, two rack mounted servers host the redundant vCenter application, one in data center 1 and the other in data center 2. Both are capable of managing both resource pools of five blades each in both of the data centers while keeping the configuration continuously synchronized. For those SAP applications with mid-range and premium service levels, replicated storage is assigned.
Dynamic Resource Scheduling and High Availability options are set to “on”, host monitoring is enabled. Currently no data center local fail-over capacity gets reserved – this enables spreading of the VMs of the resource pools over as many hosts as possible.
All physical hosts are connected to the User LAN, Backup LAN, and Server LAN and to the relevant Storage LUNs, giving all VMs the same connectivity. VLAN assignment to the virtual NIC of each of the VMs then permits the multi-tenancy required for the envisioned service. Figure 4 gives an example.
Virtual Machines that have a Site Recovery scenario are defined on both resource pools (per data center). Only one side is active at each time, but the inactive VM can be started rapidly in case a data center fails (“cold standby”). Each SAP application landscape consists of a Development, Quality Assurance, and Production environment with different resiliency requirements. When the Production landscape is running in one data center, Development and Quality Assurance are assigned to the other. In case of a fail-over of the Production environment, scripting is in place to stop the Development and Quality Assurance environments so that computing and memory resources of their VMs can be reused for a failed-over production environment.
From a capacity assignment point of view, this means that each of the VMs for Development and Quality Assurance are permitted to only consume a maximum of 50 percent of the size of the Production environment so that their combination would yield the exact amount of capacity required for a Production environment failover. However, this scenario will only happen when all resources of a data center can no longer be accessed from a user perspective (e.g. data center power down, enclosure failure, network failure in the data center, etc.). Detection and resolution of such events and incidents are performed with the appropriate data center monitoring tools and processes and will kick-off an automatic failover process. Currently, an automatic failback is not supported with the current vSphere version 4.x integration and hence needs to be prepared and executed manually. A future migration to vSphere 5 will allow us to automate a failback scenario as well.
Storage Setup
Each SAP system Type 1 and 2 will receive a differentiated storage setup. A Type 1 setup, with three fully virtualized instances of the SAP stack, has the LUN assignment shown in Figure 5.
Here, all virtual machines that will host Type 1 SAP installations are assigned two LUNs in order to separate regular file systems from performance-critical database accesses. (One for the SuSE Linux OS and 2 GB of swap space and all SAP and Oracle files systems and one LUN for the Oracle Database data file systems). The first LUN gets a default size based on the available VM template for this type and the second LUN gets the size that is determined on the basis of an SAP database sizing.
Figure 5. Type 1 fully virtualized LUN setup
The first LUN is further subdivided into two VMDKs (VMware’s storage format for virtual machines and their file systems). The first VMDK is used for OS and 2 GB of swap space while the second VMDK contains all SAP and Oracle file systems plus the rest of the swap space. This swap space is always 2.5 times the size of the internal memory of the VM (based on best practice SAP requirements) minus 2 GB (already accounted for in VMDK 1). The second LUN results in a new VMDK which is used for the Oracle SAP data file systems that contain the database data files.
The Type 2 setup on the other hand (with Distributed Instance virtualized, RDBMS on Superdome) is depicted in Figure 6. In this case, all virtual machines that will host the Type 2 SAP environments get only a single LUN that contains the SuSE Linux OS, swap space, and all SAP file systems. The LUN gets a default size based on the available template for this SAP system type. The LUN is further subdivided into two VMDKs. The first VMDK is used for OS and 2 GB of swap space while the second VMDK contains all SAP file systems and the rest of the swap space. This
Figure 6. Type 2 LUN setup
swap space is always 2.5 times the size of the internal memory of the VM (based on best practice SAP requirements) minus 2 GB (already in VMDK 1 for OS Swap requirements).
Since this type of SAP system does not have a database installed on the VM, there is no requirement for a second LUN.
Figure 7. Physical and virtual switches Networking Setup
This reference implementation utilizes a mixture of physical and virtual switches that can be consistently managed in a unified manner (see Figure 7).
As every VM gets assigned its own Nexus 1000V virtual switch, communication to the blade enclosure external Nexus 5000 happens through pass through modules. A dedicated connection to the HP Superdome environment gets implemented through
a separate Nexus 2000 switch to minimize latency as much as possible. This requirement is especially important for the Type 2 setup with 3-tier SAP environments due to its heavy networking traffic between application and database layer but can also benefit application to application interaction when the first application is running on a VM and the other on the HP Superdome.
Capacity Management
The capacity model is based on a linear relationship between this throughput metric and the SAP Application Performance Specification (SAPS) metric. With each of the five virtualization hosts accounting for 48 VMware-GHz and previously calculated 30,000 SAPS associated to each of the same hosts yields a ratio of 1200 VMware-GHz for 750 SAPS or 400 VMware-MHz for a minimum increment of 250 SAPS results.
Based on SAP Best Practices, a default step of 2 GB of memory per 250 SAPS gets used while disallowing memory overcommitment. As overcommitting of CPU resources is tolerated, a ratio of 10 percent additional CPU throughput for Production environments is assumed. For any 5000 SAPS VM this means, for instance, that a VM with a minimum capacity of 5000 and maximum capacity of 5500 SAPS for 40 GB of memory can be created. Dynamic changes in capacity can be implemented immediately if average and peak performance measures suggest, with billing automatically adjusted to any capacity change. The usage of the available capacity is balanced between the two resource pools as much as possible, but other criteria may take precedence such as proximity requirements between applications due to high communication needs. If extra capacity is required for a short period of time this is established solely with VMware features to increase the CPU capacity only, since this can be switched on and off online.
Capacity planning of the total resource pool is currently a manual process with ordering of extra physical hardware done on the basis of monitoring agreed-on utilization limits.
Provisioning New Tenants
Practically applying this virtualized SAP reference
implementation in a private cloud context requires the creation of organizational processes to set up tenants on this platform: End user administrators shall be empowered to create, destroy, and resize the compute environment for their businesses autonomously and independently of other tenants on the same platform.
While in its most advanced private cloud form, such processes are going to be implemented through a portal and orchestrated via appropriate automation, they are fundamentally equivalent to manual processes orchestrated through responsible human handlers as well. Motivated by the need to keep implementation efforts below certain limits, a forms-based procedure represents the current way of self service but is planned to be replaced in the near future.
The following set of information currently represents the necessary and sufficient data being collected in order to trigger the SAP virtual application deployment for a certain required sizing.
SAP System Templates
Requesting a VM for an SAP system is done by submitting a request form that needs to contain the following types of information to setup the VM, the operating system, its kernel parameters, bootstrap user IDs, and file systems:
Standard information: • Customer Name
• System ID (3 character Oracle SID) Service-level related information:
• Required number of SAPS
• Regime (standalone, bronze, silver, or gold) Standard technical provisioning-level information:
• Type of SAP system:
-ABAP Central instance with database -Java Central instance with database
-ABAP+Java Central Instance with database -ABAP Dialog instance
• Dialog Instance Sequence Number (only allowed in case type of SAP system is ABAP Dialog Instance)
• Data Center
• Networking information for User LAN, Server LAN, Management LAN, and Backup LAN
Non-standard technical provisioning-level information: • Non Default user IDs
• Non Default file systems • Non Default NFS mounts • Non Default Descriptors • Non Default Kernel Parameters • Non Default Firewall Settings
With this amount of information submitted, all necessary
settings on the platform can be processed and the new tenant organization can be set up by any standard process from standardized virtual machine templates for each of the various Central and Dialog Instance versions including the database if required.
Summary and Outlook
Virtualization of SAP environments with VMware and the latest Intel technologies has brought a lower TCO through industry standard infrastructure components, a better use of available capacity, and a more automated set of procedures for provisioning, monitoring, and capacity management. This paper was intended to show the possibilities on an architectural and setup level for any IT professional interested in the benefits and possibilities of utility-based SAP computing.
For Capgemini, this implementation represents a first step in bringing more flexibility, lower TCO, and lower risk in the management of the SAP environments to its customers. In the future, Capgemini plans to enhance this architecture on a number of fronts: first and foremost, a continuous refresh to the latest software versions will see us adopt vSphere 5, allowing us to benefit from MCA-R support and upleveling on failover and failback automation. Continuous hardware refresh to future Intel Xeon processor based platforms will additionally enhance total throughput and the total operational efficiency of the environment. Furthermore, with virtualization well established, customer self-service portals integrated with the appropriate provisioning and setup automation will enable a further cloud-oriented use of this open architecture, very much in line with Capgemini’s support for the goals of the Open Data Center Alliance7.
To learn more about deployment of cloud solutions, visit www. intel.com/cloudbuilders
References
1. SAP-Intel Co-Innovation Lab. SAP and VMware: Virtualizing Resource-Intensive Applications. http:// www.intel.com/content/www/uk/en/virtualization/ virtualization-xeon-sap-and-vmware-virtualizing-resource-intensive-applications-paper.html?wapkw=sap
2. Intel. Transform Mission-Critical Computing. http:// www.intel.com/content/www/uk/en/processors/xeon/ xeon-e7-transforming-mission-critical-computing-brief. html?wapkw=mca-r
3. DRAM errors in the wild: a large-scale field study. Schroeder, Bianca, Pinheiro, Eduardo und Weber, Wolf-Dietrich. New York, NY, USA : ACM, 2009. Proceedings of the eleventh in-ternational joint conference on Measurement and modeling of computer systems (SIGMETRICS '09).
4. Intel. Intel® Xeon® Processor E7 Family for New RAS Servers: White Paper. http://www.intel.com/content/www/ uk/en/processors/xeon/xeon-e7-family-ras-server-paper. html?wapkw=xeon+ras
5. SAP. SAP Quicksizer. https://service.sap.com/quicksizer
6. SAP. SAP Standard Application Benchmarks. http://www.sap. com/solutions/benchmark/index.epx
7. Open Data Center Alliance. http://www opendatacenter alliance.org
Disclaimers
∆ Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See www.intel.com/ products/processor_number for details.
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROP-ERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined.” Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized er-rata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or by visiting Intel’s Web site at www.intel.com. Copyright © 2012 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon inside, and Intel Intelligent Power Node Manager are trademarks of Intel
Corporation in the U.S. and other countries.
