Migrating from UNIX SMP Servers to Linux Clusters
The Cost-Effective Alternative – PolyServe Clusters of Intel Architecture-based
Servers
An Intel and PolyServe White Paper
The increasingly compelling price/performance and absolute performance advantage of Intel Architecture-based servers, coupled with the huge upswing in the popularity of Linux, is leading major data centers to migrate mission-critical applications from traditional large UNIX systems to Intel Architecture-based servers running Linux. Linux is very similar to UNIX – very little retraining of staff is involved – and almost all key UNIX software applications are now available in compatible Linux versions.
The movement to migrate from UNIX to Linux is accelerated by the availability of
technologies like Oracle9i Real Application Clusters (RAC) and PolyServe Matrix Server, which allow workloads to be scaled out across multiple cost-effective Intel Architecture-based servers. Workloads that in the UNIX world required large SMP servers can now be run on groups (clusters) of inexpensive, but very high performance Intel Architecture-based systems.
Figure 1. UNIX SMP Server to Linux Cluster Migration
The key driver, however, behind interest in UNIX-to-Linux migrations is cost. Data centers are primarily interested in Linux to help drive down the overall cost of hosting and supporting
applications. The fact that Intel Architecture-based servers are much less costly than UNIX systems is interesting, but is not by itself completely compelling. Initial hardware purchases make up only a small fraction of overall IT spending, which, of course, also includes other fixed asset capital costs, maintenance and support costs, downtime costs, administration costs, consulting costs, as well as other human capital costs.
This paper describes proven offerings available today that allow Linux servers to deliver a better than 50% cost savings versus traditional UNIX architectures, while delivering the same mission-critical reliability and availability of the most robust platforms. Any consideration of UNIX-to-Linux migration must identify the associated requirements and challenges, their impact on the overall total cost of ownership (TCO) model for such a migration, and what hardware and software elements are needed to provide a complete solution for deploying mission-critical applications on Linux in the enterprise data center.
These are the questions that typically need to be addressed:
• Costs (Up-Front and TCO) – If I move to a cluster of Intel Architecture-based servers running Linux and PolyServe Matrix Server, isn't that going to cost more initially, and a lot more over time, than my current UNIX platform environment?
• Performance – Can clusters of Intel Architecture-based servers running Linux and PolyServe Matrix Server provide as much performance as the large UNIX systems we run today?
• Availability – Can clusters of Intel Architecture-based servers running Linux and PolyServe Matrix Server provide at least as much system uptime (.9999) as my current UNIX systems?
• Robustness – Are clusters of Intel Architecture-based servers running Linux and PolyServe Matrix Server really a robust enough environment to run my business? This paper addresses these questions and discusses how the combination of PolyServe Matrix Server and Intel Architecture-based servers running Linux deliver a fault tolerant, scalable, enterprise-caliber platform for running mission-critical data center applications.
A Fault Tolerant, Scalable Linux Architecture
By adopting Linux, the typical enterprise data center will leverage many of the IT
infrastructure technologies, capabilities and components already present today in the UNIX data center. Specifically, enterprise Linux deployments will utilize switched Fibre Channel networks, shared storage, and rack-mount servers to build a completely modular and fault-tolerant infrastructure for mission-critical applications. For example, a typical Linux data center deployment would include several clusters of Intel Architecture-based servers
connected via Fibre Channel to a Fast-T storage area network (SAN). Each cluster might be responsible for running a mission-critical application, such as an ERP application, an ecommerce site, or a business analytics data warehouse.
Figure 2. Topology of an Enterprise Linux Deployment
There are several critical benefits to this data center topology. First and foremost, this architecture lends itself to independent scaling of servers and storage. New servers can be quickly added to a cluster if the processing needs exceed the current capacity. Moreover, with the presence of advanced shared data clustering software, such as PolyServe Matrix Server, these additions or deletions of server resources can be completed quickly on demand and online with no interruption of application services to the existing cluster members. In addition, Matrix Server also provides the ability to provision additional storage capacity on demand and online with no interruption of service.
Second, the hardware and software topology is completely fault tolerant and redundant at the application, middleware, server, network and SAN levels. That is to say, a component failure at the application, middleware, server, Ethernet network, SAN network or storage level will not result in a service outage. Each server has multiple I/O paths to shared storage. Each server has multiple connections to the Ethernet networks. PolyServe Matrix Server provides the clustering software needed to take advantage of this redundant topology. PolyServe Matrix Server migrates application and middleware services on server failures, reroutes Ethernet traffic if a NIC or network subnet fails, and allows applications to transparently take advantage of the multiple physical paths to shared storage.
Availability in Linux Clusters
Building Reliable Architectures from Industry-Standard Building Blocks
There are three critical dimensions of availability: server availability, application availability and data availability. Server availability refers to how redundant and fault tolerant the overall server capability is to failure of an individual server component and whether individual server components can be added or removed without impacting the applications running on the server. Application availability refers to whether the application service is functioning and accessible to clients and can perform the full set of requested operations. Data availability refers to whether the actual business data (e.g., bits and bytes) on the storage subsystem is accessible to servers and applications/middleware and is effectively protected from failure (e.g., via data backups, snapshots, disk mirrors).
Redundant, hot-swappable hardware offerings and system management capabilities address the first dimension of server availability. PolyServe Matrix Server addresses the later two dimensions of availability – application and data availability. Matrix Server proactively monitors the health of all levels of the system stack: the applications, the middleware, the server hardware, the network connectivity and the storage connectivity. If there is a failure at any of these levels, Matrix Server initiates a failover procedure to restore application service. Matrix Server provides flexible failover and application management policies that allow administrators to move applications to spare (unused) servers or to another already-utilized server. Matrix Server monitors the wellness of applications and their underlying resources and initiates prescriptive actions that enable data center operators to achieve the same level of application availability that is currently delivered on SMP UNIX architectures.
Matrix Server supports the ability to build completely fault-tolerant SAN fabrics with no single point of failure. This capability is crucial for ensuring that clusters of Intel Architecture-based servers always have access to mission-critical data on the SAN. PolyServe Matrix Server provides Multi-Path I/O (MPIO) functionality for applications and middleware. Therefore, a server with two or more HBAs can be connected to multiple Fibre Channel switches. PolyServe’s MPIO functionality ensures that if there is a failure anywhere in the SAN fabric, the data path will be transparently rerouted through an alternative data path to the shared storage. Thus, an application can maintain the same, if not better, level of data availability as with high-end SMP UNIX deployments.
Matrix Server also provides networking fault tolerance with no single point of failure by supporting transparent NIC and network subnet failover on each node in the cluster. In addition, enterprise-class storage subsystems provide advanced volume management and data protection benefits that strengthen data availability for Linux-based servers.
Horizontal Scaling in Linux Clusters
Bringing Big System Performance to Clusters of Small Systems
Scaling applications and middleware across nodes is commonly referred to as horizontal scaling. Horizontal scaling differs from the traditional SMP scaling, or vertical scaling, in many ways. Although the core of a horizontally scaled application is usually a small multiprocessor
system, adding processors is done by economically adding multiprocessor nodes to the cluster as opposed to adding processors to the system.
Clustered databases, such as IBM DB2 v8 and Oracle9i RAC along with PolyServe’s Matrix Server software, allow data center operators to implement horizontal scalability. In fact, data center operators can now deploy Intel Architecture-based clusters that are capable of handling the OLTP and data warehousing workloads that have traditionally only been handled by large UNIX SMP machines, each costing hundreds of thousands of dollars. The clustering software advances in PolyServe Matrix Server have allowed Linux to move beyond edge deployments (e.g. web serving) to deployment of core mission-critical DBMS-based applications in the enterprise data center.
To illustrate the tremendous horizontal scaling benefits of these solution stacks, PolyServe developed the largest known Linux database implementation – nicknamed the “Tens.” As part of the Tens project, which was audited by IDC, PolyServe constructed a 10-node, 10 TB database Oracle9i RAC cluster and simulated a demanding transaction-centric workload of 10,000 concurrent users. The Tens project demonstrated the viability of the Linux ecosystem as a platform for hosting large data center workloads running against a VLDB. For more information about the Tens project, refer to the Tens white paper at:
http://www.polyserve.com/pdf/tens_whitepaper.pdf
The choice for users between dual-processor (DP) and multi-processor (MP)-based systems is primarily a function of workload characteristics. For general guidance, if the system reaches its peak workload with less than 60% utilization using a single processor, an Intel® Xeon™ processor DP platform is likely to be sufficient, since it has at least 90% additional performance with an extra processor. If on the other hand, a second processor is required to achieve this performance from the outset, the user is at risk of inadequate headroom. In this case, an Intel Xeon processor MP- based system would likely be the most appropriate choice.
Typical Intel Xeon processor MP-based servers available today are four- to 32-way systems, with processors running at 2.8 GHz, along with 2MB of third-level cache. Performance estimates of these systems are good or better than equivalent four-processor systems based on PowerPC or SPARC processors. Research also indicates that the jump from DP and MP offers good scalability from two to four processors. A four-processor MP system will perform 70-90% more than a two-processor system on the same workload.
Cost of Operations in Linux Clusters
Managing Many Servers at the Same Operational Cost as a Single Server
The common rule of thumb in information technology (IT) has always been that more servers equates to more management and operational expense. Moreover, technologies like
clustering have been perceived as complex and difficult to configure and maintain, thus resulting in additional costs because of the higher chance of operator error and the need to hire highly skilled IT administrators to manage these systems. However, these rules of thumb and observations are based on the old traditional UNIX high availability (HA) software
offerings. Many of these offering, such as SteelEye Lifekeeper and Veritas Cluster Server, exist on Linux, too, but cannot deliver the benefits of a solution like PolyServe Matrix Server that are needed to support high node count production deployments of mission-critical applications, such as Oracle9i RAC.
PolyServe Matrix Server significantly reduces the operational cost of managing a cluster of servers because it provides a “single system image” (SSI) across the cluster and provides the administrator with tools to manage the servers, SAN fabric and storage as a single coherent entity. SSI is absolutely pivotal in reducing the operational expense of cluster computing. SSI means that all servers share the same binaries, data and configuration information. Most of an administrator’s time is spent ensuring that these pieces of data are synchronized across the cluster (e.g. installing applications and OSes across all nodes in the cluster, applying patches to applications and the OS across the cluster, changing configuration settings across all nodes in the cluster, backing up data for each of the disparate systems). Without SSI, an administrator must perform all these operations on each individual machine (e.g., Oracle9i RAC has over 100,000 files that would need to be installed and updated on each node in the cluster). Therefore, as the size of the cluster increases, the complexity, repetition and likelihood of an operator error greatly increase.
Moreover, a SSI allows an administrator to patch, update and install once without the need to apply the same operation to the entire cluster. All nodes automatically gain access to the update and are synchronized. To quote Bill Ubelacker, director of system services at the Burlington Coat Factory, “PolyServe makes the whole cluster function as one coherent whole.”
PolyServe Matrix Server enables high node count cluster deployment and greatly reduces the configuration and ongoing management of cluster computing. Unlike traditional HA clustering software, PolyServe Matrix Server provides unique benefits in the following areas:
• Managing the Group – Matrix Server provides a GUI management console and command line interface that allows an administrator to operate and manage the cluster of servers, the application services and the SAN fabric as a single entity.
• Shared Application Homes – Matrix Server allows applications such as IBM DB2 or Oracle9i RAC to store their binaries and configuration information in a shared file system. Therefore, administrators need only install, update, patch or change the configuration in a single place, not on each node in the cluster.
• Unified View of Storage Resources – Matrix Server provides a single unified view of all file systems, I/O paths and LUNs in use in the cluster.
• Backing Up Storage, Not Servers – Because the cluster’s data is stored in the shared file system, administrators no longer need to back up individual servers, they can back up the data stored in the shared storage system, greatly simplifying the backup task and the time needed to complete it.
• Provisioning Applications – Applications are now installed into a shared file system. With PolyServe, a server can be easily changed or provisioned by simply mounting a new file system and starting the new application service. No longer does an administrator have to physically install a new application on the server before it can be run. With PolyServe, nodes can be easily added or removed from a cluster and new file systems mounted to allow them to host new applications without a lengthy installation process.
Robustness of Linux and the Systems Software Stack
The ultimate indicator of robustness is that large Fortune 500 customers choose to deploy their critical business systems on the technology. UNIX reached this point many years ago and has become the de facto standard for mission-critical applications. Due to rapid
advances in the open source community, Linux has become a viable option as a platform for mission-critical computing needs.
A prime example of the robustness and confidence in the Linux platform is demonstrated by the actions of Burlington Coat Factory (BCF), a large Fortune 500 retailer. BCF needed to migrate 40 Oracle databases off their legacy IBM/Sequent NUMA-Q UNIX servers. BCF evaluated using Sun UNIX servers versus a cluster of Linux servers with PolyServe Matrix Server software. After a thorough evaluation, BCF chose the combined Linux and PolyServe solution because of the substantially lower TCO with better reliability and manageability. See the BCF case study for more details at: http://www.polyserve.com/pdf/BCF_case_study.pdf Mike Prince, CIO of BCF, was impressed by the overall robustness and level of support delivered by PolyServe on the Linux platform offering: “The PolyServe team has proven its ability to deliver enterprise-class system software products as well as the support that’s required for mission-critical software in our data center.”
The Financial Model
A comparison of Linux and UNIX is not complete without an analysis of the cost saving that can be realized by deploying Linux. An effective measure of the true cost of deploying infrastructure not only accounts for the upfront capital expenses that are incurred, but also accounts for the operational costs along with the cost of system availability and the
opportunity cost of the capital spent on infrastructure over a three-year period. By calculating these costs, a CIO can determine the TCO of an Intel Architecture- and Linux-based
infrastructure in comparison to a SMP UNIX-based infrastructure.
There are six key factors that greatly influence the TCO of a Linux cluster in comparison to a large UNIX SMP system:
• Cost of Platforms – Due to the volume economics of Intel Architecture-based servers, these servers offer dramatic price/performance savings in comparison to large proprietary SMP UNIX systems.Intel Architecture-based servers – especially in two-processor blades or 1U rack-mount configurations – offer drastically lower costs per unit of processor power than any traditional UNIX system. For example, a well-equipped Linux cluster composed of eight two-way 2.4GHz Intel Xeon processor MP servers lists for around $60,000, while a 16-way Sun Sunfire 6800 lists for about $650,000.
• Cost of Availability – UNIX systems have a cost of availability of roughly 100% the base system cost required to support a given application. In order to achieve .9999 of system uptime, a UNIX system must be clustered with a passive failover server, resulting in the purchase of another expensive UNIX machine. Linux clusters have a cost of availability roughly equal to 15% of the overall cluster cost required to support
a given application. In a Linux cluster, nodes will fail independently of the whole. Using PolyServe Matrix Server, an eight-node Linux cluster could have one or two passive backup servers, resulting in the purchase of one or two additional small cost-effective Intel Architecture-based servers.
• Cost of Growth (Meeting New Demand) – Linux clusters cost less to grow, since nodes can be added at any time, even to a running application. Linux clusters can be horizontally scaled using PolyServe Matrix Server software. This means that a CIO only has to make capital expenditures when the demand on the system dictates. In UNIX environments, a CIO must provision their UNIX servers to the peak load forecasted over the useful life of the system. In short, a CIO must purchase a much larger UNIX machine that will go largely unutilized until the system load increases.
• Cost of Capital (Paying for Demand Just in Time) – Because nodes can be added at any time, it is not necessary to overprovision servers at the time of first
deployment. That stranded capacity is a cost of capital expense in two ways: capital is spent before it is needed, and capital is spent inefficiently. By buying capacity later, when it is needed, a data center not only wins in the time value of money, but also by getting more (faster clocks, newer versions) for the same amount of capital.
• Cost of Maintenance – Maintenance rates for UNIX systems are substantially more expensive than the maintenance rates for Intel Architecture-based servers. UNIX maintenance rates tend to be a percentage of the cost of the hardware on an annual basis. Therefore, the fact that a CIO purchases more expensive machines not only affects up-front costs, but also translates into larger recurring costs over the life of the system. Intel Architecture-based server maintenance rates tend to be fixed cost and are much lower than UNIX system maintenance rates.
• Utilization Rate Benefits – It is commonplace to find that most UNIX machines have utilization rates between 15-20%. Low utilization represents a great waste of capacity and money. Historically, a single application has been dedicated to a single UNIX machine. However, this is not true in Linux clusters using PolyServe Matrix Server software. In a Linux cluster where nodes can be easily moved from one application cluster to another, resources can be more efficiently and effectively utilized across applications. Thus, a Linux cluster can have a much higher utilization rate than the typical 15% for UNIX systems. PolyServe Matrix Server allows CIOs to reap greater flexibility in how assets are deployed in the enterprise data center and also improves the asset utilization of data center resources.
Financial Model Summary: Three-Year TCO
CIOs can save between 50-60% of total project costs by deploying a data center architecture that leverages PolyServe software and Intel Architecture-based servers. The graph below compares the three-year TCO of a combined Linux and PolyServe Matrix Server solution against the TCO of a Sun UNIX solution. The financial model compares the costs of a cluster of 10 two-way 2.4GHz Intel Xeon processor servers to the costs of an active-passive cluster of Sun 16-way Sunfire 6800 servers.
$0.20 $3.61 $0.06 $0.39 $1.60 $4.67 $1.53 $0.23 $1.22 $-$1.0 $2.0 $3.0 $4.0 $5.0 $6.0 $7.0 $8.0 $9.0 $10.0 Co s t i n $ M Linux Cluster SunFire 6800
Cost of Capital
Downtime Costs
Consulting & Support
Human Capital Costs
Capital Expenditures
Figure 3. Three-Year TCO Comparing Sun Fire 6800 to PolyServe-Linux Cluster
The Path Forward
This white paper has described an approach and the benefits of using clusters of Intel
Architecture-based Linux servers with PolyServe Matrix Server as a new architecture that can significantly drive down the costs of IT infrastructure, both in terms of capital expense and more importantly, in terms of operational expense.
For many enterprise data centers, the path forward to an important new IT infrastructure solution like Linux and PolyServe Matrix Server will require a proof of concept and a set of tests designed to prove the technical and business assumptions that make that new architecture attractive. Because the move from UNIX to Linux is motivated largely by economics, it is essential to define a project plan that encompasses not only the key technology requirements, such as performance and availability, but also the key business drivers, such as cost of operations, cost of availability and cost of growth.
PolyServe has found the following elements to be crucial in such a proof of concept:
• Scaling: Test scalability from two to 10 servers (e.g., of Oracle9i RAC)
• Availability: Test failover of a single node and multiple nodes in a Linux cluster; test network failures; test SAN path failures
• Operational Costs: Build and test shared application homes; build and test shared SAN backup
• Online Growth: Test the addition/removal of nodes to/from a running cluster
• Application Provisioning: Test the ability to move a server from one online application to another
The economics and performance of Intel Architecture-based clusters for mission-critical workloads are compelling. These clusters offer customers an opportunity to improve the overall service availability, asset utilization and asset flexibility in their data center at a price
PAGE 10 of 10 (877)-476-5973 www.polyserve.com point that cannot be matched by UNIX servers. PolyServe and Linux offer an extremely cost-effective scalable, available Linux platform solution for CIOs who want to reduce costs and gain additional operational flexibility in their data center.
###
PolyServe, Inc. (877) 476-5973 (within U.S.)
20400 NW Amberwood Drive (503) 617-7592 (fax)
Suite 150 (503) 617-7574 (outside U.S.)
Beaverton, OR 97006 www.polyserve.com
Copyright 2004 by PolyServe, Inc. All rights reserved. PolyServe and the PolyServe logo are trademarks of PolyServe, Inc. All other trademarks mentioned in this document are the property of their respective owners. This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any software, software feature, or service offered or to be offered by PolyServe, Inc. PolyServe, Inc., reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact PolyServe corporate headquarters for information on feature and product availability. The PolyServe Matrix Server product uses software developed by Spread Concepts LLC for use in the Spread toolkit. For more information about Spread, see http://www.spread.org. PS-Intel_unix_migration_wp_v4
