Designing WANs for Today
that Position You for
Over the last few years enterprise IT organizations have steadily increased their reliance on the Internet as an adjunct to the enterprise WAN1 for applications such as supporting both remote office workers and a rapidly growing mobile workforce. As cloud computing continues to gain momentum and enterprise IT organizations evolve their cloud computing strategies, the volume of WAN traffic carried by the Internet will grow significantly. The increased importance of the Internet as an alternative or an adjunct to the enterprise WAN is being driven by a number of specific factors. Those factors include the fact that:
§ Whereas components of IT, such as CPUs, memory and LAN bandwidth, follow Moore’s Law, traditional WAN services don’t. The unit cost for a traditional WAN service such as MPLS (Multi Protocol Label Switching) has not dropped appreciably over time and these unit costs are now one or two orders of magnitude higher than the unit cost of the Internet using access technologies such as DSL and cable.
§ Cloud services, including Software-as-a-Service (SaaS) and Infrastructure-as-a-Service (IaaS) solutions, are increasingly being embraced by IT organizations as an agile, cost-effective means of expanding the range of IT services that they provide. Most of these cloud services are accessible only over the Internet.
§ One of the next major steps in the evolution of cloud computing is for IT organizations to construct private cloud computing data centers. Many enterprises will choose to locate their private cloud data centers either at traditional co-location facilities or within the data centers of cloud service providers. They will be motivated to do this in order to take advantage of the inexpensive, high-bandwidth Internet connectivity and management out-tasking available at these sites.
§ In the near future, IT organizations will start to make significant use of hybrid cloud
computing solutions. In many cases, these solutions will be based on distributing IT services over a combination of both private and public cloud data centers. Most of these public cloud data centers will be accessible only over the Internet.
The goal of this white paper is to describe how a new technology enables IT organizations to re-think how they design their WANs. This new design is one in which traditional WAN services, Internet access services from multiple ISPs and the Internet core are unified to provide a seamless WAN service that features both the high reliability that is often associated with traditional WAN services and the scalability and low cost that is associated with the Internet. To help achieve that goal, interviews were conducted with three IT professionals. One of the interviewees was Soheila Soheil. Soheil’s title is VP, Partner Programs Bay Area Internet Solutions (BAIS, Inc.). The other two interviewees work for enterprise organizations and they
1 The phrase “enterprise WAN” and “traditional WAN” will be used interchangeable to described WANs based on
can’t be mentioned by name or company in this white paper. One of those two interviewees is the director of voice and data at a company in the real estate business. The other is the IT technology manager for a U.S. Government organization. These two interviewees will be referred to in this white paper as The Real Estate Director and The Government Technology Manager, respectively.
Out-Tasking in the Era of Cloud Computing
The emergence of cloud computing has accelerated the trend for IT organizations to use third parties to provide traditional IT functions. In the current environment, one of the most common forms of out-tasking is hosting enterprise applications and web servers at collocation facilities. Many of the providers of managed hosting services either already have become, or are in the process of becoming, Cloud Computing Service Providers (CCSPs), whereby they offer a variety of public cloud computing services. These services can be divided into a number of categories, including:
One approach to providing SaaS solutions is based on the solution being delivered to the customer directly from an Independent Software Vendor’s (ISV’s) data center. Another approach to providing SaaS solutions is for an ISV to host their application at an IaaS provider’s data center.
• IaaS Services
One example of an IaaS service is basic storage. Another example is physical and/or virtual server instances that are typically combined with storage, load balancers and possibly other physical or virtual appliance-based services such as security.
• Cloud Network Services (CNS)
As described in a recently published market research report2, Cloud Network Services is a burgeoning wave of solutions that deliver a service that would normally be provided by the IT department. This includes services such as VoIP, VDI, collaboration, unified
communications, management, optimization and disaster recovery/business continuity. CCSPs may layer CNS solutions on top of IaaS solutions in part to offer on-demand scalability.
• Cloud Data Center Services
One example of this category of service is when an IT organization outsources both its private data center infrastructure and the management of that infrastructure to a CCSP. The hosted private cloud data center would be based on dedicated resources that can be located
either at a site controlled and managed by the enterprise or at a site controlled and managed by the CCSP.
Another example of this category of service is when a CCSP operates a highly virtualized, multi-tenant public cloud data center and uses it to provide enterprise customers with a Virtual Private Data Center (VPDC) that is co-managed by the enterprise IT organization and the CCSP. One of the advantages of this particular service is that hybrid cloud computing solutions can leverage VPDCs in conjunction with private cloud data centers to deliver the dynamic on-demand scalability that is one of the major value propositions of cloud
The services described above are expected to experience rapidly growing levels of acceptance by the enterprise market. For example, according to Gartner3, the SaaS market had worldwide revenues of $10.0 billion in 2010 and is projected to reach $21.3 billion by 2015. Gartner4 also
estimates that the IaaS market will grow from $3.7 billion in 2011 to $10.5 billion in 2014. One thing that all these services have in common is that they are typically accessed from enterprise sites over the Internet. As a result, CCSPs typically ensure that their sites have high-speed, low-cost Internet access. While cloud services can theoretically be accessed using MPLS or other traditional WAN services, most CCSPs don’t currently provide this option.
Traditional WAN Services
As previously noted, IT organizations have traditionally designed and built enterprise WANs using services such as Frame Relay, ATM or MPLS. Frame Relay was based on an earlier technology (X.25) and was first deployed in enterprise networks in the last 1980s. It offered a cost-effective alternative to the use of private lines, especially for IT organizations that that were satisfied with having a Committed Information Rate (CIR) that didn’t exceed T1/E1 speeds. However, Frame Relay doesn’t inherently support Quality of Service (QoS) or packet prioritization, which are critical for many delay-sensitive applications.
In the early to mid 1990s, some IT organizations began to implement ATM in part to address the limitations of Frame Relay. Unlike Frame Relay, ATM was advocated as a single unifying technology that would span the LAN and the WAN. However, ATM never fulfilled this promise and its decline began when the IEEE (Institute of Electrical and Electronics Engineers) and IETF (Internet Engineering Task Force) developed a number of standards that essentially eliminated the theoretical competitive advantages that ATM had in the LAN and the WAN.
Roughly a decade ago, service providers began to deploy MPLS as a unifying network core technology. MPLS offers a number of key advantages to service providers, including:
§ QoS and traffic engineering § Layer 2 and Layer 3 VPNs
§ Support for meshed topologies, providing any-to-any connectivity
3SaaS Revenue Growth 4 Qas.com
§ Emulation of legacy WAN services, including both Frame Relay and ATM
As a result of these advantages, MPLS is rapidly displacing Frame Relay and ATM services. For example, The 2010 Cloud Networking Report5 included the results of a survey in which the
respondents were asked to indicate if they intended to increase or decrease their use of WAN services such as Frame Relay, ATM and MPLS. Those results are shown in Table 1.
Service % Decrease % Stay the Same % Increase
Frame Relay 33.9% 64.1% 1.7%
ATM 22.3% 71.4% 6.3%
MPLS 4.8% 43.2% 52.0%
Table 1: Expected Change in the use of WAN Services
MPLS, however, is a very complex technology. Whereas many IT organizations implemented technologies such as Frame Relay themselves, virtually all IT organizations that utilize MPLS do so as a managed service. One of the potential advantages of accessing a managed service, such as MPLS, is that the service is accompanied by a Service Level Agreement (SLA). Unfortunately, in most cases the SLAs are weak. In particular, it is customary to have the SLAs be reactive in focus; i.e., the computation of an outage begins when the customer opens a trouble ticket. Generally, the carrier’s SLA metrics are calculated as network-wide averages rather than for a specific customer site. As a result, it is possible for a company’s data center to receive notably poor service in spite of the fact that the network-wide SLA metrics remain within agreed-to bounds. In addition, the typical level of compensation for violation of SLAs is quite modest. The 2010 Cloud Networking Report also documented the results of a survey in which the respondents were asked to indicate which of the following best describes the SLAs that they get from their network service providers for services such as MPLS.
• The SLAs go a long way towards ensuring that we get a quality service from the network service provider.
• The SLAs are better than nothing but not by much. • The SLAs are not worth the paper they are written on. Their responses are shown in Figure 1.
The$SLAs$go$a$long$ way,$34%$ The$SLAs$are$be6er$ than$nothing,$53%$ SLAs$are$not$worth$ the$paper,$13%$ 0%$ 10%$ 20%$ 30%$ 40%$ 50%$ 60%$
Figure 1: The Effectiveness of SLAs
The fact that two-thirds of the survey respondents indicated that the SLAs that they receive from network service providers are either not worth the paper they are written on, or that the SLAs they receive are not much better than having none, indicates that for the majority of IT organizations the SLAs that are associated with MPLS don’t provide any measurable value.
The traditional approach to providing Internet access to branch office employees has been to carry the Internet traffic on the organization’s enterprise WAN to a central site where the traffic is handed off to the Internet. The 2010 Cloud Networking Report contained the results of a survey question in which the survey respondents were asked to indicate how they currently route their Internet traffic and how that is likely to change over the next year. Their responses are contained in Table 2.
Internet Traffic Routed to a Currently Central Site
Will be Routed to a Central Site within
a Year 100% 39.7% 30.6% 76% to 99% 24.1% 25.4% 51% to 75% 8.5% 13.4% 26% to 50% 14.2% 14.2% 1% to 25% 7.1% 6.7% 0% 6.4% 9.7%
Table 2: Routing of Internet Traffic
The data in Table 2 indicates that IT organizations are reducing the amount of Internet traffic that they backhaul to a central site. This is driven by the increase in demand to access the Internet coupled with the prohibitive cost of adding more traditional WAN capacity.
As shown in Figure 2, enterprises that centralize Internet access will typically provide access to the Internet in general, and to public cloud services in particular, via a centralized
high-bandwidth Internet connection at one or more data centers.
Figure 2: Centralized Internet Access
The advantage of centralizing Internet access is that it enables IT organizations to exert more control over the Internet traffic and it simplifies management, in part, because it centralizes the complexity of implementing and managing security policy. The primary disadvantage of centralized Internet access is that the backhauled Internet traffic consumes a significant amount of expensive enterprise WAN bandwidth. In addition, this approach to Internet access is likely to add significant delay compared to providing Internet access locally. Another option for enterprises preferring centralized Internet access is to extend the MPLS VPN to include the site of the cloud service provider, as shown in Figure 3. This option has the advantages of extending the reliability and QoS strengths of the MPLS VPN to include the site of the cloud service provider. The Internet traffic is still backhauled but the cloud service is accessed directly via the MPLS VPN. The disadvantages of this option are increased WAN transmission costs and limitations in the range of accessible cloud services because of the CCSP's preference for the Internet as the access method. From the CCSP’s perspective, a single Internet connection is much simpler and more economical than are high bandwidth connections to the MPLS networks of multiple carriers. In addition, the high fixed costs of these WAN services can detract from the overall cost-effectiveness of an on-demand cloud service.
Figure 3: Connecting to Cloud Services via MPLS
Local Internet access, as shown in Figure 4, generally requires each remote site to have its own security implementation. This includes firewalls and possibly other security appliances, such as Intrusion Prevention Systems (IPSs) and virus scanners. In the traditional IT environment, IT organizations that choose to implement local Internet access typically accepted the responsibility of implementing distributed security solutions in order to avoid the high WAN transmission costs that are associated with backhauling Internet access traffic over an enterprise WAN.
A New Approach to WAN Design
As mentioned in the introduction, a new technology is enabling IT organizations to re-think how they design their WANs. The key enabler of this new approach to WAN design is the ability to wrap a layer of intelligent abstraction around a physical WAN that is comprised of a number of parallel network connections. Throughout this white paper, this new approach to WAN design will be referred to as either The New Approach or The New Approach to WAN Design.
The key components of The New Approach are shown in Figure 5. As illustrated, at each site the parallel network connections could include multiple ISP connections to the Internet, possibly using multiple access technologies (e.g., DSL, cable, 3G/4G), as well as enterprise WAN services, such as Frame Relay and MPLS. The New Approach can be used to
supplement an enterprise WAN service or it can be based entirely on multiple Internet connections. A key component of the design is the ability of the appliance at each site to choose the most appropriate end-to-end path. In the Internet portion of the design, multiple consumer-grade network connections in parallel dramatically improve the availability, delay, jitter and packet loss characteristics of the public Internet. As a result, the Internet is
essentially converted into a low-cost, highly reliable, enterprise-class WAN with service quality that compares very favorably with the service quality of traditional WAN services. The Real Estate Director commented that they used to have an MPLS network that connected their users to their two data centers. They are in the process of eliminating all of their MPLS circuits and replacing them with a WAN that is comprised entirely of low-cost Internet services. He added that one of the key factors that drove them to adopt The New Approach to WAN Design was the business need for significantly more WAN bandwidth and the high cost that was associated with that bandwidth. He stated that, “It was not economically feasible to increase bandwidth sufficiently on the MPLS platform but now that is completely achievable using a different approach to WAN design.” When asked about the quality of his new WAN, The Real Estate Director said that it was better than the MPLS service that he was using. He justified that statement by adding that he can get additional Internet bandwidth notably faster than he can get additional MPLS bandwidth and the Mean Opinion Scores (MOSs) of the VoIP traffic that he carries on his new WAN are better than they were on the MPLS network.
Data Center ISP$A$ ISP$B$ ISP$C$ ISP$D$ INTERNET$ Cable xDSL $MPLS$ Redundant WAN Virtualization Appliances Remote Office T3/OC-3/GE T3/OC-3/GE T1 WAN Virtualization Appliance
Figure 5: The New Approach to WAN Design
One of the underlying techniques that enables the WAN (depicted in Figure 5) to provide a high level of service quality is adaptive path selection. Adaptive path selection algorithms make an instantaneous selection of the best path for each application type on a packet-by-packet basis. If there is a failure or congestion in one of the paths, traffic can be re-directed to a different path in as little as a few hundred milliseconds. Application-aware adaptive path selection among
multiple paths provides a form of virtual QoS for the public Internet. The path selection
algorithms also provide load distribution across all paths to ensure that the maximum advantage is taken of all the available bandwidth.
Because of adaptive path selection, the availability of a WAN that consists of multiple parallel paths is very high, even if the availability of each component path is only moderately high. For example, Figure 6 depicts a system that is composed of two components that are connected in parallel. Network(A( 99%(Reliable( Network(B( 99%(Reliable(
Figure 6: Two Networks Connected in Parallel
The system depicted in Figure 6 is available only when the two networks are unavailable. Assuming that each network is a diversely routed DSL or cable access line and that one of the
access lines has an availability of 99 percent and the other has an availability of 98 percent, then the system has an availability of 99.98 percent. Alternatively, if both access lines have an availability of 99 percent, then the system is available 99.99 percent of the time. This level of availability is equal to or exceeds the availability of single vendor Frame Relay or MPLS networks.
The Government Technology Manager stated that his organization used to have an MPLS network and they "loved it." Unfortunately, the network was also expensive and inflexible. He stated, “It was possible for us to have multiple MPLS providers, but it was difficult to do and it resulted in a lot of finger pointing on the part of the providers.” According to The Government Technology Manager, they completely replaced their MPLS network with a WAN similar to the one depicted in Figure 5 and in three months they saved enough on WAN bandwidth to pay for the necessary WAN appliances. He stated that the new WAN gives him “lots of flexibility” and that none of their VoIP users have noticed any difference in the quality of their voice calls. In order to implement a WAN like the one depicted in Figure 5, WAN appliances are placed physically or logically in-line with the Internet access routers at every site and encrypt and encapsulate each packet traversing the Internet. Encryption is optional for packets sent over enterprise WAN connections. The encapsulation header includes a timestamp, sequence number and an IP address of the destination WAN appliance that corresponds to the far end ISP that will be used by a particular packet that traverses the WAN.
Three Alternative Design Scenarios
This section of the white paper describes three scenarios in which The New Approach to WAN Design offers significant benefits for accessing the Internet and for cloud computing services.. The first scenario involves an enterprise that has implemented centralized Internet access and wants to use the Internet to access a wide range of public cloud network services, such as SaaS and IaaS. The advantage of using The New Approach to WAN Design in this scenario is that it makes Internet backhaul far less expensive, increases capacity and adds scalability. For example, as shown in Figure 7, all of the backhauled traffic between the remote sites and the central data center will benefit from reliability, security and QoS features that are integral to The New Approach. The reliability of the WAN between the central data center and the public cloud services’ sites will be very high because of the dual ISP connections; e.g., ISPs A and B. However, the unique benefits of The New Approach to WAN Design will not generally be available on this portion of the end-to-end path because of the impracticality of the CCSP provisioning a dedicated WAN appliance for each service subscriber.
Do IPS & Firewalling
centrally versus the remote office Aggregate bandwidth to/from the remote office increased, and scalable
Figure 7: Scalable and Economical Internet Backhaul
The second scenario involves an enterprise that has implemented a private cloud computing data center either at a central site of its own or at a co-location facility. Soheila Soheil stated that some of the advantages of using a co-location facility are that it frees an IT organization from having to worry about operational and security issues and it allows their customers to be up and running in a very short time frame without a massive expenditure of capital. Soheil added that they connect their co-location facility to multiple Tier 1 providers and are “carrier neutral.” This, coupled with avoiding local loop charges, drives very competetive pricing. As a result, one of the services that they offer their customers is to switch the customer’s traffic between service providers without having to wait for the installation of new facilities. The price of Internet bandwidth at co-location facilities varies based on factors, such as the location of the facility, provider of the bandwidth and the amount of bandwidth that is purchased, but should generally fall in the range of $5/Mbps to $35/Mbps.
In the second scenario, the full benefits of The New Approach to WAN Design can be derived for all of the intra-enterprise traffic, including the traffic between the enterprise users and the private cloud resources, as well as for the server-to-server traffic between the data centers. As shown in Figure 8, the economic advantages of The New Approach are further enhanced when the private cloud data center is built at a co-location facility where Internet access costs are generally considerably lower than they are at an enterprise’s site. In this scenario, all of the Internet traffic can be backhauled to the enterprise’s private cloud data center. If the data center is housed at a co-location facility and if there are a number of geographically dispersed co-location sites, directing the backhauled Internet traffic to the nearest co-location site can minimize the propagation latency that is associated with centralized Internet access.
HQ / Data Center ISP$A$ ISP$B$ ISP$C$ ISP$D$ Cable xDSL Remote Office T3/OC-3/GE T3/OC-3/GE WAN Virtualization Appliance ISP$E$ INTERNET$ Data Center @ Colo T3/OC-3/GE WAN Virtualization Appliance Redundant WAN Virtualization Appliances Web$Site$@$ Internet$Core$
Scalable Reliable & Secure Internet Access Cheap Internet Bandwidth
$5 - $35 / Mbps / Month
Figure 8: Private Cloud Data Center at Co-Location Facility
The third scenario is one in which the enterprise has its own data center and has also subscribed to a public cloud data center service, such as an outsourced private data center located on the CCSP’s premises or a Virtual Private Data Center hosted in a CCSP's multi-tenant data center. With both types of outsourced data center services it should be possible to extend The New Approach to WAN Design to include the CCSP’s data center by having WAN appliances provisioned at these data center sites. Figure 9 shows how a hybrid cloud computing environment could be supported by a WAN comprised of dual ISP Internet
connections at all sites, plus an MPLS connection at the remote sites and the enterprise’s data center. In this design, all general-purpose Internet traffic can be economically backhauled to the enterprise’s data center, while all intra-enterprise cloud traffic can transmit over one of the Internet links.
Public Cloud Data Center Service
INTERNET% Enterprise Data Center INTERNET% Diverse IP Networks Virtual WAN
WAN Virtualization appliances at all sites
Private Data Center at Colocation Facility
Figure 9: Hybrid Cloud Computing
Because cloud services are typically accessed over a WAN, the adoption of cloud computing will place a significant volume of new traffic on the WAN. Unfortunately, in many cases the cost, availability and performance of the WAN will be an impediment to the ongoing adoption of cloud computing. MPLS, for example, is usually regarded as being highly available and providing acceptable performance. It is, however, also usually regarded as being expensive. The use of the Internet with access technologies, such as DSL and cable, is usually regarded as being low cost, but in many situations it is not regarded as providing acceptable performance nor high availability. Because of these characteristics, neither of these services on their own will be able to effectively support cloud computing.
In order to eliminate the WAN as an impediment to the adoption of cloud computing, IT organizations need to re-examine the WAN technologies and services that they use. One of the advantages that The New Approach to WAN Design provides is that it enables IT organizations to supplement the benefits of an enterprise WAN service, such as MPLS, with inexpensive Internet bandwidth. Another advantage that The New Approach provides is that it also enables IT organizations to dynamically combine multiple consumer-grade Internet connections into an enterprise-class WAN that has service quality that compares very favorably with the service quality of traditional WAN services.
One of the key use cases for The New Approach to WAN Design is for IT organizations that want to continue to backhaul their Internet traffic to one of their data centers. In this case, instead of backhauling the traffic using an expensive enterprise WAN service, such as MPLS, the traffic can be backhauled on lower cost services. A second key use case is when an IT organization has implemented multiple cloud data centers, either at their own sites or at co-locations facilities. In this case, the Internet traffic can be backhauled to the closest site and the inter-data center traffic can also benefit from The New Approach because it is possible to place WAN appliances in each data center. In those cases in which a data center is housed at a co-location facility, the IT organization will be able to leverage the low cost Internet access that is available at most co-location facilities. A third key use case is when an IT organization has one or more of its own data centers and it also uses one or more public cloud data center services. Similar to the situation in the preceding use case, the Internet traffic can be backhauled to the closest site and the inter-data center traffic benefits from the advantages of The New Approach.
The New Approach to WAN Design is already being used to reliably and inexpensively connect remote offices to private data centers. Enterprises that build their private network this way also position themselves to achieve the same predictable performance they expect in their private environment when using Internet-based public cloud services, now or in the future.