Public report
Overview of recent changes in the IP
interconnection ecosystem
May 2011
17038-93
Contents
1 Executive summary 1
1.1 Internet trends 1
1.2 Evolution of Internet interconnection 2
2 Introduction 4
3 Internet trends 7
3.1 Internet traffic has grown exponentially 7
3.2 Internet traffic has become globalized 9
3.3 The value of Internet content and applications has increased dramatically 11
3.4 Bandwidth-intensive types of Internet traffic have become predominant 13
3.5 Cloud computing services have become widely utilized 16
3.6 Internet connections are a key feature for many new devices and services 17
3.7 Conclusion 18
4 Evolution of Internet interconnection 20
4.1 Description of the early Internet 20
4.2 Internet exchange points (IXPs) 21
4.3 Internet service providers (ISPs) 23
4.4 Content providers 25
4.5 Impact on backbone providers 28
5 Conclusion 34
Annex A: Introduction to the IP interconnection ecosystem Annex B: About Analysys Mason
Copyright © 2011. The information contained herein is the property of Analysys Mason Limited and is provided on condition that it will not be reproduced, copied, lent or disclosed, directly or indirectly, nor used for any purpose other than that for which it was specifically furnished.
Analysys Mason Limited 818 Connecticut Avenue NW Suite 300 Washington DC 20006 USA Tel: (202) 331 3080 Fax (202) 331 3083 [email protected] www.analysysmason.com
Registered in England: Analysys Mason Limited Bush House, North West Wing, Aldwych London WC2B 4PJ, UK
1
Executive summary
Last year saw the 15th anniversary of the commercialization of the Internet backbone as we know it today, when the US National Science Foundation Network (NSFNET) Backbone Service was decommissioned in favor of the current commercial Internet backbone. On that day, April 30, 1995, the Netscape browser had just been introduced, AOL was the most popular ISP in the US and was still charging hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been released. The Internet was mainly used for file transfer, email was simple text, and the Web was not yet multimedia.
Over the intervening 15 years, there have been significant changes in the Internet ecosystem, including an enormous increase in Internet traffic and bandwidth along with a corresponding dramatic increase in the value generated by Internet content. During this period, interconnection arrangements between all of the players in the ecosystem have continuously evolved to meet the challenges of the changing Internet ecosystem, driven by commercial considerations and not regulation.1
1.1
Internet trends
The changes in the Internet ecosystem are mainly due to the following interrelated trends: Internet traffic has
grown exponentially
There has been an explosion of Internet access and in the traffic generated by each individual user, for the reasons described below. As a consequence, market players have adapted by modifying their network architectures and revamping their relationships with suppliers and peers.
Internet traffic has become globalized
From its historical origins in the US, Internet usage has grown significantly around the world, and correspondingly, Internet content creation has also globalized. As a result, the proportion of Internet traffic originated and terminated outside the US has significantly increased over time. Additionally, the amount of Internet traffic exchanged in US-based Internet exchange points (IXPs) is now much less predominant, as many countries and regions have expanded their Internet traffic exchange capabilities through one or more local IXPs.
The value of Internet content has dramatically increased
15 years ago, money flows were mostly directed to ISPs and backbone providers for Internet access and traffic, while content was mostly free or had limited value. This has changed significantly, as content has gained in value thanks to an improved quality of access and the corresponding availability of premium content. As a result, content providers have been able to leverage the value of their content in order to reduce their costs for
1
The only instances of regulatory intervention in IP interconnection have been during merger proceedings, when agencies have imposed conditions ranging from divestiture to temporary limits on changes in peering agreements.
delivering their traffic (for instance, by building their own infrastructure to distribute content around the country closer to end-users).
Bandwidth-intensive and quality needy types of Internet traffic such as video have become
predominant
Bandwidth-intensive and quality needy types of traffic, such as video, have experienced a massive growth, which has had a major impact on bandwidth requirements and the direction of traffic flows. There has been a significant increase in the amount of traffic being downloaded from centralized sites such as YouTube, while at the same time the popularity of peer-to-peer services has resulted in significant traffic between end users. These increases in overall traffic, along with changes in traffic patterns, have led ISPs and content providers to adapt their network policies and architectures. Cloud computing
services have become widely utilized
Cloud computing services, ranging from online email to business applications, have been embraced as barriers to their adoption have been progressively lifted, thanks to the improvement of Internet access, the development of security safeguards, and the resulting increase in awareness and confidence in cloud computing. This has increased overall usage of the Internet, while also making consumers more sensitive to latency and other quality measures that impact access to their cloud applications.
Internet connection became a
mandatory feature for many new consumer
electronic devices
Many new devices have emerged that incorporate Internet access: mobile devices with features adapted to data traffic, netbooks and tablets designed to enable Internet access to cloud-computing applications, and new video streaming devices relying on broadband. As a result, many traditional communications services have migrated to the Internet, for example TV over IP and Voice over IP (VoIP). This has led market players to increase mobile access, install additional network capacity, and establish new forms of interconnection.
The changes described above, individually and in sum, have led to significant changes in the underlying Internet ecosystem as it was established 15 years ago.
1.2
Evolution of Internet interconnection
15 years ago, the Internet ecosystem had a relatively simple structure, with clear and hierarchical relationships between end users and content providers buying Internet access from ISPs, ISPs buying Internet transit from backbone providers, and backbone providers peering with each other.2
At the time, given the historical development of the Internet, much of the global traffic originated, terminated, or transited the US.
But Internet players have steadily adapted to the changing market conditions described above in a number of ways, in order to carry greater traffic volume with lower latency, and to reduce their transit
2
costs, thereby changing the traditional hierarchy of the ecosystem in the process, while also making it truly global. Internet Exchange Points (IXPs) facilitated increased connectivity
The development of IXPs around the world has benefited backbone providers by facilitating distributed interconnection between peers and transit customers, which improves the quality of service and reduces traffic carrying costs. However, the increase in utilization of IXPs also helped in marginalizing the role of the backbone providers, as former customers began to interconnect directly with one another at the IXPs.
ISPs and content providers increasingly connect directly
Many “route around” options have emerged for ISPs and content providers to exchange Internet traffic, while avoiding Internet backbone transit costs. These include secondary peering arrangements between ISPs to exchange traffic directly, much of it from peer-to-peer applications. Increased traffic has also led content providers to negotiate paid peering arrangements with ISPs to deliver content to their end users. Backbones have adapted to these changes by selling only partial transit to ISPs and content providers, which have arranged much of their traffic exchange directly. At the same time, peering relationships have evolved based on volume, traffic flows and routing.
Content providers have a variety of options for delivering traffic
Increases in content traffic have led to traffic imbalances, such that ISPs delivered much more traffic from content providers to their end users. This has led to paid peering arrangements to cover the cost of the traffic. Content providers have leveraged their expanding financial resources to deliver traffic directly to ISPs by building private networks or using third-party content delivery networks (CDNs) to deliver content directly to caches at the edge of ISPs‟ networks, further reducing transit costs and the resulting revenues for backbones.
From the backbone provider‟s point of view, these changes led to a reduction in demand for transit services, and an increase in competition from former customers who now have a number of choices for delivering and exchanging traffic. Further, backbones must compete vigorously on the price of transit in order to generate the traffic volume to continue to peer with one another. This has resulted in an increase in the level of competition for Internet transit services, as evidenced for example in the fall in transit prices over the past five years, with no sign of respite.
In conclusion, in the 15 years since the commercialization of the Internet backbone, the Internet ecosystem has proven itself to be able to develop and sustain interconnection in the absence of sector-specific regulation. It has also shown itself to be able to adapt well to rapid and profound market changes without regulatory intervention, which in telecommunications is typically much slower to implement changes in interconnection arrangements and has issues with implicit subsidies and arbitrage. As a consequence of these changes, today‟s Internet ecosystem is no longer hierarchical, but rather a dynamic web of interconnections between a variety of Internet players.
2
Introduction
This paper marks the 15th anniversary of the commercialization of the Internet backbone as we know it today. There have been significant changes in the Internet ecosystem over the last 15 years, resulting in enormous increases in Internet traffic, supported by a massive expansion of the Internet bandwidth, and accompanied by a major increase in the value generated by Internet services and applications. Through these changes, one feature remains prominent : interconnection arrangements between all of the players in the ecosystem have continued to evolve to meet the challenges of the changing ecosystem, driven almost entirely by commercial considerations and not regulation.3
On April 30, 1995, the US National Science Foundation Network (NSFNET) Backbone Service was decommissioned in favor of the current commercial Internet backbone. The NSFNET operated a national backbone network allowing regional networks to connect at “supercomputing” sites.4
At this time, the objective was to create an open network linking academic researchers, and allowing them access to distant supercomputers over the network at no cost. When the NSFNET network went online in 1986 and allowed commercial traffic (and not just NSFNET traffic) to be carried, it consisted of six sites interconnected with leased 56kbit/s Digital Data System (DDS) links – this was upgraded to 45Mbit/s links in 1991.
At the time that the NSFNET was decommissioned in 1995, Netscape – the first commercial Web browser – had just been introduced, AOL was the most popular ISP in the US and was still charging hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been released. The Internet was used mainly for file transfer, email was simple text, and the Web was not yet multimedia. Since then, traffic has exploded, as shown in Figure 2.1below.
3
The only instances of regulatory intervention in IP interconnection have occurred during merger proceedings, when agencies have imposed conditions ranging from divestiture to temporary limits on changes in peering agreements.
4 From 1987 to 1995, the NSFNET Backbone was designed, managed, and operated on behalf of the NSF by Merit Network, Inc., a non-profit corporation governed by public universities partnering with IBM, MCI, and the State of Michigan.
Figure 2.1: Monthly backbone traffic in the US [Source: Minnesota Internet Traffic Studies; Analysys Mason Estimates]
This increase in traffic comes as a result of a virtuous circle of more powerful computers, the availability and adoption of broadband access, the increase in bandwidth available for carrying Internet traffic, and the introduction of multimedia content (particularly video), along with new ways to share that content, such as peer-to-peer. Figure 2.2 below shows how new categories of Internet video have arisen in the last five years and are projected to become the dominant category of content over the next few years.
Figure 2.2: Consumer Internet traffic forecasts [Source: Cisco Visual Networking Index]
Note: in the figure above, the “File sharing” category includes traffic from peer-to-peer applications such as BitTorrent and eDonkey, as well as web-based file sharing.
The NSFNET was the backbone over which regional networks exchanged traffic. When it was closed
0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 T B /m o n th 0 2,000 4,000 6,000 8,000 10,000 12,000 2006 2007 2008 2009 2010 2011 2012 2013 2014 PB p e r m o n th
Web and Email File Sharing
Internet Gaming Internet Voice
Internet Video com. Internet Video to PC
traffic exchange. Instead, four Network Access Points (NAPs) across the country were designated as the points at which traffic could be exchanged by the new commercial backbones. Such interconnection was not regulated, making it possible for commercial arrangements known as peering and transit to be negotiated between the backbones and ISPs. The introduction of commercial backbones not only modified the structure of the Internet infrastructure ecosystem, but also accommodated to changes to the type of Internet traffic carried, as this opening increased greatly the carrying and exchange of commercial traffic in addition to the university and research traffic previously carried.
This document aims at describing changes in the Internet ecosystem since the commercialization of the Internet backbone. We analyze the changes in terms of the type of Internet traffic carried, the infrastructure supporting the Internet traffic, as well as the evolution of the different players and their inter-relationships. We also assess how commercial models adapted to these changes, and analyze the impact of these changes on competition between the backbones.
The remainder of this document is laid out as follows:
Section 3 provides an outlook of the main trends over the past 15 years that have impacted the Internet backbone
Section 4 analyses the evolution of the Internet backbone, with a focus on interconnection arrangements, and describes the current market situation from point of view of each of the main types of Internet players
3
Internet trends
The Internet ecosystem has evolved significantly over the past 15 years, with corresponding impacts on interconnection arrangements. This evolution is partly due to a number of trends that are all interrelated, but that can be described as follows:
Internet traffic has grown exponentially
Internet traffic has globalized
the value of Internet content and Applications has increased dramatically
bandwidth-intensive types of Internet traffic such as video have become predominant
cloud-computing services have become widely utilized
Internet connections are a mandatory feature for many new consumer electronic devices and services.
Figure 3.1: Trends that have impacted the Internet ecosystem over the past 15 years [Source: Analysys Mason]
We provide in this section an analysis of these trends, which will help to better understand the context of the evolution of Internet interconnection described in the following section.
3.1
Internet traffic has grown exponentially
The past 15 years have seen an explosion of Internet usage around the world, particularly from Internet broadband subscriptions that allow end users to use their connections to access the new high bandwidth services such as video streaming. At the end of September 2009, there were more than 75 million fixed broadband subscriptions in the US, representing a penetration of around 60% of households, and significant growth from the end of 2001, when there were only around 10 million broadband subscribers, as shown in the Figure 3.2 below.
Globalization of Internet traffic Value of Internet Content Peer-to-peer and video streaming Cloud computing Online devices Volume of Internet traffic
Figure 3.2: Growth in number of broadband subscribers in the US [Source: Analysys Mason Research]
In the past 10 years, this dramatic increase in the adoption of fixed broadband has been accompanied by an explosion of mobile broadband usage. According to CTIA, wireless data revenues in the US rose from US$211 million in 2000 to US$41.5 billion in 2009.5
Similar growth has characterized markets in Europe and much of Asia as well, resulting in demand for Internet services that is distributed much more evenly across the world than 15 years ago, as shown in Section 3.2.
Not only has the number of Internet access users experienced a significant growth, but also the bandwidth usage by each individual user has exploded. This can be explained in part by the increase in the availability of even higher capacity Internet links, and the growing availability of bandwidth-intensive applications such as video downloads (see section 3.4), the development of cloud-computing services (see section 3.5), and the take-up of a large variety of devices connected to the Internet (see section 3.6).
These combined effects have resulted in an exponential growth of Internet traffic in general, and in the traffic carried by Internet backbone providers in particular, as shown above in Figure 2.1. The fast growth in traffic has naturally generated significant changes in the Internet ecosystem, as market players have adapted to this demand increase by modifying their network architectures (e.g. expanding their networks by deploying fiber and upgrading their capacity to accommodate higher traffic volumes and traffic that is more latency sensitive, and creating new routes to carry the traffic more efficiently), as well as revamping their relationships with suppliers and peers, as described below. 5 Source: http://www.ctia.org/media/industry_info/index.cfm/AID/10323 0 10 20 30 40 50 60 70 80 90 2001 2002 2003 2004 2005 2006 2007 2008 2009 Q3 M ill io n s 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Pe n e tra ti o n a s % h o u se h o ld s DSL Cable modem
Residential FTTB Broadband FWA
3.2
Internet traffic has become globalized
For historical reasons, in the early 90s Internet usage was biased towards the US, as was Internet traffic exchange at the time that the NSFNET was decommissioned in 1995. This situation evolved significantly over time, as demand for Internet access exploded – first in all the other developed countries, and then in developing countries. Indeed, today a significant portion of Internet users are to be found in large markets like China and India: for example, China today boasts more than 400 million Internet users, and other emerging Asian nations are also fuelling the growth of traffic in this region.
Figure 3.3: Evolution of the distribution of Internet users by region[Source: ITU]
The increase in the number of Internet users outside the US led to a corresponding increase in Internet traffic in other regions of world, with Internet content being created, hosted and accessed from outside the US. As a result, the Internet network topology began to evolve, so as to enable a more efficient carrying of traffic. The historical use of US-based IXPs to exchange foreign traffic was not only inefficient, but also led to a call for intervention that went by the rubric of International Charging Arrangements for Internet Services (ICAIS). In this debate at the Asia-Pacific Economic Cooperation (APEC) forum and the International Telecommunication Union (ITU), countries underserved by Internet infrastructure sought to address the issue of bearing the costs of connecting to US-based IXPs for traffic exchange.
To deal with these issues, national and regional networks started to interconnect to each other and exchange their traffic at regional IXPs such as AMS-IX in Amsterdam and LINX in London. These IXPs exchange a significant amount of traffic among a growing number of members. For instance, AMS-IX has 389 connected networks exchanging 1091.9Gbit/s of traffic at peak times, while LINX
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 9 9 5 1 9 9 6 1 9 9 7 1 9 9 8 1 9 9 9 2 0 0 0 2 0 0 1 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9
North America Europe and Central Asia
East Asia and Pacific Latin America and Carabbean
Middle East and North Africa South Asia
has 379 members with a peak traffic exchange rate of 614Gbit/s.6
This traffic is not just national, but also international, and the result is that the host countries have turned into significant international hubs. As shown in Figure 3.4 below, the Netherlands, followed by the UK, has a distinct edge in the amount of international Internet bandwidth per population compared with a wide variety of countries, based in no small part on the significant role of Dutch and British IXPs in attracting international backbones and content providers. Similar activity is taking place in Asia and to a lesser degree in other parts of the world, as IXPs act to help to localize traffic and attract a surrounding ecosystem of backbones and content providers.
Figure 3.4: Growth in international Internet bandwidth per head of population [Source: Telegeography, ITU, Euromonitor, company data]
Consequently, the role of the US as a hub for international exchange of Internet traffic has diminished significantly. Figure 3.5 below shows that even though Latin America still relies heavily on the US for Internet bandwidth, regional networks in Europe, Asia and Africa are less and less dependent on US-based facilities to carry and exchange traffic. Africa‟s Internet traffic is now focused more predominantly on Europe, while Europe‟s traffic is mainly intra-regional, which is also becoming the norm in Asia.
6
In October 2010 AMS-IX had 389 networks connected, measured by autonomous system numbers, with a peak traffic exchange of 1091.9Gbpit/s (source: www.ams-ix.net). LINX provides general statistics showing that in October 2010 it had 379 members exchanging 614Gbit/s peak traffic (source: www.linx.net).
00 20 40 60 80 100 120 140 160 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 k b it/ s Australia USA Canada Germany France Singapore UK Netherlands
Figure 3.5: Share of Internet bandwidth connected to US and Canada, by region [Source: Telegeography]
Figures represent Internet bandwidth connected across international borders as of mid-year. Domestic routes are excluded.
Today, 89 countries possess their own IXPs to exchange national and regional Internet traffic, and thus lower the costs associated with international Internet access. 7
It is interesting to note that this evolution has not required specific regulatory intervention, but is the result of commercial solutions adopted by national and regional networks.
3.3
The value of Internet content and applications has increased dramatically
As new Internet services have developed, the positioning of the different stakeholders in the Internet ecosystem has evolved (see Annex A for an introduction to the IP interconnection ecosystem). 15 years ago, ISPs and backbone providers were at the top of the hierarchy, and money flows were mostly directed to them for Internet access and traffic, while content was mostly free or of limited value. This has changed significantly in recent years as the value of content has risen, thanks to an improved quality of service and the development of premium content.
Online music provides a good illustration of the evolution of content values in the past decade. Initially, music labels were reluctant to sell their music online, for a variety of reasons. As a result, the first widespread distributor of online music was the peer-to-peer network Napster. This was free, but content was largely pirated and copyright issues led to its shutdown in 2001. Following this, labels became comfortable with the level of protection being offered by digital rights management (DRM), and began to make their music available online, leading to the emergence of paid downloads. A good
7
Source: Packet Clearing House Report on Internet Exchange Point Locations (https://prefix.pch.net/applications/ixpdir/summary/) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2002 2003 2004 2005 2006 2007 2008 2009 2010
illustration is the success of the media platform iTunes Store, launched by Apple in April 2003: as of February 2010, 10 billion songs have been sold on this platform.8
Other significant value-added services have emerged from mobile Internet access, with the explosion of the mobile paid applications market. The success of Apple‟s App Store can illustrate this trend: as of January 2010, 3 billion applications (both free and paid-for) had been downloaded through the store.9
More generally, the Yankee Group estimates that the total revenues for the mobile paid applications industry in the US alone reached US$1.6 billion in 2010, against US$537 million the preceding year. This rise in content value will continue in the coming years, as this market growth will be supported by devices such as the new iPad designed for the consumption of online multimedia content. According to the same source, application revenues are forecast to reach US$11 billion by 2014.10
The massive growth in Internet advertising revenue illustrates well the growth in content value. Figure 3.6 below provides an estimate of the evolution of the fixed and mobile Internet advertising markets in North America. The Internet advertising market is estimated to have been worth around US$25 billion in 2009, compared with approximately US$10 billion five years before, and is projected to grow to more than US$35 billion by 2013. It should be noted that a large part of these revenues are captured by content providers such as Google.
Figure 3.6: Internet advertising market in North America [Source: PricewaterhouseCoopers LLP, Wilkofsky Gruen Associates] 8 Source: http://www.apple.com/pr/library/2010/02/25itunes.html 9 Source : http://en.wikipedia.org/wiki/ITunes_Store#Video_2 10 Source : http://www.lesechos.fr/info/comm/020439905748-applications-mobiles-l-institut-yankee-group-a-revu-a-la-.htm 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9 2 0 1 0 2 0 1 1 2 0 1 2 2 0 1 3 U SD mi lli o n
Wired Internet advertising Mobile Internet advertising
An interesting example of the evolution of the value of content is ESPN360.com, a broadband network for live sports programming in the US and several other countries. ESPN has essentially adopted a pay-TV model, in which ISPs must sign up and pay ESPN for offering the service, rather than charging end users directly. The customer pays for access as a part of its ISP fees, whether it uses the service or not, and may choose an ISP based on the availability of ESPN360.
This example suggests the extent to which the relative positions of content providers and ISPs have evolved in the Internet ecosystem. In particular, content providers have been able to leverage the attractiveness of their content in order to reduce their supply costs for accessing the Internet. Furthermore, as we will see in Section 4.4, content providers can use their fast-growing revenues to invest in their own network infrastructures, in order to further reduce the transit costs by developing self-supply solutions, and thus avoid paying other market players to deliver their traffic.
3.4
Bandwidth-intensive types of Internet traffic have become predominant
The nature of Internet traffic has changed dramatically in recent years, and in particular there has been a massive growth in the proportion of video traffic as a percentage of total Internet traffic. Video traffic is distributed either via peer-to-peer services or streamed from a centralized server. Projections show that video will remain a major driver for traffic increase (see Figure 2.2) and sustain the growth in Internet traffic. We focus in this section on the growth of streaming video and peer-to-peer traffic, which not only has a major impact on the bandwidth requirement for Internet traffic, but also on the traffic patterns, as video streaming modifies the traffic pattern from traditional two-way communication to one-way content delivery).
3.4.1 Sustained growth in video traffic
The consumption of video media consists mostly of Internet video to PC (i.e. free or pay-TV viewed on a PC), and in a more limited scale in Internet video to TV (the same as video to PC, but streamed to the TV set). Thus today, 82% of Internet users in the US watch videos online, and an average user views 182 videos per month.11
YouTube, a video-sharing website on which users can upload and view videos, is a leading symbol of this Internet explosion. YouTube was founded in February 2005 and bought by Google in 2006. In only five years, this company achieved an almost unimaginable growth: by October 2009, it was serving more than a billion videos per day worldwide.12
In November 2009, 12.2 billion videos were viewed on YouTube on a monthly basis in the US alone.13
Not only has YouTube managed to draw a very large audience, it has also succeeded in attracting a large amount of “free” user-generated content: YouTube announced on March 2010 that users were uploading some 24 hours of video to the platform every minute.14
This represents double what was 11 Source : http://website101.com/statistics/Internet-stats-video-stats-social-media-usage-traffic-email/#ixzz0jJUhngj3 12 Source : http://bits.blogs.nytimes.com/2009/10/09/youtube-were-bigger-than-you-thought/?hp) 13 Source: http://bits.blogs.nytimes.com/2009/10/09/youtube-were-bigger-than-you-thought/?hp) 14 Source: http://youtube-global.blogspot.com/2010/03/oops-pow-surprise24-hours-of-video-all.html
uploaded only two years before (of course, some of this material is commercial, and YouTube has had to address the copyright issues that arose first with Napster).
Figure 3.7: Hours of video uploaded on YouTube platform per minute [Source: YouTube]
But YouTube, while significant, is not the only player in the Internet video market: there are other major video content providers that are experiencing similar growth levels. Another example would be Hulu, a website offering commercial-supported streaming video of TV shows and movies from several major TV channels and studios. In November 2009, Hulu was streaming 924 million videos per month in the US alone, according to the research company Comscore.15
In the UK, the BBC‟s online iPlayer is estimated to have received well over a billion requests for programming in 2010, with 114 million requests in July 2010 alone.16
Other services such as Netflix stream video directly to the TV. According to Sandvine, streamed audio and video represent almost 43% of data consumption in North America during peak periods, with Netflix video alone accounting for over 20% of download traffic during peak times.17
In addition to these video-streaming sites, a significant amount of video is consumed using peer-to-peer networks.
15 Source: http://www.comscore.com/Press_Events/Press_Releases/2010/1/November_Sees_Number_of_US_Videos_Viewed_Online_Surpass_30 _Billion_for_First_Time_on_Record 16
This includes both video and radio programming. See http://www.bbc.co.uk/blogs/bbcInternet/img/Publicity_Pack_July_2010.pdf.
17
Source: Fall 2010 Global Internet Phenomena Report, Sandvine, p.9. 0 5 10 15 20 25 30 Q 2 2 0 0 7 Q 3 2 0 0 7 Q 4 2 0 0 7 Q 1 2 0 0 8 Q 2 2 0 0 8 Q 3 2 0 0 8 Q 4 2 0 0 8 Q 1 2 0 0 9 Q 2 2 0 0 9 Q 3 2 0 0 9 Q 4 2 0 0 9 Q 1 2 0 1 0 H o u rs
3.4.2 Sustained growth in peer-to-peer traffic
Peer-to-peer networks allow a distributed community of users to share digital content or resources, including video.18
But contrary to a classic client–server architecture in which all network content is held on a server in a central location and accessed via a client on the end-users‟ devices, peer-to-peer resources are located in and provided by devices at the edge of the network by “peers”. Peer-to-peer quickly became a popular way for Internet users to share video and music files, as it was particularly adapted to transfer large files over the Internet. Indeed peer-to-peer networks use the resources provided by users (bandwidth, storage space, and computing power) to increase the total capacity of the system at little or no cost to the originator of the peer-to-peer network. In contrast, in a typical client–server architecture, clients make demands on the system, but do not supply any resources (as such, as more clients join the system, the owner must provide more resources to meet the demand, or congestion will impact performance).
Peer-to-peer services themselves are not a recent Internet application: they have their origins in many of the early Internet services, such as Internet Relay Chat (IRC) which was developed at the end of the 1980s. The widespread popularization of peer-to-peer goes back to Napster in 1999. Napster‟s success was striking, as only two years after the launch of operations it was reaching 26.4 million users. As much of the material distributed over Napster violated copyrights, the service was closed down after a lawsuit instigated by the Recording Industry Association of America (RIAA).19
Nonetheless, peer-to-peer networks have become more and more popular, and have overcome the perceived drawbacks of Napster‟s network (centralized servers distributing the files and requests to the peers) by relying on a decentralized or a hybrid architecture: without central servers, the technical risks are minimized, operational costs are reduced (database management in particular is distributed to peers) and the ability of authorities to shut down the service following legal decisions is much more limited. Kazaa and Gnutella 0.6 are notable examples of such new peer-to-peer networks.20
As broadband access grew, and music began to shift to commercial sites such as iTunes, peer-to-peer networks were increasingly used for the exchange of higher-bandwidth video traffic. The result of the popularization of peer-to-peer was a significant change, in which users became content providers as well as consumers, and the flow of content shifted from a centralized location to become more distributed. Unlike the streaming of a video from a centralized website such as YouTube, the downloading of the same video using a peer-to-peer network has a very different impact in terms of network utilization: in this case the traffic goes from one or more end users to another end user (and both ways, as peer-to-peer is based on reciprocal sharing).21
18
Even though the rise in peer-to-peer traffic is relatively recent, peer-to-peer services themselves should not be considered to be a recent Internet application. Peer-to-peer networks have their origins in many of the early Internet services: for instance, the Internet Relay Chat (IRC), which was developed at the end of the 1980s.
19
See http://papers.ssrn.com/sol3/papers.cfm?abstract_id=504062
20
See http://en.wikipedia.org/wiki/Timeline_of_file_sharing
21
The difference may be most clearly understood by comparing YouTube with a peer-to-peer network. In either case, the content may be user-generated, but with YouTube the originator uploads it once to YouTube, from which others download it. In contrast, with peer-to-peer, each time a user downloads the content it is resent from the originator (or other peers).
As a big proportion of traffic goes directly from ISP end users to ISP end users, and large file-sharing traffic requires very high bandwidth, ISPs have responded by adapting their network policies and architectures. In particular, they have had to find alternative routes to deliver this video traffic. Combined with the general growth of Internet traffic, it started to make economic sense for ISP to adopt solutions like secondary peering to address this traffic issue (see Section 4).
3.4.3 Forecasts
Growth in video traffic is not expected to slow down in the medium term, and according to Cisco, the sum of all forms of video (IPTV, video on demand, Internet video, and peer-to-peer) will account for over 85% of global consumer traffic by 2013. Internet video excluding peer-to-peer (free or pay TV and video on demand delivered on a PC or a TV) will account for over 55% of all consumer Internet traffic by this time.22
It appears therefore that video traffic will be the major driver for the Internet traffic growth in the coming years, in proportions that will require significant investments in Internet infrastructures and technology development to carry the traffic and sustain the quality of service (via an increase in available bandwidth).
It is interesting to note that in relative terms peer-to-peer is actually declining as a percentage of overall IP traffic, in favor of video streaming from centralized servers. This evolution can be explained by the growing availability of alternative ways to watch online premium content (e.g. YouTube, iPlayer and Hulu.com) using centralized servers, and the relative drawbacks of peer-to-peer downloads compared to video streaming. From video-streaming websites, content is available with high quality, is often free, and can typically be downloaded quickly and easily. By contrast, peer-to-peer content, while also free, is not immediately available (content is always downloaded in random chunks, which makes it difficult to estimate the download completion time), there is no quality guarantee, and users are subject to an additional risk resulting from possible copyright violations.
3.5
Cloud computing services have become widely utilized
Online services have also emerged significantly over the past 15 years: examples include data storage, and remote applications ranging from email or online photo albums, to suites of business applications including spreadsheets and word processing. Because the content and applications are centralized in the network, rather than distributed at its edges, this is often referred to as “cloud computing”.23
Although the concept of cloud computing is not new, it has become more popular recently as the barriers to take-up have been progressively lifted, thanks to the improvement of Internet access (e.g. development of broadband, availability of cheap dedicated high capacity links), the improvement of
22
Source:http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-481360_ns827_Networking_Solutions_White_Paper.html
23
As defined by the National Institute of Standards and Technology, cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. (See http://csrc.nist.gov/groups/SNS/cloud-computing/)
security (in particular, enhanced encryption techniques), and the increase in awareness and confidence in cloud-computing techniques.
YouTube is a good example of a popular cloud-computing application, where users store videos on YouTube‟s servers, and access them remotely in a streaming mode. The email services offered by Microsoft and others can also be seen as a very successful popularization of cloud-computing services. For instance, Google‟s Gmail became available to the general public on February 7, 2007 (after an initial invitation-only beta release on April 1, 2004), and by December 2009 had already gained 176 million users.24
This began a transformation of services that were formerly performed at the edge of the network and perhaps involved some incidental Internet traffic, into ones that were performed in the network itself, with integral Internet access and only intermittent processing at the edge. In addition to the examples discussed above, applications such as word processing and spreadsheets began to be offered as Internet services, threatening traditional software models. From the perspective of this paper, this increased overall usage of the Internet, and also made consumers more sensitive to latency and other quality measures that impacted their access to their cloud applications.
3.6
Internet connections are a key feature for many new devices and services
The past 15 years have seen the development (and the wide acceptance by the general public) of a large variety of devices that are connected to the Internet to enable access to the new services described above, as well as additional services that were unleashed in response to the new usage patterns that resulted from these devices.
New connected devices
The release in 2007 of the first popular smartphone – Apple‟s iPhone – marked the take-off of mobile Internet access. More and more mobile devices then started to offer specific features adapted to Internet browsing and data traffic (e.g. mobile applications). By June 2009, there were approximately 6.4 million active iPhone users in the US, with 88% of iPhone users accessing the Internet and 75% of them downloading mobile applications.25
According to Validas, iPhone users in the US each consumed 273MB of data per month on average, and 12% of iPhone users consumed at least 500MB of data per month (as of February 2010).26
Most vendors today are offering similar large touch-screen devices with embedded Internet functionalities.
Netbooks are getting a mass-market status, and according to the research firm ABI Research, 35 million netbooks were shipped worldwide in 2009.27
Netbooks are specifically designed to connect 24 Source: http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htm 25 Source : http://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/ 26 Source : http://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/ 27 Source : http://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analyst
remotely to the Internet, either via a WiFi connection or a 3G connection, and are thus optimized for cloud computing with relatively little memory for installed software.
Tablets can also support new usage intimately related to Internet connection, and the launch of Apple‟s iPad and other similar products supports the view that expectations are high for this market, with more than 14 million iPads being sold in 2010.28
New video-streaming devices have also emerged that exclusively rely on a broadband connection: examples include Vudu, the Boxee Box (so-called “Social Media Center”), etc. These boxes directly enable video streaming to the TV, and video file-sharing in competition with pay-TV, and are beginning to be embedded into other devices such as game consoles or TV sets.
New online multimedia services
Second, many new services related to communications and multimedia have been released that substitute traditional telecom services with online equivalents. This evolution can be explained by the fact that an embedded Internet connection can enable advanced features and reduce the costs of carrying traffic.
In addition to these services offering video on demand, broadcast TV services can also be provided over an Internet connection (IPTV), in particular as part of the triple-play offerings of telecom operators.
Voice over IP (VoIP), which consists in delivering voice communication over IP networks, has a tremendous potential, as it allows customers to make free or very cheap voice calls. So far, quality issues have restrained a wide acceptance of VoIP, but increases in fixed network capacity are changing the situation at a rapid pace. Moreover, the development of cheap mobile broadband offers (“unlimited” data plans with mobile subscriptions) opens the way to VoIP over mobile: for instance, the Line2 application for iPhone enables users to call using the data connection instead of the traditional “minutes” included in the bundle and paid for by the user.
In summary, the development of electronic devices and services relying on Internet access has generated significant changes in the Internet ecosystem, as market players have adapted to this demand increase by modifying and investing in their network architectures: rolling out mobile broadband networks, installing additional network capacity, and developing new forms of interconnection as described below.
3.7
Conclusion
As explained previously, and illustrated in Figure 3.8, Internet traffic and revenues have been fed by the simultaneous growing demand for Internet-based services and the development of Internet-based solutions to answer and sustain this demand.
28
Figure 3.8: Supply and demand feed the growth of Internet traffic and revenues [Source: Analysys Mason]
It is not only Internet traffic that has exploded over the past 15 years: so has the Internet market as a whole. It is interesting to note that, while there has been a significant increase in the revenues generated by the e-commerce market – and to a smaller scale by the online advertising and the device manufacturing markets – there has been a more measured increase in revenues from Internet access itself. As shown in Figure 3.9 below, the revenues per month per Internet user generated by e-commerce and online advertising significantly outweigh those from Internet access.
Figure 3.9: Evolution of the Internet ecosystem in terms of revenues per Internet user per month [Source: PWC, US Department of Commerce, Bureau of economic analysis, Euromonitor]
Mobile broadband Netbooks Willingness to pay for content Peer-to-peer Cloud computing Video streaming Interne t traf fic and revenue s Demand Supply 0 20 40 60 80 100 120 2000 2001 2002 2003 2004 2005 2006 2007 2008 $ p e r I n te rn e t u s e r p e r m o n th Online advertising E-Commerce Devices Internet access
4
Evolution of Internet interconnection
As discussed in the previous section, the trends in Internet usage, particularly high bandwidth video traffic, have led to significant increases in traffic. This, in turn, has led to a significant increase in bandwidth capacity demands, which provides a corresponding increase in costs, and concerns over latency, since streamed video content needs to be delivered with minimal delay. At the same time, content is increasingly monetized, providing content providers with a means to help ensure high-quality delivery to their users. Here we describe how the main Internet stakeholders have adapted to these new bandwidth demands, how new players have emerged, and how interconnection arrangements have evolved accordingly, changing the traditional hierarchy of the Internet ecosystem in the process.
4.1
Description of the early Internet
The Internet ecosystem 15 years ago had a rather simple structure, with relatively clear and hierarchical relationships between the different players: end users and content providers bought Internet access from their local ISP; ISPs bought Internet transit from a limited number of backbone providers to get access to the whole Internet; and backbone providers peered with each other at the US-based NAPs to ensure the full extent of their Internet connectivity. The result was a hierarchy focused largely in the US, with the backbones at the top, interconnecting with one another via peering, and selling the resulting Internet access to downstream ISPs, who in turn sold access to their end users. Figure 4.1 below illustrates the architecture of the early Internet 15 years ago. Annex A provides more background on the relationships between the different players.
Figure 4.1: Simplified architecture of the early Internet [Source: Analysys Mason]
Content provider & aggregator Backbone 1 End-user Backbone 2 Backbone 3 ISP 1 ISP 4 ISP 3 ISP 2 Internet Access Transit Peering
All traffic passes through the backbone providers, that carry
the traffic between the ISPs
In these years, the online market was still very young and dial-up was the predominant means of Internet access, imposing technical constraints on the services that could be provided. For these reasons, content providers had a limited size. In this context, we should note that it is only 15 years ago that the first of today‟s major Internet companies started to emerge: Amazon (founded in 1994), Yahoo (1995), eBay (1995), Google (1998), etc.
The Internet ecosystem 15 years ago had a rather simple structure characterized by relatively clear and hierarchical relationships, with end users and content providers buying Internet access from ISPs, ISPs buying Internet transit from backbone providers, and backbone providers peering with each other.
4.2
Internet exchange points (IXPs)
The nature of Internet exchange has been changing for the past 15 years. When the NSF decided to commercialize the Internet backbone, it designated four dispersed Network Access Points (NAPs), in San Francisco (operated by PacBell), Chicago (BellCore and Ameritech), New York (SprintLink) and Washington DC (MFS). The NAPs were used for public peering, whereby traffic was exchanged using a common switch, which soon congested due to the large volume of traffic. As the historical home of the Internet, a significant amount of international traffic passed through the US at the time, either to connect with US providers, or for transit, sometimes back to the same country of origination.29
At the same time, the operators of the NAPs had the potential to favor their own services: for instance, when the MAE-East NAP was owned by MCI WorldCom, it required the use of MCI circuits to access MAE-East services.30
As a result, a migration began towards peering at IXPs around the world. IXPs, such as Equinix, began to open large data centers, where operators could host their servers and/or connect to one another using direct cross-connections. In some countries, non-profit IXPs run by a consortium of members were established – today several of the largest IXPs use this model, including LINX in London, and AMS-IX in Amsterdam. The move to IXPs enabled what is known as private peering, whereby service providers do not use a common switch and thus can control congestion over each bilateral link with other service providers. The service providers could also sell or purchase transit over these direct cross-connections at the IXPs. As a result, these IXPs became focal points for the entire Internet ecosystem, where the stakeholders could buy and sell their services to one another at relatively low cost.
There are clearly strong network effects that support the growth of an IXP. The more service providers there are in an IXP and the more traffic they exchange, the more attractive it is for other service providers to locate at the same IXP in order to arrange connectivity with the largest number of
29
As all international ISPs at the time had to connect to the US for international transit, they sometimes used these links to exchange domestic traffic, in order to avoid the cost of directly connecting to each other ISP in the country at a time when many domestic telecoms markets were not yet competitive. This flow of traffic out of the country and then back in is often referred to as tromboning.
30
service providers. However, the growth in traffic described above has provided a strong countervailing pressure that has caused IXPs to proliferate across populated areas of countries and regions. Having a dispersion of IXPs has benefits for backbone providers in a peering relationship, in terms of reducing latency, ensuring quality of service (via redundancy) and controlling costs:
First, where the two parties in a transmission are in the same region, having a nearby IXP enables the traffic to remain local, rather than tromboning to a more distant location. This enhances the quality of service thanks to reduced latency. A certain level of redundancy is also provided by there being multiple IXPs in different locations in the same region, giving some protection to connectivity in case of natural disaster, for instance.
Second, where the two parties in a transmission are in different regions, having dispersed IXPs enables backbone providers to effectively share traffic loads, whereby one provider will carry the originating traffic and the other will carry the return traffic.
This second benefit results from what is sometimes referred to as deflection routing or more commonly hot-potato routing, whereby each backbone will exchange the traffic with a peer at the earliest exchange point (for further details see section A.2.3 of the Annex). When the traffic ratio (the ratio of traffic flowing between peers in one direction compared to the traffic flowing in the other direction) is reasonably even, the networks will evenly share the costs associated with carrying traffic exchanged by their users, and thus the agreement is considered fair.
As a result, in their publicly available peering policies, backbone providers may require interconnection in up to six different points in the US, for instance (typically with dedicated connections in IXP locations), in order to reduce carriage requirements and balance traffic loads. Backbone providers will also specify a maximum traffic ratio in order to limit the traffic imbalance between them and fairly split the costs of carrying traffic.31
These types of requirements are designed to reflect comparable costs and benefits, so that the peering arrangement is mutually beneficial to both parties.
While IXPs have clearly been beneficial to backbone providers in lowering costs and making more efficient use of capacity, they have also enabled other trends whereby providers rely less on backbones, as discussed below.
The development of IXPs has benefited backbone providers by improving the quality of service and reducing traffic carrying costs. However, the benefits rely on roughly even traffic ratios between the peers, as any significant traffic imbalance jeopardizes the peering principle that each party should bear a fair share of the costs of carrying the traffic.
31
4.3
Internet service providers (ISPs)
ISPs have reacted to the changing mix of traffic described above in several ways, each of which reduces their reliance on traditional backbones for transit services:
The rise in peer-to-peer usage has dramatically increased the traffic that ISPs‟ customers are exchanging between each other (acting in the role of both consumers and suppliers of content). This has led to what is referred to as secondary peering.
At the same time, the rapid growth of centralized video content and traffic in sites such as YouTube has led ISPs to connect directly to content providers. This typically results in a form of interconnection known as paid peering.
We describe each of these changes in turn.
First, the changing mix of traffic led the largest ISPs to reevaluate their need to purchase IP transit from upstream backbone providers. Traditionally, IP transit was used as a means to access traditional content providers, which may be connected to the upstream transit provider or another backbone. On the other hand, IP transit is not necessary to access the customers of another ISP, particularly if that ISP is located in the same IXP. As a result, many ISPs began to peer with one another, as shown in Figure 4.2.
Figure 4.2: The principle of secondary peering [Source: Analysys Mason]
This had two impacts for the ISPs. First of all, it lowered the cost of transit, because the traffic exchanged via secondary peering no longer needed to be sent through the existing transit relationship(s). As transit is based on a measure of traffic volume, this lowered costs. One estimate is that up to 40% of a typical ISP‟s traffic in the US was peer-to-peer that could be exchanged using secondary peering, with a corresponding reduction of cost.32
In addition, secondary peering reduced latency for peer-to-peer traffic, as the traffic passed directly between the ISPs rather than via one or
32
The Evolution of the US Internet Peering Ecosystem, William B. Norton, Equinix White Paper 11/19/2003. Norton was the Co-Founder and the Chief Technical Liaison for Equinix. (See http://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdf)
Backbone 2 Backbone 1 ISP 1 ISP 2 Peering Transit Transit Secondary peering IXP
more upstream backbones. ISPs that have engaged in such secondary peering are sometimes said to purchase only partial transit from backbones – namely only those destinations that they cannot access via peering.
Second, the rapid growth of centralized video content and traffic in sites such as YouTube has led to a different form of interconnection known as paid peering. The genesis of paid peering between an ISP and a content provider is similar to that of secondary peering between two ISPs, namely it is an attempt to directly connect in order to reduce transit costs as well as the latency of transmissions. However, there is a fundamental difference resulting from the ratio of traffic involved. When two ISPs peer, it is likely that they will exchange roughly equal amounts of traffic, as their end users are likely to have an equal propensity to generate and consume each other‟s traffic, be it peer-to-peer or interactive video games or just email. On the other hand, when an ISP and a content provider peer, the traffic ratios will not be balanced, as for instance the request for a video consumes very little bandwidth while the video itself takes far more bandwidth: we estimate that the streaming of a three-minute video on YouTube generates around 35 times more downlink traffic than uplink traffic.33
As a result, the ISP will deliver far less traffic to the content provider than it carries in return.
When the imbalance of traffic between service providers rises above a certain traffic ratio, the convention is that the generator of the traffic pays the provider that carries the traffic for the imbalance, in order to cover the cost of capacity needed. Nonetheless, paid peering is advantageous for both parties. Previously, both the ISP and content provider used a transit connection to a backbone to exchange traffic, and each party paid for the traffic that they sent and received. Now, by directly interconnecting, they each lower their transit payments accordingly. Indeed, in this case the content provider would have paid transit fees for the traffic that it delivered to the backbone, while the ISP would have paid similar transit fees for receiving that traffic from the backbone. With paid peering, the content provider pays the ISP directly for delivering the traffic, probably at a rate lower than it would have paid for transit.
As a result of these actions on the part of ISPs to lower their transit fees, interconnection has begun to look like Figure 4.3 below. In order to reduce transit fees, ISPs have begun peering with one another, typically at IXP locations for efficiency. At the same time, content providers have begun to interconnect directly with ISPs, using paid peering to compensate for the imbalance of traffic across the connections. This not only further lowers the ISPs‟ transit fees, but also results in fees from the paid peering. These new interconnections come at the expense of backbones, which, as a result, end up selling less transit to the ISPs and content providers.
33
The video streaming request sent by a user to YouTube generates around 430 kbytes of uplink traffic, while the resulting video stream generates around 15 Mbytes.
Figure 4.3: Interconnection possibilities that emerged in the past 15 years [Source: Analysys Mason]
Many “route around” options have emerged for ISPs and content providers to carry Internet traffic while avoiding Internet backbone transit costs. These options include secondary peering between ISPs and partial transit agreements. Traffic imbalances have also led to the development of paid peering, which is advantageous for ISPs and content providers.
4.4
Content providers
Paid peering with ISPs is only one of the means that content providers are using to lower their costs – efforts that go hand in hand with ways to reduce latency and thereby improve the experience for their users. Another technique used for this purpose is called web caching. A web cache used by an ISP stores a copy of a web page or video requested by an end user. The ISP can then use the cache to satisfy subsequent requests for the same content, instead of reloading it from the original source. In other words, if 500 users view the same popular video, it can be downloaded once from the original content provider over the transit connection, and the next 499 times it will be viewed from the web cache within the ISP‟s network, significantly lowering the amount of traffic over the transit connection, and therefore reducing transit payments.
Web caching may be used by ISPs, as described above, but this technique is more typically used either by third-party content delivery networks (CDNs) or by larger content providers, which are effectively building their own delivery networks in order to deliver content closer to (or from within) the ISPs‟ networks. Commercial CDNs effectively compete with backbones in order to deliver their clients‟ content directly to ISPs. CDNs ensure that the content of their clients is distributed to end users with good performance, independently of the location of the end user. To fulfill this role, a CDN typically owns a network of caching servers, connected by fiber, which feed copies of content to ISPs
Content provider & aggregator Backbone 1 End-user Backbone 2 Backbone 3 ISP 4 ISP 3 ISP 2 IXP Internet Access Transit Peering Secondary peering
Partial transit Paid peering ISP
Single entity
at a limited number of points of presence (PoPs) where networks interconnect.34
The largest CDNs include Akamai (founded in 1998) and Limelight (2001). According to one estimate, CDNs deliver close to 10% of all Internet traffic.35
See Figure 4.4 below for a stylized diagram of how a CDN interacts with the other players in the Internet ecosystem. The use of CDNs to carry content traffic effectively reduces the traffic to be carried by the backbone provider, and avoids any traffic imbalance issues. However, at the same time it reduces the transit revenues of backbone providers.
Figure 4.4: Structure of a content delivery network [Source: Analysys Mason]
Content providers themselves may use a similar configuration to build a content delivery network for their own content. For instance, Google has embarked on a Google Global Cache project with similar goals to a CDN, in this case to locate Google servers within ISP networks.36
The difference from a CDN is that Google servers are dedicated to Google content. Other content providers such as Microsoft are also building such networks to deliver content, such as Microsoft‟s video games. The difference in the traffic carried over the backbone can be significant: Arbor Networks estimates that Google traffic represents at least 6% of global Internet traffic, and more than 60% of that traffic is delivered to ISPs over direct connections rather than via third-party providers.37
Figure 4.5 shows how the percentage of Google traffic using private peering has increased in recent years.
34
CDNs usually try to connect directly to ISPs for cost and performance reasons, but may need in some instances to buy transit from backbone providers (when they do not own the fiber network), and also buy transit from ISPs (when they cannot get free peering from large ISPs).
35
ATLAS Internet Observatory, 2009 Annual Report, Arbor Networks, Inc., University of Michigan, Merit Network, Inc., Pre-Publication Draft, page 15.
36
See http://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdf
37
Source: “How Big is Google?”, Craig Labovitz, http://asert.arbortnetworks.com/2010/03/how-big-is-google/
Content provider Backbone Content Delivery Network End-user Backbone network B Backbone network A Regional ISP Mobile operator
Aggregation & Access Aggregation & Access
Regional ISP
Figure 4.5: Percentage of Google traffic using direct peering [Source: Arbor Networks38]
Even smaller content providers can also use technology to lower their costs, avoiding not just transit charges, but also the use of CDNs. For instance, Vudu delivers high-quality video on demand to end users‟ TVs via a dedicated set-top box that is external or integrated into a TV or Blu-ray player. Each device has a large hard drive, which contains the first few seconds of each title in order to enable the title to begin as soon as it is chosen. The rest of the chosen content is pieced together via an Internet connection to the device, using peer-to-peer networking from other Vudu devices that have stored that content. As a result, a user is able to watch high-definition movies with no delay with as little as a 4Mbit/s broadband connection. In this way, Vudu uses its customers‟ own broadband connections to deliver content to other users in lieu of a more expensive CDN or backbone service.
As a result of the volume of traffic that they deliver, and the associated cost of delivering that traffic, content providers have a strong incentive to build out their networks in order to reduce their transit requirements and deliver the traffic directly to the ISP network, if not from within the ISP‟s network. As a result of the increasing monetization of content, providers also have the means to engage in such network build-outs. But even if CDNs are increasingly used or built to deliver Internet content to ISPs, in particular to deliver the growing share of video traffic for which caching is appropriate, backbones and/or ISPs still provide significant infrastructure to deliver the content. As explained earlier, CDNs typically distribute their caching servers into specific PoPs to connect with backbones and ISPs‟ networks. The selection of these points by the CDNs is optimized in terms of coverage, i.e. CDNs will primarily set up PoPs in regions concentrating significant amounts of Internet traffic. However, backbones and ISPs then ensure the delivery of content to the end users from these PoPs to the end users, regardless of final location.
38 http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/ 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Ap r-0 8 J u n -0 8 Au g -0 8 O c t-0 8 D e c -0 8 F e b -0 9 Ap r-0 9 J u n -0 9 Au g -0 9 O c t-0 9 D e c -0 9 F e b -1 0 Pe rce n ta g e
Overall, the traffic carriage effort supported by backbones and ISPs is significant in comparison with the CDNs. CDNs typically connect at tens of PoPs: for instance, Limelight notes that in 2010 its network consisted of 75 PoPs where it interconnected globally, in order to deliver traffic to more than 900 ISPs and backbones. This is broadly similar to the number of PoPs that each single major backbones typically possesses.39
However, the backbones and/or ISPs carry the Internet traffic further into the network, in
