Key findings and conclusions:
Product Category:
Edge and Core Routers
Vendor Tested:
Cisco Systems
Products Tested:
Cisco 12000 and 10000 Series Internet Routers
and Cisco 7500 Series Router
LAB TESTING SUMMARY
REPORT
April 2002
Report 240402
• Cisco Nonstop Forwarding with Stateful Switchover drastically reduces mean time to repair (MTTR)
• Delivered zero route flaps with BGP, OSPF, IS-IS and static routes during recovery
• Maintained state and showed zero link flaps during recovery for multiple protocols including Frame Relay, ATM, HDLC, PPP and Ethernet
• Cisco 12000 experienced no packet loss during route processor switchover, resulting in uninterrupted voice and video transmission
isco Systems engaged Miercom to conduct analysis and performance validation tests of its Cisco Nonstop Forwarding (NSF) with Stateful Switchover (SSO) capabilities, which are part of Cisco Globally Resilient IP Technologies. The goal of testing was to validate switchover times during a route processor failure on three Cisco edge routers: the Cisco 12000 Series Internet Router, Cisco 10000 Series Internet Router and Cisco 7500 Series Routers, each of which displays different strengths at the edge. Tests were conducted with 600 interfaces; 2,000 OSPF or IS-IS routes; and 65,000 BGP routes on each router .
We measured mean time to repair (MTTR) times in seconds on all three routers during simulated hardware and software failures. Results showed that the Cisco 12000 Series Internet Router recovered instantly with zero packets lost; the Cisco 10000 Series Internet Router recovered in an average of 1.63 seconds, and the Cisco 7500 Series Router recovered in an average of 6 seconds. (See chart below for comparisons between Cisco NSF with SSO and Cisco’s previous best recovery mechanismRoute Processor Redundancy+.)
C
Recovery Time Comparisons:
Cisco Route Processor Redundancy+
versus Cisco NSF with SSO*
*A comparison of average mean times to repair (MTTR) from system failure and begin passing traffic. Cisco NSF with SSO is compared to Cisco Route Processor Redundancy+, the best form of redundancy available on the Cisco 12000 and 10000 Series Internet Routers and 7500 Series Routers prior to the release of Cisco NSF with SSO.
MTTR (seconds)
40 30 20 10 0
37
0
35
Cisco 12000
1.63
30
6
Cisco 10000 Cisco 7500
About the testing …
The test bed was constructed in a network with sufficient port density and variety to demonstrate that Cisco NSF with SSO works simultaneously across a wide array of interfaces and protocols. We conducted three series of tests on each of the following:Cisco 12000 Internet Router, Cisco 10000 Series Internet Router and Cisco 7500 Series Routers. Cisco NSF with SSO was configured on each tested router and on all adjoining routers. It was disabled only when running baseline tests using Route Processor Redundancy+.
In all tests, an Ixia 1600 was configured to transmit bi-directional traffic over all the links and ports of the Unit Under Test (UUT). The UUT was always configured for Cisco NSF with SSO. At the start of each test, the Ixia route advertisements, traffic flows and pings were started.
Once traffic was verified as flowing, a failure was induced in the primary route processor and its impact observed on the traffic flows, links and routes. At the conclusion of the test, the primary route processor automatically restarted and established itself as the secondary processor.
All tests were first conducted with IS-IS, and upon completion, the routers were reconfigured and all tests rerun using OSPF. As a second verification, a ping was set up from Enterprise Access1 (ENT1) to Core1 via the UUT. The ping was monitored during the switchover to measure the seconds of packet loss, One ping per second (per/sec) was sent. Link and route flaps were monitored through observation on the ENT and Core router consoles. In the final series of tests, the UUT was reconfigured to disable Cisco NSF with SSO and operate in Route Processor Redundancy+ mode. Tests were then re-executed, and the recovery time of a ping from ENT1 to Core1 via the UUT was observed and compared to recovery times that were observed when running Cisco NSF with SSO.
Cisco 12000 Series Internet Router:
The service provider distribution layer (Core1, Core2) consisted of two Cisco 12008 Internet Routers.The enterprise access layers (ENT1, ENT2, ENT3) consisted of two Cisco 12008 Internet Routers and a Cisco 7200 Series Router. The UUT was a Cisco 12410 Internet Router. On layers 1 and 2, ENT1, a Cisco 12008 Internet Router connected 81 T1s, 24 DS0s and 12 DS3s via three channelized DS3 links and a channelized OC-12 link to the UUT, which in all tests was configured for Cisco NSF with SSO. ENT2 (a Cisco 12008 Internet Router) connected 3 x OC-3 ATM links with 125 Permanent Virtual Circuits (PVCs) each, for a total of 375 sub- interfaces. ENT3 (a Cisco 7200 Internet Router) connected 81 T1 links and 24 DS0 links via 3 channelized DS3 links to the UUT. There were 2 x OC-12 POS connections from the Core1 Core2 layers to the UUT. The total port count to the UUT was 599 ports, plus 1 loopback interface. The ATM interfaces were running aal5snap encapsulation. On the serial connections, 210 connections were using PPP and 12 were using HDLC, both with “keepalives” enabled. If SSO failed to reestablish and control the serial links, the timers caused link flaps. In addition to the routes required to pass traffic on all interfaces, an additional 65,000 overlapping BGP background routes were injected into Core1 and Core2 from the Ixia, forcing the processor to handle route re-convergence in a failover condition. The UUT handled 65,000 overlapping background BGP routes; 2,000 IS-IS or OSPF routes; and 1,311 BGP routes. In addition to the traffic streams, video and voice over IP (VoIP) streams were also set up to traverse the UUT. During the failover, the video stream was closely monitored for any disruptions or degradation of quality. An interactive conversation was also held on the VoIP system to determine if there was any disruption or degradation during failover.
Cisco 10000 Series Internet Router:
The Core consisted of a 10005 router, while a 10008 Internet Router represented the enterprise access layer (ENT1). The UUT was a 10008 Internet Router. On layers 1 and 2, 622 interfaces were connected to the UUT via 5 channelized DS3s from the ENT1. These enterprise interfaces (ENT1) were configured as 504 T1s using PPP encapsulation and 112 T1s using Frame-Relay encapsulation. Six additional clear channel DS3 interfaces had HDLC encapsulation. The Core router was connected to the UUT using a Gigabit Ethernet interface. In addition to the routes required to pass traffic on all interfaces, an additional 65,000 BGP background routes were injected into the core from the Ixia, forcing the processor to handle route re-convergence in a failover condition. The UUT handled 65,000 background BGP routes; 2,004 ISIS or OSPF routes; and 1,232 BGP routes.Cisco 7500 Series Router:
ENT1 consisted of a Cisco 7513 Internet Router while Core1 and Core2 each included two Cisco 7513 Internet Routers. On layers 1 and 2, a total of 576 interfaces were connected to the UUT via an ATM OC-3, channelized T3 and E3 connection from ENT1. The enterprise interfaces were configured as 336 ATM PVCs and 112 serial connections from the T3 link and 128 serial connections from the E3 link. The core was connected to the UUT via 4 x OC-3 POS interfaces, two from each router. In addition to the routes required to pass traffic on all interfaces, an additional 65,000 BGP background routes were injected into the core (32,500 to each router) from the Ixia, forcing the UUT processor to handle route re-convergence in a failover condition. The UUT handled 65,000 background BGP routes; 2,000 ISIS or OSPF routes; and 1,152 BGP routes.Test-bed Setup
Copyright © 2002 Miercom Core/Edge Routers Page 2
Copyright © 2002 Miercom Core/Edge Routers Page 3 Background
Cisco Software Release 12.0(22)S with Cisco NSF with SSO provides full and automatic recovery from catastrophic route processor failures in routers equipped with redundant route processors (RPs). The technology is now available on Cisco 12000, 10000 and 7500 series edge and core routers, which were tested in this review. (See “About the testing,” page 2, for a detailed description of the test methodology and implementation.) Cisco NSF with SSO will be supported for Cisco 6500 and 7600 routers in future Cisco IOS Software releases.
Cisco NSF with SSO allows uninterrupted IP packet forwarding while the IP routing protocols converge in the background. It is available for BGP, OSPF and IS-IS routing protocols. Tests were conducted using all three.
Benefits: The major benefit of Cisco NSF with SSO is dramatically reduced downtime in the event of route processor failure, which can drive
down operational expenses and reduce impact on in-service traffic. In addition, the higher levels of availability facilitate the deployment of IP-based services, such as voice, video and other “mission- critical” services, including virtual private networks.
A proper deployment of Cisco NSF with SSO can result in the elimination of link and route flaps, packet loss and associated downtime in the event of a failure in hardware or software.
Target environments: Cisco NSF with SSO is typically deployed in large IP networks with “single points of failure.” This includes service provider edge access sites, which supply connectivity to enterprise customers; enterprise edge environments with high-speed links; and businesses providing crucial services with uninterrupted access, such as Web hosts, commercial-class voice, stock trading, etc. In these environments conventional redundancy techniques, such as multihoming, are unavailable or too expensive to implement.
The Advantages of Single Route Processor versus Dual Route Processor
Single Route Processor Dual Route Processor
Failure detection:
Trouble-shooting required to isolate router failure Immediate failover to secondary RP upon hardware or software failure in primary RP
Hardware replacement:
Part replacement requires dispatching a technician and possibly reconfiguring the network if router is replaced
Standby router assumes role of primary router, eliminating any downtime for repairs
System initialization:
Cisco IOS must be loaded on the route processor and – in the case of a distributed system - on any of the various line cards within the chassis. The Cisco IOS configuration file must be parsed, and memory allocation for any configured services must occur.
Cisco IOS software is pre-loaded on standby RP.
Checkpoint facility provides initial exchange of configuration file upon start-up, and subsequent configuration changes are communicated to the standby router as they occur. Line cards in distributed system never reset.
Link re-establishment:
All link-layer protocol state is lost during failure; Layer 2 link flaps may require significant work to re-establish state at higher layers of the OSI model.
Link-layer protocol information shared between primary and standby RPs is constantly updated; link state maintained across an RP failure; link re-establishment is not required Routing convergence:
No packet forwarding until routing protocols re-converge Dependence upon converged routing protocol to maintain packet forwarding during RP switch-over eliminated Resumption of forwarding:
Packet forwarding resumes only after all routing protocols are converged and best-path calculations performed
Packet forwarding is uninterrupted before, during and after
RP switchover
Copyright © 2002 Miercom Core/Edge Routers Page 4 Performance Results
Cisco Systems contracted Miercom to test the IP resiliency features in Cisco IOS Software Release 12.0(22)S on the Cisco 12000 Series Internet Router, Cisco 10000 Series Internet Router and Cisco 7500 Series Router. Testing focused on Cisco NSF with SSO features, which are designed to greatly decrease MTTR.
During testing, we created a simulated network consisting of a network core, unit under test (UUT) and an enterprise edge network. On all platforms, 65,000 BGP and 2,000 OSPF or IS- IS routes were injected into the UUT via the core network. From the enterprise, about 600 links were made via ATM and serial connections. Traffic was passed bi-directionally between the core network and enterprise devices through the UUT. A failover was then induced in the primary RP of the UUT, and the effects on the traffic and neighboring routers were observed.
On all three routers, results showed a dramatic improvement in failover times when compared to Route Processor Redundancy+ mode, the
best form of redundancy available on these Cisco routers prior to the introduction of Cisco NSF with SSO. All three routers had MTTR times over 30 seconds when running in Route Processor Redundancy+ mode, but MTTR times ranged from zero seconds to a high of only 6.57 seconds with Cisco NSF with SSO. (See graph, page 1.)
The goal of testing three Cisco routers was to demonstrate the Cisco NSF with SSO would work simultaneously across a wide variety of interfaces and protocols in a network with no packet loss, route flaps or link flaps.
Cisco 12000 Series Internet Router: Cisco NSF with SSO dramatically reduced failover times compared to those observed using Route Processor Redundancy+ mode. Specifically, failover occurred with no interruption to data streams and with no packet loss. In addition, an interactive VoIP call and video stream showed no discernable disruption of either voice or video communications during the RP switchover.
Furthermore, neither the enterprise nor the core router reported route or link flaps. (Tables 1 and 2 illustrate specific results when running IS-IS and OSPF routing protocols.)
Table 1: Cisco 12000 MTTR Using IS-IS Routing Protocol
Run Failure Type Packet Loss Video Phone Link Flaps Route Flaps
1 Software induced 0 No impact - 0 0
2 Software induced 0 No impact - 0 0
3 Software induced 0 No impact - 0 0
4 Software induced 0 No impact No impact 0 0
5 Software induced 0 No impact - 0 0
6 Pulled card 0 No impact - 0 0
7 Software induced 0 No impact - 0 0
Table 2: Cisco 12000 MTTR Using OSPF Routing Protocol
Run Failure Type Packet Loss Video Phone Link Flaps Route Flaps
1 Software induced 0 No impact - 0 0
2 Software induced 0 No impact - 0 0
3 Software induced 0 No impact No impact 0 0
4 Software induced 0 No impact - 0 0
5 Pulled card 0 No impact - 0 0
6 Software induced 0 No impact - 0 0
Copyright © 2002 Miercom Core/Edge Routers Page 5 Performance Results – continued
Cisco 10000 Series Internet Router: Results of testing showed a dramatic improvement in failover times when Cisco NSF with SSO was deployed. The Cisco 10000 Series Internet Router demonstrated an average failover time of 1.63 seconds with Cisco NSF with SSO, compared to an average 35-second failover time when tested in Route Processor Redundancy+ mode. In addition, the enterprise and core routers reported no route or link flaps during any of the tests in which Cisco NSF with SSO was enabled.
Cisco 7500 Series Router: Results of testing also showed a dramatic improvement in failover times when using Cisco NSF with SSO. The Cisco 7500 Series Router demonstrated an average failover time of 6.0 seconds, compared to an average 30-second failover time when tested in Route Processor Redundancy+ mode. No route or link flaps were reported during any of the tests.
Cisco, Cisco IOS Software, Cisco Systems, and the Cisco Systems logo are registered trademarks of Cisco Systems, Inc., and/or its affiliates in the U.S. and certain other countries.