Perry T. Eidson
Technical Operations Communications Engineer Emory University Technical Services
404-727-0248
Internet2 Fall 2008 Member Meeting October 14, 2008
Founded in 1836 at Oxford, Georgia by
the Methodist Episcopal Church
Expanded to Atlanta Campus in 1915
Through Generosity of Asa Candler, Founder of Coca-Cola
2008 Enrollment – 12,755
Ranked 18th 2008 U.S. News and World
Report Overall – Individual Schools May Rank Higher
Nine Schools
Emory College – 1836 Oxford College – 1836 School of Medicine – 1854 School of Nursing – 1905
Candler School of Theology – 1914 School of Law – 1916
Roberto C. Goizueta Business School – 1919 Graduate School – 1919
Rollins School of Public Health – 1990
Affiliates Include Carter Center and Yerkes National
Total Employees, Emory and Emory
Healthcare – 23,733 (largest private employer in Atlanta area)
2008 Combined Budget - $3.0 billion
Endowment as of August 31, 2007 - $5.6
billion (17th among U.S. universities)
$411.2 million in sponsored research for
Emory Healthcare Largest Health Care
Provider in Georgia
4 Owned Hospitals With Major Presence
at Grady Memorial Hospital
(Fulton-DeKalb Hospital Authority), VA Medical Center Atlanta and Children’s Healthcare of Atlanta-Egleston
Major and Minor Emory Clinic Locations
Major Data Center Supporting Academic
and Healthcare Needs
◦ IBM z/890 Mainframe Running z/OS
◦ IBM P590 Upgrading to P595 AIX Multiprocessor Partitionable Machines
◦ 6 Egenera Bladeframes
◦ HP C-class Blade Servers
◦ VAX/VMS Machines
◦ 750 Terrabyte SAN Network Growing to 1+ Petabyte
◦ Cytrix Environment for Healthcare
Several Schools and Affiliates Have Their
Own Networks/Servers That Connect to University Backbone
Dual 10G Backbone – Cube Core
Topology
44k+ data ports
32K+ voice ports
Virtual Core Networks and Routers
Provisioned Through Large Routers/ Firewalls
Metro WAN – 2002
◦ 527 Megabits to Major Locations
◦ Mix of T-1, 10 Meg NMLI, 100 Meg NMLI
◦ Additional Locations Not Included in Fiber Project Due to Minimal Requirements or Excessive Distance Not Cost Effective
◦ Mixed Vendors, Routers Everywhere
Evolved From Previous 5 Legacy
Organizations – Separate Billing/Contracts
Network Communications Created in
2000 Combining 5 Separate Entities
◦ University Telecom
◦ University Network Operations
◦ Emory Clinic Telecom
◦ Hospital Telecom
◦ Healthcare IS Networking
Combined Entity Had Size, Scope,
Expertise and Financial Wherewithal to Tackle DWDM/Fiber Project
2001 Began Evaluation of Equipment and
Fiber Vendors
2002 Tested Systems in Lab, Identified
Dark Fiber Vendor
2002 Awarded Contracts
◦ Multi-month Fiber Construction
2003 Fiber Complete/Equipment Installed
First Cut-over Due to NMLI Failures
◦ Added 2 Hops per Leg to Enable Cut to 1 Gig
DWDM Circuit Prior to Core Upgrade From 100 Meg to 1 Gig
◦ Latency Dropped Enough So End User
Customers Called NetCom to Ask What Changed – Much Better Terminal Response Visible!
First Design Called for Three Separate
Rings
◦ Traffic Separated by Function
Core Traffic – HIPPA and FERPA Regulations
Border Traffic – Access to Outside Entities
◦ Rings Took Separate Paths
◦ SONET Ring for TDM Links
Final Design – All Rings Took Same Path
and Met SOX at Telecom Hotel (56 Marietta)
Cox Hall NDB Briarcliff Campus ECLH Grady 56 Marietta CMONT* North* Main Campus *Not on DWDM Rings – Campus Fiber
At Cutover, Amortized Monthly Fiber
Ring Cost Less Than Cost of Leased Circuits
Bandwidth Increased From 527 Meg to 17
Gig
Latency Decrease and Bandwidth Increase
Resulted In Remote Locations Moving From WAN to LAN Environment
Moved From Routers at Most Locations
to Central Routers and Switches Everywhere Else
Moved From Physically Separate Cores to
Virtual Cores Utilizing 10 G Duplicated Physical Pipes – Large Core Routers
Providing VRF
Moved From Multiple Physical Firewalls to
Address Power, Space and Cooling
Constraints in Primary Data Center
Issues With Off Site Tape Backups
◦ Extended Period of Time to Recover
◦ Support of Healthcare and Research Require
Immediate Recovery
Multi-year Effort to Identify Appropriate
Location
◦ Location Relative to Main Campus
◦ Cost
◦ Network Availability
Multi-Phase Project
◦ Secure Site and Mirrored Critical Data
◦ Create Business Continuity Infrastructure
Hot-Standby
White ST
Former A&P Meat Packing Warehouse
◦ Bunker Like Construction
Dual Commercial Power Feeds
N+1 Diesel Generator Sets For Backup
Power
Redundant Cooling With Well as Primary
Source and City Water as Backup
Multiple Air Conditioning Units
Multiple Power Distribution Units
Fiber Paths to Leased Space Kept Separate
by at Least 8 Feet
Separate DC Plants in Emory Leased Space
Separate DWDM Nodes – 1 Each on Core
and Border Rings
Dual Master Switches Cross Linked via Fast
Spanning Tree for Immediate Recovery in Event of Link Failure
DC-A DC-B D WDM Cor e D WDM Bor der A C-DC Data Switch A C-DC Data Switch
Core Fiber Ring
Border Fiber Ring
Current Network – 30 Gig to SDC
◦ Dual 10G IP Links From Separate Core Routers
Separate Rings – Opposite Directions
A & B DC Feeds From Separate DC Plants
◦ Dual 4G Fiber Channel – Transparent Transponders
Separate Rings – Opposite Directions Bonded Into Single 8G Link
Seamless Fallback/Recovery If One Side Impaired
◦ Private 1G Link for PACS System
Special Need for Flat Non-Routed Network
IP Latency Well Under 1 Millisecond
Fiber Channel Latency – Microseconds
◦ Calculated Latency Based on Speed of Light
◦ Actual Response Slightly Slower – But Only
Just Slightly
◦ SRDF Read/Writes Look Same at Remote
Center
150 Terabytes Storage Currently Online
◦ 130 Terabytes IP Based
◦ 20 Terabytes Fiber Channel
PACS System
◦ Active-Active Between Campus Data Center
and SDC
◦ Sun Hardware – Multiple Servers
Near Future (end of year)
◦ EMC DMX4 at Campus
Current DMX3s Moved to SDC
◦ P595 Upgrade for Campus
Current P590 Moved to SDC
◦ Integration to Storage Grid Complete
2 Locations on Campus for Storage Plus SDC
◦ Schools Independent IT Departments Using
Major Outpatient Clinic
New Hospital
Link to Oxford Campus – 50 Mile Link
From 56 Marietta Telecom Hotel
Internet Links
◦ 10 G to Internet2
◦ 1 G Link to AIX Public Peering Point
◦ 2 – 1 G Links to Separate Commercial
59 Exec Park
Cox Hall
59 Exec Park
NDB Cox Hall
Combination of DWDM and Campus
Fiber Enables CDP
DWDM Bandwidth and Latency Brings
Remote Locations “On Campus”
Criticality of Data Sets
◦ Multi-Generational Studies Both Human and
Animal Irreplaceable
DWDM Growth Has Been Explosive
◦ 17 Gig at Install – 32X Displaced Circuits
◦ 114 Gig Currently – 6X Installed Capacity