Ken Rabedeau, CTO Energy Systems Integration Division
Sept 8, 2011 UTC Region 9, Newport, OR
Design and Operational Considerations
for Electrical Grid Protection Systems Built on
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
Current TDM Environment
A Collection of Hard Mapped Circuits
SONET
OC48
SONET
OC3
SONET
OC12
SONET
OC3
SONET Cross-Connect Hard external patchEMS
System 1
EMS
System 2
TPR TPR Protected SONET Cross-ConnectSONET
OC3
Current TDM Environment
A Collection of Manually Hard Mapped Circuits
SONET
OC48
SONET
OC3
SONET
OC12
SONET Cross-Connect Hard external patch Device Config to Select PathEMS
System 1
EMS
System 2
TPR TPRX
Node or Fibre Maintenance Temporary re-route Protected SONET Cross-ConnectCurrent TDM Environment
Recap
Design-wise:
• Heavy Engineering effort up front, circuit by circuit
• Point-to-point, fixed path, 1 failover path maximum
• Dedicated Network Resources
• Stranded Capacity
• Consistent performance
Operationally:
• Blind: no end-to-end, pro-active monitoring capabilities
• Binary: works or doesn’t
• Difficult and time consuming to troubleshoot
• Complex and onerous design documentation
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
Why MPLS?
No Compromises!
1.
No single point of failure
2.
Applications Security – VPN’s
3.
QoS – Engineered Prioritization of data streams
4.
End-to-End Management
5.
Interface breadth
(Ethernet, RS-232, x.21, T1, fractional T1, DS0, E&M, G.703, C37.94 and more)
MPLS Enables
The Future IP-Centric Communications
Separated service networks
Each service has its own network A mix of networking technologies
Optimization Simplification
Converged service network
All services in one network
Network transformation to provide the required communications foundation for the emerging smart grid
Traditional Drivers & Challenges
Transporting TDM over Packet Networks
•
Network Operator Drivers
• Achieve lower cost base transport per T1
• Avoid proportional scaling of costs with number of T1s needed • Convergence over single packet network for all services
• Future-proof
• Lower OPEX with fewer networks to manage
•
Network Operator Challenges
• Operational transition from a Layer 1 network to a IP/MPLS network • New packet-orientated equipment/design concepts
• Statistical nature/QoS
• OAM differences between Layer 1 (SONET/PDH) network to IP/MPLS networks
• Network synchronization
• No synchronous interface to transport timing
• Use of new evolving timing over packet technologies
IP/MPLS Environment
• Operationally:
• Capable of end-to-end, pro-active monitoring (SLA)
• Historical and real-time path information is a necessity for management and
troubleshooting
• Path asymmetry is a risk
• Ability to bridge multiple systems into homogeneous network Δt kV 7750 SR IP/MPLS TPR 7705 SAR 7710 SR Substation TPR 7705 SAR Substation E&M RS-232 Ethernet G.703 C37.94
• Design-wise:
• Design once methodology (Service Catalogs)
• Multiple failover backup paths (FRR) • Priority Access to Shared Resources and
Engineered performance (H-QoS / QoS / RSVP-TE)
• Efficient Capacity Utilization
E&M RS-232 Ethernet G.703 C37.94
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
??ms ??ms
SONET
OC3
Components of Latency
Where is the Biggest Culprit?
SONET
OC48
SONET
OC3
SONET
OC12
TPR TPR 16ms <10ms? Network Latency1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
Fine Tuning Latency in IP/MPLS Routers
Packet Switched Network (PSN) GigE GigE Network• Fixed delay (physical limits)
• Packet transfer delay based on link speeds and distances from end to end
• Variable delay (design)
• the number of and type of switches
• queuing point in the switches
• QoS is key to ensure effective service delivery
DS1/E1 LIU DS1 / E1 Data Si g Packetization Packetization
• As TDM traffic from the Access Circuit (AC) is received, it is packetized and transmitted into the PSN
• Two modes of operation:
• CESoPSN (RFC5086) for structured nxDS0/64k channels • SAToP (RFC4553) for unstructured T1/E1 Access Circuit
TDM Packets moving in this direction
• Synchronization and timing is reconstituted DS1/E1 LIU Data Si g Jitter Buffer Playout • TDM PW packets are received from the PSN and stored into its
associated configurable jitter buffer
• Play-out of the TDM data back into the AC when it’s at least 50% full
DS1 / E1 Access Circuit
Example End-to-End Latency
Calculation for a TDM PW
Packet Switched Network (PSN) DS1 LIU DS1 LIU DS1 Data Si gPacketization GigE GigE Data
Si g Jitter Buffer Packetization PD Network ND Playout JBD DS1 Access Circuit Access Circuit
TDM Packets moving in this direction
The total end-to-end latency is calculated by summing the packetization delay (PD), network delay (ND) and jitter buffer delay (JBD) as shown here:
Total Latency = PD + ND + JBD
– e.g. PD of 2 ms (16 T1 frames/packet), ND of 3 ms, JBD of 4 ms (JB size 8 ms) Total Latency = 2 + 3 + 4
= 9 ms
TDM Packetization over IP/MPLS
Latency Characteristics
The two principal services are used for structured and unstructured connections
CESoP — Circuit Emulation Service over Packet
Provides fractional services (nxDS0)
SAToP — Structure Agnostic TDM over Packet
Provides unstructured T1/E1 services
Two services are collectively referred to as Circuit Emulation Services (CES)
Services are transported over an MPLS Network using Pseudowire point-to-point tunnels
CES IWF CES IWF
TDM
The CES Interworking Function (IWF) applies the proper encapsulation to the nxDS0
or T1/E1 traffic
Pseudowires (PWE3) identify the specific CES
connection
MPLS Tunnels transport traffic from point A to B
Flexible configuration of buffers within the CES IWF allows control
of packetization, latency and jitter which meets the requirements for TDM services.
MPLS Tunnel
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design & Latency Recap
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
Latency Optimization Link/Path Design
Addressing Variable Delay
Unnecessary packet processing by IP/MPLS routers will add latency. MPLS traffic engineering capability enables deterministic and predictable performance.
Unnecessary packet processing by IP/MPLS routers will add latency. MPLS traffic engineering capability enables deterministic and predictable performance.
Transport Domain Fiber / SONET / Microwave / DWDM IP Domain Service Aggregation Routers IP Domain Core Service Routers IP Domain Service Aggregation Routers
Fine Tuning Latency in MPLS Routers
Recap
Packet Switched Network (PSN) GigE GigE Network • Fixed delay• Packet transfer delay based on link speeds and distances from end to end
• Variable delay
• the number of and type of switches
• queuing point in the switches
• QoS is key to ensure
DS1/E1 LIU DS1 / E1 Data Si g Packetization Packetization Access Circuit DS1/E1 LIU Data Si g Jitter Buffer Playout DS1 / E1 Access Circuit 2.5ms latency is feasible Increase in bandwidth Decrease latency
Decrease in jitter buffer
Decrease latency Decrease jitter tolerance
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
Flexibility in Timing is a Necessity
External Synchronization
Line Synchronization
Timing over Packet
(Adaptive Clock Recovery, IEEE 1588v2 PTP, NTP) Synchronous Ethernet Synchronous Ethernet Client PRC L2 or L3 PSN PDH, SDH, NTR, I-frame L2 or L3 PSN
Reconstituting the TDM signal demands highly accurate clocking capabilities from the hardware. Flexibility to work with a variety of clocking sources and modes is a significant factor to implementation.
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
MPLS Service Catalog for Teleprotection
Pre-define Services and Utilize Templates
MPLS Payload Size
Jitter Buffer Size
RSVP-TE SAToP (RFC4553) CESoPSN (RFC5086) Network Delay Playout Buffer Size H-QoS QoS FRR
Teleprotection Service over MPLS (Service Catalog)
Design Once and Replicate
Synchronc-piization
pe VL
L
Teleprotection Key Requirements End-to-end latency less
than 16ms (typical 10ms) Low jitter
1.
Current TDM Environment
2.
IP/MPLS Environment
3.
Design Considerations
a)
Components of Latency
b)
Fine Tuning Latency in MPLS Routers
c)
Latency Optimization Link/Path Design
d)
Timing
e)
MPLS Service Catalog for Teleprotection
4.
Operational Considerations: Next Generation
Network Management
Operational Considerations
Next Generation Management Platform - Requirements
Migration to IP/MPLS networking for Teleprotection is enabled
by next generation, advanced network management
platforms.
Key functionalities for consideration should include:
1.
Ease of Troubleshooting
IP/MPLS is extremely dynamic, does the network manager provide real-time
and historical control plane information for service paths? Is this information
presented in an interactive graphical display?
2.
Latency Monitoring and Alarming
Is the network manager capable of pro-actively testing and alarming on
conditions where the Teleprotection parameters are not met?
3.
Path Symmetry and Alarming
Certain Teleprotection schemes are bi-directional in nature and are sensitive
to variations between transmit and receive circuit performance. Can the
network management platform monitor and alarm if there is an asymmetrical
circuit condition in the network?
7705 SAR
Next Gen Network Management
• Performs OA&M tests and reports on results • Raises alarm if pre-set SLA threshold crossed
• Alarms if asymmetrical condition exists on teleprotection circuit • Detects network topology and records path changes
Alarm 7705 SAR 7705 SAR 7705 SAR 7705 SAR 7750 SR 7705 SAR 7750 SR 7750 SR TPR TPR 7750 SR
Real-time and historical information presented in a graphical format
combined with the capability to pro-actively test and alarm on SLA
violations are needed to facilitate ease of Operations.
Operational Considerations
Next Generation Management Platform - Example
Forward
Path
Reverse
Operational Considerations
Consistent End-to-End GUI
•
Wizard based service
provisioning for
Services/Tunnels.
• Deployment of a multiple-site
service can be created and applied in one operation.
• Mapping services to both physical
& logical entities to ensure the correct QoS
• Simplifies service creation for new
and existing customers
• Real-time Config Database
Operational Considerations
Control Plane History and Auditing
To
June 18, 2009 8:35 AM
From
June 18, 2009 08:00 AM Select Time Interval to Investigate
OSPF adjacency added in this interval
(the only control plane event in that interval)
Length of history is dependent on the number of objects kept in the database and the rate of change in the network
Major change in checkpoints infrastructure. CPAM now tracks
*all* changes, not just snapshots.
Green: new link
Red: deleted link
Yellow: modified link (filterable)
Purple: flapping link (flap count)
Overlay time this event happened
Can drill down to see how many times and
Take Aways
Current TDM
Network
difficult to
scale?
Current
Network
or segments are
out of capacity?
Troubleshooting
is
time
consuming?
Multiple Networks
Multiple Teams?
Expensive
leased
circuits?
Bell Labs Whitepaper:
Ken Rabedeau
[email protected]
Backup
Material
Network Architecture - TDM
Base Station OC-3 RTU SCADA collection Omni PCX Operations Billing System Collaboration tools Public Internet LMR Management OC-3 Base Station RTU Base Station RTU Base Station RTU NxT1 NxT1 NxT1 NxT1Network Architecture – IP/MPLS
Base Station RTU SCADA collection Omni PCX Operations Billing System Collaboration tools Public Internet LMR Management GigE Base Station RTU Base Station RTU Base Station RTU Broadband IP Traffic Radio + Data Broadband IP Traffic Radio + Data Broadband IP Traffic Radio + Data Radio + Data Broadband IP TrafficCircuit Emulation Services Over MPLS/GRE
for T1/E1 Private Line Transport
Leverage a Transformed PSN Infrastructure for Legacy Services Structured & Unstructured
T1/E1 transport over IP/MPLS or GRE Tunnels
Highly Scalable T1/E1 fan-in
T1/E1 T1/E1 T1/E1 T1/E1 T1/E1 T1/E1 7705 SAR 7705 SAR 7705 SAR 7705 SAR Comprehensive Synchronization Solutions plus embedded OAM
and management
7750 SR
STM-1,OC-3 ch. PBX
