Communication Network Modeling and Simulation
for Wide Area Measurement Applications
Yi Deng, Hua Lin, Arun G. Phadke, Sandeep Shukla, James S. Thorp, Lamine Mili Bradley Department of Electrical & Computer Engineering
Virginia Polytechnic Institute & State University Blacksburg, Virginia, 24061, USA
{yideng56, birchlin, aphadke, shukla, jsthorp, lmili}@vt.edu
Abstract— In the recent years, Phasor Measurement Unit (PMU)
based Wide Area Measurement System (WAMS) has been receiving ever increasing attention from the academia as well as from the industry. Power utilities have been designing and implementing WAMS to provide more intelligent monitoring, control, and protection of the power grid. In order to achieve real-time operations in the modern power systems, construction of an economic and efficient communication infrastructure is a necessity, and various utilities have been laying fiber optical network along their transmission and distribution right of way to leverage the abilities of PMUs in providing greater visibility over a larger area of the grid – thereby providing opportunities for better control and stability. The choice of network architecture, protocols, and various measures for quality of service guarantees must be made by the network architects at the utilities. In this paper, we present a methodology based on profiling data traffic for various WAMS applications according to their communication requirements, and then creating simulation models and scenarios to obtain various parameters for specific architectural and protocol choices. Our simulation results are encouraging in the sense that under modest choices all the applications meet the timing and bandwidth requirements. However, the main contribution of this work is the methodology that would allow the utilities to evaluate various communication infrastructure choices while deploying WAMS.
I. INTRODUCTION
The Smart Grid, which is in the rapid development and deployment process, achieves its challenging objectives using a large number of new techniques [1]. Wide area measurement technology, as one of the cornerstone techniques for power system dynamic analysis, has been proven to be an effective method for large-scale power grid protection and control. Wide area measurement system (WAMS) which utilizes Global Positioning Systems (GPS) based Phasor Measurement Units (PMU) has the capacity of capturing the steady state and transient state information in real-time [2]-[4]. When the distributed synchrophasor data are accumulated in a centralized monitoring and controlling data center, they eventually support various applications in terms of state estimation, event optimal control, systematic protection, etc. [5], [6].
A high-speed and intelligent communication infrastructure is the key to make time-critical WAMS applications feasible in practice. Early communication technologies like power line carrier and microwave communication have their own
limitations in reliability, scalability and robustness [7]-[9]. However, recent adoption of optical fiber communication in power system allows the end-to-end data transmission latency low enough to meet the communication needs. The properties in terms of closed transmission media, lightweight physical composition, ultra-high bandwidth, and low-loss light propagation make the optical fiber very attractive in WAMS applications.
In this paper, the communication infrastructure of WAMS is modeled subject to the communication requirements of WAMS applications. In section II, we describe the background of communication requirements for WAMS and depict its potential network architecture. WAMS applications are then classified into categories according to their communication characteristics. SDH over IP with MPLS support is proposed as the main protocol used in this architecture. In section III, the WAMS network is modeled in OPNET software hierarchically, and the simulation profiles used in OPNET for WAMS applications are introduced. The simulation results are presented and discussed in section IV. Finally, the full paper is concluded in section V.
The main contributions of this work can be summarized as follows: The choice of network architecture, protocols, and various measures for quality of service guarantees must be made by the network architects at the utilities. In this paper, we present a methodology based on profiling data traffic for various WAMS applications according to their communication requirements, and then create simulation models and scenarios to obtain various parameters for specific architectural and protocol choices. Our simulation results are encouraging in the sense that under modest choices of network medium, architecture and protocols – all the applications meet the timing and bandwidth requirements. However, the main contribution of this work is the methodology that would allow the utilities to evaluate various communication infrastructure choices while deploying WAMS.
II. COMMUNICATION TECHNOLOGIES FOR WIDE AREA MEASUREMENT SYSTEMS
A. Overview of WAMS
In general, wide area measurement in power systems is a highly distributed application. The advances in communication technologies, protocols, and quality of service differentiated Proceedings of the 2nd IEEE International Conference on Smart Grid Communications (Smart-GridComm), Brussels, Belgium, October 17-20, 2011
service models enable us to share large volumes of data across geographically distributed power grid assets and equipments via high-speed, low latency network. WAMS thus require mission-critical communication infrastructure capable of collecting synchronized real-time measurements from distributed PMUs [10]-[12]. A fortunate confluence of the latest technologies for high-speed data acquisition, real-time data transmission, efficient data concentration and high-performance data processing are making real WAMS implementations possible. A typical WAMS consists of a GPS based high-precision synchronized clock source, multi-level reliable high-performance communication networks, centralized or a hierarchy of compute-intensive data processing center(s), and multi-function decision making and control executing center(s) – which all together form an integrated technology platform.
In Figure 1, hardware architecture of a practical WAMS is sketched. It is composed of five main components: substations with Phasor Data Concentrator (PDC) denoted by blue dots, substations without PDC denoted by red dots, centralized Super Phasor Data Concentrator (SPDC) denoted by a black dot, relay protection office denoted by a green dot, and high-performance backbone networks denoted by a cloud shape.
In these substations, PMUs, Relays, and Intelligent Electronic Devices (IEDs) are connected with each other using an Ethernet. IEC61850 GOOSE is being increasingly deployed as the communication protocol between devices within the substation. As the data switching within a substation is modest, such a shared media access protocol can support the communication traffic quite well. The PDCs gather phasor measurement data which are acquired from distributed PMUs, and then realign and reformat them for an SPDC. In the SPDC node, the SPDC equipment is used to calculate and analyze the uploaded data. There are also a large amount of data storage equipments in charge of logging the measurements. The
Fig. 1. One possible hardware architecture of WAMS
System Control Center (SCC) sends control action messages to critical relays during urgent situations through high-bandwidth network. The relay protection office is attended by engineers who can manipulate the control actions in the case that the characteristics should be modified after long period of usage or the measurement alerts. By using high-performance routers, the substations, SPDC, and relay protection office are all bound together as an integrated system.
B. Communication Medium for WAMS
Communication infrastructure is essential for WAMS applications which collect phasor measurement data from remote locations. Channel capacity and latency are usually the most significant performance-related factors in any communication task. The data streams created by the PMUs are quite modest so that with the right communication technology, the channel capacity is rarely a limiting factor in most WAMS applications. On the other hand, some applications may require low latency – in particular, time-critical applications for real-time control of power systems. However, not all applications require low latency, for example, post-mortem analysis applications which require PMU data to analyze the power system performance during major disturbances [13]. The channel capacity and latency of different communication media, such as power line, satellite, microwave, and optical fiber etc., vary significantly. Among these, the key features of fiber optics that have been high channel capacity, high data transfer rate, low transmission loss, immunity to electromagnetic interference and low cost, make it the most suitable candidate for WAMS applications [7], [14]-[16]. Therefore in this work, we assumed optical fiber communication infrastructure for WAMS application enabling. Our simulation based analysis is realistic as per our experience with a number of electrical utilities in the United States – the optical fiber communication is adopted as the main technology for the WAMS backbone networks for most of them.
C. Communication Protocols for WAMS
Considering the real-time performance and reliability, SDH in optical fiber networks is becoming mainstream for WAMS. Developed from Synchronous Optical Network (SONET), SDH was adopted by International Telecommunication Union Telecommunications Standardization Sector (ITU-T) as an international standard for the second generation of digital transport technology in 1988. In the frame structure of SDH, the overhead section occupies about 2.96% of the frame size so that the payload transmission efficiency will be quite high. Furthermore, there are maintenance and management sections within the frame structure which help form a robust network for WAMS communication. On the higher level, IP-related protocols are popular in packet-switching networks. Compared to circuit-switching networks, packet-switching networks are more flexible and efficient. Also, in order to introduce differentiated Quality of Service (QoS) guarantees, the multi-protocol label switching (MPLS) scheme – an Internet Engineering Task Force (IETF) standard based on Cisco’s tag switching has been established. This is a connection-oriented structure integrated into the otherwise connectionless IP network. One major function of MPLS is Resource Reservation Protocol – Traffic Engineering (RSVP-TE), which manages the flowing pass for every IP package and avoids data aggregation
at the congested node. Another function of MPLS is the priority scheme to supply the IP precedence or label based QoS [17]. MPLS Virtual Private Networks (VPN) helps divide the different WAMS application services into differentiated priority channels. Conclusively, the IP over SDH with MPLS seems to be the most appropriate candidate for WAMS – and in our simulation study – we employ this.
III. WAMSAPPLICATIONS MODELING AND SIMULATION With the development of WAMS, the feasibility of several applications on it has been proposed and proved through theoretical analysis. The communication needs for each kind of WAMS application have been discussed in [5]. From the recent research studies, it is clear that with fiber optical infrastructure, MPLS over SDH, the communication requirements for each application are quite modest. Nonetheless, the collective performance still could be uncertain when those applications run simultaneously on the communication infrastructure creating a lot of data and control traffic – possibly affecting latency, and reliability. In order to study these, as well as to create a methodology for evaluating various network architecture, protocol and communication media choices – we model the communication networks of WAMS applications in OPNET software and do profile based simulations to compute various timing parameters. This way, one can investigate various communication network topologies, channel capacities, transmission delays, link throughputs and bandwidth utilizations.
A. A Short Introduction to OPNET
The networking simulation tool we use is the OPNET Modeler [18], a powerful and comprehensive modeling and simulation software which is dedicated to communication network research and development. The hierarchical network modeling architecture which corresponds to actual protocol layer, device layer, and network layer can enable the accurate simulation of WAMS-like, end-to-end, system-level network architecture.
B. WAMS Application Categories
WAMS supports various operations of power system such as monitoring, protection, and control. In the literature, there are 12 frequently used WAMS applications [5]. According to the communication needs, these applications can be classified into four different data transmission profiles: periodic transfer without acknowledgements, large amount of burst transfer without acknowledgements, small amount of burst data transfer without acknowledgements, and burst transfer with acknowledgement. Mapping to OPNET’s pre-defined transaction profiles, these four communication profiles can be modeled as: the video conference, file transfer protocol (FTP) data transfer, print operation, and remote login with response respectively. The corresponding relationship between WAMS application types and OPNET application profiles is listed in Table1.
This classification can distinguish the time critical applications from other applications. Take the application ‘supervision of backup zone’ as an example – designed to prevent mis-operation of back-up protection zones – the PMUs on remote buses need to monitor the apparent impedance of the
TABLE I. APPLICATIONS CATEGORY
No WAMS Application Types Application Profiles 1 Periodic transfer without acknowledgements Video conference
2 Large amount of burst data transfer without acknowledgements FTP data transfer
3 Small amount of burst data transfer without acknowledgements Print operation
4 acknowledgement required Burst transfer with Remote login with response
transmission lines. When a false fault is picked up by backup relays, the PMUs around the back-up relays should send messages to backup relays to prevent the false trips. This action belongs to the fourth type of applications since by acquiring the acknowledgement the sender can make sure that the critical action has been executed.
C. WAMS Communication Network Structure
We model the entire WAMS communication network in a hierarchical manner from local inner-substation networks, to last mile access networks and to wide area backbone networks. A skeletal network structure is shown in Figure 2.
The octagon shape nodes Ri (i=0, 1, 2, .., n-1) represent one of the n routing nodes (RN) on the backbone network which in this case, has a ring topology. The routing nodes may consist of various numbers of high-performance routers. There are two kinds of circle shape nodes Si,j and S’i,j which represent the PMU-equipped substation without PDC and with PDC respectively. Usually, there is only one S’
i,j node in a regional area. The main function of the PDC is to aggregate and align the PMU data based on time stamps. Substations in the same regional area are connected to one of the routing nodes on the backbone ring. Figure 2 only shows the subsidiary substations for one routing nodes, Ri, but in fact, other routing nodes also have their own group of substations. The star shape node P represents the SPDC in WAMS. There is only one SPDC in WAMS which is in charge of system monitoring and control at the highest level.
Physically, this example ring topology backbone network is an optical fiber network using SDH. The communication links between two routing nodes (Ri ↔ Ri+1) denoted by LH are modeled as 155.520Mbps SDH STM-1. This type of link is also used to connect the SPDC node to the routing node (P ↔ Ri). The communication links between substations and routing node (S ↔ Ri or S’ ↔ Ri) denoted by LL are modeled as 2Mbps E1 links which should be sufficient for all applications.
Fig. 3. The system infrastructure of the entire system.
The WAMS communication infrastructure constructed in OPNET is shown in Figure 3. In total, 120 PMUs are placed in the system which can cover most areas of the eastern US grid. In each substation, we place two PMUs, various relays, and circuit breakers over a 100Mbps Ethernet. In OPNET, the paradigms of these equipments are represented by workstations and servers. The SPDC node is composed of server-based SPDC and workstation-based SCC. The subnet of the routing node is composed of six high-performance routers as shown in Figure 4. The routers connect to substations located in its region and to other routing nodes. The dotted line between R2 and R3 indicates that the routing node subnet is able to expand to a larger scale by adding more routers.
Fig. 4. The architecture of subnet of routing node IV. SIMULATION RESULTS
First, we create simulation scenarios for various WAMS applications individually in OPNET. We simulate all the applications on the same network infrastructure but with different prototype application profiles. The network statistics for each application including data flow, transmission throughput, real-time indicator, reaction time, and end-to-end delay are significantly different based on the simulation results. Some of the applications are latency critical and some are not. Then, a hybrid scenario is simulated where all possible WAMS applications run simultaneously on the network infrastructure.
A. Individual Simulations 1) Power System Monitoring:
As one of the most important applications of WAMS, the power system monitoring needs all PMUs which are installed in substations to upload the measured data to the centralized
SPDC. The main three monitoring applications are “state estimation”, “seams between state estimates” of two adjoining Independent System Operators (ISO), and “instrument transformer calibration” for all-PMUs estimator.
Although these three monitoring applications are different and independent, their communication characteristics are very similar and in fact can be captured using the same network simulation profile. In the “seams between state estimates” application, by utilizing the uploaded all-PMUs measurement data including boundary reference buses in both areas, the SPDC can distinguish the two state estimators and the differences of reference angles. As for “the instrument transformer calibration”, all needed information has been transmitted and stored to the SPDC periodically and these pre-stored data will be re-fetched every 12 hours when doing calibration calculation. Hence, the communication data flows in these two applications are all the same with the all-PMUs state estimation application so that only one simulation profile is needed.
The packet end-to-end (ETE) delays of the power system monitoring applications are plotted in Figure 5. The entire transmission delay can be divided into two stages. The first stage is the aggregation delay from local PMUs to regional PDCs. The second stage is the gathering delay from distributed PDCs to the centralized SPDC. From the curves, we can see that the regional transmission delays shown by color lines are around 20ms and the wide area transmission delays denoted by black lines are 40ms, therefore the entire transmission delays are approximately 60ms.
Fig. 5. ETE delays for monitoring applications
TABLE II. THROUGHPUTS AND UTILIZATIONS FOR OTHER CONNECTIONS Segment Direction Throughput
(Mb/s)
Utilization (%)
Regional From RN to PDCs 0.449 22
Wide Area Between Routing Nodes 0.549~2.693 0.36~1.8
Wide Area From RN to SPDC 5.386 3.6
The throughput and link utilization are shown in Figure 6 and Table II. From the statistics of simulation data, the heaviest loaded channel is the communication link between SPDC and its nearest routing node. The number of local PMUs can be controlled in a predictable manner. Therefore, the communication throughput between PMUs and PDCs is also controllable. If more PMUs are installed into the WAMS architecture, the throughput of this communication channel will increase accordingly and will become a bottleneck for the communication.
2) Power System Protection:
In this category, the communication network architecture take charge of delivering the real-time control commands. Typical protection applications include adaptive dependability and security (ADS), monitoring approach of apparent impedances (MAAI), adaptive out of step (AOS), supervision of back-up zone (SBUZ), adaptive loss of field (ALF), intelligent load shedding (ILS), intelligent islanding (II), etc. Depending upon the communication latency needs, they are categorized into non critical applications which can tolerate up to one second delay, and time-critical applications that the maximum acceptable delay should be below 50ms.
The simulation results of the maximum end-to-end delays and the maximum response time for all the protection applications are listed in Table III. In the ADS application, the system control center decides the current state of the system, and then sends voting messages to nine distributed critical relays asking for acknowledgements; In the MAAI application, the apparent impedance – calculated by PMUs will be sent to the relay engineer office without requiring acknowledgements to warn the relay engineers regarding the change of relay tripping characteristics; In the AOS application, the PMUs which are installed outside the generator, upload the measurements and track rotor angles, and to determine coherent groups they send the information to the PDC with no acknowledgements; In the SBUZ application, PMUs which fall inside the back-up zones can monitor the apparent impedance and send decisions to back-up relays with acknowledgements, and eventually prevent mis-operations which could lead to cascading failures and blackouts; In the ALF application, in order to revise some drift data which are introduced by long time usage or unexpected environment changing, the SCC will send the adjustment orders to the loss of field relays with acknowledgements; In the ILS application, the PMUs which are used to monitor the tie-line power flows will take charge of the supervisory control, and send measurement data to PDC without acknowledgements; In the II application, since the instability is inevitable, the system control center has to make decisions to trip or block the related circuit breakers with acknowledgements needed.
TABLE III. DELAYS AND RESPONSE TIME OF PROTECTION APPLICATIONS Apps Direction Maximum ETE Delay (ms) Maximum Response Time (ms)
ADS SCC to 9 Critical Relays 40 79
MAAI 3 PMUs to Relay Engineer Office 42 N/A
AOS 2 Generator PMUs to PDC 44 N/A
SBUZ 10 PMUs to Back-up Relay 25 55
ALF SCC to 3 Loss of field Relays 45 89
ILS 4 Tie line PMUs to PDC 44 N/A
II SCC to 9 Circuit Breakers 46 90
As shown in Table III, after analyzing the simulation results of all the applications and comparing with the communication needs, the maximum end-to-end delays and response time within the ring topology fiber-optic network are all below the time constraints – 50ms, 100ms respectively.
3) Power System Control:
The simulations for power system control applications mainly focus on the control of sustained oscillations and large oscillations. The simulation results for these two applications are shown in Figure 7.
Fig. 7. The ETE delay for power system control applications
Using PMUs to damp the low frequency inter-area oscillations is one of the most attractive applications of WAMS. In our simulation, we assume that there are 5 control devices and a total of 25 remote PMUs in the system. This application involves wide area communication where measurement data might travel hundreds of miles. The large oscillations control application, which is mainly used for preventing transient instability, gathers and allocates measurement data and control signals for data communication.
B. Hybrid Simulation
In practice, all the applications mentioned above are going to run simultaneously no matter what the transaction types are. In a worst scenario, there will be data traffic associated with all these applications at the same time. In order to guarantee the accuracy and effectiveness of PMU measurement data, these time-critical applications must have constraints on the end-to-end transmission delay. In the worst case scenario, we simulated these independent applications all working at the same period of time. The hybrid simulation results of end-to-end delays are shown in Figure 8.
Fig. 8. The hybrid simulation results of packet end-to-end delays The end-to-end delays between distributed PDCs and centralized SPDC are about 40ms which is the same as the delays of all-PMUs state estimation application. The end-to-end delays from local PMUs to regional PDCs are around 20ms which are slightly different with those individual applications.
V. CONCLUSIONS
Wide area measurements applications enabled by the wide spread deployment of PMUs and communication networking will be one of the most important aspect of intelligent power grid of the near future. Rapid PMU deployments have begun in the United States, in European Union, and in China. Algorithms for various WAMS applications have been proposed, and implemented in prototypes, wide area closed loop control for stability have been proposed, and experimented with. However, many of these applications require real-time delivery of large volume of real-time data that the PMUs collect 30 times a second or faster. The choice of the proper network topology, transmission media, and protocols will play important role in fulfilling the idea of wide area monitoring and control. Choice of network architecture, protocols etc., are hard to make without knowing the exact volume of data, requirements in latency and bandwidth, quality of service etc. A framework in which such choices can be made via realistic simulation studies is proposed and demonstrated here. Although, in this paper we only present results for a specific network topology, specific data bandwidth choice, and protocol choice – these can be easily varied in the modeling and simulation framework of OPNET to compare and choose by a utility company. As the WAMS applications and deployments get more dense, and wide spread, the network traffic associated with the various applications will grow, and new architectural choices, newer protocols and transmission media etc., must be chosen. This paper demonstrates one framework to do that. Also, in this work, four traffic profiles are used to obtain the simulation models and scenarios. The results obtained in the simulation study presented here is quite encouraging – all applications and the simultaneous execution of all WAMS applications meet the latency and bandwidth
constraints. In the future, we will integrate OPNET with a power system simulation environment so that real data traffic can be simulated from PMU models – providing further accuracy to the results.
REFERENCES
[1] U.S. Department of Energy Office of Electricity Delivery & Energy Reliability “Smart grid research & development multi-year program plan 2010-2014”.
[2] A.G. Phadke, J.S. Thorp, “Synchronized Phasor Measurements and Their Applications”, Springer, 2008.
[3] J.D.L. Ree, V. Centeno, J.S. Thorp, A.G. Phadke, “Synchronized phasor measurement applications in power systems”, IEEE Trans. Smart Grid, vol. 1, no. 1, June 2010.
[4] A.G. Phadke, “The wide world of wide-area measurement”, IEEE power & energy magazine, September/October 2008.
[5] A.G. Phadke, J.S. Thorp, “Communication needs for wide area measurement applications” Critical Infrastructure (CRIS), 5th International Conference, September 2010.
[6] S.H. Horowitz, A.G. Phadke, “Power System Relaying, Third Edition”, Research Studies Press Limited, John Wiley & Sons Inc, 2008.
[7] V.C. Gungor, F.C. Lambert, “A survey on communication networks for electric system automation”, Comupter Networks 50 (2006) 877-897. [8] M. Shahraeini, M.H. Javidi, M.S. Ghazizadeh, “comparison between
communication infrastructrues of centralized and decentralized wide area measurement systems”, IEEE Trans. Smart Grid, vol.2, no. 1, March 2011.
[9] M. Shahidehpour, Y. Wang, “Communication and Control in Electric Power Systems, Applications of Parallel and Distributed Processing”, IEEE Press, John Wiley & Sons, Inc, 2003.
[10] M.G. Adamiak, A.P. Apostolov, M.M. Begovic, C.F. Henville, K.E. Martin, G.L. Michel, A.G. Phadke, J.S. Thorp, “Wide area protection – technology and infrastructures”, IEEE Trans. Power Delivery, vol. 21, no. 2, April 2006.
[11] J. Bertsch, C. Carnal, D. Karlsson, J. Madaniel, K. Vu, “Wide-area protection and power system utilization”, Invited Paper, Proc. of the IEEE, vol. 93, no.5, May 2005.
[12] V. Terzija, G. Valverde, D. Cai, P. Regulski, V. Madani, J. Fitch, S. Skok, M.M. Begovic, A.G. Phadke, “Wide-Area monitoring, protection, and control of future electric power networks”, Invited Paper, Proc. of the IEEE, vol. 99, no.1, January 2011.
[13] M. Chenine, E. Karam, L. Nordstrom, “Modeling and simulation of wide area monitoring and control systems in IP-based networks”, IEEE Power & Energy Society General Meeting, July 2009.
[14] A-R.A. Khatib, “Internet-based Wide Area Measurement Applications in Deregulated Power Systems”, Ph.D. Dissertation Virginia Tech, July 2002.
[15] B. Naduvathuparambil, M.C. Valenti, A. Feliachi, “Communication delays in wide area measurement systems”, System Theory, Proc. of the 34th Southeastern Symposium on, 2002.
[16] K. Hopkinson, X. Wang, R. Giovanini, J. Thorp, K. Birman, D. Coury, “EPOCHS: A platform for agent-based electric power and communication simulation built from commercial off-the-shelf componets”, IEEE Trans. Power System, vol. 21, no. 2, May 2006. [17] H.G. Perros, “Connection-oriented Networks SONET/SDH, ATM,
MPLS and Optical Networks”, John Wiley & Sons, 2005. [18] OPNET Technologies, Inc. “http://www.opnet.com”.
