How can I. implement a high-availability system? Develop your project. System Technical Guide High Availability solutions

How can I …

implement a high-availability

system?

Develop

your project

System Technical Guide

High Availability solutions

Disclaimer

This document is not comprehensive for any systems using the given architecture and does not absolve users of their duty to uphold the safety requirements for the equipment used in their systems or compliance with both national or international safety laws and regulations.

Readers are considered to already know how to use the products described in this document.

The STG Collection

System Technical Guides (STG) are designed to help project engineers and Alliance System Integrators during the development of a project. The STGs support users during the architecture selection and the project execution (design, configuration, implementation and operation) phases, with an introduction to the system operating modes.

Each STG is a starter kit that provides users with:

•

Technical documentation

•

Application examples

•

Object libraries

Each STG addresses one or several customer challenges within the proposed solution using the offer from Schneider Electric.

All explanations and applications have been developed by both Schneider Electric experts and system integrators in our solution labs. The contributions from the system integrators help the kit’s content address the needs of our users.

All STGs are illustrated with industry-specific applications to give more concrete examples of the methodology.

The STGs are not intended to be used as substitutes for the technical documentation related to the individual components, but rather to complement these materials and training.

Development Environment

Each STG has been developed in one of our solution platform labs using a typical PlantStruxure architecture.

PlantStruxure, the process automation system from Schneider Electric, is a collaborative system that allows industrial and infrastructure companies to assess their automation needs while at the same time addressing their growing energy management requirements. In a single environment, measured energy and process data can be analyzed to yield a holistically optimized plant.

Table of Contents

1. Introduction...7

1.1. Purpose ... 7 1.2. Customer Challenges ... 7 1.3. Prerequisites ... 8 1.4. Project Methodology... 8

2. Selection...11

2.1. Redundancy Basics ... 12 2.2. Operational Principles... 14 2.3. Selection Criteria ... 25 2.4. Selected Architecture ... 25 2.5. Conclusion ... 31

3. Design...35

3.1. Introduction... 35 3.2. SCADA System ... 35

3.3. Premium Hot-Standby System... 36

3.4. Quantum Hot-Standby System... 54

4. Configuration ...69

4.1. SCADA ... 69

4.2. Control and Field Network ... 82

4.3. Premium Hot-Standby PAC Station ... 88

4.4. Quantum Hot-Standby PAC Station ... 98

4.5 Advantys STB Ethernet I/Os... 114

5.2. Quantum PAC ... 129

5.3. Conclusion ... 142

6. Performance...143

6.1. Performance test protocols ... 143

6.2. Premium PAC Architecture... 145

1. Introduction

1.1. Purpose

The purpose of this document is to provide recommendations, guidelines and examples to help implement a high availability automation system. A System Technical Note: “How can I increase the Availability of a system” detailing the theoretical basics of high-availability has already been issued.

This guide focuses on the different ways to increase availability through redundancy at various layers of the automation system and proposes a methodology to implement an efficient high-availability automation system.

Each step of the implementation of the solution, from the selection to the operation, is broadly described and illustrated with examples.

The solutions described in this STG are fully part of the PlantStruxure control system. The High availability System Technical Guide is structured in two parts:

•

Part 1: Redundant architecture with single attachment.

•

Part 2: Complex architecture including dual network attachment and dual ring.

1.2. Customer Challenges

For customers in industries where availability and reliability are major concerns, the challenges are:

•

Provide a high level of availability.

•

Attain a high level of reliability.

•

Short recovery following system unavailability.

•

Maintainability facilitated by redundancy.

•

Switchover time adapted to critical processes.

This STG suggests approaches to these challenges and highlights specific areas using SoCollaborative Engineering products in line with PlantStruxure Control System.

1.3. Prerequisites

We recommend the reader have knowledge of the following SoCollaborative software:

•

Unity Pro

•

Vijeo Citect

We also recommend the reader become familiar with the System Technical Note: “How can I increase the Availability of a system” and to have knowledge of the Premium/Quantum/M340 Schneider PACs.

1.4. Project Methodology

This STG describes the project methodology and includes the following phases: Selection, Design, Configuration, Implementation and Performance.

This guide is illustrated using 2 architectures (Premium and Quantum). Their features are described in the Selection phase. Each architecture shows a specific feature necessary of Hot-Standby application.

Beginning with process analysis and user requirements, we identify and develop common functionalities for all the architectures. These key functions are explained in the Design, Configuration and Implementation phases.

Finally, the Performance phase summarizes the results of different tests performed on the 2 architectures.

Here are the phases described in this document:

•

I. Selection: In this phase, the selection procedure to define a redundant

architecture is presented: ƒ Basic of redundancy ƒ Operational Principles ƒ Description of architectures

•

II. Design: This phase covers the operational principles of the different

components of high availability architecture: ƒ SCADA system

ƒ Network ƒ PAC Station

• Specifications and constraints • I/O system

ƒ DFB library

•

III. Configuration: In this phase, the configuration information of the different

components of the architectures is detailed.

•

IV. Implementation: Using the same architectures and components, information

about final customization to address the project requirements is provided.

•

V. Performance: A summary of the architectures performance in response to

2. Selection

During the selection phase, an optimal architecture is chosen as well as the most appropriate components of the project, according to your specific requirements. Several architectures and systems are presented in this chapter in order to address a wide range of functions and needs. Also, the way to select among these architectures given project needs is presented.

2.1. Redundancy Basics

This chapter describes redundancy general principles and its application in an automation system.

The following PlantStruxure architecture is a representative example to illustrate the different layers where redundancy can be implemented.

Control Network

PAC PAC PAC

Ethernet Ethernet Ethernet Profibus DP Ethernet Field Devices Field Network PACs Station Data Servers SCADA Clients

The diagram also represents a wide range of hardware setups demonstrating the ability to achieve various redundancy levels.

2.1.1. Redundancy Layers

Availability can be increased in an automation system at different layers:

•

SCADA system:

The SCADA system has to handle data acquisition, graphics, events, alarms, trends, and reports. SCADA server redundancy enhances the likelihood these services will continue to operate without loss of data in case of system interruption. Different software and hardware configurations allow different levels of availability.

•

Control Network:

A well defined topology and management of the control network increase network availability and reliability. Thus, in turn, makes communication between the SCADA system and the PAC stations more reliable. Several network topologies and network

protocols are available to achieve the optimal level of availability and to fit the whole system needs.

•

PAC station:

According to your needs in terms of I/O number and topology, you can choose among a Quantum or a Premium Hot-Standby PAC system. The field network type is also an element to consider before choosing the PAC station, as well as the I/O system (local or distributed).

•

Field Network:

Redundancy can also be applied to the field network. As was the case for the control network, a well defined topology and management of the network increase field network availability. Device redundancy is also implemented to increase availability of the field equipment.

2.1.2. Redundancy level

We can differentiate several levels of redundancy according to their performances in term of availability. A summary of these levels is presented in the following table: Redundancy Level State of the standby system Switchover performance

No redundancy No standby system Not applicable

Cold Standby The standby system is only powered up if the default system becomes

inoperative.

Several minutes

Large amount of lost data

Warm Standby The standby system switches from normal to backup mode.

Several seconds

Small amount of lost data Hot-Standby The Standby system runs together with

the default system.

Several milliseconds No lost data

2.2. Operational Principles

Depending on the level of availability required, redundancy is applied in various ways. This chapter discusses Hot-Standby applications and describes the basics of

redundancy in each layer of an automation system. Moreover the chapter describes the various options available in terms of redundancy and availability.

The selected architectures used in the following parts of this STG are described in the Chapter 2.4.

2.2.1. SCADA

The main operating principles of the different SCADA servers are described in the following paragraphs.

General Principles

The different servers of the SCADA system (Alarm, Report, Trends, and I/O servers) can either be installed on the same computer or on different computers allowing for more reliability. For a redundant configuration, each server (Primary) is associated with its redundant server (Standby) installed on a different computer.

For example, the picture below describes servers installed on redundant computers (Primary and Standby)

I/O Server redundancy

In a redundant SCADA system, an I/O device is associated with the Primary and Standby servers. The Primary server accesses periodically the I/O device to read and write tags. The Standby server only checks the communication with the I/O device. At startup, if the Primary I/O server can not establish a connection with the I/O device, the SCADA system switches to the Standby I/O server.

During operation, if the Primary I/O server stops communicating with the I/O devices, the system then switches to the Standby I/O server. The following diagrams illustrate 2 cases: a broken network cable and a server that has stopped communicating.

When the I/O server defined as Primary returns to operational state, the SCADA system returns control back to the Primary server.

Alarms / Trends / Reports Servers (ATR) redundancy

The management of the Alarms, Trends and Report servers (ATR servers) by the SCADA system follows the steps listed below:

If the Primary ATR server stops operating, the system switches to the Standby ATR server.

When the ATR server defined as Primary returns to operational state, any clients connected to the Standby ATR server remain connected to the Standby server.

For example, the following picture describes a server reconfiguration initiated by a switchover: I/O, Trends and Reports servers are working on the Primary SCADA server, and the Alarms server is working on Standby SCADA server.

2.2.2. Network

Various topologies and protocols are used to increase the availability of the network (control or field network). The principle is to create different paths to access devices. In case of a network element on the main path stops functioning, another path is used.

The following table illustrates the main network topologies.

Architecture Limitations Advantages Disadvantages

Bus The traffic must flow serially, therefore the bandwidth is not used efficiently.

Cost-effective solution If a switch becomes inoperative,

communication is lost. Star

Tree

Cable ways and distances

Efficient use of the

bandwidth, as the traffic is spread across the star. Preferred topology when there is no need for redundancy.

If the main switch becomes inoperative, communication is lost.

Ring

Dual Ring

Behavior similar to Bus.

Auto-configuration if used with self-healing protocol. Possible to couple other rings for increasing redundancy.

The availability of auto-configuration depends on the protocol used.

Ring topologies are mainly used to increase the level of network availability. Network redundancy management protocols, such as Hiper-ring or MRP, are used for network recovery in case of part of the network cease to function.

2.2.3. PAC Station

Hot-Standby Definition

A Hot-Standby system is used when downtime cannot be tolerated. It delivers high availability through redundancy and always consists of two units with identical

configurations. One of the two units acts as the Primary CPU controller, and the other acts as the Standby CPU controller. One controller must be set in the Primary CPU state and the other must be in the Standby CPU state or offline. The redundant unit takes the control when the main one encounters an anomaly.

The Primary PAC updates inputs, manages Hot-Standby, runs the program while transferring data to the Standby PAC and updates outputs. Thus, the switchover between the Primary and the Standby PAC occurs without any loss of data.

As described on the diagrams above, for each execution cycle, the outputs update only takes place when the data transfer AND the program execution are completed. Therefore, it is important to properly define the amount of data to be transferred from the Primary to the Standby PAC to minimize the wait time induced by a data transfer longer than the execution time of the program execution.

On the diagram on the left, the cycle execution is optimized: the data transfer is performed faster than the program execution.

On the diagram on the right, the longer data transfer induces a wait time that slows down the cycle execution.

Primary and Standby PACs

Assuming that the configuration of the system is correct, the first PAC to be powered up is automatically recognized as the Primary one. Therefore, you can define the PACs role by controlling the sequence order in which they are powered up. When two redundant CPU PACs are switched on simultaneously, the firmware automatically affects the Primary status according to the MAC address. The PAC with the lower MAC address is defined as the PAC A, that is the Primary at the powering up of the system.

Hot-Standby System Programming Elements

This paragraph describes programming basics, useful to know when implementing a Hot-Standby system.

System Words

ƒ %SW60: Command Register

The command register defines the operating parameters of a Hot-Standby application for both the Primary and Standby CPU.

The System Word %SW60 can be used to read and write the command register of Hot-Standby System.

•

The diagram below illustrates the Quantum System Word %SW60: Disables LCD Invalidate Keypad - bit0 = 0

Enables LCD Invalidate Keypad - bit0 = 1

Sets Controller A to OFFLINE mode - bit1 = 0

Sets Controller A to RUN mode - bit1 = 1

Sets Controller B to OFFLINE mode - bit2 = 0

Sets Controller B to RUN mode - bit2 = 1

Forces Standby offline if there is a logic mismatch - bit3 = 0

Does not force Standby offline if there is a logic mismatch - bit3 = 1

Allows exec upgrade only after application stops - bit4 = 0

Allows exec upgrade without stopping application - bit4 = 1

MSB 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 LSB

bit5 = 0 - No application program transfer

bit5 = 1 - Application program transfer requested

bit8 = 0 - Swaps Modbus port 1 adress during switchover

bit8 = 1 - Does not swap Modbus port 1 adress on a switchover

bit9 = 0 - Swaps Modbus port 2 adress during switchover

bit9 = 1 - Does not swap Modbus port 2 adress on a switchover

bit10 = 0 - Swaps Modbus port 3 adress during switchover

bit10 = 1 - Does not swap Modbus port 3 adress on a switchover

•

The diagram below illustrates the Premium System Word %SW60: Sets Controller A to OFFLINE mode - bit1 = 0

Sets Controller A to RUN mode - bit1 = 1

Sets Controller B to OFFLINE mode - bit2 = 0

Sets Controller B to RUN mode - bit2 = 1

MSB 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 LSB

OS versions Mismatch (this bit can be used to permit temporary differences between the firmware versions on the respective Hot-Standby PACs)

ƒ %SW61: Status Register

The Hot-Standby Status Register is a readable register located at system word %SW61 and is used to monitor the current status of the Primary CPU and Standby CPU.

•

The following diagram illustrates the Quantum System Word %SW61:

This PAC in OFFLINE mode - bit1= 0 . bit0= 1

This PAC running in primary CPU mode - bit1= 1 . bit0= 0

This PAC running in standby CPU mode - bit1= 1 . bit0= 1 Other PAC in OFFLINE mode - bit3= 0 . bit2= 1

Other PAC running in primary CPU mode - bit3= 1 . bit2= 0

Other PAC running in standby CPU mode - bit3= 1 . bit2= 1

The remote PAC is not accessible - bit3= 0 . bit2= 0

PACs have matching logic - bit4 = 0

PACs do not have matching logic - bit4 = 1

This PAC's switch set to A - bit5 = 0

This PAC's switch set to B - bit5 = 1

MSB 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 LSB

bit7 = 0 - Same PAC OS version

bit7 = 1 - Different PAC OS version

bit8 = 0 - Same copro OS version

bit8 = 1 - Different copro OS version

bit12 = 0 - Information given by bit13 is not relevant

bit12 = 1 - Information given by bit13 is valid

bit13 = 0 - NOE address set to IP

bit13 = 1 - NOE address set to IP+1

bit15 = 0 - The hot standby has not been actived

•

The following diagram illustrates the Premium System Word %SW61: This PAC in OFFLINE mode - bit1= 0 . bit0= 1

This PAC running in primary CPU mode - bit1= 1 . bit0= 0

This PAC running in standby CPU mode - bit1= 1 . bit0= 1 Other PAC in OFFLINE mode - bit3= 0 . bit2= 1

Other PAC running in primary CPU mode - bit3= 1 . bit2= 0

Other PAC running in standby CPU mode - bit3= 1 . bit2= 1

The remote PAC is not accessible - bit3= 0 . bit2= 0

This PAC set as Unit A - bit5 = 0

This PAC set as Unit B - bit5 = 1

CPU-sync link OK - bit6 = 0

CPU-sync link NOK - bit6 = 1

No processor OS version mismatch - bit7 = 0

Main processor OS version mismatch - bit7 = 1

No Copro OS version mismatch - bit8 = 0

Copro OS version mismatch - bit8 = 1

MSB 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 LSB

bit10 = 0 - No monitored ETY OS version mismatch

bit10 = 1 - Monitored ETY OS version mismatch

bit13 = 0 - Configured IP or Modbus Adress

bit13 = 1 - Configured IP or Modbus Adress +1

bit15 = 0 - The hot standby has not been actived

bit15 = 1 - The hot standby is active

Application Program or Unity Pro configuration Checksum mismatch beetween Remote PAC - bit4 = 1

bit9 = 0 - All in-rack (Monitored and no-monitored) ETY modules have the minimun version

bit9 = 1 - At least one ETY does not have the minimun version

No application Program or Unity Pro configuration Checksum mismatch beetween Remote PAC - bit4 = 0

ƒ %SW62…65: Reverse Register:

System Words %SW62/63/64/65 are reverse registers reserved for the reverse transfer process. The reverse registers can be written in the application program (first section) of the Standby CPU controller and are transferred at each scan to the Primary CPU controller.

Non-Transfer Area

The Non Transfer Area is a defined memory zone which is not transferred during the update of the Standby CPU controller.

The Premium PAC has a 100 words predefined zone (%MW0 to %MW99) whereas the Quantum PAC the size of the zone is defined by the user (%MW1 to %MWx). First Section (section 0)

In a PAC redundancy system, the execution of the application program is different according to the PAC in which the execution takes place. The main difference is that the whole application program is executed in the Primary PAC whereas the Standby PAC only executes the first section (section 0). This point is very important as many settings of the system are defined in the section 0.

Hot-Standby Management

Switchover Conditions

A Hot-Standby system is designed to provide uninterrupted service. This feature requires continuous monitoring of different equipment.

Note: Concerning both Premium and Quantum, the switchover is performed only if the Standby PAC is operational and ready to take over control from the Primary PAC. A Hot-Standby system continuously monitors the key components in order to detect any stoppage in operation. Additional monitoring is performed by the application for more specific requirements.

The monitoring by the system initiates a switchover on the following occurrences:

•

Premium system

ƒ Fault on power supply

ƒ Fault CPU (firmware, hardware) ƒ Halt, Stop, Offline CPU

•

Quantum System ƒ Fault power supply

ƒ Fault CPU (firmware, hardware) ƒ Halt, Stop, Offline CPU

ƒ Fault CRP module

For the Premium and Quantum Hot-Standby architectures presented in this guide, additional equipment (Network Controller and Device Network) are monitored by the application in order to increase the availability.

To perform this specific monitoring, we need to develop Derived Function Blocks (DFBs) that monitor the system, control and process the anomalies, and handle the switchover.

DFBs Libraries

The Unity Pro Quantum system library offers EFBs to manage a Hot-Standby system. These EFBs allow the handling of command (%SW60), status (%SW61) and reverse (%SW62 to 65) registers.

The Unity Pro Premium library does not include pre-designed EFBs. Consequently, we have developed a user-defined Hot-Standby DFBs library. This library is described in the next chapter.

Network monitoring is integrated in our architectures. This functionality is handled neither by the Premium PAC nor by the Quantum PAC (except for the Monitored ETY module on a Premium system). Therefore, we have developed a specific Hot-Standby DFBs library for each configuration:

•

Premium

ƒ ETY_Monitor (Ethernet)

•

Quantum

ƒ NOE_Monitor (Ethernet) ƒ PTQ_Monitor (Profibus)

We have developed an events synthesis block that processes the output of these DFBs, while offering the possibility to mask some defaults, and a switchover

management block which controls the availability of the Standby PAC before initiating a switchover.

2.3. Selection Criteria

Various levels of performance can be attained with different architectures and using different components. It is crucial to select the right configuration that most closely fits your needs in terms of availability, cost and maintenance.

Before implementing a high-availability system, consider the following points:

•

What is the general availability level to reach for the whole system?

•

How many devices can stop functioning, yet have the system remain operational?

•

What is the maximum allowed downtime for the entire system?

•

Are there any constraints (existing system, topology) that imply the use of specific tools and equipment?

•

What is the size of the system and are additional extensions planned?

•

What is the topology? Centralized? Distributed?

•

Are there some areas of the process with priority needs?

2.4. Selected Architecture

We have selected two representative architectures to illustrate in this guide, one for Quantum PAC and one for Premium PAC. All the other layers are common for the two architectures. These architectures are intended to represent a medium range automation system with high availability needs in terms of process control and medium availability in terms of SCADA and network control.

Each layers in the system can tolerate one non-operating device and still remain operational. The implementation of the different layers is described in the following chapters.

2.4.1. SCADA

The resources used by our application are moderate, so all the servers (Alarms, Trends, Reports and I/O) can be installed on one computer. In order to withstand one non-functioning device, the redundant servers are installed on a second computer. Two clients connected to the network allow the control and monitoring of the process. This configuration is adapted to our needs in terms of performance and redundancy level.

2.4.2. Control Network

The ring architecture is chosen for its redundancy capability. Four ConneXium switches handle the ring architecture. One switch or cable segment can cease operating with no impact on the communication through the network. The MRP redundancy management protocol is chosen for its performance in recovery time.

2.4.3. PAC station

The Hot-Standby architecture allows that one PAC stops operating without loss of data. As was the case with the control network, if the Primary PAC ceases to operate, the Standby PAC takes over from the Primary.

The following diagram sums up the selection of the PAC station according to specific requirements of the application.

Time Critical Application ?

Y

Application requiring multiple and/or scattered

I/O Stations ? Y

N

In-Rack and/or distributed I/O system

In-Rack I/O Stations only !

Local I/O Station N Redundant I/O Modules required ? Premium HSBY Y N Premium or Quantum HSBY Remote I/O Stations Quantum HSBY Premium or Quantum HSBY

The selected architectures, Premium and Quantum, are detailed in the following paragraphs.

Note: The number of In-Rack I/Os in the process is decisive for dimensioning the system. As a Premium Hot-Standby system does not handle extension racks, the use of In-Rack I/Os is limited. This means that, beyond a given number of In-Rack I/Os, a Quantum PAC that handles Remote I/Os will be used instead of a Premium PAC. Note: Only the Quantum PAC station provides redundancy solutions for a Profibus network.

Premium

The chosen redundant PAC station is a Premium Hot-Standby architecture with redundant analog and digital inputs and outputs. The 2 units are synchronized via an Ethernet link.

IP:172.20.104.5

MASK: 255.255.0.0 IP:172.20.104.6

PAC A MASK: 255.255.0.0IP:172.20.101.57 PAC B

IP:172.20.101.58 MASK: 255.255.0.0 ABE7 2 5 0 0 ABE7 JM o Modules

Analog Output Analog Input

Digital Output Digital Input

From Control Network

To Ethernet Field Network

Sync-link C ncept MASK: 255.255.0.0 ABE7 Connection Blocks

Quantum

The chosen redundant PAC station is a Quantum Hot-Standby architecture with a shared Remote I/Os module. The 2 units are synchronized via an optical fiber link (Sync link).

2.4.4 Field Network

Profibus DP

The first part of the field network is composed of a Profibus DP daisy chain managed by 2 redundant Profibus Master modules. Each extremity of the daisy chain is connected to a redundant Quantum PAC. The control of the device on the chain is possible even if one of the PACS ceases to operate. The Standby Profibus master PAC then handles the control of the chain.

Ethernet

The second part of the field network is built around an Ethernet ring to bring

redundancy to the field devices. The Ethernet ring is built in the same manner than for the control network. 3 Connexium switches run MRP on the ring and connect different devices connected on Ethernet.

2.5. Conclusion

The 2 following diagrams present the whole Quantum and Premium architectures from the SCADA to the field network. These architectures will be used subsequently in the document to illustrate redundancy runtime principles and performance reviews. The second part of this document will use the same architectures, but including dual ring structures and dual attachment.

IP: 172.20.104.10 IP: 172.20.104.11 IP: 172.20.104.12 IP: 172.20.104.20 IP: 172.20.104.21 IP: 172.20.104.22 IP: 172.20.104.34 IP:172.20.104.5

MASK: 255.255.0.0 IP:172.20.104.6MASK: 255.255.0.0

PAC A PAC B

Client 1

SERVER 1 SERVER 2

IP: 172.20.101.1 IP: 172.20.101.2

IP:172.20.101.57

MASK: 255.255.0.0 MASK: 255.255.0.0IP:172.20.101.58

SW1 SW2 SW3 SW4 IP: 172.20.101.30 SW10 SW11 SW12

Client 2

ABE7 2 5 0 0 ABE7 JM Conc

Analog Output Analog Input

Digital Output Digital Input

IP: 172.20.101.31 Manager Manager

Redundant Premium

Architecture

Sync-link ept Modules ABE7 Connection Blocks

IP: 172.20.104.10 IP: 172.20.104.11 IP: 172.20.104.12 IP: 172.20.104.20 IP: 172.20.104.21 IP: 172.20.104.22 IP: 172.20.104.34 IP:172.20.104.1 MASK: 255.255.0.0 IP:172.20.104.2 MASK: 255.255.0.0 PAC A PAC B

Client 1

SERVER 1 SERVER 2 IP: 172.20.101.1 IP: 172.20.101.2 IP:172.20.101.110 MASK: 255.255.0.0 IP:172.20.101.111 MASK: 255.255.0.0 SW1 SW2 SW3 SW4 IP: 172.20.101.30 SW10 SW11 SW12 Remote I/O Manager Manager

Redundant Quantum

Architecture

Client 2

IP: 172.20.101.31

3. Design

3.1. Introduction

The design part of the STG covers the operational principles of the different components of high availability architecture.

After a short review of the SCADA set up, the following points concerning the PAC station will be detailed:

•

What is a Hot-Standby System (Premium and Quantum)

•

Parts and tools of a Hot-Standby System (Premium and Quantum)

•

Specifications and constraints of a Hot-Standby System (Premium and Quantum)

•

Distributed and In-Rack I/Os (Premium and Quantum)

We will also describe the DFBs used in our Hot-Standby library and why they have been developed.

The SCADA and Network parts are more detailed in the Configuration chapter

3.2. SCADA System

3.2.1. Architecture Presentation

The architecture is composed of 2 redundant Vijeo Citect servers, 2 clients and 2 Hot-Standby PACs (Quantum or Premium). The communication between these

components is achieved through an Ethernet ring.

Each Vijeo Citect server handles I/O, Alarms, Trends and report server functionalities. In our hardware configuration, we choose to install the IO and ATR (Alarm, Trends, and Reports) servers on one computer. The performance of these computers is sufficient in terms of CPU and disk space to handle our application.

3.3. Premium Hot-Standby System

This chapter describes the different features and specifications of a redundant Premium system.

3.3.1 Premium PAC Specifications

Primary and Standby PACs

The Primary PAC executes the application program, controls Ethernet network and In-Rack I/Os and synchronizes the Standby PAC at the beginning of each program cycle.

The Standby PAC does not run the whole program but only the first section (section 0). Moreover, it does not handle the redundant In-Rack and Ethernet I/Os but just checks the state of the Primary PAC.

In case of an anomaly, the Standby PAC takes over the control from the Primary PAC (see switchover time measurements in Performance chapter).

Primary and Standby PACs permanently exchange data in order to check the system integrity via the synchronization link.

A Premium Hot-Standby system necessarily comprises Monitored ETY modules (one in each rack). These modules handle the diagnosis of Premium CPU redundancy configuration status. This diagnosis is achieved through Sync ETY link.

Note: Sync ETY and Synchronization link are different and are not used for the same purpose.

Monitored ETY modules

As for the CPUs, the position in the rack and the firmware version of the Ethernet modules must be identical.

Note: A firmware version 4.0 or earlier is required.

The monitored ETY module allows the swap of the Ethernet services as well as the automatic permutation of the IP addresses between Primary and Standby TSX ETY. ETY modules are linked with Ethernet switches (one switch per ETY) or via an Ethernet crossover cable. An optical connection is also possible in the case of a long distance communication.

Sync ETY link also allows handling of Ethernet I/O devices with the proper Ethernet I/O Scanning service configuration.

In order to initiate a switchover when a Sync ETY link stops operating on the Primary PAC, Ethernet I/O Scanning service must be configured on the monitored ETY module. In addition to the service activation, an I/O Scanning line must also be declared. If the service is not configured in the monitored ETY module, a switchover will not occur if a Sync ETY ceases to operate.

In case a monitored ETY module ceases to function, the CPU sends a status modification command to all the configured ETY modules populating the X-Bus and the monitored ETY module populating the Standby PAC to switch their IP addresses.

Hardware Constraints

The following table lists the only modules that can be used in a Premium Hot-Standby configuration:

Power Supply All available power supply modules Rack Non-expendable Racks only Ethernet

Communication TSX ETY4103 or TSX ETY5103 (firmware version v4.0 or earlier)

Modbus Communication

- Modbus communication module TSX SCY21601 (firmware version 2.3 or earlier) equipped with multiprotocol communication board TSX SCP114 (firmware version 1.7 or earlier) (slave or master)

-Modbus communication module TSX SCY21601 (firmware version 1.1 or earlier) (Master Modbus only)

Note: The TSX SCY 21601 associated with multiprotocol communication board TSX SCP114 allows the redundant Premium PAC systems to run as Modbus Slave or Master. This configuration allows using Modbus Masters from other suppliers. TSX SCY 11601 module can only be used in Modbus Master. Digital I/Os No restrictions apply

Software Constraints

The following constraints apply at the application level

•

The use of event tasks is not recommended. An event might be lost if it occurs just before or during the switchover.

•

The use of FAST tasks handling dedicated outputs is not recommended as output status modifications might be lost during the switchover.

•

The use of counting modules is not recommended. Following the frequency, some pulses might be lost during the switchover.

•

The use of fronts is not recommended. They might not be accounted during the switchover.

•

The use of the SAVE_PARAM function is not recommended in a CPU redundancy application. This function erases the initial value of a module parameter saved in the program code. This code is not transferred from the Primary PAC to the Standby. More generally, explicit instructions like WRITE_CMD and WRITE_PARAM must be well defined before use.

•

Initial values declared with a recorded attribute (for example DFB variables) can not be replaced with actual values: Do not use%S94 bit.

•

Following inherited functions blocks can not be used: PL7_COUNTER PL7_DRUM PL7_MONOSTABLE PL7_REGISTER_32 PL7_REGISTER_255 PL7_TOF, PL7_TON, PL7_TP PL7_3_TIMER

3.3.2. Premium Hot-Standby DFBs Library

The following table summarizes the different DFBs created for our application.

DFB FUNCTION

HSBY_RD Reading Command word (%SW60) hot-standby system

HSBY_WR Writing Command word (%SW60) hot-standby system

HSBY_ST Reading Status word (%SW61) hot-standby system

ETHERNET ETY_MONITOR Monitoring ETY Ethernet Module

SYNTH_FAULT Synthesis Fault monitored elements

SYNTH_OR_ETY Synthesis Fault ETY module (Logic OR)

SYNTH_AND_ETY Synthesis Fault ETY module (Logic AND)

SWITCHOVER SWITCH_MANG Switchover Managment SYSTEM

SYNTHESIS

System DFBs

In order to manage the different registers of a Premium Hot-Standby system, we have created blocks that allow reading and writing registers %SW60 and %SW61.

•

HSBY_RD_P: Read the command register %SW60

BOOL Run Mode Controller A BOOL Run Mode Controller B

BOOL OS Versions Mismatch

HSBY_RD_P

PLCB_RUN PLCA_RUN Offline_if_OS_Mismatch

•

HSBY_WR_P: Write the command register %SW60

Manual Control BOOL Manual_Control_Enable

Command Run Mode Controller A BOOL PLCA_RUN

Command Run Mode Controller B BOOL PLCB_RUN

Forced Command OS no Mismatch BOOL Offline_if_OS_Mismatch

HSBY_WR_P

This block allows sending switch commands from the program (PLCA_RUN,

PLCB_RUN), also, in order to be able to update the CPU OS, the ETY module or the coprocessor, it allows to set the OS mismatch bit to 1 to avoid switching in offline mode.

A dedicated input allows sending switch orders for example, during maintenance activities.

•

HSBY_ST_P: Hot-Standby system, status check

This block allows to process data from register %SW61. It gives information about each PAC role (Primary, Standby, and Offline), OS version, and so on.

BOOL Hot-Standby System active

BOOL This Pac is PAC A

BOOL This Pac is PAC B

BOOL This Pac is Offline BOOL This Pac is Primary BOOL This Pac is Standby

BOOL Remote state Pac undefined BOOL Remote Pac is Offiline BOOL Remote Pac is Primary

BOOL Remote Pac is Standby

BOOL Identical Logic Pac A et Pac B

BOOL CPUs synchronized

BOOL Same CPUs OS

BOOL Same Copro OS

BOOL ETY version ok

BOOL Monitored ETY OS Mismatch

LOGIC_OK Mon_ETY_OS_OK CPU_SyncLink_OK CPU_OS_OK Copro_OS_OK ETY_minVersion REMT_UNDEF REMT_OFF REMT_PRI REMT_SBY THIS_OFF THIS_PRI THIS_Sby THIS_ISA THIS_ISB HSBY_ST_P HSBY_Active

Ethernet link monitoring DFBs

ETY_Monitor: Ethernet module monitoring

External default, Ethernet cable unplugged BOOL BLK Fault BOOL Module Fault Module Error BOOL MOD_ERROR

Command Run Mode Controller A (T_COM_X103) IODDT COM_ETY5103 COM_ETY5103 IODDT

Monitoring Rate value INT Monitoring_Rate Enable BOOL Reading Pulse READ_STS function Pulse computer in the Standby Section BOOL Pulse

Monitoring Rate current value INT RateEt RateEt INT ETY_Monitor

The “ETY_Monitor” DFB monitors the status of the Ethernet link provided by the TSX ETY 5104 (or TSX ETY 4103). We use as inputs the BLK and MOD_ERROR

information from IODDT T_GEN_MOD.

•

BLK: external default, Ethernet cable unplugged

•

MOD_ERROR: Module error

The IODDT T_GEN_MOD is updated by the READ_STS function. This function reads the status word of a ETY module. The execution rate is controlled by the Monitoring Rate parameter configured by the user (see Chapter 5: Implementation).

BOOL BLK Fault BOOL

BOOL MOD_ERROR

IODDT COM_ETY5103 COM_ETY5103 IODDT

INT Monitoring_Rate Enable BOOL Reading Pulse READ_STS function EN ENO

BOOL Pulse %CHx.X.MOD CH

INT RateEt RateEt INT

ETY_Monitor

READ_STS

ETY3_State T_GEN_MOD + MOD_ERROR BOOL EXCH_STS INT STS_IN_PROGR BOOL EXCH_RPT INT STS_ERR BOOL MOD_FLT INT MOD_FAIL BOOL CH_FLT BOOL BLK BOOL CONF_FLT BOOL NO_MOD BOOL EXT_MOD_FLT BOOL MOD_FAIL_EXT BOOL CH_FLT_EXT BOOL BLK_EXT BOOL CONF_FLT_EXT BOOL

NO_MOD_EXT BOOL Module absent or power down (only FIPIO extension)

Internal fault: Module failure (only FIPIO extension) Faulty channel(s) (only FIPIO extension)

External fault: Terminal Block (only FIPIO extension)

Hardware or software configuration fault (only FIPIO extension) External fault: Terminal Block

Hardware or software configuration fault Module absent or power down

FIPIO extension module fault Error while reading module status Module Faults

Internal fault: Module failure Faulty channel(s)

Module error Exchange status

Status parameter read in progress Channel report

The MOD_ERROR bit is set to 1 when an ETY module ceases operation. One frequent cause is the cessation of communication of a device on the I/O Scanning which, in our case, should not initiate a switchover. Therefore, in order to filter this occurrence, we use the T_COM_X103 function to monitor the I/O Scanning status and validate the MOD_ERROR value.

When implementing a Hot-Standby system, this block is used once for each ETY module in the configuration.

Switchover Management

SYNTH_FAULT: Performs the defaults synthesis

Synthesis Fault ETY Module BOOL Faulty_ETY

Synthesis Fault SCY Module BOOL Faulty_SCY

Synthesis Fault Scada BOOL Faulty_SCADA

Fault Mask word WORD Fault_Mask

Fault_Synth INT Synthesis Fault Word

Fault BOOL OS Versions Mismatch

SYNTH_FAULT

This block aims at processing the faults that would lead to a switchover. We find in input the results of the ETY and SCY modules failure detection. “Faulty_SCADA” is an input pin in the case of the communication between the SCADA and the PAC is monitored.

This DFB also processes:

•

Battery faults

ƒ %S67 = application memory card battery ƒ %S68 = processor battery

ƒ %S75 = data storage memory card battery

•

CPU fault

ƒ %S12 = CPU running

•

General In-Rack I/O fault

ƒ %S119 = fault of one or several I/O modules in the rack

•

Slots 3 to 10 fault

ƒ %SW160 = operating status of Premium modules installed on station 1 The faults processing is performed using the mask value set on the input pin “Fault_Mask”. This mask allows to select which fault to take into account according the configuration and to the user’s settings.

Each fault corresponds to one bit of the “Fault_Synthesis” word: BIT Element monitored

Bit 0 Battery Fault

Bit 1 Fault CPU

Bit 2 General In-Rack I/O fault

Bit 3 Fault on Slot 3

Bit 4 Fault on Slot 4

Bit 5 Fault on Slot 5

Bit 6 Fault on Slot 6

Bit 7 Fault on Slot 7

Bit 8 Fault on Slot 8

Bit 9 Fault on Slot 9

Bit 10 Fault on Slot 10

Bit 11 Ethernet Adapter(s) ETY Fault

Bit 12 MODBUS Adapter(s) SCY Fault

Bit 13 SCADA Fault

The result of this synthesis is saved in a word and set as an output on the

“Fault_Synth_Plc” pin. If there is at least one fault, the output pin “Fault” is set to 1. During the implementation of the system, this block is used twice: once for the Primary PAC and once for the Standby PAC.

In order to be able to compute the status of several ETY modules, logical “OR” and “AND” processing DFBs have been created:

BOOL FLT_ETY_1 BOOL

BOOL FLT_ETY_2 BOOL FLT_ETY_3

BOOL FLT_ETY_1 BOOL

BOOL FLT_ETY_2 BOOL FLT_ETY_3 SYNTH_OR_ETY FAULT_ETY SYNTH_AND_ETY FAULT_ETY

SWITCH_MANAG: Approve or deny a switchover

INT PRIM_DIAG

Synthesis Fault word Standby INT STBY_DIAG

Switchover Number Reset BOOL SWITCH_NB_Reset SWITCH_NB UNIT Switchover request

Manual Switchover BOOL FORCE FORCE BOOL Manual Switchover

Synthesis Fault word Primary

SWITCH_MANAG

The “Switch_Manag” DFB manages and counts switchover queries. The switchover approval is computed from the Primary and Standby PACs diagnosis coming from the “Fault_Synthesis” DFBs as seen above.

A switchover is allowed if:

•

The Standby PAC diagnosis is OK.

•

More than 30s have elapsed since the previous switchover.

Note: The time delay before the switchover takes place can be adjusted using variables of the DFB (Delay_Time_Before_Switchover). This delay is set to 1s by default.

The switchover counter can be reset using the input pin “Switch_N_Reset”. For maintenance reasons, the input pin FORCE allows a manual switchover of the system.

During the implementation, this block is used only once.

Switchover Time

Remote Pac is Primary BOOL Remote_is_Primary Sw_Timer TIME Switchover Time

This Pac is Primary BOOL This_is_Prima

Switch_Over_Time

The time gap during the switchover is a very important feature of the Hot-Standby system. A DFB has been defined to measure this time. The principle is based on the measurement of the time when the Primary PAC loses its Primary status and when the Standby turns Primary. This block, placed in the section 0, processes the system word %SW61 information and uses the ITCNTRL block function which allows event time measurements. The accuracy of the switchover time depends on the PAC scan time, for more accuracy, other measurement can be performed as described in the performance chapter.

3.3.3. In-Rack I/O System

This paragraph describes the management of the I/Os populating the main rack. Inputs acquisition is performed locally by both Primary and Standby PACs, whereas the Primary PAC outputs are mirrored on the Standby PAC (provided that there is no specific action programmed in the section 0).

Redundant Digital I/Os Implementation

Digital input and output signals are connected to the PAC through an ABE7 connection block. These signals are multiplexed/de-multiplexed by a Telefast connection device as seen on the above diagram (ABE7 ACC11 for the inputs and ABE7 ACC10 for the outputs). Exceptions detected on Digital inputs cannot initiate a switchover.

The digital I/Os implementation is illustrated on the diagram below.

Digital Outputs in the section 0

As the Standby PAC executes the first section (section 0) of the application program and then applies the object image %Q received from the Primary PAC, it is

important not to modify the redundant output status in this section. A

modification of the output bits in the section 0 can lead to an inconsistent status of the outputs as they are modified twice in the same MAST task.

Digital Outputs fall-back mode

In general, the outputs fall-back mode must be similar to their current mode in order to avoid an operation discrepancy during the switchover.

Pulse Triggered Actuators

Digital output redundancy implies a distortion of the command signal. Output modules are connected in parallel of the physical output, via a connection block. The result of a command is based on the length of the pulse and the delay after which the pulse is applied on the Standby PAC.

These mechanisms have to be taken into account in order to handle In-Rack digital output redundancy.

Positive pulse trigger

As shown on the above diagram, the length of the output is longer than the Pulse Time. This does not have any impact on the device behavior.

In the case where the delay is greater than the pulse length and using an actuator with a low response time, the signal received by the actuator might be composed of 2 commands, as shown on the diagram below:

Negative pulse trigger

As shown on the above diagram, the length of the output is shorter than the Pulse Time. This does not have any impact on the device behavior unless it cannot handle a shorter command.

The following diagram presents the case where the delay is greater than the pulse time of the output signal:

Because the delay is greater than the Pulse Time, the device will not receive any command.

Analog Inputs implementation

Analog signals are connected to the PACs through a signal duplicator. For our application, we use a JMConcept TELIS9000U2 module (which replaces reference JK3000N2)

The table below describes the signal range handled by the TELIS9000U2:

Standard scales

0/1mA; 0/10mA; 4/20mA; +/-1mA; +/-10mA; +/-20mA

User defined scales

from -22mA to 22mA

Standard scales

0/100mV; 0/1V; 0/5V; 1/5V; 0/10V; 2/10V; 2/10V; 0/50V 0/100V; 0/200V

User defined scales

from -110mV to 110mV; from 2V to 11V; from -200V to 220V PT100; PT1000

Ni100; Ni1000

Thermocouple J, K, R, S, T, E, B, N, W3, W5, NiMo

Potentiometer from 100Ω to 100kΩ

Resistance 0/200Ω; 0/1kΩ; 0/10kΩ;

Sensor Power Supply 2 or 3 wires, 24V - 29mA max

Output 1 Current 0/20mA; 4/20mA; from 0 to 20mA

Output 1 Voltage 0/10V; +/-10V - from 0 to 10V

Output 2 Current 0/20mA; 4/20mA; from 0 to 20mA

Output 2 Voltage 0/10V; +/-10V - from 0 to 10V Digital Output USB connector in front panel

RS485 Modbus Jbus isolated from input and output 1

Relay Output Relay: 1RT; 2RT; 3RT; 4T; 1RT & 1T INPUTS

OUTPUTS Current (continuous)

Voltage (continuous)

Probe

As seen on the next figure the signal from the process is duplicated and wired on both PACs thanks to the JMConcept module.

Analog Outputs implementation

The implementation is handled by a low-level commutation interface, in our case a JMConcept GK3000D1 module.

The principle is to select the output coming from the Primary PAC. This selection is performed by 2 relays controlled by a PAC digital output. The management of the relays can be performed either by 1 non-redundant output or by 2 redundant outputs to increase reliability. In our architecture, we choose to manage the relays with 2 redundant digital outputs.

The following table describes the GK3000D1 interface relays logical operation:

Relay A 1 0 1 0

Relay B 0 1 1 0

Output Channel Analog Input 1 Analog Input 2 last correct channel

Analog Input

Analog Input 1 4/20 mA Analog Input 2 4/20 mA

Digital Input 1 on optocoupler 30V max Digital Input 2 on optocoupler 30V max Analog Output 4/20 mA

Digital Output RS 485 isolated from input Modbus, Jbus

Digital link allows programing of the module Digital link allows acquisition of the measurements

INPUTS

OUTPUTS

Analog Outputs controlled by non-redundant Digital Outputs

The 2 PACs analog outputs are connected to a GK3000 D1 module. This module, controlled by digital signals, routes to its output, using 2 relays, one of its 2 inputs. In our case, the digital signals that control the GK3000D1 module are 2 digital outputs of the PACs. The main benefit of this solution is that it only uses 1 digital output to route analog outputs.

A wiring example is presented in the diagram below:

The Primary PAC must set to 1 the digital output that controls the relay in order to route its own analog output to the output channel of the commutation interface. The Standby PAC then sets to 0 the digital output that controls the other relay.

Note: This logical operating principle must be coded in section 0. The fall-back mode of the digital output module must be set to 0.

For example, according to the diagram above, if PAC_A is the Primary PAC, the digital output connected on the relay A is set to 1 while PAC_B digital output

connected to relay B is set to 0. This leads to route the analog signal A on the output of the communication interface.

Note: This solution is not used in our architecture.

Analog Output Digital Output

Analog Outputs controlled by redundant Digital Outputs

An analog output redundancy is performed, thanks to a GK3000D1 communication interface and redundant digital outputs, using 2 digital outputs per PAC. Each relay of the GK3000D1 interface is connected to a digital output.

Note: This logical operating principle must not be coded in the first section (section 0). For example, according to the diagram above, if PAC_A is the Primary PAC, the digital output No 0 (relay A) is set to 1 and the digital output No 1 (relay B) is set to 0. Thus, the analog signal ANA_ A is routed to the output of the communication

If… Then… PAC A is Primary Digital output number 0 is set to 1 (relay A) Digital output number 1 is set to 0 (relay B)

PAC B is Primary

Digital output number 0 is set to 0 (relay A) Digital output number 1 is set to 1 (relay B)

Analog Output Digital Output

3.4. Quantum Hot-Standby System

This chapter describes the different features and specifications of a redundant Quantum PAC system.

3.4.1. Quantum PAC Specifications

Primary and Standby PACs

Primary PAC runs the whole application program including the first section. It handles remote I/Os and updates the redundant PAC after each program cycle.

If the Primary PAC stops, the Standby PAC takes over the control from the Primary PAC in one cycle.

Standby PAC only runs the first section of the application program, checks the CPU and CRP modules availability and does not handle remote I/Os.

Note: CRP modules are needed in a Quantum Hot-Standby configuration, even if the system does not use remote I/Os. In this case, the two CRP modules are linked for monitoring purposes.

Local I/Os can be configured and used in a Hot-Standby Quantum system. However, they are not saved on the Standby system. They can be written in the Standby system with different values using the first section of the program (section 0). Moreover, the output modules management can only be performed using variables that are not transferred from Primary to Standby PAC.

Hardware Constraints

•

Non-compatible modules:

The following modules can not be used in a Quantum Hot-Standby configuration: 140N WM 100 00

140N OC 771 00

Note: Modules 140NOE77100 and 140NOE77110 are not available anymore.

•

Local I/Os:

The term “local I/Os” refers to the I/O modules located in the local rack, which do not take part in the redundancy system.

When no programming has been specified, the Standby CPU outputs state is imposed by the Primary CPU. Standby CPU outputs can be programmed in the section 0 (which is the only section executed in the Standby CPU) and they take the specified status unless they are forced in the Primary CPU.

Standby CPU inputs status (%l readable in the application and from the animation table) does not reflect the hardware Standby CPU inputs status, but instead is related to the Primary CPU inputs status.

The Standby module byte status indicates:

- the Primary module status in the case of a mixed module

- the Standby module status in the case of an output module (DDO)

It is important to manage the locals I/Os in the section 0 only by using the %MW words of the non transfer area.

•

Remote I/Os:

Communication modules cannot be used in a Remote I/O rack within a Hot-Standby application.

Software Constraints

We recommend not using TIMER events as they are not synchronized in Quantum Hot-Standby applications.

3.4.2. Quantum Hot-Standby DFBs Library

The following table summarizes the different DFBs created for the Quantum application.

DFB FUNCTION

ETHERNET NOE_MONITOR Monitoring NOE Ethernet Module

PROFIBUS PTQ_MONITOR Monitoring PTQ Ethernet Module

SYNTH_FAULT Synthesis Fault monitored elements

SYNTH_OR_NOE Synthesis Fault NOE module (Logic OR)

SYNTH_AND_NOE Synthesis Fault NOE module (Logic AND)

SYNTH_OR_PTQ Synthesis Fault PTQ module (Logic OR)

SYNTH_AND_PTQ Synthesis Fault PTQ module (Logic AND)

SWITCHOVER SWITCH_MANG Switchover Managment SYNTHESIS

Ethernet link monitoring DFBs NOE_Monitor

The NOE_Monitor DFB monitors the Ethernet link hosted by the N module.

OE

llows The monitoring is managed by the MBP_MSTR block function integrated in the NOE_Monitor DFB. The MBP_MSTR block a performing operations on communication networks, for example, the extraction of the local statistics of the NOE module.

ENABLE ACTIVE ABORT MBP_MSTR ERROR SUCCESS CONTROL DATABUF

This extraction rate is controlled by the “Monitor Rate” parameter configured by the user. Extracted data is available on output pins as shown on the diagram on the left.

BOOL MSTR_ACTIVE MSTR_ACTIVE BOOL

BOOL MSTR_DONE MSTR_DONE BOOL

BOOL MSTR_ERROR MSTR_ERROR BOOL

INT RateEt RateEt INT

INT Error_Count Error_Count INT

ARRAY[0..9]OF INT MSTR_Control MSTR_Control

MSTR_DataBuf

BYTE Slot Data_TCP DIAG_TCP

INT MonitoringRate LED_APPL BOOL

INT Retries LED_LINK BOOL

BOOL Pulse LED_RUN BOOL

NOE_Failure BOOL CFG_PORT BOOL ETH_100M BOOL OPTICFIB BOOL FULL_DUP BOOL FAULT BOOL EQUP_TYP UINT IP_AD_1 INT IP_AD_2 INT IP_AD_3 INT IP_AD_4 INT MAC_ADD_1 WORD MAC_ADD_2 WORD MAC_ADD_3 WORD MAC_ADD_4 WORD MAC_ADD_5 WORD MAC_ADD_6 WORD NOE_Monitor ARRAY[1..9]OF INT ARRAY[1..37]OF WORD

NOE Local statistics

In order to diagnose the health status of the NOE module, we check the status of the module and the link thanks to respectively Led RUN and Led Link. The number of faults is given by the MPB_MSTR. This number is compared to the “retries” value, if the number of faults is greater or equal than the “retries” value the NOE_Failure is set to 1.

The block implementation is associated to a data structure NOE_Monit. NOE_Monit Struct + MSTR_Active BOOL MSTR_Done BOOL MSTR_Error BOOL MSTR_RateEt INT MSTR_Error_Count INT Failure BOOL Fault BOOL Led_Appl BOOL Led_Cfg_Port BOOL Led_Eth_100M BOOL Led_Full_Dup BOOL Led_Link BOOL Led_Ooptic_Fib BOOL Led_Run BOOL EQUP_TYP BOOL MAC_ADD_1 WORD MAC_ADD_2 WORD MAC_ADD_3 WORD MAC_ADD_4 WORD MAC_ADD_5 WORD MAC_ADD_6 WORD IP_AD_1 INT IP_AD_2 INT IP_AD_3 INT IP_AD_4 INT EQUIP_TYPE UINT

+ MSTR_Control ARRAY[1..9] OF INT

+ MSTR_DataBuf ARRAY[0..37] OF WORD

+ MSTR_Data Diag_TCP

During the implementation the block is used as many times as the number of NOE modules.

Profibus link monitoring DFBs PTQ_Monitor

Fault Timeout TIME Fault_Timeout PTQ_FAULT BOOL General PTQ fault

Master PTQ Status WORD Master_Status Faulty_Active BOOL Active PTQ fault Master PTQ Operating WORD Master_Operating_State

Active PTQ Status BYTE Active_Status Faulty_Passive BOOL Passive PTQ fault Nb slaves seen by active PTQ BYTE Active_NbSlave

Passive PTQ Status BYTE Passive_Status Faulty_Passive_UDP BOOL Passive PTQ fault via UDP Nb slaves seen by passive PTQ BYTE Passive_NbSlave

Passive PTQ Status via UDP BYTE PassiveP_Status_UDP Nb slaves seen by passive PTQ via UDP BYTE Passive_NbSlave_UDP

PTQ_Monitor

A Quantum Hot-Standby system does not handle automatically the PTQ-PDPMV1 module redundancy. However, this module provides enough information about the health status of the Active and Passive Master (Active = Primary and Passive = Standby) to be able to develop a DFB to manage redundancy.

The health status information circulates through the Profibus network. That means that, if the Profibus link is lost, the modules are not able to communicate their health status. Therefore, we use the Ethernet UDP capability (“PTQ Link Message”) to set up the PTQ module redundancy function. An Ethernet crossover cable is needed to link the two modules. Thus, the Primary module can access the health status of the Standby module even if the communication link is lost.

Note: The link between the 2 PTQ modules can also be established using an Ethernet switch. In that case, the user can transfer the module configuration without unplugging the cable. This solution is also used when the distance between the 2 PACs is too important.

Data used by the PTQ_Monitor DFB

1. Data circulating through the Profibus network

•

ProfibusCRC32: Profibus Master configuration

This word describes the configuration of the Profibus parameters and of the slave devices.

•

PTQModuleCRC32: PTQ-DPPMV1 configuration.

This word describes the configuration of the PTQ module (mapping, and so on).

•

Profibus Master Operating State

This word describes the status of the master module.

0x0000 Offline 0x4000 Stop 0x8000 Clear 0xC000 Operate

•

ProfibusMasterModuleStatus: Profibus master module’s operating status

Bit 2 Application Status 0 - Application Stopped 1 - Application Running Bit 8 Data exchange

0 - There is no data exchange with any of the assigned slaves

1 - There is Data Exchange with at least one of the assigned slaves

Bit 9

Slave input frozen/cleared

0 - A slaves inputs in the IN area are cleared in a slave is not in Data Exchange

1 - A slave's inputs in the IN area are frozen if a slave is not in Data Exchange

Bit 12 Reset

0 - No action

1 - A reset is requested by the PROFIBUS Master module because a new database has been downloaded

•

HSBY Passive Status (Byte) – from Profibus interface

Bit 0 = PA

This bit indicates the state of the local master.

0 - Active master. Master is controlled by the Primary PAC

1 - Passive master. Master controlled by the Standby PAC

Bit 1 = SO

This bit indicates if the local master recognizes any of its assigned slaves as "offline".

0 - At least one slave is "offline" 1 - All slaves OK

Bit 2 = CE

This bit indicates if the local master has recognized an exception response.

0 - No exception response active 1 - At least one exception response active

Bit 3 = DB

This bit indicates if the local master has detected a database mismatch.

0 - Database OK 1 - Database mismatch

Bit 4 = OD

This bit indicates when the data in the output data area of the DPRAM is updated after a switch over.

0 - Output data is not updated

1 - Output data is updated (Once this bit is set, it remains set for the remaining session until the any bus is either reset or HSBY state changes to "Not Connected") Bit 5 – 6 = not used

Bit 7 = COM

This bit indicates if the counterpart is present

0 - Counterpart not present 1 - Counterpart is present

•

HSBY Passive number of slaves (Byte) - from PROFIBUS interface

Slave number seen by the Passive module

•

HSBY Active Status (Byte) - from PROFIBUS interface

Bit 7 = HS

This bit indicates that the Hot- standby functionality is enabled.

0 HSBY disabled. Module operates as "stand alone" master or HSBY state equals "Not connected".

2. Data circulating through UDP mailer

The data available via UDP mailer only apply to the Passive module. It is identical to the data found on Profibus network:

•

HSBY Passive Status UDP - from UDP HSBY Server

•

HSBY Passive number of slaves UDP - from UDP HSBY Server

•

HSBY Passive PROFIBUS CRC32 UDP - from UDP HSBY Server

•

HSBY Passive User Configuration CRC32 UDP - from UDP HSBY Server

3. Operation

The Active PTQ module is declared non-operational in the following cases:

•

exception response

•

application discrepancy

•

inactive Hot-Standby system

•

module in “Run” state

•

loss of communication with the slave devices

•

Master Operating State equals “ 0xC000=Operate”

The Passive PTQ module is declared non-operationel in the following cases:

•

Exception response (Profibus and UDP) and Passive_number_of_slaves_UDP equals 0

•

missing Counter part (Profibus and UDP) and Passive_number_of_slaves_UDP equals 0

•

module in Active mode and Passive_number_of_slaves_UDP equals 0

The Ethernet link which supports the UDP service is also monitored. If the status word PTQ_Passive_UDP equals 0, we consider the link as non-operational

In short, our DFB asks for a switchover when the Active module is non-operational and the Passive one is operating normally.

While implementing the Hot-Standby system, this DFB is used as many times as there are PTQ modules.

Switchover Management

Defaults Synthesis

Synthesis Fault NOE Module BOOL Faulty_NOE

Synthesis Fault PTQ module BOOL Faulty_PTQ Synthesis Fault Scada BOOL Faulty_SCADA

Fault Mask word WORD Fault_Mask

Fault_Synth INT Synthesis Fault Word Fault BOOL OS Versions Mismatch SYNTH_FAULT

This block aims at processing the faults that would lead to a switchover. We find in inputs the results of the NOE and PTQ modules failure detection. “Faulty_SCADA” is an input pin in the case of the communication between the SCADA and the PAC is monitored.

This DFB also processes:

•

Battery events

ƒ %S67 = application memory card battery ƒ %S68 = processor battery

ƒ %S75 = data storage memory card battery

•

CPU non-operating ƒ %S12 = CPU running

•

General In-Rack I/O non-operating

ƒ %S119 = event of one or several I/O modules in the rack

•

Slots 3 to 10 non-operating

ƒ %SW180 = operating status of Quantum modules installed on station 1 The faults processing is performed using the mask value set on the input pin

“Fault_Mask”. This mask allows to select which event to take into account according to the configuration and the user’s settings.