images of the train data to the corresponding pulse in the pulse sequence and also to the SRAM cell in which the image has been stored.
This Image Descriptor block is followed by the Detector Specific data block. The content of this block is not further specified and can be filled with any meta information which may be important for the specific detector. In case of the DSSC detector, the Detector Specific data contains the sensor values, provided by the SIB and special trailer words which are described in section 7.1.1.
4.3 Karabo - The XFEL Control Software Framework 4.3.1 Introduction to SCADA Systems
In large physical experiments, like the EuXFEL, the global control and supervision framework faces enormous challenges. Thousands of devices provide real time data in several hundred thousands of I/O channels. Also large numbers of computers and whole computer clusters have to be connected and integrated into one large system. Software frameworks which provide solutions for these tasks are called SCADA systems (Supervisory Control And Data
Acquisition). Central services which have to be provided by a such a global control system
are:
• Device configuration
• Device monitoring and process visualization • Data management
• State awareness • Automation
• Data logging / archiving • Alarm handling
In general, a SCADA system unifies distant computers, devices and user interfaces in one large framework and provides access for large numbers of users. Very important for any SCADA system is its reliability and robustness. The realization of the distributed system requires platform independent communication between processes on the same and also on distant systems. This is typically implemented in so called Middleware applications which hides the complexity of the operating system and its hardware and infrastructure by providing a special communication protocol. Since the Middleware layer describes the central communication layer for all devices, certain central tasks like logging and data management functionality are in general included in available applications. A typical configuration for a SCADA system is visualized in figure 4.6.
During the last decades different commercial and open source frameworks became available and are utilized both industrial and in science. Prominent examples for open source SCADA systems used in physical experiments are EPICS [49], TANGO [50, 51] or DOOCS [52]. Since every system has its own advantages and no system implements all
4. The XFEL Beamline Environment Standard PC Hardware µTCA , ATCA Embedded Systems User Interface Control Software DAQ / Database Commercial
Intruments Equipment Technical Hardware Custom Network e.g.
Ethernet
Client Client Client
Device Server Device Server Device Server
Figure 4.6.:Typical setup of a SCADA system. A central network connects device servers and
client applications. The device server is a specialized application that provides distant access to the hardware components. The device server can run on any hardware, in large experiments often µTCA or ATCA systems are used, also embedded systems or standard PCs are possible.
features of the others, the choice of the SCADA system to be utilized in large experiments is always a highly discussed topic. Moreover, the comparison of the various systems is very difficult since they are not implementing the same abstraction layers. Some interesting publications which try to compare selected SCADA systems are for example [53] and [54].
Figure 4.7.:EPICS Logo [49].
The central part of the EPICS (Experimental Physics and Industrial Control System) framework is the common high speed network protocol, called Channel Access (CA), which is specially designed for high bandwidth and soft real-time network communication. It can be used as an interface by almost any kind of application to connect to the global EPICS network. The EPICS environment brings a large library of software tools to integrate a variety of commercial and custom devices.
SCADA systems like EPICS and also TANGO (see below) are developed in large consortia and used by multiple large experiments like DESY, SLAC or KEK as examples for EPICS[49]. The enormous amount of invested development time in the order of more than hundreds man years has made them grow to reliable and well proven systems during the last decades.
4.3. Karabo - The XFEL Control Software Framework
Figure 4.8.:DOOCS Logo [52].
At DESY, a widely used system is Distributed Object Oriented Control System (DOOCS) [52]. It is developed since 1997 as a device based control system written in C++ with a focus on object oriented programming. Thus, hardware devices are represented in the framework as software devices, which are executed on device servers. One device object summarizes all methods and parameters which are required for its usage. A device server runs as a distinct process and can control an arbitrary number of devices. This allows independent execution of different tasks by different servers. The communication between devices is also controlled by the device servers which are connected by a network.
In a second hierarchy layer, hardware control devices can be controlled and summarized by middle-layer devices. In the middle-layer, much automation or safety functionality can be implemented independently of the client layer. The network communication is based on a server-client architecture model in which a central name server is used to manage the names of the available devices and servers for remote access. Properties and methods of distant devices can directly be accessed via Remote Procedure Calls.
User interaction is implemented by graphical clients executed on the respective worksta- tion. To create graphical interfaces, DOOCS implements its own JAVA tool called JDDD[55]. Due to the successful implementation at DESY at HERA and the predecessor X-ray FEL FLASH, both the electron accelerator and the long undulators of the EuXFEL are also to be controlled with DOOCS [56].
Figure 4.9.:TANGO Logo [57].
The TANGO control software is a half commercial SCADA system, which is developed by a European consortium and has been first presented in 1999 [51]. Its first development and application started at ESRF in Grenoble, France. It was developed on top of the TACO architecture which first introduced the concepts of software devices that are running in device servers similar to DOOCS. Like in DOOCS, TANGO also provides Middle Layer devices for further abstraction.
TANGO is based on the Common Object Request Broker Architecture (CORBA), which is a specification for application oriented Middleware applications with the focus on object oriented programming. The specification enables platform independent communication between software written in different languages and running on different computers and platforms. In the CORBA specification, the network device communication is implemented by an object request broker (ORB) which provides the interface to remote procedure calls (RPC) on distant (c++/JAVA/Pathon) class objects. The TANGO network layer is also often called the TANGO software bus after the various hardware buses which allow to connect
4. The XFEL Beamline Environment
large numbers of hardware devices to one communication network. Its modern architecture also implements mechanism for save execution in multiple threads. Further mentioned is always the high scalability of the TANGO control system and its light-weight device servers which can run also on embedded systems.
Until today, TANGO grew to a large consortium of 45 members and partners from both industry and research facilities all over Europe. Besides the ESRF, further large experiments use TANGO control. Just to name a few, there are several large synchrotrons e.g. Soleil in Paris, France, Elettra in Trieste, Italy, Solaris in Krakau, the first synchrotron in Poland and also Petra III at DESY in Hamburg, Germany, which are working with and participating in the TANGO control system. Despite the existing well proven SCADA systems, at XFEL the decision was made to develop an own software framework for beamline control and data management which will be presented in the following.
4.3.2 The Karabo Software Framework
For the control of the photon beamlines at the EuXFEL as well as the experiments and the associated detectors, XFEL develops an own new SCADA system. Its name is Karabo, which is the Sotho4word for ’I am the solution’. The first version of Karabo has been released in 2013[58],[59].
Detailed information about the design concepts of the Karabo framework are presented in the reference presentation [60], which is constantly updated. Because the implementa- tion of the DSSC Karabo devices is an important section in this thesis basic features and possibilities of the Karabo framework will be presented in this section.
For operation of the large 2D detectors a variety of different systems have to be controlled: power supplies, high voltages, vacuum systems, and of course the detector control and the data acquisition. The data acquisition is one of the biggest challenges for the Karabo framework because of the large amounts of data which are produced by each 2D Megapixel detector of up to 16 GByte/s. Besides the data storage also data correction, calibration and data analysis will be implemented in Karabo. First results of near to real time data handling of detector data in Karabo have been already presented in [61].
The network communication in Karabo is based on a central message broker similar to the TANGO control system. The architecture is shown in 4.10. Device communication is implemented in Karabo by the Open Message Queue (OpenMQ) [62], which is a message
orientedMiddleware platform. OpenMQ is an implementation of the Java Message Service
(JMS) [63] specification and runs as integral part of the Glassfish [64] application server. By using enterprise ready systems for these critical parts of the Karabo framework, it benefits from a well proved system.
The Hash Object
The central data structure in Karabo is the Hash object. A Hash is a fully recursive container class which implements key-value association for data access, where a key is a unique string and value can be of any type. The Hash supports hierarchical data structuring and preserves insertion order. For fast data access it is optimized for random key-based look-ups.
4Language from South Afrika, Lesotho and Botswana
4.3. Karabo - The XFEL Control Software Framework
Figure 4.10.:The Karabo architecture [58].All devices are connected (dashed lines) to the central
message broker, which supervises the message based device-to-device communication. The broker provides automated logging features and also alarm handling. The device server applications (indicated by rectangles) manage the access to all installed devices on the system on which the device server is executed. Central tasks of the framework functionality are implemented by special device servers like the GUI server or the data logger server. High-bandwidth point-to-point (p2p) connections (solid lines) can be defined to efficiently transfer large amounts of data (see text) between devices without interfering the central broker. The obligatory description of the device properties allows the Karabo GUI to automatically generate a user accessible representation of the device. This enables easy and quick integration of new devices into the graphical interfaces of the Karabo framework. The Karabo GUI and also the IPython, an interactive command line interface for python, provide full access to devices which are installed in the framework. This also includes adding of new devices, interactively generating of graphical user interfaces and controlling and monitoring of the devices.
4. The XFEL Beamline Environment
Efficient Data Transfer in Karabo (p2p channels)
In contrary to the application oriented Middleware of TANGO, in which distant procedures are directly called and executed, the message oriented Middleware (MOM) architecture allows for a much looser coupling of the single components. This means by implementing message queues, clients can automatically operate at their own pace while the messaging infrastructure absorbs all messages at production rate in order to guarantee zero message loss. Thereby, communicating clients can either be executed on same machine in different threads or on different machines in the same network. The OpenMQ implements both point-to-point (p2p) connections for high bandwidth data transfer and publish-subscribe messaging for general device access. All critical aspects of producer-consumer problems, like race conditions, have already been solved and safely implemented in the OpenMQ standard. The standard also supports topics, which can be assigned to device servers and consequently all devices which are managed by this server. All devices within a topic can be associated with a particular access level, which can be used, for example, to allow only a certain group of devices to change parameters of devices in this topic.
While in general all device communication like parameter access and data logging is realized by the publish subscribe mechanism and all communication messages pass the central broker, the direct p2p messaging does not interfere the broker. In general a p2p connection sends data via TCP packets. If two devices are executed on the same system data is transferred directly between the processes. The p2p connections can dynamically be connected and disconnected during run time. Multiple senders and receivers are supported per channel and several data handling policies can be selected. So it is always possible to decide whether data should be copied or received in shared mode. In shared mode there are two configuration modes:
• Round robin: Equally distribute the data. If the next input input line is not ready, the writing output channel will be blocked until this input channel becomes free. • Load-balanced: Data is sent always to the next available input channel. If the data
can not directly be handled one of the available policies (see below) can be defined. Very efficient if data recipients are expected to have different processing times. In any case, if the counter part of an input or an output is currently not available there has to be defined which will happen to the data. For each output, the following options are possible:
• throw an exception, which must be handled in the code.
• queue the data. Data will be appended to a queue until all memory is filled. • wait until an input becomes available. Effectively a blocking write at the output. • drop the data at the output and proceed without blocking.
The queue option can be very efficient when working with multiple senders and receivers but it can also gather significant amounts of memory if the amount of queued packets becomes too large. Consequently, it should be used carefully.
4.3. Karabo - The XFEL Control Software Framework
The data, which is transferred in a p2p channel, is always a Karabo Hash structure. Its format has to be declared at the output as well as at the input and the sender has to ensure that the transferred data fits to the declared structure. The Karabo framework can not check if the format of the transferred data is correct. Therefore, the developer has to make sure that the implemented data formats fit together.
The input and output channels which are provided by Karabo mean easy integration of complex concurrency functionality without facing the difficulties of race conditions and efficient data distribution. Karabo already provides an easy-to-use solution for this problem.
Additionally, Karabo supports event driven inter-process and device-to-device communi- cation. Therefore, an own event loop has been implemented which is also known from the signal-slot model of the Qt library [65, 66].
Karabo Devices
A Karabo device is described by its properties and slots. Properties can be of various types like strings, integers, booleans or vectors and are always accessed by their ’key’ string. Also various features like a description, alarm limits etc. can be assigned to each property. The slots define the access to the device methods. The description of the device properties and slots is mandatory and has to be added to the device Schema. Like in a Karabo Hash object, the device Schema can be nested and hierarchically structured in order to group things which belong together. The Schema defines the interface of the device to the outside within the Karabo framework. While all properties, similar to member variables, are accessible within all methods of a device, they also provide a self-description of the device for the integration into the Karabo framework. For example, other devices can access the properties described in the device schema.
A certain category of devices has to be mentioned at this point: the Middle Layer devices. These devices are specialized to access properties and slots of other devices. For simplified development, a special Python Middle Layer API has been introduced to describe these Karabo devices. This API reduces coding effort and provides convenient device control. Middle Layer devices are mainly used for simplified device usage and automated device control.
Karabo GUI
The GUI is written in C++ using Qt widgets. It defines the central control interface to the Karabo framework. By reading the given device Schema, the Karabo GUI generates automatically user accessible fields, check boxes and buttons. This automatic interface generation provides one of the big advantages of the Karabo framework because this allows instant testing, debugging and user access to a new device without the need of additional programming effort. The Karabo GUI also allows easy composition of more convenient graphical user interfaces by just drag and drop of device properties and buttons into the central scene field. More information about the Karabo GUI and its usage can also be found in the Karabo Documentation as well as in various tutorials.
4. The XFEL Beamline Environment Figure 4.11.: S creens hot o f the GU I [67]. (a) The liv e N avigati on, with co mputers, serv ers, devi ce cl asses and devi ces. (b) The list of properti es and slots of a devi ce, aut omati ca lly gen erated fro m its Schema. (c) A Scen e, sho wing sev era ldevi ce properti es. (d) The Project, whi ch can be used for gro uping of devi ces and scen es. (e) The message Logs. (f) Access to onlin e document ati on. 42
4.3. Karabo - The XFEL Control Software Framework
Karabo Summary
The comparison of different control systems is difficult since each system has its own advantages and no system implements all the features of the others. This is why there exists tools that blur the boarders between the single systems. Special TANGO to EPICS and TANGO to DOOCS converters are available which can be used to make TANGO or EPICS devices available in the DOOCS clients and vice versa. At the end, with Karabo, a system has been created which combines most advantages of the other systems in one homogeneous system. Karabo combines the software framework, the user interface and the programming interface in a single application.
The standardized framework to control everything provides a certain advantage in terms of staff training and integration of different components.
Preliminary studies [61] show that in theory Karabo can provide sufficient performance for complex calibration pipelines which offer near-to-real time image data calibration and scientific data analysis.
In the authors point of view, the mandatory description of the device properties and methods which is simultaneously used to generate a user interface and a programming interface defines one of the big advantages of Karabo. This feature allows instant generation of a device in the Karabo Framework without the additional effort to write a user interface.
Software development in a large framework is always more complicated than in a small standalone application. But with some experiences in programming languages, Karabo can easily be learned and the Framework and the GUI bring a lot of useful tools and