DigiLog – An Electronic Logging System for Operations
Alex Baumgartner1
SOLENIX GmbH, Kappel, 4616, Switzerland Marta Pantoquilho2, and Sebastian Martin3
SOLENIX Deutschland GmbH, Darmstadt, 64293, Germany Alessandro Donati4
European Space Agency, Darmstadt, 64293, Germany
The logging process of spacecraft and ground station operations activities at the European Space Operations Centre (ESOC) is mainly based on paper logbooks. While this traditional approach is very flexible and practical, the disadvantage is that all the valuable information contained in the logbooks is not accessible for further use in an efficient way. DigiLog is a project at ESOC with the objective to validate the usefulness of an electronic logging system for operations. The system is web-based and supports multiple missions and users in parallel with detailed access permissions. The system is used by selected test-bed missions since some months and has received throughout positive responses. The full advantages of an electronic logging system will be shown in the future by follow-on projects in the areas of data mining and automation. The paper introduces the system, its functionalities, results of the field tests and provides an outlook to the future usages of the system.
Nomenclature ECC = ESTRACK Control Centre
ESA = European Space Agency
ESOC = European Space Operations Centre ESTRACK = European Space Tracking Network FCT = Flight Control Team
MCS = Mission Control System SPACON = Spacecraft Controller
I. Introduction & Objectives
HIS paper introduces an electronic logging system that is currently under development and testing at the European Space Operations Centre (ESOC). Although the paper focuses on the use of the system for operations of satellite missions, the concept is of generic interest for any logging process and the implemented system will also be used in other areas.
T
The logging of activities and events is an important task in mission operations. It allows keeping track of a mission’s history, including details of the activities, occurrences of events and anomalies etc. The log is also a fundamental input to the anomaly investigation process and supports the smooth functioning of a team working in shifts. Therefore accurate logging of activities is an important duty of a Flight Control Team (FCT).
Currently, activity logs at ESOC are paper-based, i.e. traditional log books are used. One reason for this is that
SpaceOps 2008 Conference (Hosted and organized by ESA and EUMETSAT in association with AIAA)
AIAA 2008-3300
log books is that all the information contained within the books is not accessible electronically and thus can not be post processed directly or used by an electronic device.
In 2007 a project was started at ESOC with the objective to validate the usefulness of an electronic logging system for operations, to identify suitable technologies, and to capture detailed user and system requirements. It was decided to implement a prototype system to be tested under real conditions.
In the beginning of the project different concepts were evaluated, including paper-based and electronic hybrid systems, purely electronic systems, and systems with different hardware components, e.g. digital pens or tablet PCs. The various concepts offered different advantages in the areas of flexibility, mobility, efficiency, reliability and more.
The user-preferred solution for the prototype system was a purely electronic web-based system due to the simplicity of the concept, the flexibility and independence in the development, and the low implementation costs. The focus of the prototype system was to be on flexibility, simplicity and efficiency, since these are the main advantages of the paper log. The main concern of the FCTs was that an electronic system would be cumbersome to use, which is of particular importance during times of high activity, e.g. launch and early orbit phases (LEOP) and critical maneuvers.
DigiLog is under development by the Advanced Mission Concepts & Technologies Office1, OPS-HSC, at ESOC. The office consists of a small team of researchers with the aim of identifying new technologies for innovative operations concepts to comply with the challenges of future missions.
Representatives of ESA currently flying missions have been involved during the requirements elicitation and early prototyping phase. Two missions were chosen to serve as test users and drivers of the prototype development.: XMM-Newton and INTEGRAL. The intention was to use DigiLog during the development phase in parallel to the operational paper logs, i.e. as a shadow system.
II. Current Status
Today (status April 2008) the system is still a prototype, but it is stable enough to be used operationally. In fact, the system is used operationally by the missions XMM and INTEGRAL. Other FCTs, as well as the ESTRACK Control Centre (ECC), responsible for operating the ESA ground stations, have expressed their interest and are currently test-using the system.
Most of the functionalities requested by the mission users have been implemented. Further adaptations to the specific needs of different groups of users (such as the ECC users) might be implemented in the future.
A. Functionalities
Being a web-based system the users can access DigiLog through a web-browser. The user is authenticated on a login page by username and password. After the successful authentication, the user has to select a log book (a user can have access to several log books from the same mission or from different missions). The main page of DigiLog contains the log itself, i.e. a list of configurable length containing the log entries. New log entries can be written using a rich text editor and additional controls. The editor supports formatting of the text, e.g. to highlight important parts in different colors.
Each log entry contains as a minimum the following information: event date and time, message type (e.g. information, warning and error), author, and the log entry text which is a free text field. In order to speed up the process of entering a new record into the log, the timestamp is by default the time of submission of the entry. In addition, standard entries which are often used, such as procedure names are provided as selectable text snippets to be automatically included into the free text field entry. Attachments, e.g. files and screenshots, can be added to a log entry as well as comments and reviews. In order to keep track of the entries at any given time, the writing of new entries is made on the same page where the last entries are displayed, as depicted in Figure 1.
Figure 1: Screenshot of the main page of DigiLog. The writing of entries is done at the top of the page. After submitting the new entry it is added to the list of entries below.
To facilitate the handling of large numbers of entries and to keep the site responding fast when browsing through a logbook, the number of entries displayed can be limited. This entries pagination includes limitation by rows or day of year of entry, and the entries can be sorted by any of the columns, i.e. date, message type, author etc.
After an entry has been inserted it cannot be modified anymore. This behavior is adapted from the characteristic of a paper logbook and it ensures that data is not volatile and cannot be lost or tempered in any way. However, it is possible to add reviews to an entry. The review is a new version of a log entry and it is used to correct errors in an entry, e.g. typos and wrong information. The main page of the logbook always displays the latest version of a log entry, i.e. the last review instead of the original entry. Any entry can have several reviews and it is always possible to see the history of a log entry. Each review must be accompanied by a mandatory justification for that review. All entries (original or review) contain the identification of the user that created it. In addition to the reviews it is also possible to add a comment to a log entry in order to provide supporting information.
The system is designed to support many logbooks in parallel, i.e. virtual logbooks. In fact DigiLog could support many missions in parallel, including multiple satellites in one mission, and multiple logbooks per satellite, e.g. LEOP logbook, simulation campaign logbook, routine mission logbook etc.
There is a direct print functionality from the main page of DigiLog for printing the currently visible entries in a printer friendly format. It is possible to print directly to an installed printer or to a PDF document. In order to facilitate information exchange from within the logbook or across different virtual logbooks, DigiLog offers search and filter functionalities. Log entries can be filtered and searched based on any of the log entry fields. The exportation of a logbook is possible to tab separated files or HTML file formats.
The search, filter and export functionalities can be applied to one log book or to a set of logbooks (provided that the user performing the operation has at least read access to all of them). In order to promote the exchange of data between missions and to facilitate data analysis across missions, it is possible to visually merge the entries of several logbooks, i.e. display the contents of several logbooks mixed together.
The Logbook Management section allows to create and change the recurrent text entries, open and close logbooks and to create new logbooks depending on the role of the administrator.
A responsible person of each mission is assigned as logbook administrator to the corresponding logbook to minimize and divert the management effort from a single global administrator. Global administrators are assigned within the development staff with the ability to create and delete logbooks and to assign new local administrators. B. Architecture
The DigiLog pages are provided to the user from a web server using PHP to dynamically generate the content of the pages. The PEAR libraries are used as a layer to the database and MySQL serves as database to store the entries, users and settings. The architecture schematic is depicted in Figure 2.
The database is replicated in near-real-time to a second MySQL database on another server. The second server is shut down on a daily base to back-up the database to external drives.
This approach increases the system availability and reliability. However, it is not an optimal backup strategy as
e.g. in case of a master database failure, the system would still need a downtime to copy the redundant database back to the primary system. While accidental deletion of logbooks or entries is prevented by the database table constraints, an accidental deletion of the database would be replicated to the back-up database as well. In such an event, the database can only be rolled back to the time of the last daily backup to the external drives. A real operational system should provide real-time, incremental backups and a hot failover in case of server or database errors, i.e. switch seamlessly and automatically to another server without users noticing any system outage or effect. Currently the users keep a paper logbook in the drawer in order to write the log entries in case of a system failure, until the system is recovered. With a hot failover server, this would not be necessary.
Figure 2: Digilog current Architecture: two servers ensure real time data replication.
Currently the system can only be accessed from inside the ESA networks, as a security precaution. However, ideally the system should be accessible from the outside in order to allow users to read the log remotely, e.g. while a FCT member is on mission or for the mission scientists that are stationed at a different location. This access will be provided by a third server containing a replication in near real time of the Master database. This server will be available only for reading purposes. With this solution, depicted in Figure 3, we aim at 1) ensuring that the entries are inserted only from the operational environment inside ESOC, mimicking the paper log book security, and 2) reducing the risks of hacking attacks to the system, since the database itself will only accept reading queries and the data will only flow in one direction: from the master server into the outside server.
III. Results of field tests
During the development of DigiLog, importance was given to simplicity and user-friendliness, flexibility, and efficiency in adding new records. These are the main advantages of a paper logbook and the main concern of the FCTs regarding an electronic system. The system has been in use by the missions XMM and INTEGRAL since
Master Server Slave Database Master Database ESA Networks Clients Slave Server Outside Database Outside Server Internet Clients Data flow direction
Figure 3: DigiLog future Architecture: three servers ensure real time data replication and outside access from a secured and controled environement.
Legend:
and the left pages are for additional notes, which add a lot of flexibility to the logbooks. These additional notes can be used for different reasons, e.g. to provide further information that does not fit into the line of the log entry, to record information that should not be part of the official log, to draw sketches etc. DigiLog (currently) does not provide such functionality. Additionally, a paper logbook does not have system outages, and is not susceptible to hacker attacks, i.e. a paper log is more error resistant. However, the field tests also show that the advantages of having an electronic system outweigh the disadvantages. If we additionally consider the potential of future applications based on DigiLog, the advantages over the paper logbooks are enormous.
The users have been involved in the project from beginning (in the spirit of incremental iterative development). This ensured that the development was going in the right direction, i.e. all developments were according to the users’ needs. The vast majority of the currently available functionalities are useful and actually being used by the FCTs. The only exceptions are the functionalities related to multiple logbooks, e.g. merging, searching across different books etc. This is due to the fact that there are currently only two missions using DigiLog and the amount of records is still relatively small.
Some of the concrete advantages of DigiLog are presented in the following list.
• No more deciphering of handwritten entries. All information is stored in machine readable format.
• The work place is more organized and there are no logbooks on the desktop. The computer required to enter the records was there anyway.
• No need for large archives of the logbooks. Everything is stored electronically.
• Remote access to the logbook. This includes various advantages. It is not required to go to the mission control room in order to read the logbook. The information in the logbook can be accessed even from outside ESOC. Therefore the FCT does not have to send this information to external parties anymore. Remote access also counter-balances the fact that it is not possible to carry around an electronic log system. (In addition, the logbook can be printed if a hard copy is needed.)
• Possibility to copy and paste error messages from other systems, e.g. from the Mission Control System (MCS). These messages are often cryptic and manually copying such messages is an error-prone process. • Possibility to add attachments, e.g. screenshots, instead of printing, cutting and gluing it into the logbook. • Often repeated entries can be selected for auto-insertion, therefore reducing the amount of text to be
written.
• Specific information in the log can be found easier, due to the search and sort functionalities.
Apart from these direct advantages and benefits, the electronic log system opens the door for many other application and benefits, as described in the section IV.
The field tests also yielded that the current architecture of the system is not ideal. The system should provide a hot stand-by in case of failures. Due to available resources, the current system needs several manual steps to repair the system in case of an error. Managed on a best effort basis, several hours of downtime could occur.
C. Challenges
During the project, different challenges had to be overcome. Apart from the ones related to the software development, two particular challenges were met in this project.
ESOC uses different networks for different purposes and each one has its own policies and restrictions. It was difficult to find and optimal network configuration that allows to fully exploit the possibilities of DigiLog. For example, in order to copy and paste messages from the Mission Control System, it is necessary to access DigiLog from the machine that is running the MCS clients. This means that the system has to be accessible from the Operations Network which is very restrictive. On the other hand, the system was also to be accessed from outside the ESA networks. It is possible to find an ideal configuration using several servers and mirror the content from one to the others. However, being a prototype project the required resources for this were not available, and some of these capabilities had to be restricted.
During the project it became apparent that different FCTs have different procedures and habits for logging
IV. Next steps The next steps in the project are described in the following list.
• Refactoring: DigiLog is currently undergoing a refactoring process. This process will clean up and streamline the source code.
• Back-up and redundancy: as more resources become available, a more solid back-up and redundancy strategy will be implemented. This will involve additional servers as mirrors and hot-stand-bys, and real-time back-ups.
• New functionalities and bug fixes: the project’s task list still contains some functionalities to be implemented and software bugs to be fixed. This will be done in parallel to the refactoring process. One example of these new functionalities is a field for generic notes and comments, i.e. not attached to a single log entry. This should resemble the left page of the paper logbooks.
• New users: DigiLog is currently under evaluation by other FCTs and by the ECC. New users most probably also require new functionalities.
• Network configuration: with more servers becoming available in the frame of the back-up and redundancy upgrades, it should also be possible to find a better network configuration that supports the full capabilities of the system in terms of (remote) access.
D. Possible future work
DigiLog has big potential for future applications and for extensions. Especially, when more missions will use the system, a wealth of valuable information will be collected that will be useful not only for statistical analysis and quality management of mission operations, but also for data mining in general, across missions and mission phases. By applying state-of-the-art artificial intelligence algorithms to the logs it might be possible to identify cause-effect relationships between certain events and anomalies, or patterns in the behavior of the spacecraft2.
Part of the logging process could be automated by a process reading the MCS logs and inserting relevant information into the log.
Some missions create daily operations reports or pass reports after ground station passes. Most of the inputs for these reports are taken from the logbooks. In the frame of another project3 at OPS-HSC, an automatic generator of mission operations reports is developed. By connecting the two systems, it will be possible to automate – to a large extent – the creation of the daily- and pass-reports.
If the requirement for using paper logbooks persists for some users, it is possible to start using digital pens to write the log entries in the books. Digital pens write on paper and at the same time they record what is written. When put into the cradle connected to a machine, the information is automatically transferred to the machine. The use of these pens could be embedded into DigiLog, i.e. the users enter the log records handwritten into the logbooks and as soon as the pen is put into the cradle, the records are transmitted to the electronic system. First tests with digital pens have been performed and provided promising results. Particularly the handwriting recognition worked very well.
V. Conclusion
DigiLog, an electronic logging system has been introduced in this paper. It is a prototype application developed by the Advanced Mission Concepts and Technologies Office at ESOC. It is currently in use by the Flight Control Teams of the missions XMM and INTEGRAL, and under evaluation by a number of other missions. The field tests showed that DigiLog provides many advantages over the traditional paper logbooks. These advantages outweigh by far the drawbacks in the areas of flexibility and simplicity. Many of the initial concerns of the FCTs could be cleared by developing appropriate user interfaces and functionalities.
DigiLog opens the door to many future applications that will bring additional value to operations, such as data mining, automation of reports and automation of logging.
To make optimal use of the system it is required to implement standard logging procedures applicable to all missions using the system. Furthermore, a more suitable hardware and network configuration should be put in place to provide access to the system from all the relevant networks and to ensure appropriate back-up services and redundancy.
The feedback from the test users on the current system is throughout positive and it is expected that in the near future several other missions will start to use the system.
Acknowledgments
The authors thank the members of the Flight Control Teams of the missions XMM, Mars Express and INTEGRAL for their support in the project. In particular Stephan Anstoetz, Norbert Schmidt and Paolo Lippi provided invaluable feedback during the development of the system.
References
1
Donati, A., Martinez-Heras, J. and Baumgartner, A., “Advanced Technology Infusion in Support of Mission Operations: The ESOC Experience,” Proceedings of the 7th International Symposium on Reducing the Costs of Spacecraft Ground Systems
and Operations (RCSGSO), Moscow, Russia, 2007.
2
Martinez-Heras, J. and Donati, A., “Data Mining at the Service of Space Operations,” Proceedings of the2006 SpaceOps Conference, Rome, Italy, 2006.
3
Baumgartner, A. and Donati, A., “Automating Reporting Tasks in Operations,” Proceedings of the 2006 SpaceOps Conference, Rome, Italy, 2006.
