THE REQUIREMENTS FOR A COTS SOFTWARE COMPONENT: A CASE STUDY

THE REQUIREMENTS FOR A COTS SOFTWARE COMPONENT: A

CASE STUDY

Ljerka Beus-Dukic, Andy Wellings

York, YO10 5DD, United Kingdom E-mail: {ljerka,andy}@cs.york.ac.uk

Abstract

The goal of the GUARDS project is to design and develop a generic fault-tolerant computer architecture that can be built from predefined standardised components. The architecture favours the use of commercial off-the-shelf (COTS) hardware and software components. However, the assessment and selection of COTS components is a non-trivial task as it requires balancing a myriad of requirements from end-users and the preliminary architecture design. In this paper, we present the requirements and assessment criteria for a specific COTS software component, the operating system kernel. As an interface specification constitutes a major compatibility criterion for the selection of COTS components in GUARDS, a particular emphasis is placed on operating system conformance to the POSIX 1003.1 standard. We discuss the general lessons learned from the assessment process and raise a number of questions relevant to the assessment of any COTS software component.

1. Introduction

Traditionally, computing architectures deployed in safety-critical real-time applications were specialised to meet the specific requirements of the application domain for which they were built. Often these proprietary architectures rely on specially developed hardware and software. This practice has led to very costly and inflexible solutions that can not cope with rapid technological changes of the underlying hardware and software.

The GUARDS1 project is an ESPRIT project with an aim to design and develop a Generic Upgradable Architecture for Real-time Dependable Systems. Instances of this generic architecture should be configured from reusable hardware and software components and used in diverse application domains: nuclear, railway and space. To minimise cost and to maximise flexibility, the architecture favours the use of commercial off-the-shelf (COTS) components.

The assessment and selection of COTS components was an important activity in the GUARDS project, with a high potential impact on the project objectives. The selection of the right COTS component proved to be a non-trivial task and required careful consideration of multiple criteria and careful balancing between application requirements, component’s technical characteristics, financial and contractual issues.

Genericity is a fundamental objective of the GUARDS project. This objective served as a basis for the elaboration of hardware and software components at levels of abstraction high enough to support various kinds of real architectures and application domains. Reusability and composability were simultaneously enforced so that the final products can be assembled from predefined standardised components (COTS and GUARDS hardware and software components).

In this paper we present the issues related to the requirements specification, assessment and selection of one of the most important COTS software components in GUARDS – the operating system kernel. First, we explain how we obtained the requirements for the operating system. Then, we describe the general and specific GUARDS applications’ requirements which have guided the selection process. Section 4 presents,

1

GUARDS is partially financed by the European Commission as ESPRIT project 20716. For more details on the project see http://www.cs.york.ac.uk/~ljerka/guards.html.

in brief, the characteristics found in POSIX conformant operating systems. Then in Section 5 we discuss how these characteristics match the GUARDS requirements and highlight some of the issues discovered during the assessment process. Finally, in Section 6 lessons learned during and after the procurement process are reported.

2. Requirements Acquisition

At the start of the project, a requirements engineering method based on a semi-formal approach and developed by one of the project’s partners (Siemens Österreich), was seriously considered for the requirements specification process. However, due to the limited time end-users could spend learning how to use the tool which supports the method (RUTH − Requirements Engineering Through Hypertext) this idea was abandoned.

The requirements for an operating system kernel, as a specific COTS component, were acquired from different sources. Some of the general project’s requirements (e.g. composability and ability to be certified) were known from the conception of the project as they were included in the fundamental objectives of the project and as such could be found in the first document which describes the project (the project’s proposal). Other general requirements (e.g. stability, support for different programming languages) and the application specific requirements came out of the documents resulting from the end-users’ requirements identification task. As none of the authors participated in the requirements workpackage of the project, these documents did not contain explicit statements of the operating system requirements so they had to be extracted at a later stage. On the other hand, GUARDS architecture requirements could not have been known before this architecture was decided during the design task. In the meantime, with the limited set of known requirements (agreed at the meeting with the end-users), we conducted market research to identify candidate COTS kernels.

Using information available in textbooks, journals and on the Internet (both vendor independent sources and vendor supplied), we produced a kernel survey and offered it to the users. Two (out of three) end-users made their choice of the operating system during the design phase of the project. We have not participated in the actual selection process as our role is strictly advisory. However, once end-users selected operating systems, we helped them establish further requirements on the product (using a questionnaire) such as optional features which need to be included (e.g. memory management hardware and software). Furthermore, we were also in the position to check the claims made by vendors about the characteristics of their product (e.g. POSIX compliance, performance).

3. Kernel Requirements

The main requirement on the operating system was to choose a standard off-the-shelf operating system. This requirement imposed an early design choice since GUARDS fault-tolerant architecture must provide support for redundant hardware and enforce barriers to error propagation from low integrity to high integrity software. There is no commercially available operating system providing such support. Hence, a

1

12 3C

2

Application Fault-tolerance and integrity management Inter-channel communication network Inputs and unconsolidated outputs Consolidated outputs Output data consolidation Software integrity levels Channel redundancy Intra-channel lanes Local operating systems 4 3 I 2 1 M COTS components

Generic GUARDS components Application-specific components

Figure 1. GUARDS architecture

decision was taken to build fault-tolerance and integrity management on top of the operating systems of a pool of processors (see Figure 1).

3.1 Architecture Requirements

There are several requirements on operating systems kernels which are important to the GUARDS architecture:

• hardware transparency − this includes portability to a variety of hardware platforms, no or very little isolated machine-dependent code, and transparent extension to network operation. Upgrading of the underlying hardware should ideally only require porting of the kernel. Also a change of kernel shall have as little effect as possible on the GUARDS specific hardware components.

• scalability − a kernel can be scaled down for embedded systems and scaled up for large development workstations.

• facilities to support distribution (shared memory, network transparency).

• real-time support − an operating system kernel has to provide a set of basic facilities for real-time applications: clock access, support for different process priority levels, scheduling algorithms, interrupt handling, and pre-emption.

• facilities for spatial and temporal firewalling2 − an operating system kernel has to provide an environment in which application tasks can execute free from interference from other application tasks.

3.2 Application Specific Requirements

The industrial partners in the GUARDS consortium3 have very different requirements and constraints resulting from the diversity of their application domains:

The railway application is a control system in charge of train movements and protection which supports both vital and non-vital functions. Therefore

• vital computation and data must be protected from the effects of errors in the lower integrity non-vital software and data.

The nuclear submarine application is a part of an instrumentation and control system and requires:

• physical segregation of redundant hardware;

• a high degree of evolvability due to long deployment time (several decades); and

• unmodified COTS operating system(s).

The space application is a system for guidance, navigation and control of autonomous spacecraft with critical phases. Hence the requirements for

• reliability (15-year mission); and

• configurability (reconfiguration needed to optimise components in a critical phase). 3.3 General GUARDS Requirements

A number of general end-user requirements for the GUARDS operating system had to be taken into account during the assessment and selection process. These requirements are:

• runs on a wide range of target processors

• supports a wide range of boards/computers

• has interface compliant to standards (e.g. POSIX)

• runs in a flexible development environment

• has facilities for the development of specific device drivers

• is modular (i.e. different options could be added in)

• is configurable (e.g. a set of services offered can be tailored)

2

A kernel has a spatial firewalling capability if a memory space with exclusive rights of access can be allocated to a process. Temporal firewalling capability means that the operating system provides facilities for temporal isolation of tasks and interrupts (e.g. accurately monitoring of task execution time, stopping a task when its alloted execution time has expired, etc).

3 _{The GUARDS consortium consists of three end-user companies: Technicatome (France), Ansaldo Segnalamento}

Ferroviario (Italy) and Matra Marconi Space France; two technology-provider companies: Intecs Sistemi (Italy), Siemens AG Österreich PSA (Austria); and three academic partners: LAAS-CNRS (France), Pisa Dependable Computing Centre (Italy) and the University of York (United Kingdom).

• has support for different programming languages (e.g. C/C++/Ada)

• is certified/certifiable

• is UNIX compatible

• is designed for real-time (i.e. clocks, timers, interrupts available)

• has MMU support

• has multi-process support

• has multi-thread support

• is stable

• is popular (in particular in the specific application domain)

• has long-term support

This long list of requirements contains a mixture of functional and non-functional requirements, some of which are perhaps more important than others, but nevertheless accurately reflects end-user expectations from an operating system and its vendor.

4. A Criterion for GUARDS Kernel Selection

To meet most of the envisaged requirements of the GUARDS architecture and applications, a decision was made by the project’s consortium that the GUARDS operating system kernel has to be compliant to the POSIX standard. This decision stems from the accepted project’s strategy that interface specification constitutes a compatibility criterion for the selection of COTS components.

POSIX, the Portable Operating System Interface, based on the UNIX operating system, is an evolving set of standards that is being produced by the IEEE and standardised by ANSI and ISO. POSIX is the standard for the source code portability of applications from one system to another. On a system conforming to a particular version of POSIX, the applications which use the POSIX functions should be able to compile and run with a minimum of modification effort.

The first POSIX standard POSIX 1003.1 (IEEE 1990) defines the nucleus of common operating system functions that could be used to create portable programs between different platforms. The operations included are for process creation and deletion, signals, basic I/O and terminal handling. There are two extensions of POSIX which are particularly appropriate for GUARDS. The first addresses those services which are required by real-time applications (POSIX 1003.1b (IEEE 1993)). The second (POSIX 1003.1c (IEEE 1995)) addresses the notion of concurrent threads executing in a single address space (processes in POSIX execute in separate address spaces).

The POSIX 1003.1b specification provides standardised interface for the following functions:

• shared memory

• memory locking and protection

• asynchronous and synchronous I/O

• prioritised asynchronous I/O

• preemptive priority scheduling

• semaphores

• priority interrupts

• clocks and timers

• message queues

POSIX 1003.1b is an interface for a single processor. It does not address explicitly either multiprocessor or distributed systems (Gallmeister 1995). Some mechanisms provided, e.g. message queues, extend well to a

network model while some of the other mechanisms do not extend so well, e.g. shared memory. The interfaces of POSIX 1003.1c are specifically targeted at supporting tightly coupled multitasking environments, including multiprocessors. The specific sets of functions cover thread management and thread synchronisation (mutexes and condition variables).

The choice of a single requirement (the POSIX 1003.1/ 1003.1b/ 1003.1c conformance) proved to be very useful as it provided a narrower range of operating systems for the initial assessment. However, it did not take long to discover that even this single “core” requirement (Finkelstein, Spanoudakis et al. 1996) had to be relaxed to the POSIX 1003.1/ 1003.1b conformance only, as the choice of operating systems conforming to the POSIX 1003.1c interface standard was limited.

5. Discussion

The two operating systems chosen for GUARDS prototype applications are VxWorks and QNX. Both operating systems claim compliance to POSIX 1003.1/ 1003.1b only, and consequently a number of required functions from POSIX 1003.1c are not included (e.g. support for threads, mutual exclusion and synchronisation). However, both operating systems have not implemented all of the POSIX 1003.1b functionality (see Table 1). Note that this information, for example, is not made readily available for a potential user. In addition, how many users know what compliance or conformance to a particular POSIX standard actually means? According to the POSIX standard document “a conforming implementation shall support all required interfaces defined within” this standard. The statements regarding POSIX conformance provided by vendors range from “X conforms to the POSIX 1003.1 … and has been implemented according to drafts of the POSIX.1b and POSIX.1c …” or “Y supports many POSIX 1003.1b basic system calls, including …” to “Z includes POSIX.1 (certified) and many POSIX.1b real-time services”. What is the relevance of many when we want to know which ones?

Microkernels

POSIX functionality GUARDS

VxWorks QNX

pthreads ✔ ✖ ✖

fork ✔ ✖ ✔

semaphores ✔ ✔ ✔(partially)

mutexes and condition variables ✔ ✖ ✖

message passing ✔ ✔ ✔

signals ✔ ✔ ✔

timers ✔ ✔ ✔

asynchronous I/O ✔ ✔ ✖

priority scheduling ✔ ✔ ✔

shared memory objects ✔ ✔(not POSIX) ✔

memory locking ✔ ✔ ✖

Table 1. GUARDS profile functionality for different microkernels

In order to be able to draw a complete list of the GUARDS requirements for the operating system substantial work had to be done on the preliminary design of the GUARDS architecture and subsequent

applications. One of the results of this work is the GUARDS computational and scheduling model (Beus-Dukic and Wellings 1998). The GUARDS computational model consists of the mechanism for releasing executable functions and the mechanism for synchronisation and communication between these functions. The scheduling model dictates the way in which these functions are executed at run-time. The role of an operating system kernel is to provide the run-time support generic to all GUARDS applications conforming to the computational and scheduling model. Thus, it is necessary to know exactly which of the POSIX functions are deployed in a specific operating system under assessment.

A more general issue is to what degree one can rely on the scarce information provided by vendors in their product descriptions and how much effort (time and money) one has to spend before gaining sufficient insight into the specific product feature. In our experience getting specific information on the product is a long and tedious process and in the case of this particular feature of the operating system (details of POSIX conformance) was performed after the product had been purchased.

Comparison

Another issue is a need to make common attribute comparisons of COTS software components. At the moment it is often difficult to make such comparisons. One would assume that choosing an operating system, the oldest COTS software component, would be devoid of such a problem but unfortunately that is not the case. For example, one has to compare one vendor’s “typical scheduling latency” with another’s “Suspend/Switch/Resume/Switch time”; or one vendor’s “typical context switch time” with another’s “observed and raw context switch time”. Given this, how can a fair comparison be made of various operating systems?

Domain Knowledg

e

The use of COTS software components for safety-critical real-time applications is of major concern (Sinclair 1995), especially for the components which form an integral part of the end application (e.g. operating systems). The requirements of this domain, such as fault-tolerance and compliance to certification standards, have to be considered amongst other requirements and appropriate decisions made. Unfortunately, there is little assistance from standards and guidelines on justifying the quality of COTS software (Arlat 1997). When assessing different operating systems we found that the certification requirement was particularly difficult to match as only one vendor claimed to have a certified product. Since COTS software components can play very critical roles, other strategies for tolerating design faults have to be considered (e.g. GUARDS uses diversified redundant COTS software components).

6. Lessons Learned

First of all, the end-users did not expect us to recommend a specific COTS product. What they needed was a set of requirements for the operating system kernel which would help them in their own selection process. Nevertheless, the exercise proved to be useful and taught us a few lessons:

Lesson 1

Mandatory requirements must be described in sufficient detail to enable component assessment.

To reduce the overall time spent on assessing components from different vendors it is necessary to have detailed information about the mandatory requirements for the component (preferably from a vendor-independent source). For example, the POSIX compliance claims made by vendors of different operating systems was only possible to check by comparison with the existing POSIX standards.

Lesson 2

Enough details about how mandatory requirements are met must be obtained from a COTS component vendor prior to component selection.

It is important to gather enough evidence to support vendor claims about meeting mandatory requirements before the selection is made. The component vendor should cooperate in this stage but, if this is not the case, it could be a sign of future problems in communications with the vendor. An additional source of evidence could also be another user or a group of users already using the component.

Lesson 3

Choices made by other users from the same application domain must be included as valid requirements during the component selection.

Standardisation activities in a particular application domain often result in a set of specific requirements which need to be included in the requirements of any system built for that domain. Consequently, the choice of components is reduced to the set that satisfies the general requirements of the application domain. The component selection time and effort can be significantly reduced if we know how these specific requirements are met by the components selected by other users from the same application domain. This lesson is not a new one for the industrial users in the application domains with a good tradition of cooperation (e.g. space domain) as they will not hesitate to choose components already chosen by other users from the domain (e.g. VxWorks kernel).

Lesson 4

Legacy and future applications’ requirements must be included in a set of requirements before component assessment.

It is very useful to know in what kind of legacy applications a particular component was used or is still being used. This will inevitably influence the decision-makers who are likely to look beyond component use in a current application, examining both the past and the future.

Lesson 5

Requirements for component framework must be part of requirements for the component.

Component specifications need to list the frameworks that are either required or could optionally be interfaced with. A component has to be compatible with other components in the framework. Choosing component on the basis of the component framework could be an effective selection criteria when looking for components that need to support a chosen component framework. In the case of an operating system, the choice is often dictated by the operating system availability on the target hardware.

Lesson 6

Requirements need to be described in a way that will allow efficient retrieval of information provided by component vendors.

Despite recent technological advances, which enable easy access to information, locating the components and component vendors based on the requirement profile is still a problem. Components are often described in a vendor’s ‘language’, which does not necessarily correspond to the ‘language’ used by a potential customer. For example, during the assessment of real-time kernels we often had to make ‘translations’ between the characteristics we were looking for and the ones given by different vendors. Not only did they often not directly correspond, but also they were occasionally buried inside other component characteristics.

7. Conclusion

The GUARDS architecture pledges the use of COTS components wherever feasible. This stems from the project’s fundamental objective to develop a generic architecture for the range of safety-critical real-time

applications. We have presented the requirements for a single COTS software component – the operating system kernel. Although, we have chosen a single criterion for the selection of commercially available operating system kernels – conformance to the POSIX standard – we have found that some of the POSIX functionality needed for the GUARDS architecture was missing and alternative solutions have to found. Throughout the process of assessment and selection we have also came across a number of issues, some of which need to be addressed by the research community.

8. References

Arlat, J. (1997). Preliminary Definition of the GUARDS Validation Strategy. LAAS-CNRS Report 96378, GUARDS Report D3A1/AO/5002/E.

Beus-Dukic, L. and A. Wellings (1998). Time-Related Dependability Mechanisms in GUARDS. Proceedings of the DASIA’98 Conference on Data Systems in Aerospace, Athens, Greece, 25-28 May 1998 (SP-422, July 1998), ESA Publications Division, pp. 335-40.

Finkelstein, A., G. Spanoudakis, et al. (1996). Software Package Requirements & Procurement. 8th Int. Workshop on Software Specification & Design, IEEE CS Press.

Gallmeister, B. O. (1995). POSIX.4, ISBN 1-56592-074-0, O’Reilly&Associates.

IEEE (1990). Portable Operating System Interface: Part 1: System Application Program Interface (API) [C Language]. IEEE. Std 1003.1-1990.

IEEE (1993). Portable Operating System Interface: Amendment 1: Realtime Extension [C Language]. IEEE. Std 1003.1b-1993.

IEEE (1995). Portable Operating System Interface: Amendment 2: Threads Extension [C Language]. IEEE. Std 1003.1c-1995.

Sinclair, I. J. (1995). The Use of Commercial Off-the-Shelf (COTS) Software in Safety-Related Applications. HSE Contract Research Report 80/195.