Establishing an Environment for Continuous Integration and Test of Flight Software

(1)

FSW

FSW--09, November 200909, November 2009

Establishing an Environment for Continuous

Integration and Test of Flight Software

Robert Klar, Christopher Mangels, and Randal Harmon Robert Klar, Christopher Mangels, and Randal Harmon

Southwest Research Institute

(2)

Integration and Test

Integration and Test is an important part of the

development process and represents a significant

investment in terms of cost and time

In recent organizational estimates, our software integration andIn recent organizational estimates, our software integration and

testing activities have accounted for more the 40% of the total

planned effort

Testing tends to be done late in the project lifecycle, limitingTesting tends to be done late in the project lifecycle, limiting its its

effectiveness

Setting up a test environment for a flight subsystem can be Setting up a test environment for a flight subsystem can be

challenging

Ground Support Equipment (GSE) often is more complex Ground Support Equipment (GSE) often is more complex

than the flight subsystem itself

(3)

FSW

Improving Integration and Test

Why put emphasis on improving integration and test?

Since integration and testing activities represents a large partSince integration and testing activities represents a large part of of

the overall planned activities, improving Return On Investment

here has a big effect.

W. Edwards Deming put forth the notion that W. Edwards Deming put forth the notion that ““You cannot test You cannot test

quality into a system.

quality into a system.””

This is true. However, by analyzing and understanding our testinThis is true. However, by analyzing and understanding our testing g

processes, we are able to collect information that can be used t

processes, we are able to collect information that can be used to o

improve our design and implementation processes. This can improve our design and implementation processes. This can improve quality.

improve quality.

Planning for integration and test should be an important

part of the software development plan.

Espouse the principle of Espouse the principle of ““Design for TestDesign for Test””

Hardware/Software resource planning is important (to avoid Hardware/Software resource planning is important (to avoid

Marching Army costs)

(4)

Continuous Integration and Test

Continuous Integration

Idea that emerged in Extreme Programming community [1]Idea that emerged in Extreme Programming community [1]

With multiple developers, components for a particular software With multiple developers, components for a particular software

item are being developed or modified in parallel. Over time, th

item are being developed or modified in parallel. Over time, this is may result in a very difficult integration (

may result in a very difficult integration (““Integration HellIntegration Hell”” [3]).[3]).

To avoid this problem, integration is done more frequently.To avoid this problem, integration is done more frequently.

A plethora of tools exist today for automating continuous A plethora of tools exist today for automating continuous

integration.

Bitten, Bitten, BuildBotBuildBot, , CruiseControlCruiseControl, , TinderBoxTinderBox, etc., etc.

Because there are so many choices, it is important to find Because there are so many choices, it is important to find

one that works well for you.

Continuous Test

Including testing with each integration is a good practiceIncluding testing with each integration is a good practice

(5)

FSW

Establishing an Environment

Desirable Characteristics for a CI&T Environment

Simple. The system needs to be simple to deploy and use. If Simple. The system needs to be simple to deploy and use. If

not, it will not provide any advantage to improving the I&T

process.

Reusable. We have many small independent software Reusable. We have many small independent software

development efforts. We do not want to

development efforts. We do not want to ““reinvent the wheelreinvent the wheel”” with with each one.

each one.

Cost effective. Since we are involved in many small programs, Cost effective. Since we are involved in many small programs,

expensive tools are difficult to procure. Open source tools and

low

low--cost commercial tools are attractive for this reason. Tools cost commercial tools are attractive for this reason. Tools with recurring license costs can be problematic.

with recurring license costs can be problematic.

Maintainability. Since space programs need support for many Maintainability. Since space programs need support for many

years, we need to make choices to guard against obsolescence.

Flexible. Science Instruments and Spacecraft Bus Subsystems Flexible. Science Instruments and Spacecraft Bus Subsystems

have different needs. We need to accommodate both.

(6)

Establishing an Environment

Plan for testing

–

do not let it evolve!

Resources required for testing are available at different pointsResources required for testing are available at different points in in

the schedule. This is true of most programs.

It is desirable to maintain consistency at each level of integraIt is desirable to maintain consistency at each level of integration tion

to avoid rework

Testing progress needs to be measurableTesting progress needs to be measurable

Number of test cases totalNumber of test cases total

Number of test cases completed successfullyNumber of test cases completed successfully

Need to be able to quickly adapt when problems are foundNeed to be able to quickly adapt when problems are found

Consider backup options for test environmentsConsider backup options for test environments

Setup a regular schedule for integration and test.

(7)

FSW

CI&T Case 1: Instrument (Juno

-

UVS)

Detector Electronics Detector Assembly OAP Mirror Grating Entrance Baffle Fail-Safe Door Detector Door Scan Mirror & Assembly Slit Aperture Door Pump-out Tube Ta Radiation Shield

UVS is an Ultraviolet Spectrometer for the Juno Mission (JPL). The spinning Juno spacecraft will perform science observations for about one year around 2017 from thirty highly elliptical 11-day polar orbits around Jupiter. UVS main observations will be performed +/- 3 hours of perijove each orbit.

(8)

CI&T Case 1: Instrument (Juno

-

UVS)

Host Workstation Telemetry and Command System with Python Interpreter (GSEOS) Keil51 Environment UVS Flight Software TCP Socket Serial I/F Sim Sim Input Host Workstation Telemetry and Command System with Python Interpreter

(GSEOS) _{UVS Engineering}

C&DH Board Terminal Server (ser2net) RS485 to RS422 Converter TCP

Socket Serial Serial

UVS Flight Software Sim Input Host Workstation Telemetry and Command System with Python Interpreter

(GSEOS) UVS Engineering or Flight Model UVS Flight Software TCP Socket Serial Spacecraft Emulator (SCE) UVS Detector Electronics High-speed Serial Signal Generator Analog

Plan for integration

in stages.

Using the same Using the same

Host environment, Host environment, allows for allows for progression and progression and regression regression

(9)

FSW

CI&T Case 1: Instrument (Juno

-

UVS)

Build environment is simpleBuild environment is simple

C language and makeC language and make

Our CI&T environment should also be simpleOur CI&T environment should also be simple

Evaluated CI tools such as Evaluated CI tools such as CruiseControlCruiseControl and Antand Ant

Not a good fitNot a good fit

Instead chose to base CI on Python and Instead chose to base CI on Python and croncron, and CVS, and CVS

Integration and test by Integration and test by ““Nightly ScriptNightly Script””

Rebuilds software and starts execution of Test ScriptsRebuilds software and starts execution of Test Scripts

Runs until stoppedRuns until stopped

(10)

CI&T Case 1: Instrument (Juno

-

UVS)

2009/11/03 03:59:26:TM

2009/11/03 03:59:26:TM--HK: 669760: Command accepted: Control Pixel StimulatorHK: 669760: Command accepted: Control Pixel Stimulator 2009/11/03 03:59:26:TM

2009/11/03 03:59:26:TM--HK: 669760: Command executedHK: 669760: Command executed 2009/11/03 03:59:26:TM

2009/11/03 03:59:26:TM--HK: 669760: Detector STIM activatedHK: 669760: Detector STIM activated 2009/11/03 03:59:26:TSTOK: : 010: all commands executed 2009/11/03 03:59:26:TSTOK: : 010: all commands executed 2009/11/03 03:59:26:TC : : Control Pixel

2009/11/03 03:59:26:TC : : Control Pixel Stimulator(StimStimulator(StimOff): 0xb3000000Off): 0xb3000000 2009/11/03 03:59:34:TM

2009/11/03 03:59:34:TM--HK: 669768: Command accepted: Control Pixel StimulatorHK: 669768: Command accepted: Control Pixel Stimulator 2009/11/03 03:59:34:TM

2009/11/03 03:59:34:TM--HK: 669768: Command executedHK: 669768: Command executed 2009/11/03 03:59:34:TM

2009/11/03 03:59:34:TM--HK: 669768: Detector STIM deactivatedHK: 669768: Detector STIM deactivated 2009/11/03 03:59:34:TSTOK: : 020: all commands executed 2009/11/03 03:59:34:TSTOK: : 020: all commands executed 2009/11/03 03:59:34:TFAIL: 669768: 030: No STIM counts availa

2009/11/03 03:59:34:TFAIL: 669768: 030: No STIM counts available cannot test event count rates.ble cannot test event count rates. 2009/11/03 03:59:39:TSEQ : :

2009/11/03 03:59:39:TSEQ : :

>>>:============================================================

>>>:==================================================================================== 2009/11/03 03:59:39:TSEQ : : >>>: Test Series Summary:

2009/11/03 03:59:39:TSEQ : : >>>: Test Series Summary: nightly_testsnightly_testshas been executing for 92 minuteshas been executing for 92 minutes 2009/11/03 03:59:39:TSEQ : : >>>: Summary results:

2009/11/03 03:59:39:TSEQ : : >>>: Summary results: 2009/11/03 03:59:39:TSEQ : : >>>: 56 Failures and

2009/11/03 03:59:39:TSEQ : : >>>: 56 Failures and 12 Warnings in 33 Tests12 Warnings in 33 Tests 2009/11/03 03:59:39:TSEQ : : >>>: 82.50% tests complete

2009/11/03 03:59:39:TSEQ : : >>>: 82.50% tests complete(33 of 40)(33 of 40) 2009/11/03 03:59:40:TSEQ : : 2009/11/03 03:59:40:TSEQ : : >>>:============================================================ >>>:==================================================================================== 2009/11/03 03:59:40:TSEQ : : 2009/11/03 03:59:40:TSEQ : : >>>:============================================================ >>>:==================================================================================== 2009/11/03 03:59:40:REMRK: : Attempting to start test s

2009/11/03 03:59:40:REMRK: : Attempting to start test sequence: 0160_checkmemequence: 0160_checkmem 2009/11/03 03:59:40:REMRK: : Test sequence started: 016

2009/11/03 03:59:40:REMRK: : Test sequence started: 0160_checkmem0_checkmem

(11)

FSW

CI&T Case 2: Central Instrument Data

Processor (MMS

-

CIDP)

Magnetospheric MultiScale (MMS) Mission

Constellation of 4 identically spacecraft in Constellation of 4 identically spacecraft in

variably spaced tetrahedron (1 km to several

variably spaced tetrahedron (1 km to several RR_E_E))

Ground contacts must be multiplexed in time Ground contacts must be multiplexed in time

in order to retrieve data from all 4 spacecraft

each day

http://stp.gsfc.nasa.gov/missions/mms/mms.htm

Overall Objectives: To discover the Overall Objectives: To discover the

detailed physics of the reconnection

process including its controlling factors,

its spatial distribution, and its temporal

behavior.

(12)

CI&T Case 2: MMS

-

_-

CIDP

_CIDP

Layered Software Architecture benefits testingLayered Software Architecture benefits testing

Architecture can be verified in layerArchitecture can be verified in layer

““Nightly ScriptNightly Script”” development is indevelopment is in--progressprogress

(13)

FSW

CI&T Case 2: MMS

-

_-

CIDP

_CIDP

Plan for integration in

stages.

Using the same Host Using the same Host

environment, allows for

progression and regression

Need to configuration Need to configuration

manage test environment

software as well as the

Flight Software!

We use CVSWe use CVS

Host Workstation (Linux) Telemetry and

Command System with Python

Interpreter (GSEOS)

SpaceBel SPARC Sim. CIDP FSW TCP Socket SpW I/F Sim Sim Input C2I Worksation (Linux) Host Workstation (Linux) Telemetry and Command System with Python Interpreter (GSEOS) TCP Socket

CIDP to Inst. I/f Card (PCI) CIDP FSW Input Sim / C2I I/F Loopback Engineering or Flight CIDP Host Workstation (Linux) Telemetry and Command System with Python Interpreter (GSEOS) CIDP FSW S/C Simulator Instrument Simulator TCP Socket TCP Socket (IGSE Prot.) SpW Serial Serial Serial

(14)

Challenges Ahead

Automated Continuous Testing generates large volumes of test datAutomated Continuous Testing generates large volumes of test dataa

Data analysis tools are necessary to make use of the test resultData analysis tools are necessary to make use of the test results s

in a timely manner

Currently test development significantly lags code developmentCurrently test development significantly lags code development

““How can I test it if I have not built it yet?How can I test it if I have not built it yet?””

Looking for applications of TestLooking for applications of Test--Driven Development [2]Driven Development [2]

Test Code is developed before a feature is implementedTest Code is developed before a feature is implemented

Test is run to verify a FAILTest is run to verify a FAIL

Feature is implemented and test run againFeature is implemented and test run again

Currently good quality simulation environments are expensiveCurrently good quality simulation environments are expensive

Adaptations of public domain emulators (Adaptations of public domain emulators (BochsBochs, QEMU, etc.) , QEMU, etc.)

provide promise to reducing costs here making complete virtual

software environments more affordable

(15)

FSW

Questions?

(16)

References

1.

1. Martin Fowler. Martin Fowler.

h

http://www.martinfowler.com/articles/continuousIntegration.htmlttp://www.martinfowler.com/articles/continuousIntegration.html. .

2.

2. Kent Beck.Kent Beck. TestTest--Driven Development: By ExampleDriven Development: By Example. Addison. Addison-

-Wesley. 2002.

Wesley. 2002.

3.

3. Ward Cunningham.Ward Cunningham.

http://c2.com/cgi/wiki?IntegrationHell