Agile Systems Integration Process
Jason D. Winningham, David J. Coe, and Jeffrey H. Kulick
Department of Electrical and Computer Engineering
University of Alabama in Huntsville, Huntsville, Alabama, USA
Abstract - This paper presents an adaptation of an Agile
development process specifically tailored towards software systems integration projects. The proposed Agile Systems Integration Process has been piloted in a senior-level software engineering studio course using an HO gauge model of a Positive Train Control system as the cyber-physical target hardware. The software developed by the students included five components: train scheduling, interactive display control, walk-about tablet manual controller, track geometry and connectivity input and safety monitoring software. Approximately 30 students were divided into ten teams with two teams competing to build each of the five components. This paper discusses what aspects of the proposed Agile systems integration process worked successfully and several important lessons learned to improve future performance.
Keywords: Agile, systems integration, positive train
control, agile in an educational environment
1
Introduction
The classical approach to software development is called the Waterfall process, which is shown in Figure 1 below [1]. This paradigm is often used in industries that have large complex systems that have to be sustained over many years such as aircraft, weapons systems, and cyber-physical systems. Oftentimes the process is proscribed by a certification organization such as the Federal Aviation Administration (FAA) for aircraft systems. The currently followed processes, DO-178C calls for explicit identification of the phases and artifacts of each stage and formal reviews either by the FAA or it’s representative called a DER (Designated Engineering Representative) [2]. Each stage of development has formally identified documents that must be produced as part of the states: requirements document, design document, code and unit test documents, integration documents.
At the production of each artifact, the quality of the artifact is assessed. A formal review is performed including a requirements review, design review, unit test review, integration test, and independent verification and validation review. The goal of these quality control steps is to identify defects as early as possible since the defect removal cost increases exponentially the longer the defect remains in the product [3].
The advantage of the Waterfall approach is that much thought must be given to each stage, starting with the
requirements and ending in the verification and validation, that the code meets the requirements and the requirements are the correct one for the system. However, as we all know, system requirements change as a system evolves. Over the multi-year development cycle of a weapon system, for example, technology evolves and threats emerge that were unforeseen at the time the program started.
Figure 1 – Classic Waterfall Process [1]
A modified Waterfall model, called the spiral model has evolved to allow for a cyclic evolution of complex products [4]. As shown in Figure 2, each phase is run to completion with a delivered product following which the requirements are re-evaluated, the design re-considered and new requirements and design elements added. During the multiple cycles of the spiral process full-fledged artifacts are produced including requirements, design, code and test, and integration test documents.
While these processes are appropriate and even required by many industries, the new more responsive Agile approach has been developed for applications that do not have the need for detailed artifacts or reviews. Agile development processes are often used by small teams of developers for short lifetime or short turnaround products such as cell phone applications or internal research and development projects. In an Agile process such as Scrum, requirements are replaced by user stories and the development cycle is very short, called sprints. Often these sprints are just a few weeks long. At each sprint the development team consults with the customer, identifies part of the product that can be produced in the next sprint, and develops and tests the code for that sprint. A key element of Agile processes is the delivery of an implemented and tested partial product at the end of each sprint.
Figure 2 – Spiral development process [4]
One huge advantage of the Agile approach is that defective or misunderstood requirements are very quickly identified since the intended interpretation of the requirements are revealed within a short period. There is no question that defective or conflicting requirements are rapidly identified and remediated.
As development of large cyber-physical systems have evolved, which have traditionally used a Waterfall process, developers have been using modeling, simulation, and hardware in the loop for systems of systems integration to deal with the misalignment of development times and delivery dates for completed subsystems. So it is not uncommon for a complex system to be built completely virtually, instantiated in a hardware-in-the-loop framework (HWIL), and have component parts replaced piece by piece (when available). Even after the component parts are available, virtual implementations may continue to be used due to limited availability of hardware or software components. Thus complex missile systems may have started with some legacy hardware, simulation models of threats, sensor behavior, command and control components, and synthetic threat model generators. As the system progresses, components of the HWIL system will be replaced with real hardware/software as part of the system integration activities. Parts, such as threat simulations may remain in their initial form as Agile developed components so that they may be rapidly adjusted to model evolving threats and sensor capabilities. So there is a need for systems of systems integration with both fully Agile developed components and mixed systems with traditional waterfall and Agile developed subsystems.
In this framework the University of Alabama in Huntsville evolved an existing agile development senior software design studio into a systems of systems integration experience using Agile-developed parts by multiple teams
of developers. Below we first present a description of the senior software studio course, the cyber-physical system utilized during the pilot study, and the Agile systems integration process as employed in the pilot study.
2 Agile senior software studio course
Our senior-level software engineering studio course was chosen to pilot the Agile systems integration process. This one semester course was originally created to provide an immersive team software development experience for undergraduate computer engineering students at the University of Alabama in Huntsville (UAH). In this course, teams of three to four students design, implement, and test a non-trivial software product. As the course was initially formulated, students employed a Waterfall methodology that included the preparation of several milestone documents including the Software Requirements Specification (SRS), Software Design Document (SDD), and Software Test Plan (STP) in addition to development and testing of the software itself. After several offerings of the course, it became clear that one semester was too little time for some students to develop a high quality non-trivial product using a Waterfall methodology.
In early 2011, a more Agile process derived from Scrum was adopted for use in the course. In developing this process, it was important to retain some elements of a Waterfall process such as traceability and test coverage analysis since many graduates from UAH remain in north Alabama developing safety-critical systems such as helicopters, missiles, and radar systems. It was also important to reduce documentation demands so that students had more time to implement and test their products.
At the beginning of the course, students are divided into teams of three to four students. Every student on the team develops his or her own complete proposal for a team-sized project that satisfies the list of project constraints, which include development of an open source product, use of real-time interactive graphics, the use of socket-based networking, interaction with an SQL database, and mandatory use of Subversion repository. Given the constraints, students typically pursue development of a video game or simulation. The project proposal includes the following sections. • Project Overview • Market Research • Fundamental Requirements • Storyboard • Derived Requirements o High-level use cases o Non-functional requirements • Traceability
proposals, each team must select the project they want to pursue from the set of the project proposals produced by members of that team.
The modified Scrum process includes the following five steps, which are further explained below.
• Sprint planning • Sprint execution
• End of sprint live demonstration • Sprint retrospective
• Peer evaluation
During the semester, students have time to execute four to five sprints.
In the sprint plan, students must include detailed use cases only for those use cases selected for the sprint backlog. Use cases not selected for the sprint backlog remain on the product backlog as high-level (unelaborated) use cases. Additionally, students must include a detailed set of acceptance tests for all use cases on the sprint backlog (test-driven development). Note that by requiring detailed use cases instead of user stories, the use cases may serve as both requirements and as test scripts for conducting the end of sprint acceptance tests.
At the end of each sprint, students must conduct a live demonstration of the new functionality implemented. For the live demo, the team must check out their source code from the Subversion repository, compile it, and execute the set of acceptance tests along with any additional tests suggested by the instructor or customer. Responsibility for conducting the demonstration rotates through the team members so that everyone must demonstrate competence and participation in the project.
The traditional sprint retrospective provides an evaluation of the process as executed during that sprint. The retrospective for the modified Scrum process has been augmented with additional information including a labor report itemizing the time spent by each team member and productivity statistics such as source lines of code produced and use cases attempted versus completed. It also includes Subversion repository statistics gathered using the StatSVN utility to indicate who is or is not contributing to the project [5]. Finally, the retrospective includes a summary of the acceptance test results (pass/fail).
At the end of each sprint, each student completes a peer evaluation form (derived from [6]) in which they must assess the performance of each team member on four criteria: Quality of Work, Quantity of Work, Level of Teamwork, and Timeliness of Work. The performance scale ranges from 1 (little or no contribution) to 5 (exceptional). Written comments are required to explain any below average assessment of 1 or 2. After completion of the source code development sprints, a final “documentation” sprint is performed during which
the as-built product.
While the implemented and tested source code within the Subversion repository was the primary course deliverable, students were also required to deliver a set of requirements, which consisted of the set of as-implemented use cases (updated from those in the sprint plan), as well as a test report containing the acceptance tests developed for each sprint (from the sprint plan) and the test results (from the sprint retrospective). Finally, as a measure of the thoroughness of their testing efforts, students are also required to deliver a test coverage report that summarizes the amount of delivered code per module that remains untested.
3 The cyber-physical system
For the pilot systems of systems integration study, students were asked to develop train scheduling, display control, walk-about tablet manual controller, and safety-monitoring software for our newly enhanced Positive Train Control (PTC) test bed.
Figure 2 – Block diagram of system
The components of the system as identified in Figure 2 include:
1- a train scheduling system that computes routes for trains to follow and issues commands to trains and switches as the train progresses down the route;
2- a safety monitoring system that implements what is tradiditionally called vital logic and is concerned with local safety issues such as trains heading towards a collision;
3- a display system that displays for hte user the state of the tracks, train occupancy, train routes, switch settings, and safety violations.
4- a track layout system that allows the entry into the train control system layouts of tracks, switches, and detection sections, and
5- a tablet based controller for manual remote control of the layout's locomitives and switches.
As previously presented [7-10], the UAH Software Safety and Security Engineering Laboratory has developed HO gauge model train cyber-physical systems of increasing complexity to facilitate our teaching and research programs in software safety and security. Our PTC test bed models real world positive train control systems currently being deployed nationwide in the United States. A key feature of these real world PTC systems is the centralized safety monitoring system that can remotely control trains in order to prevent accidents.
The five primary software components of the system (routing, safety, display, layout and manual trablet control) are interfaced to each other and the track via a SQL server wiich maintains a set of SQL tables that reflect the track status (as read from the track) and track commands (as issued by the various control components)
The trains and track switches are controlled by and observered by a set of Digtrax LocoNet hardware which silultantously monitors locomotive location while issuing commands to the locomotives (speed, direction) and switches (through, bypass). The component that reaches across the SQL database and LocoNet hardware is called the SQL-LocoNet bridge.
As shown in Figure 4 below, our PTC test bed has been recently re-engineered to increase the total number of individually-controllable blocks of track to over 100 blocks. The track layout consists of two different train systems, a main line track consisting of the outside ovals and rail yards and the interurban system that consists of the figure-eight loops connected by track that crosses the main line track at four crossover points. The added complexity of these crossovers present points of interference that must be accounted for in both the scheduling system and the safety-monitoring systems.
An Arduino-based block occupancy detection and safety system has recently been integrated into the PTC test bed. Redundant occupancy detection is employed to prevent collisions in case of hardware failures. As with real-world trains, current sensing is employed to detect the presence of a locomotive within a specific block of track [11]. As previously reported, fine grain localization will be provided wirelessly by inertial measurement unit/tie counting
transponders located on each train [9]. The re-integration of fine grain localization into the enhanced PTC test bed has been left to the next offering of the software studio course.
Figure 4 – Panoramic photograph of PTC test bed [9]. As a safety precaution, the operating rules of real train systems require that only one train may occupy a given block of track, except under certain special circumstances. Figure 5 below shows one of the two Arduino boards that comprise the safety system. Each block of track in the PTC test bed has its own power supply that has been routed through a relay mechanism to allow the safety system to shut off that block’s power supply to prevent an impending collision. Figure 6 below shows the custom-designed wiring consolidation circuit board (purple) that integrates multiple off-the-shelf Digitrax controller boards (green) with the arrays of relays used by the safety system to shut off power to blocks of track. Thus, even with a defect in the scheduling software failure, the trains should never collide. Short circuit protection is offered by fuses located in series with each relay.
Figure 6 – Photo of safety system relay arrays.
4 Pilot Systems of Systems Integration
Study
4.1 Setup
As discussed earlier, the software development task was divided up into five parts with an SQL database acting as the interface mechanism. By making the interface between the components a database, it was possible for each team to develop and test their code as it evolved to ensure it met the interface description of the database tables. The large enrollment in the course (30+ students) resulted in a total of ten development teams of 3-4 students each. To increase the likelihood that a fully functional, five part system would be developed, two teams, an A team and a B team, were assigned to develop competing implementations for each of the five parts.
A key requirement for the fully integrated system was that all parts delivered must be interoperable with each other so that all teams would have to interact with each other to develop appropriate component interfaces. For example, the train scheduler developed by the A team must be interchangeable with the train scheduler developed by the B team in the fully-integrated system. Moreover, any part 1 component must be compatible with all parts 2 – 5.
4.2 Systems Integration Plan
The intention at the onset of the project was to use continuous integration which is a primary property of both agile integration and traditional integration processes. In continuous integration, components and builds of components are integrated as they become available. Sketches of a component's capabilities are to be serially replaced by their ever refined version as time goes on. If some component lags the others in developmental maturity, legacy components could continue to be used. The goal was
with the code that was developed after each sprint. Because the integration was managed by the SQL-LocoNet bridge, an asynchronoously updated database, students could mix and match old and new (tried true or developmental) components as they wished.
5 Observations and Discussion
Although the systems integration plan seemed reasonable at the time, a number of issues arose that led to a somewhat different execution.
Agile processes call for replacement of formal documentation by personal communication in the form of daily meetings and bi-weekly retrospective/prospective planning meetings. The students in the class are somewhat different than your typical undergraduates in that most of them have full time jobs in the local industrial and research community that is supported by the US Army's Redstone Arsenal and NASA's Marshall Space Flight Center. Typically the only one-on-one physical meetings occurred after class which ended at 7pm two times a week.
Generally the meetings that did occur were too short and too dense (30 students attended class regularly of the 32 enrolled). As a result the absence of formal documentation and the absence of regular inter and intra team meetings resulted in numerous teams moving forward with development before fully developed ICDs describing the interface were created. Conomintantly, students developed unit test methodologies for their personal incarnation of the data base. Although several class sessions were dedicated to the emerging bifurcation of the database standards, differences remained until the very end of class when integration testing was performed as a formal requirement.
Another issue that arose was the continuous evolution of the target hardware. An early version of the hardware was available at the beginning of the term for student use. However, the final complete layout with all its whistles and bells was not available until the second half of the term. The students failed to appreciate the fact that such issues arise frequently in the development of real world systems.
During the study, the university was also closed unexpectedly due to several snow days. The university was closed for up to two weeks equivalent workdays which was unprecedented in the previous 25 years history of the authors. Students used these periods of reduced communication to forge ahead on their own vision of the interfaces, which resulted in more aggravation and breakage of code.
Finally, some methodologies used by students, such as copy and paste of large code segments rather than writing a parameterized function or developing table driven code, resulted in at least one team producing 25,000 source lines of code. While the code was highly repetitive, each defect detected required the repetitive repair of each block of copied code for each defect detected, which likely increased
the probability that one of the students might have missed applying the repair in one or more places.
6 Conclusions
The Agile course upon which this paper is based is undergoing evolution as we speak and the success of the endeavor has only been demonstrated a few days ago. Of the 10 team development projects, 9 were completed all the way through systems integration. The goal to have students develop a large (each component ran thousands of source lines of code) multicomponent project which involved component development and system integration based on agile processes was much harder than expected. Even though the course is ongoing, several important lessons have been learned for future enhancements to the course.
Lesson Learned: Face-to-face communication must occur Students’ rapid and eager acceptance of low-documentation processes was not coupled with the need for much greater verbal and meeting mediated communication. Agile replaces detailed specification and design documentation with face-to-face communication. The integration plan hinged on these face-to-face communications to resolve component interface issues early on. Since some teams failed to conduct these face-to-face discussions and other teams failed to abide by the decisions made during these discussions, interface issues continued to arise even in the last sprints. For interfaces between teams the old fashioned signed off ICDs would probably have been a better implementation. For processes that require significant communication within and between team members communication vehicles such as Skype and Google drive should be set up at the very earliest stage. Moreover, points must be assigned to these communications as incentive for students to engage in them.
Lesson Learned: All code is owned by the team
Students who have not worked in an environment where breakage is a measure of the success of the project rather than the loss of an individual’s personal code contribution to the project. Agile is predicated on far more rework and scrapping of codes that are developed earlier in the process. Students feel that each line of source code is fully formed work product are loathe to scrap or rework them. Agile processes gain their ability to identify and rectify early errors only through much larger breakage in the work product than processes with great planning and breakage should be looked upon as progress rather than lost children.
Lesson Learned: Practice continuous integration within and across components.
Continuous integration is required for success of agile based systems of system integration. Not only must each team develop working, testable product each sprint, these
must be integrated on a continuous basis to enforce acceptance of interface specifications rather than a wild west approach to interface design (my way or else).
7 Acknowledgements
We would like to thank Dr. George Petznick, our resident train expert, for his assistance in the design and construction of the Positive Train Control test bed.
8 References
[1] Stephen R. Schach, Object-Oriented & Classical Software Engineering, 7th edition, McGraw-Hill, 2007 [2] RTCA DO-178C, “Software Considerations in Airborne Systems and Equipment Certification”, December 13, 2011.
[3] S.H. Kan, S.D. Dull, D.N. Amundson, R.J. Lindner, and R.J. Hedger, “AS/400 Software Quality Management,” IBM Systems Journal, vol 33, no. 1, 1994.
[4] B.W. Boehm, “A Spiral Model of Software Development and Enhancement,” IEEE Computer, vol. 21, May 1988.
[5] StatSVN utility. http://www.statsvn.org/
[6] David Kelly and Mary Sadowski, “Peer Evaluation within a Team Design Project”, ASEE Conference Proceedings
[7] David J. Coe, Joshua S. Hogue, and Jeffrey H. Kulick, "Software Safety Engineering Education," 2011 International Conference on Software Engineering Research and Practice (SERP'11), WORLDCOMP 2011, July 18-21, 2011, Las Vegas, NV.
[8] Travis Cleveland, David J. Coe, and Jeffrey H. Kulick, "Video Processing for Motion Tracking of Safety Critical Systems," 2013 International Conference on Software Engineering Research and Practice (SERP'13), WORLDCOMP 2013, July 22-25, 2013, Las Vegas, NV. [9] Scott Schiavone, Sjohn Chambers, Sunny Patel, Lee Ann Hanback, David J. Coe, Jason Winningham, B. Earl Wells, George Petznick, and Jeffrey H. Kulick, "UAH OnTrack: Precision Navigation System for Research on The Software Safety Issues of Positive Train Control," 2014 International Conference on Software Engineering Research and Practice (SERP’14), WORLDCOMP 2014, July 21-24, 2014, Las Vegas, NV.
[10] David J. Coe and Jeffrey H. Kulick, "Positive Train Control: Concepts, Implementations, and Challenges," 2014 International Conference on Software Engineering Research and Practice (SERP’14), WORLDCOMP 2014, July 21-24, 2014, Las Vegas, NV.
[11] American Railway Association, The Invention of the Track Circuit, New York, 1922.
