Case Study :
Software Disasters -1
WELL-KNOWN: DENVER INTERNATIONAL AIRPORT BAGGAGE HANDLING SYSTEM
BY DHRUV P & UDIT P
Background
City of Denver – 1990 New state-of-the-art airport – 140km2 area
Capacity : > 50 million passengers annually Largest in the USA
Critical component : Automated Baggage handling System
Reduce aircraft turnaround time by 30 minutes
Baggage handling system at a
glance
88 airport gates in 3 concourses
17 miles of track and 5 miles of conveyor belts 3,100 standard carts + 450 oversized carts
14 million feet of wiring
Network of more than 100 PC’s to control flow of carts 5,000 electric motors
2,700 photo cells, 400 radio receivers and 59 laser arrays
Causes of project failure
Became a public humiliation for everyone involved Airport opening delayed by 16months
Cost the city $1.1M per day throughout the delay On the opening day,
System was used in just 1 concourse by single airline and only for
outbound flights
All other baggage was handled using conveyer belts, manual tug and
trolley system
Chronological Events
Time Event
Nov, 1989 Work starts on the construction of the airport
Oct, 1990 City of Denver engages Breier Neidle Patrone Associates to analyse
feasibility of building an integrated baggage system. Reports advises that complexity makes the proposition
unfeasible
Feb, 1991 Continental Airlines signs on and plans on using Denver as a hub
June, 1991 United Airlines signs on and plans on using Concourse A as a hub
Summer, 1991 Airport’s Project Management team recognizes that a baggage handling solution for the complete airport was required. Bids for an airport wide
Chronological Events
Time Event
Fall, 1991 Of the 16 companies included in the bidding process only 3 respond and review of proposals indicate none could be ready in time for the Oct 1993 opening. The 3 bids are all rejected
Early 1992 Denver Airport Project Management team approach BAE directly
requesting a bid for the project
Apr, 1992 Denver Airport contracts with BAE to expand the United Airlines baggage handling system into an integrated system handling all 3 concourses, all airlines, departing as well as arriving flights. In addition system is to handle transfer baggage automatically.
Contract is hammered out in 3 intense working sessions
Chronological Events
Time Event
Aug, 1992 United Airlines changes their plans and cuts out plans for the system to transfer bags between aircraft. Resulting
changes save $20m, but result in a major redesign of the United Airlines portion of the system. Change
requests are raised to add automated handling of oversized baggage and for the creation of a dedicated ski equipment handling area
Sept, 1992 Continental requests ski equipment handling facilities be added to their concourse as well
Oct, 1992 Chief Airport Engineer, Walter Singer dies. Mr Singer had been one of the driving forces behind the creation of the automated baggage system
Chronological Events
Time Event
Jan, 1993 Change orders raised altering size of ski equipment claim area and adding maintenance tracks so carts could be serviced without having to be
removed from the rails
Feb, 1993 Target opening date shifted from 31 Oct 93 to 19 Dec 93 and soon
thereafter to 9 Mar 94
Sep, 1993 Target opening date is shifted again, new target date is 15 May 1994
31st Oct, 1993 Original target for opening 19th Dec, 1993 Second target for opening Jan, 1994 United Airlines requests further
changes to the oversize baggage input area
Chronological Events
Time Event
9th Mar, 1994 Third target for opening
Mar, 1994 Problems establishing a clean
electrical supply results in continual power outages that disrupt testing and development. Solution requires
installation of industrial filters into the electrical system. Ordering and
installation of the filters takes several months
Apr, 1994 Airport authorities arrange a
demonstration for the system for the media (without first informing BAE). Demonstration is a disaster as clothes are disgorged from crushed bags
Apr, 1994 Denver Mayor cancels 15 May target date and announces an indefinite delay in opening
Chronological Events
Time Event
May, 1994 Logplan Consulting engaged to evaluate the project
15th May, 1994 Fourth target for opening May, 1994 BAE Systems denies system is
malfunctioning. Instead they say many of the issues reported to date had
been caused by the airport staff using the system incorrectly
Aug, 1994 City of Denver starts fining BAE $12K per day for further delays
28th Feb, 1995 Actual opening
Aug, 2005 In order to save costs the system is scrapped in favour of a fully manual system. Maintenance costs were
Reasons of Failure : Strategical
Changes
United Airlines – insisted automated high speed baggage system Denver agreed to build for all concourses
Assumed : Individual airlines would make their own baggage handling mechanism
Prior to this, every airline used to build their own baggage handling system
Change : in Summer 1991,
Project Management Team realised: in-order to build an integrated system
throughout the airport, they need to take the responsibility back from the airlines.
Decision was taken little more than 2 years prior to the opening date
Reasons of Failure : Strategical
Changes
Manual baggage system was rejected
Size of airport : not feasible to handle baggage on time Labour intensive and expensive process
Diesel powered tugs would travel through poorly ventilated tunnels
Chock the tug driver
Reason of Failure : Yet, they
proceed
3 Red Lights not to proceed : Yet, they proceed
The 1990 Breier Neidle Patrone Associates report indicated the
complexity was too high for the system to be built successfully
Analysis of the three bids received indicated that none of the vendors
could build the system in time for the Oct 1993 opening
Experts from Munich airport advised that the much simpler Munich
system had taken 2 full years to build and that it had run 24 / 7 for 6 months prior to opening to allow bugs to be ironed out
Reason of Failure : Yet, they
proceed
BAE was not initially chosen for bid 3 official bids were rejected
Airport team contacted BAE
BAE didn’t have prior experience – Yet they accepted it !!!
Big opportunity to grow the business by securing other big contracts around the
globe
BAE started working on specification and prototype
Prototype misled
Said to have filled up 50000 sq ft warehouse
Demo convinced Slinger that project was feasible
BAE senior management team only communicating with Slinger
Reason of Failure : Yet, they
proceed
Slinger and BAE underestimated the complexity of the project They didn’t take the expert advise
Slinger, as a civil engineer might not understand the mathematical
complexity of the problem
Slinger, responsible for entire airport had other considerable amount
of duties
Pressure to move quickly would have caused them to put the
reasonable thinking aside
Reason of Failure : Over
Commitment
BAE agreed to deliver the project within the fixed scope, schedule
and budget
Contractual conditions and scope was decided in just 3 sessions
3 sessions were clearly not enough to get the complete understanding
of the scope and risk
Airlines were major stakeholders
Excluded from the key decisions discussions
This is always a losing strategy
When they were consulted,
Asked for significant changes which caused a severe rework or negation of previous work
Reason of Failure : Over
Commitment
BAE claimed to deliver the project in 2 years time
Really needed 3 to 4 years
Tight schedule
Team would have focused on “Happy Path” scenarios
And less time thinking on making the system fault tolerant
Full simulation of the system was avoided
Full testing of the complete system was not completed Training schedule was cut down
Mitigation Plan
Do not commit something you cannot deliver
Consult with external experts such as Logplan as BAE didn’t have prior
experience
Defining activities with the resources and duration in sequence could
have given the appropriate schedule Testing
Less coverage and less scenarios covered Major bugs cannot be identified
Reason of Failure : Ongoing
Changes
Project was already in trouble In original negotiations,
BAE : no changes would be made
Meeting stakeholder’s need became too strong – accepted the changes
Changes:
Add ski equipment racks
Add maintenance tracks : Cart can be serviced without being removed
from the tracks
Changes to oversized baggage handling
Acceptance of changes indicates
No clear understanding of impact
Reason of Failure : Ongoing
Changes
Changes to requirements/design was carried out by DIA without
consulting the airlines or BAE.
The same situation occurred with the schedule.
BAE failed to appreciate change impact – this was due to the
complexity of the system and its tightly coupled nature.
As the project was slowly implemented, it swelled in complexity .
When design changes were made, this only made things worse.
Centralized system faults, which ran through many of the baggage
system’s subsystems, were discovered.
Reason of Failure : Ongoing
Changes
By far, the biggest change was caused by United Airlines penny
pinching. They removed a complete loop of track; having just one loop.
Saved $20 million.
Required a complicated redesign at a critical phase of the project.
Other major changes included relocating the outside stations,
Continental Airlines larger baggage link and another mezzanine baggage platform.
Due to time constraints, many of the problems which required
substantial redesign were covered over by quick fixes.
Mitigation Plan
Communication should include all the team members and
stakeholders
Critical findings should be escalated
To make the system more modular and less tightly coupled,
Patchwork could have been avoided and development life cycle had
followed to resolve them and SE rule had followed
All stakeholders could have discussed and brainstormed at
Reason of Failure : Incompetence
with the physical building
Design of building was initiated prior to the baggage automation Building designers made space for the baggage handling based on
their prior experience
Baggage system team was forced to work within the constraint of
the physical structure of the airport
Change of structure would cost : $100M Sharp turns : bags ejected from carts
Speed of the carts was halved from 60car/min to 30 car/min
Mitigation Plan
Open communication strategy between DIA and airlines could
have avoided the decision about who would design the baggage system
Never make an assumption
Decision and construction of the automated baggage system and
building should have started at the same time.
Building team could have consulted/work along side with baggage
team for the baggage space related design work
PMT failed to identify the interface(allowance of space) between
Reason of Failure : Take Alternative
In Apr, 1994, the Mayor of Denver recognized that project was onfire
Logplan was consulted
Recommended to take the alternative and build the manual system
with trolley
Additional cost of $51M
Mitigation Plan
Such complex, big and risky project must have
Frequent reviews Expert assessments
Reason of Failure : Incomplete
Testing
Computer simulation was avoided
Line-balancing problem was detected 6 months after airport was opened
Whole system was consisted of more than 100 queues
Due to tight schedule, testing time was reduced Munich airport spent 2 years in testing
BAE ignored this fact
Test engineers unable to communicate on radio channel because of dead spot in radio transmission around the airport
Shared large power supply caused the fuses to trip, stopping the motors
Mitigation Plan
Running the full computer simulation could have avoided line
balancing problem
Simulation could have been used to see the redesign impacts
Spending more time for testing
Consider cases for the similar projects and learn from them such as
Munich
Take expert advise
For issues such as radio channel communication and power
fluctuation,
Reason of Failure : Reliability of the
Components
Barcode reader are not accurate every time
Photocell quality and their adjustment caused the computers
presume that there was a telecare jam
Faulty latches dump luggage on the tracks or jammed against the
side of the tunnel
Airflow flipped light or empty bags out of their telecars Sharp turns – high stress area – tracks frequently broke
Mitigation Plan
Should have investigated multiple redundancy Considered multiple failure scenarios
Reason of Failure : Politics
Oct 1992, Chief airport engineer Walter Slinger died Gail Edmond became new project head
Not given same power of autonomy
Mitigation Plan
More efficient and expert management team should have been
assigned from the beginning.
Decision making should involve by all the management team
Reason of Failure : Failure Recovery
BAE didn’t have any backup system in case of failure Alternative tug and cart system implementation caused additional
$51M
There was no way of reliving strain from the system No alternate ways for baggage
Mitigation Plan
Alternative system should exist in the case of automation failure It should have been defined at the requirement stage
Conclusion
Never underestimate the mathematical complexity of the problem Spend time in planning activity
Spend time on risk management and mitigation plan drawing activity Spend time on design of the system
Do not give over commitments in the face of massive risks and uncertainty
Make clear communications across the team and keep record of each communication
Understand and consider the implications of the change requests
If required, take them as separate iteration
WELL-KNOWN: VIRTUAL CASE FILE SYSTEM.
Virtual File System
Trilogy
UAC➙VCF
Network Infrastructure Hardware Infrastructure
FBI Information Technology Upgrade
Requirement of the UAC
Integrate five most legacy systems of the FBI Automated Case Support System. IntelPlus.
Criminal Law Enforcement System.
Integrated Intelligence Information Application. Telephone Application.
Provide point and click web based interface.
9/11 even affected FBI
In midway of VCF, the 9/11 Commission revealed all the loopholes
of the FBI hardware and software infrastructure. Report stated
“FBI’s information system were woefully inadequate”.
There was no effective mechanism for capturing or sharing its
institutional knowledge.
In the face of intense public and congressional pressure, team of IT,
accessed the web-interface developed by SAIC.
Team realized that letting the agents to just have point and click
wont do much help.
Introduction
VCF was supposed to automate the FBI’s paper-based work
environment, allowing the agents and intelligence analyst to share the vital investigative information.
Ubiquitous nature of data. Access from anywhere.
No existing commercial software package was available.
Science Application International Corp (SIAC) was the company
who TRIED to developed it by June 2004.
FBI wanted Big (in middle of UAC)
Pivot Entirely new centralized database. New Graphical User Interface.
Now VCF would combine ACS with other two applications only
The Telephone Application. Law Enforcement Application.
Multi-media Capabilities.
Allowing agents search across various investigation and would be
accessible to both, the FBI agents as well as intelligence analysts. (Including more actors).
Joint Application Development.
To formalize, requirement for VCF, SIAC embarked on a serious of
JAD sessions.
Agents and experts got to gather with a group of SIAC engineers to has
out what functions the VCF would perform.
JAD Session finally produced an exhaustive detailed requirement
document (6 month long).
SIAC agreed to deliver the initial version of the VCF in December
2003 instead of June 2004 i.e. within 22 months, using the flash cutover.
What requirement specification
said…
Matthew Patton was hired as an security engineer in the SAIC. After
the failure, he was interviewed for the failure of VCF.
Patton’s description for the SRS, suggest that the project was going off
the rails right from the beginning.
Outcome of JAD => Tried to design the system, before even figuring out what they wanted the system to do
The document violated first rule of the software planning: Keep it simple.
SRS should describe what and not how.
The SRS was of quality specifying details like “there will be a page with a button saying email on it”.
SRS had the description of the system layout, but devoid of what the system would do.
JAD session resulted into a massy and overstuffed requirement document.
Generally Customers say what the want and Consulting firm says how they
will deliver.
But one of the project manager in the SAIC, said that the failure to VCF was
the culture of the FBI.
“The culture within the FBI was, We are going to tell you how to do it too”.
Overly Specific nature of the requirement focused developers on their tiny
pieces of the puzzles.
Patton added that, there were so many individual modules, that
developers had no idea of how these pieces would fit with each other.
Writing the code to develop applications, that were already present.
What requirement specification
said (Cont’d)…
Before the start of VCF
Matthew Patton, a real hero to capture the infeasibility of the VCF.
Questioned the technical feasibility of the project.
Aired the objection regarding the feasibility to the Web Discussion
Board.
Reward: A visit by two FBI agents concerning about disclosing the
national security secrets.
Result: SAIC and FBI agreed on deeply flawed 800-Page set of system
requirement.
Before the start of VCF (Cont’d)
Disagreements among the members for VCF Interview with the FBI agents regarding the information transfer from
paper-based system to electronic-based system.
Zalmai Azmi, raised the question on VCF over current system?
Raises legal policy questions
“In an electronic world, nothing really is destroyed; it is always somewhere.”.
Software Development
In December 2002, the requirements were settled, and there was
something to start on with
SAIC adopted on a spiral development methodology, an iterative
approach to developing the software.
Build a particular function ➙ Validate by the agents ➙ if something
is wrong ➙ sent an official request to the change control board.
From December 2002 to 2003, 400 official notices were collected by
the change control board.
Software Development (Cont’d)
Comments on Validation by Agents “Oh, we’ve got to change this, That isn’t what I meant to be”.
And that’s how we started logging change request after change
One More Pivot
For about 25% completion of the VCF, FBI wanted a “page crumb”
capability added to all the screens.
This functionality gives the user, a list of URLs identifying the path taken
by the VCF to arrive at the current screen.
One of the 8 development team, created the functionality as
Audit Reports
In September 2003, U.S. General Accounting Office released a
report titled, “FBI needs an Enterprise Architecture to guide its modernization Activities”.
Randolph C. Hite, added that, “it was a classic case of not getting the
requirement sufficiently defined in terms of completeness and correctness from the beginning”.
Randolph, advised to redesign the requirement that would be in
compliance with what is already designed and produced.
Late and Limited Blueprint
Today many organizations rely on blueprint (enterprise architecture).
To guide hardware and software investment decision. It describes an organization’s mission and operations. How it uses the technology to accomplish its task.
How the system is structured and designed to accomplish the task.
According to Government Accountability Office, inspector general
Reports says…
Incompetency with the real-world Environment
13,000 Computers could not run modern software.
Most of the computers were just connected with the low speed
INTRANET links of about 56 kbps.
Network Components were not supporting the addressed protocols
used in VCF.
Conclusion “Not supporting environment”
Unrealistic Contract
FBI chose cost-plus-award fee contact
This would include the cost of all labor not based on fixed requirements Additionally to include all the unforeseen cost.
The D-Day Comes…
On 13th December, 2003 SAIC delivered the VCF to the FBI, only to
have the declared DOA.
FBI Rejected the VCF.
17 functional deficiencies. Big and small deficiencies
Big: Not providing the ability to search for indiv iduals by specialties and job tittle. Small: Button on the GUI labeled as STATE should hav e been
STATE/PROVINCE/TERRITORY.
Arbitrator’s findings (12th march,2004)
59 issues and sub issues derived from 17 original deficiencies. 40 SAIC’s error and 19 FBI’s fault.
Comments from SIAC
Because of tight schedule:
SAIC developers didn’t ensure that all the programmers were making the
change in the same way.
This inconsistency occasionally meant that different modules of VCF
handled data in the different way.
Consequently, when one module needed to communicate with other, error
sometimes occurred.
Punaro said that this didn’t compromised the system. The real killers were
Significant Management turbulence.
Ever-shifting nature of the requirement.
Trail and error approach of the FBI Agents.
Time to heal…
Aerospace Corporation., California.
Reviewed the VCF, in order to come up with what to do .
SAIC to work on electronic workflow portion of VCF, and turn it into Initial
operating Capability (IOC) ($ 16.4 million).
Create a way to translate the output from the VCF System. Check out and improve the network performances.
Develop training program.
With new management in-place, 120 SAIC Engineers began work on
IOC project in June 2004.
Agreed on strict development schedule. Acceptance criteria.
Series of control gates.
Release date for IOC
Completed before the schedule time Was an aid to the task of management.
Did not improve the productivity of most users. (Internal FBI assessment).
Fill the form electronically ➙ routed for approval ➙ To comply with paper based case management system ➙ print out, sign up and file the form.
Factors Contributing to Failure
Enterprise Architecture: Efforts and results in EA where late, limited and fall for short of what was
required.
How the interaction among the various applications needs to be automated?
How the data among the application should flow?
What about the security of data when it comes to various actors and applications?
Justification for why 17 functional deficiencies and 4000 change
requests during the development and lack of experts in enterprise architects.
Factors Contributing to Failure
(Cont’d)
Transition Plan
Tried to transfer from ACS (800 pound gorilla) to VCF in just one shot.
Data migration was a major headache.
No body knew how many interfaces where present and how they were
defined.
No Failure Recovery. No Real World Testing
Mitigation Plan
Phase roll out with Rapid Prototype Development approach.
“Smaller steps, more rapid done, rather than one big jump and make it all the way across”
Factors Contributing to Failure
(Cont’d)
Scope of the VCF
Just concentrated on the end results.
Failed to lookover the interoperability of the existing systems with VCF
Made the scope of the VCF unrealistic and very wide.
SIAC had an messy custom coded effort, trying to build all the
functionalities, and expecting all to work flawless and perfect(developed by 8 teams).
Mitigation Plan
Factors Contributing to Failure
(Cont’d)
Project Management
Some of them didn’t had any prior experience in the project
management. (As an when approach)
Sherry Higgins: 29-year veteran of AT&T, previously handling the help desk at the technology Command and Control Center.
Made Depew, as a VCF Project Manager.
Concluded that “FBI didn’t had adequate human resources and skill
base needed to deal with FBI’s modernization project”.
All the well-wishers for the VCF were rewarded with resignation
9/11 Attack Doubts on capabilities
VCF
More Complex Poor EA Less time Flush Cutover experiencedLess
PM Failure Improper start of the project Tensed Atmosphere
