DERA Guide to Software Purchasing

This guide dated 1998, supplied in draft form, comprises 43 sides of A4 paper and deals, amongst other things, with the evaluation, acquisition and management of COTS software. Its stated purpose is to provide advice on;

1. Assessing and choosing the best approaches to software purchasing. 2. Evaluation, acquisition and management of COTS software.

3. Evaluation, acquisition and management of bespoke software. 4. Understanding software contractual issues.

While item 3 is not relevant to this thesis, the other three items, particularly item 2, are fundamental to the use by MoD of COTS software. The guide starts by stating that it is not a substitute for experience and that software practice training courses are provided by DERA.

It highlights the three options available for sourcing software - ‘Do it yourself, purchase COTS, and purchase bespoke. It points out that this is unlikely to be a straight choice but will involve a degree of competition. For MoD, the first option is not generally a possibility.

It then outlines the general software procurement life cycle and emphasises the need to be aware of good purchasing principles. Under ‘choosing the right option’, it suggests among other things:

1. Always buy COTS software in preference to building it yourself or contracting out. 2. Consider integration of COTS software in preference to custom development 3. Use object libraries (e.g. Microsoft Foundation Classes) rather than write your own. 4. Look at DERA, MoD and industry for sources of similar software.

It highlights the advantages of COTS as:

1. Reduced development cost. 6. Reduced risk of failure.

2. Multiple use could lead to discounts. 7. Reduced long-term maintenance costs. 3. Reduced testing costs. 8. Reusable components for other systems. 4. Economies of scale. 9. Reduced risk.

5. Earlier delivery.

It stresses the need to spend time finding out what is available but says overall time and money will be saved. It also mentions that the use of COTS software is almost always a compromise and that COTS often requires customisation. The disadvantages of COTS software are:

1. Suppliers exaggerate the capabilities of their products making assessment difficult. 2. Requirements may only partially be met.

3. Missing functions are often promised in the next release.

4. Releases are often late (allow twice the elapsed time after the promised date).

5. Dependence on COTS products means more dependence (e.g. on operating systems). 6. Future compatibility with dependencies may not be guaranteed.

7. Some functions may not be available (and may or may not be easy to add). 8. Documentation standards are often poor (same is often true for bespoke systems).

9. Whole life costs need careful consideration. 10. Product support levels may not meet expectations.

Under the section headed ‘Managing COTS product selection and purchases’ it mentions that the purchaser has no control over the design, product management or development of COTS software. Contractual issues and integrating COTS software are seen as important factors. Assurance that software has adequate quality and reliability for its future use can be gained from:

1. Buying from a reputable and reliable supplier.

2. Using suppliers on which DERA has vendor information. 3. Testing and accepting the software on receipt.

4. Ensuring the availability of adequate support and post-development service.

The most common problems with COTS software are: 1. Failing to define requirements for the product.

2. Defining too many requirements and stating too many ‘mandatory’ requirements. 3. Having unbalanced evaluation weightings against individual requirements. 4. Having prejudices about one solution before evaluation.

5. Failing to allocate sufficient resources and experience to the procurement process.

There are also caveats against becoming over-dependant on COTS products, and that when COTS products are embedded in systems, the architecture of the main system may well be affected. There is a better chance of replacing COTS products or having some competition if international de facto standards are used to define interfaces. An exit route should be planned so that other COTS products can be bought if one supplier fails or a better product emerges.

Preferred COTS products can be used across an individual project as well as in many projects, but information must be updated as the market changes and re-evaluations run at sensible intervals. Care is needed in specifying requirements and deciding whether to use a COTS product, as well as in dealing with contract issues such as licensing, shrink-wrapped purchase, intellectual property rights, maintenance, indirect supply and standards. In addition to price and benefits, there is an exhortation to consider whole life cost implications when deciding what to purchase.

4.9.1 Review o f information

This guide, although still in draft form, is valuable as it represents an outcome of a significant effort by DERA to look as the issues involved in procuring IT including COTS IT. Clearly it provides an extremely useful cross-reference to other information gathered, albeit only for software, and could form the basis of a possible guide on the subject by DPA. Its comprehensive views include nine advantages and ten disadvantages of using such software. It also suggests four routes to obtaining quality COTS software and highlights the five most common problems that occur when taking this route. It also gives caveats on over-dependence on COTS products and the need for a careful contractual approach.

Benefits

Reduced development and testing costs as well as earlier delivery help. There are reduced chances of failure, long-term maintenance costs and risk, and reusable components for other systems.

Problems

Suppliers exaggerate product capabilities making assessment difficult. Requirements may only partially be met. Releases are often late. Missing functions are often promised in the next release. Some functions are not available. There is an ongoing dependence on COTS products. Future compatibility may not be guaranteed. Poor documentation standards are normal. Product support levels may not meet expectations. Whole life costs need careful consideration.

Other issues

The most common problems with COTS software are failing to define requirements for product, defining too many requirements, having unbalanced evaluation weightings against individual requirements, having prejudices about one solution before evaluation, and failing to allocate sufficient experienced resources to the procurement process. Caveats are made against becoming over-dependant on COTS products.

4.10 Guidance on Off-the-shelf System Acquisition

This twelve-page document provides guidance for IPTs on issues arising out of the use of COTS technology in MoD procurement and support. The guidance focuses on IT systems, since using COTS IT poses the greatest management challenges. COTS-based solutions generally offer reduced initial purchase costs, reduced acquisition times and the capacity to remain close to the state-of-the-art. For these reasons using COTS products has become a common feature of many defence procurements in the on-going drive to improve value for money and reduce whole life costs. However, this approach has its drawbacks, which are stressed so that they can be managed throughout the acquisition process. The essential issues underlying the use of COTS products are:

Obsolescence.

Lack of product control. Requirements trade-offs. Open standards,

A changing relationship with contractors. System flexibility.

Interfaces.

The document provides brief recommendations for through life management, inter-project liaison, managing the contractor, managing COTS software, safety, security, reliability and environmental issues. Of these, the sections on costs and support are particularly relevant.

4.10.1 Costs

MoD-specific requirements that lead to changes in COTS products will be key cost and risk drivers. Whole-life cost estimating for COTS-based systems is complex and unreliable. Major

COTS-based systems require as much design effort as a bespoke development and provision is needed for the redesign process that characterises COTS-based systems throughout their life. To maintain the capability of COTS-based systems, there will be a significant and probably unstable cost throughout the life of the system. Short-term savings in funding of COTS-based systems may lead to loss of skills or missed opportunities for upgrades, thus increasing overall whole life costs.

4.10.2 Support

The extent and nature of COTS IT hardware and software products in the system must be reflected in the whole life support strategy, addressing the issue of short lifetime for much COTS IT. With COTS-based systems, there will be a need for front line maintainers to access software design if some bug fixes and reconfigurations are to be successful in operational situations. The long-term support process must keep pace and reduce costs in line with current commercial practice and technology developments.

Plans (and funding) must be in place to provide relevant COTS market knowledge, to inform on upgrade decisions, provide realistic user aspirations for maintaining state-of-the art capabilities and give warnings of future obsolescence.

4.10.3 Review of information

This comprehensive DPA guide is important because it has been written for the members of integrated project teams. However, it by no means deals with all the issues associated with the use of COTS IT and, to an extent, is indicative of the gulf between DPA and the Equipment Capability customer. It starts with the benefits of using COTS IT and highlights the key issues. It then examines through life management, inter-project liaison, managing the contractor, managing COTS software, safety, security, reliability and environmental issues. On the subject of costs, it highlights the significant and probably unstable life cycle costs of COTS IT. Considering support, it emphasises the need to keep pace with current commercial practice both in process and cost reduction terms.

Benefits

Reduced initial acquisition costs and acquisition times. Ability to remain close to state-of-the-art.

Problems

Cost estimating for COTS-based systems is complex and unreliable. Short-term savings in COTS- based systems can increase overall whole life costs. Obsolescence, lack of product control, interfacing difficulties, requirements trade-offs and open standards are all difficulties.

Other issues

There is a need for requirements trade-offs and a changing relationship with contractors. To maintain COTS-based systems capability involves significant through life cost. Plans and funding must be in place to provide relevant COTS market knowledge.