ICT Software Engineering and Quality Management Tasks

(1)

ICT Software Engineering and Quality Management Tasks

COMP-70.20.00.00-0014-A-PLA

Version: A

Status: draft

2013-04-11

Prepared By: Organisation Date

Erik Allaert ALMA ICT 2013-04-11

ICT-Leads Approvals: Organisation Date

Brian Glendenning

Erich Schmid

George Kosugi

Jorge Ibsen

ALMA ICT EU-Lead

ALMA ICT NA-Lead

ALMA ICT EA-Lead

(2)

Change Record

Version Date Affected Section(s)

Change Request #

Reason/Initiation/Remarks

(3)

Introduction

The ALMA Computing Integrated Product Team (CIPT) has presented in mid-2012 its implementation plan of the Integrated Computing Team for the operations phase of the ALMA Observatory (COMP-70.05.00.00-0025-A-PLA). This plan did not only foresee the mere merging of the trilateral CIPT with the JAO ALMA Department of Computing (ADC), but it also indicated how the importance of some activities were shifting after the construction phase, strengthening particular areas that were in need of this. The reorganization of some functions and groups included the explicit extension of the Software Engineering subsystem with Quality Management. The present document describes the tasks of that group.

1.1 Scope

This document will emphasize Quality Management aspects. Some of these tasks are new since the Computing groups’ reorganization into ICT, while others were considered inherent to the CIPT SE-subsystem tasks. It is not the intention to repeat the historic backgrounds, beyond the overview in section 3.2, and the listing of activities to the extent that they are related to Quality Management. The CIPT-SE activities and plans are described in detail in the ALMA Software Engineering Plan [RD2].

1.2 References

The following documents contain additional information and are referenced in this document.

Reference Document title Document ID

[RD1] ALMA Acronyms and Abbreviations ALMA-80.02.00.00-004-A-LIS [RD2] ALMA Software Engineering Plan COMP-70.20.00.00-0002-I-PLA

[RD3] ALMA Software Engineering Practices ALMA-70.20.00.00-003-A-PRO [RD4] ALMA Software Delivery Process COMP-70.05.00.00-0012-B-PLA

[RD5] Assessment of the ALMA software development process – report 2

ESA/ESTEC (K. Hjortnaes, P. van der Plas)

1.3 Abbreviations and Acronyms

A limited set of basic acronyms used in this document is given below. A complete set of acronyms used in the ALMA project can be found in reference [RD1]. Several of the concepts these abbreviations stand for are defined and described in further sections or in the reference documents.

(5)

CVS Concurrent Versions System (version control system) ICT Integrated Computing Team

NRI Nightly Reporting Infrastructure

[S]QA [Software] Quality Assurance

[S]QC [Software] Quality Control

[S]QM [Software] Quality Management

[S]QP [Software] Quality Planning

SE Software Engineering

SE-QM Software Engineering and Quality Management

SVN Subversion (version control system) TRR Test Readiness Review

(6)



Quality Management Definitions

Software engineering (SE) is the application of a systematic, disciplined, quantifiable approach to the design, development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software. [Wikipedia]

Quality: The degree to which a component, system or process meets specified requirements

and/or user/customer needs and expectations. [After IEEE 610]

Quality Management (QM): Coordinated activities to direct and control an organization with regard to quality. Direction and control with regard to quality generally includes the establishment of the quality policy and quality objectives, quality planning, quality control, quality assurance and quality improvement [ISO 9000].

Quality Management includes the processes and activities of the performing organization that determine quality policies, objectives, and responsibilities so that the project will satisfy the needs for which it was undertaken [A Guide to the Project Quality Management Body of Knowledge. Kim Heldman; Vanina Mangano]. The Quality Management processes comprise:

 Quality Planning (QP): The process of identifying quality requirements and/or

standards for the project and product, and documenting how the project will demonstrate compliance.

 Quality Assurance (QA): The process of auditing the quality requirements and the

results from quality control measurements to ensure appropriate quality standards and operational definitions are used. SQA is a planned and systematic activity to provide confidence that a software product conforms to established requirements.

 Quality Control (QC): The process of monitoring and recording results of executing the quality activities to assess performance and recommend necessary changes.

Other authors define the terms (S)QM, (S)QA, (S)QP and (S)QC differently and in particular many use SQA and SQM interchangeably. We will take here a practical approach, and consider SQM the umbrella for all activities that

 help identify and establish the desired level of quality (based on project description, stakeholders’ requirements, procedures, guidelines, etc.).

 monitor the quality of software (adherence to standards, number/severity of issues, ...)  reduce the number of software issues by preventing their introduction in the development

process (using standards etc), or detect and remove them as early in the process as

possible; or put more generally, improve the software development process

 help establish/enforce a project-wide sense for quality. Quality is not the task of a few individuals or a single group, but it’s everyone’s job and not an independent, stand-alone discipline (similar to safety). Requires obviously managerial commitment.

(7)

Note that SQM encompasses the entire software development process, from requirements definition, design, implementation, testing, etc. up to releases. In view of the maturity of the ALMA project, not all these aspects may appear equally relevant at first sight. However, as there are still new requirements passed on to Computing/ICT, we still do need to pay proper attention to all of them.

There are different quality aspects/factors playing a role in the entire lifespan of a software product. This can be classified as follows (according to McCall’s factor model):

a) Initial implementation: product operation factors

1. Correctness: correct implementation, as specified in the requirements; this gets

certified during Verification

2. Reliability: measured by e.g. the uptime of the SW application or system as a whole

(or failure rate)

3. Efficiency: includes efficiency of resource utilization; to be improved by reducing

overheads (in software and hardware)

4. Data Integrity: lossless transmission of data, encryption where security is needed,

authorization and permission schemes.

5. Usability: how well different categories of users can understand and navigate through

the software product; HCI plays a role in this. b) Product revision factors (new revisions)

6. Maintainability: how easily defects are identified, corrected, and these corrections

can be verified. Modularity of the software, its documentation – both internally via comment statements and external via maintenance manuals – and weak coupling between modules (few dependencies) play an important role in this.

7. Flexibility: how well adaptive maintenance is supported, i.e. how well the

functionality can be expanded, modified or adjusted to changed needs, etc. This starts off with a specification/design that is open for this type of modifications.

8. Testability: this results typically in additional code that allows testers to find out if a

particular path is followed, or to report intermediated results – e.g. via logs, or test input data that should result in an expected reaction from the system.

c) Product transition factors

9. Portability: if the product will have to be migrated to another HW platform, or run on

a different/newer operating system

10. Reusability: how easily existing SW can be re-used in another product

11. Interoperability: how easily the product can work or interface with other

products/systems

The above does not consider other factors that also impact the Quality, e.g. magnitude and complexity of the project, number of developers/testers and their professional qualifications.

(8)

These are within the context of the ALMA ICT group and for the purpose of this document taken as unalterable. Also the aspects related to product transition (see item c above) are beyond the current scope of SQM in ALMA.

In this document, we will follow a pragmatic, common sense approach, rather than sticking to a particular methodology or definition. The availability for operational use of the ALMA system, including the functionality required according to planning, will serve as a prime quality indicator, and we’ll attempt to achieve this by applying some fundamental, basic principles or “best practices” – Common Sense QA (http://commonsenseqa.wordpress.com/)

(9)



Software Engineering and Quality Management Activities

3.1 High-level Objectives

The following are the high-level objectives passed on to the SE-QM group:

 Continuation of essential services provided by the Software Engineering subsystem to ALMA Computing during the ALMA construction;

 Regular assessment of software engineering tools and practices for ALMA Computing, and the introduction of new tools and practices as required;

 Investigation and evaluation of the impact of using modern methodologies to improve the quality of the software;

 Definition of quality goals for all areas of the ALMA software, and guiding the ICT Groups to achieve these goals;

 Collection of reliable statistics on the reliability, stability and efficiency of ALMA software;

 Based on these statistics identification of areas for improvement and the liaison with ICT Management and other ICT Groups to implement the required improvements;

 Study and suggest improvements to testing strategies, technology, and procedures;

 Interface to ALMA Product Assurance/Quality Assurance and ALMA System Engineering;

 Assist ICT Management with the preparations for the ALMA System Review & Operational Readiness Review;

 Documentation of the external interfaces and processes the ICT maintains with its stakeholders and suggesting improvements in these areas;

3.2 Software Engineering

The purpose of Software Engineering is to guide the software developers in their work, guiding them with a clear stated, repeatable and documentable process, providing them with all the necessary artefacts to produce high quality code and documentation thereof.

The deliverables of the Software Engineering task influence the operations of almost all other subsystems and tasks which are required to comply with processes, methodologies and procedures.

The SE activities are outlined in the ALMA SW Engineering Plan [RD2]. They are subdivided

into the following categories:

A. issuing of standards, practices, procedures, methodology, tools.

B. purchasing and distribution of tools - support for the usage of the above standards and tools

(10)

C. measuring the adoption of the above standards

D. assessing the quality of the process in general and improving it

E. adapting the process and the tools to the introduction of new technology, whenever needed, or simply keeping CASE tools up to date

F. All support in terms of system management and hardware selection/testing to fulfil the tasks corresponding to the above categories.

Software Engineering has the responsibility to systematically audit developers’ deliverables and flag any failure to follow adopted procedures and standards.

It is also the task of the Software Engineering group to improve the Software Development Process by promoting any organizational, behavioural or technical pattern, which proves of some use within the global effort of ALMA Computing.

The boundaries of the SE activities are not a priori fixed – it is a matter of definition and depends to some extent to organizational preferences. In the above list, item F corresponds primarily to the activities of system managers and network administrators: supporting the selection/testing of hardware plus the design, installation and maintenance of the computing environment (like installation of new hardware, updates of operating systems, software tools, etc.).

The deliverables can be categorized as follows:

 Procedures to be used for software delivery, integration, testing and commissioning.  Standards to be adopted by all ALMA subsystem developers (design models, source

code, templates for documentation)

 Hardware standards to be followed by all developers working for ALMA, and by the ALMA site in Chile in particular.

 Third Party CASE tools to facilitate design, requirements tracing, quality assurance and control, to be made available both to single developers and to the Integration group. These tools will have to be evaluated, tested, configured and periodically upgraded as needed.

 Support on software tools usage.

 Third party software tools for Configuration Management

 Third party software tools for replication of the software repository to the various ALMA sites.

 Common repository of current relevant technical knowledge on the Web in form of Frequently Asked Questions, generic reference materials and generic statistics and audit results.

3.3 Quality Management in general

Formal adoption of well-established Quality Assurance standards like ISO 9000 or CMMI has since the start of the ALMA project not been among the objectives of the Software Engineering task. The adoption of a written, repeatable and documentable process (as outlined in section 3.2 above) was considered in first approximation an equivalent and sufficient approach, pursuing the same goals. It was clear from the beginning though that procedures aimed at increasing the

(11)

quality of the products were going to be introduced gradually and not all in one go from the beginning. There are in fact several quality controls incorporated into the Nightly Reporting Infrastructure (NRI), exercising the ALMA code on a daily basis.

According to the above section and the “ALMA SW Engineering Plan” [RD2], Software

Engineering activities already covered in the pre-ICT era partially the following components of quality management (see also section 2 and [Wikipedia]):

 QA: organizational quality guide of standards, best practices, procedures, methodology, off-the-shelf tools

 QP: project level quality plan, committing the project to follow standards, regulations, procedure and tools, including home-made or customized ones (extending QA)

 QC: ensuring the adoption of QA and QP, by measuring, verifying, evaluating, reviewing

3.4 Software Configuration Management

Software configuration management (SCM) deals with labelling, tracking, and controlling changes in the software elements of a system. This enforces traceability of the configuration throughout the SW project’s life-cycle. Not only the source code itself needs to be controlled, but also the design, documentation, test plans, test cases and reports.

The elements of the SCM process are described in the following sub-sections.

3.4.1 Software component identification

This is to identify pieces of code that makes up a certain product/functionality (name and version number). In the case of the ALMA software, the code is grouped into subsystems and modules,

while the versions are completely controlled by the software version control product used (SVN), within the user-created branches.

3.4.2 Software version control

ALMA has recently switched over from CVS to SVN as software platform to exercise version control. The repository is also replicated to 4 different sites, with the help of a commercial replication tool (MultiSiteSVN from WANdisco), such that for the 4 executives the access is

(appears to be) local, at least for read-operations. This replication is similar to what ALMA-CIPT has been using since 2009 for CVS. Both this version control software (SVN) and the replication live up to the expectation, and there are no major technical issues hampering the software development process.

The main line of development is typically kept on the Trunk – whereby the scope of main line of development is not pre-defined, and for that a project needs to define its branching strategy.

SVN (just like CVS) itself does not impose a particular branching strategy. Two common branching strategies are:

a) basically stable

(12)

 Branches are used for development, bug fixes, experimenting, ...

 Can be applied strictly (only merge changes after QA pass) or leniently (only unit tests should pass)

b) basically unstable

 Trunk has latest code, independent of stability

 Can be applied strictly (all development takes exclusively place on Trunk) or leniently (experiments can take place on branches)

Both strategies have of course their pros and cons. In fact, ALMA’s branching strategy is somewhere in between stable and unstable:

 The trunk does have most of the latest code, but is not always stable  Branches are used for experiments

 New feature code is merged from the Trunk into the release branch (ALMA-RELEASE-B), while (some) fixes on that release branch are merged back into the Trunk.

[T1] For historic reasons, the Trunk has been accumulating differences compared to the release branch. A clear branching strategy is required, to help all developers follow the same process, and regain full control over the Trunk as the main line of development.

3.4.3 Configuration building

This is a combination of the selection of the corresponding component versions with the correct build procedure. The build procedure is typically contained in build-scripts delegating the build to the make utility and the module’s Makefile, supported by the ALMA build infrastructure.

3.4.4 Change control:

Software change control is for ALMA handled by the SCCB. The SCCB has been re-constituted in 2011 after CDR9, with responsibilities as established in the Internal Memorandum on the

Establishment of an ALMA Software Change Control Board, dated 30 September 2011. This is

chaired by the Release Manager, with participation from relevant stakeholders.

[T2] The referenced memo does not refer to QA/QM and the role this could play in the decision taking of the SCCB. QA/QM should ensure that the formal process of configuration control is followed and when/if patches are required that they are properly prepared and tested before loading onto an operation facility (see also [RD5]).

[T3] QA/QM should ensure that patches are evaluated for their applicability to the main branch (Trunk) and are merged into it for the future releases (see also [RD5]).

(13)

3.5 Collaborative Platform

3.6 Issue Tracking System

The ALMA software developers (and also other groups in ALMA) have since long been using JIRA (see http://www.atlassian.com/software/jira/overview) as a project and issue tracking tool. However, its configuration has developed organically over time, and it is not fit to track the software development and deployment process as specified in the ALMA Software Delivery Process document [RD4]. For that reason, ICT is setting up a new JIRA project, that will include

a workflow reflecting the current software delivery process, and including the requirements that come from e.g. the software acceptance.

[T4] Installation of the current JIRA version on a separate server and activation of the new ICT-project.

[T5] SE-QM will monitor issues that need re-work multiple times (test → fix → still fails → fix ...), as these could point to:

 Unclear requirements (maybe the requirements did not imply testable assertions, or they did not describe clearly and unambiguously a set of testable events)

 Not all the features of the requirement(s) are implemented  Poor error trapping / error messages

3.7 User Accounts

All ALMA software developers have user accounts for a number of tools (some licensed, others open source) and are member of various mailing lists, all used within the framework of collaborative development and common, centralized tools for ALMA software development. The creation of these accounts and the subscription to the mailing lists was not always coordinated well in the past; the same can be said about the deletion of accounts / unsubscription from mailing lists when a person leaves the project (and that can be seen as a potential security or data integrity threat).

[T6] SE-QM will coordinate the user account / mailing list management for ALMA software developers and stakeholders. The following table lists the accounts and the responsibles.

Account Description Follow-up Execution

E-mail, *nix, Windows, ...

Generic accounts giving access to local computing services Local management Local helpdesk Central ICT mailing lists (hosted at alma.cl)

An ICT member is typically member of various ICT mailing lists, e.g. ict-workers, ict-repository-users, ict-<group>. Note that ict-workers is in principle reserved for ICT

members (vs people interested in ICT activities).

(14)

Local ICT mailing lists

Executives may have ICT mailing lists for the members at their sites

Local management, SE-QM for EU Local helpdesk, SE-QM for Europe SW

repository For all the ones who need to access project2 Note that there are some external people with read-only access to the repository

(ict_repository_readonly)

SE-QM ict-repository-admins

JIRA The ALMA issue reporting and tracking system. Note that if a user leaves ICT, her/his JIRA account should be deactivated rathern

than removed

SE-QM JIRA-admin at SCO

TWiki The ALMA collaboration platform ; non-ICT users should be classified as belonging to the group NonIctButAlma or NonAlma

SE-QM TWiki-admin

at SCO ALMA-EDM Electronic Documentation Management

platform. Note that due to a limited amount of licenses, not all ICT-members have such account

SE-QM EDM-admin at SCO

STEs User accounts on STEs are managed separately, on request of IRM

SE-QM SE-QM ?

Remote access (ssh/VPN)

A member may need direct access via ssh or VPN to another executive. Typically someone at that other executive will file locally such a request to his local helpdesk.

SE-QM Remote

helpdesk/sysad mins

3.8 Testing

Note that software testing is not the same as SQA – it is rather a component of SQA (albeit a

very important one). SQA requires testing1, and testing should be based on requirements. Requirements are good for the purpose of testing (hence QA) if

1. they imply a testable assertion

2. they contain as few words as possible to clearly and unambiguously describe a single, testable event.

3. No other requirement covers the exact same event

1_“

Organisations should avoid sacrificing testing plans and timelines to compensate for development delays, budget overruns, and accelerated release deadlines. Before making the decision to eliminate or overlap test phases, carefully consider the potential impact on quality and risk” (Deloitte – see http://www.deloitte.com )

(15)

NRI is ESO’s continuous integration and quality inspection system for C/C++, Java and Python. The source code from all subsystems is extracted at close-of-business Garching time and subjected to dynamic (build and test) and static inspections (coding standards, design standards), involving several hosts. The results of these inspections are published and accessible from the web. Major violations are notified per e-mail, while violation thresholds are configurable. Performance indicators are stored in a database for trend analysis. A single cycle of an NRI run takes 24 hours. Currently NRI is set-up to alternate between checking the code of the entire ALMA software on the Trunk and the release-branch (ALMA-RELEASE-B), while some separate hosts have been configured to run NRI on specific subsystems of a specific release only. With the help of NRI we obtain a pretty detailed picture of the status of the ALMA software, and its trend. Part of that is however based on the execution of the modules’ unit test, and the fraction of code covered by these tests varies a lot across modules; on top of that, clearly not all conditions are exhaustively tested (i.e. negative testing is many times lacking), and several unit

tests are failing since a longer time (which for sure can be an issue with the test itself rather than the code under test).

There are of course some generic recommendations to follow when developing unit tests:  Key features, complex implementations and public interfaces must be tested.

 When a bug is found, there should be a test that verifies if it has been solved correctly.  Trivial methods (e.g. simple setters), do not need to be tested.

 Data used for tests should be as detailed as possible (similar to data used in an operational environment).

 The dependence on specific test data in a database should be limited.

 If an existing test fails due to a legitimate change, it should be modified or substituted by a more robust test.

 Tests should be optimised. The more time tests require, the more likely it is that the developer will avoid or interrupt testing.

[T7] SE-QM will follow-up on the failing unit tests, and aim at reducing the number of unit tests that always fail to zero. The statistics of failing unit tests will also serve as a software quality metric for the software acceptance.

[T8] SE-QM will monitor (by sampling) that bug-fixing also involves adjusting the unit tests, i.e. that the sequence followed to fix bugs is:

1. Create a test case that demonstrates the existence/effect of the problem. 2. Fix the source code

3. Run the test again demonstrating that the problem does not manifest itself anymore.

4. Include test in unit test

(16)

3.9 Software Acceptance

SE-QM will support the software acceptance process by reporting on the observed quality of the software under scrutiny for acceptance (e.g. at the TRR). It will therefore take on the following tasks (see also [RD5]):

[T9] SE-QM will certify that all material is present documenting that all new features have undergone complete verification & validation including its integration into the overall ALMA facility.

[T10] SE-QM will quantify the quality of the software (including its documentation), based on to-be-agreed criteria/metrics, indicating whether from a QA perspective the release can be accepted.

Apart from its active participation to the software acceptance process as such, SE-QM will also:

[T11] Follow-up the progress on the agreed actions and their formal close-out

[T12] Organize (or support?) a retrospective on the release, i.e. reviewing the release formally

– that means failures and successes and failures. For instance, each participant (representative stakeholders, developers and QA) could report a couple of items that were done well, and a couple to improve upon. These items can be put in a histogram, to identify the most common ones. The outcome should be a Retrospective report.

(Considering explicitly the successes may help to overcome the typical that-will-never-work attitude/perception of QA.)

[T13] Deal with the formal root-cause analysis of non-conformance reports (in particular from the acceptance process), software problem reports and thereby the fixes (see also sections 3.6 and 3.8).

[T14] Contribute to the software process improvement via a lesson learned activity. This can be e.g.; assessing the severity of issues and also why they were not detected at the previous test cycles. The goal should always be to try to find the problem at the earliest possible stage in the software life-cycle.

(17)



Reporting Line

Traditionally the Software QA function is part of/ reports to the overall Product Assurance Manager, and is hierarchically independent from the software development and test teams. This is however not how this role is filled in in the ALMA project: SE-QM is an integral part of the ALMA Integrated Computing Team (ICT); in spite of this organizational aspect, SE-QM strives of course to act and report independently – in particular in the context of the software acceptance activities.

ICT Software Engineering and Quality Management Tasks