CU9 Science Enabling Applications Development Work Package Software Requirements Specification (WP970)

(1)

Requirements Specification (WP970)

prepared by:

X. Luri, P.M. Marrese, F.Julbe, H. Enke, N. Walton,

G. Gracia, G. Comoretto

approved by: X. Luri, P.M. Marresse

reference:

GAIA-C9-SP-UB-XL-036

issue:

revision:

1 date:

2014-11-18

(2)

Abstract

This document provides the list of requirements applicable to the Gaia science en-abling applications work package (WP970). It also covers deliverable D4.1 for the GENIUS project.

(3)

Document History

Issue Revision Date Author Comment

D 0 2014-11-17 XL Code document change

D 0 2014-09-19 FJL First version of the CU9 science enabling applications SRS

(4)

Acronym List

The following table has been generated from the on-line Gaia acronym list:

Acronym Description

ASDC ASI Science Data Centre AUT AUTomated

BP Blue Photometer

CU Coordination Unit (in DPAC)

DPAC Data Processing and Analysis Consortium ESA European Space Agency

ESAC European Space Astronomy Centre (VilSpa)

GAIA Global Astrometric Interferometer for Astrophysics (obsolete; now spelled as Gaia)

GENIUS Gaia European Network for Improved User Services GPDB Gaia Parameter DataBase

HDFS Hadoop Distributed File System MAN MANual

PM Polarisation Maintaining RP Red Photometer

RVS Radial Velocity Spectrometer SAT Satellite Archive Team

SRS Software Requirements Specification SSS System Software Specification TAP Table Access Protocol

TBD To Be Defined (Determined) VO Virtual Observatory

(6)

1 Introduction

This document sets out the software requirements pertaining to the Gaia science enabling ap-plications.

1.1 Objectives

The objective of this document is to define a set of requirements for those applications developed for the GAIA catalogue scientific exploitation. Some of the listed requirements can be divided into smaller ones in order to improve its development and monitoring. The requirements will cover all funtional and technological aspects of the science enabling applications in order to produce fully functional state-of-the-art products for the GAIA catalogue scientific explotation.

1.2 Scope

This work package (WP970) includes 5 sub-work packages that cover different functional do-mains and together they cover all aspects of the software applications to a full exploitation of the GAIA catalogue.

1.3 Assumptions

Some requirements have higher level CU9 requirements. These dependencies are recorded but not repeated in this document; the relevant SRS (see next Section) should be referred to for further details. The top–level requirements specification should be considered as ‘level 0’ requirements that are unlikely to change during the development and implementation iterations, while the derived and more detailed requirements herein may subject to significant revision consistent with a pragmatic and ‘agile’ development process.

1.4 Applicable Documents

When applicable documents change a change may be required in this document. The applicable documents are listed here for clarity; a full reference list is provided at the end of the document.

(WOM-086) Software Development Plan for CU9 (WOM-033) Gaia Catalogue and Archive SRS (AB-026) Gaia data access scenarios summary

For convenience only the structure of this document follows that of the level 0 SRS (WOM-033).

(7)

1.5 Requirement Definition

The requirements set out in this SRS follow the labelling scheme:

CU9-WP97x-X-SCOPE-xxx

Where :

• WP97xis the (sub–)WP number as follows:

– WP971: Management

– WP972: Advanced data access tools

– WP973: Data Mining

– WP974: Cross-Matching

– WP975: Science Alerts

• Xis eitherS(forScientific),T(forTechnical),Q(forQuality Assurance), orM(for

Management)

• SCOPEis a four letter scope specification of the requirement following the identi-fied list of possible values as shown in the list below

• xxxis a monotonically increasing counter for every unique combination

SCOPE is a 4-letter scope specification of the requirement. In this document the following scopes have been used:

GLOB: for top-level global requirements;

ALGO: for requirements on the scientific algorithm to be applied in the data processing (this should be for the detail on how the functionality is to be achieved);

CODE: for requirements on the coding activities (this should just be for how the code is written e.g. the GPDB should be used);

COOR: for requirements on the coordination activities (this should not be used for func-tionality requested by other WPs, but that you coordinate with them in the design etc.);

DATA: for requirements on data stream handling (this can include descriptions of input and output data);

FUNC: concerning functional requirements (this describes the functionality that is re-quired by the system);

HARD: concerning hardware requirements;

PERF: for requirements on performances;

(8)

QUAL: for requirements on the quality assurance (both scientific and software, robust-ness and quality of data could go here);

RESO: for requirements on the resource management.

DOCU: for requirements on documentation.

Each requirement is presented with its own unique label and a number of attributes in the fol-lowing form:

CU9-WP97x-X-SCOPE-020 C.v Verification Status Description

Parent: Parent

CU9-WP97x-X

-SCOPE-xxx

The unique identifier of the requirement (see above).

C.v Version number of the requirement composed of major part (C) correspond-ing to the cycle (1, 2 and 3 correspondcorrespond-ing to A, B and C respectively in WOM-086) in which the requirement was created and minor part (v) corre-sponding to the version of the requirement.

Verification Envisaged validation method of requirement - this will be either AUT for automated or MAN for Manual.

Status Status identifier.

(9)

2 List of requirements

2.1 General requirements

CU9-WP971-M-PLAN-020 16.0 MAN Draft

CU9 WP970 shall operate within the agreed management structures of the CU (which in turn operates within the existing DPAC management structure).

Parent: CU9-ARC-M-PLN-020

CU9 WP970 shall develop all code and documentation within the DPAC code repository where it shall be visible to all DPAC members but modifiable only by CU9 members. Parent: CU9-ARC-M-PLN-040

CU9 WP970 shall follow the engineering guidelines laid down in WOM-086. Parent: CU9-ARC-M-PLN-060

CU9 WP970 developments shall be overseen by a System Engineering coordination group. Parent: CU9-ARC-M-PLN-060

CU9-WP971-M-COOR-020 1.1 MAN Draft

CU9 WP970 shall coordinate with other CU9 WPs in particular WP930 and WP950. Parent:

2.2 Advanced data access tools

CU9-WP972-T-FUNC-020 1.0 MAN Draft

The Gaia Archive shall provide a TAP access to the Gaia Catalogue. Parent: CU9-ITG-T-FUN-140, CU9-ITG-T-FUN-020

The Gaia Archive shall provide a SSAP access delivering BP, RP and RVS spectra cali-brated in wavelength and in flux. The delivered spectra shall be compliant with the Spec-trum datamodel (VO standard). The spectra resolution shall be specified for each specSpec-trum and standard units shall be used.

(10)

WP972 shall adapt at least one VO-tool able to display the data provided through the TAP access of the Gaia Archive.

Parent:

WP972 shall adapt at least one VO-tool in order to select and display Gaia spectra. Parent:

WP972 shall adapt at least one VO-tool in order to visualise, at least in 2D, position-nal/astrometric Gaia data.

Parent: CU9-ADV-T-FUN-040

All VO-tools adapted by WP972 shall be able to access the Gaia Catalogue through the Gaia Archive using one of the implemented VO protocols.

Parent: CU9-ITG-T-FUN-140

All VO-tools adapted by WP972 shall be able to interact each other using SAMP, in order to process different types of Gaia data in the adequate tools.

For instance: selecting a Gaia object inside Aladin and sending the position to SPLAT in order to display its spectra.

WP972 shall deliver at least one spectra matching tool/service able to work on the spectra provided by the Gaia Archive.

Parent: CU9-ITG-T-FUN-140, CU9-ITG-T-FUN-040

All spectra matching tools/services provided by WP972 shall be able to match Gaia spectra to users spectra. A users spectrum may be provided in different formats, but at least a

(11)

2.3 Data Mining

Data mining framework will allow the community to perform complex queries through an easy to use interface. This User Interface must be based on a web technology preferably fully integrated or into the rest of the Gaia archive access and querying tools or compatible with them.

Parent: CU9-CIF-T-MAN-020

Data mining framework must implement a basic set of MlLib (Machine Learning Libs) as a basic building blocks to build more complex use cases on top of them. These complex and more common use cases will also be implemented in the framework.

The framework must integrate a job management platform to handle task execution. This job management system must implement the job submitting policies necessary such as: ’synchronized execution’ or ’batch execution’ in a prioritized queue policy, with the nec-essary cluster resource management. The usage of the cluster and its resources should be monitored by the appropriate tools.

Advanced users will also be allowed to submit their own custom applications to the cluster though a proper entry point to the framework. The submitted job must also be integrated into the job management system for its execution.

Parent: CU9-CIF-T-MAN-040, CU9-ITG-T-FUNC-200

The data mining framework must implement the security policies necessary according to the SAT (Science Archive team) and ESA security standards. Also it should be integrated into the rest of the security framework (single sign on service) with the rest of the Gaia Archive web services and interfaces.

(12)

Data mining task results must be displayed through the client using advanced data display-ing tools (includdisplay-ing graphical features -2D/3D-). Interaction with the visualization Work Package (WP980) must be established.

Parent: CU9-ADV-T-FUN-020, CU9-ADV-T-FUN-040

CU9-WP973-T-DATA-020 1.0 MAN Draft

Gaia archive data must be provided into a HDFS (Hadoop Distributed File System) o sim-ilar distributed file system together with its relational version and both have to be in synch. Also, Archive metadata must be provided in order to be user by the data miming framework and its query system. A preferential file format to be used by the data mining framework must be evaluated and proposed by this work package.

Parent: CU9-CAT-M-PLN-080

2.4 Cross-Matching

In the following by External Catalogues we intend the catalogues for which the cross-match will be pre-computed and will be part of the Gaia releases. Those will be catalogues withNstars

greater than a few hundred millions, observed in the optical/near infrared and public available. The External Catalogues, the cross-match algorithm and the output will be release specific, so the following requirements will need to be fullfilled for each release.

In order to calculate the cross-match of the Gaia Catalogue with External surveys, the latter must be homogenized in order for a single algorithm to run on several different catalogues. It is thus necessary to have both original catalogues (either full or a sub-set of the fields needed by cross-match and validation) and a cross-match specific version of the same catalogues.

CU9-WP974-S-DATA-020 16.0 MAN Draft

Define External Catalogues (original catalogues and cross-match specific catalogues) input through a definition of the data models.

CU9-WP974-S-DATA-040 16.0 MAN Draft

Define cross-match output through a definition of the data models (full output will be used for the cross-match validation, sub-set will be made available to final users).

(13)

Prepare and deliver to ESAC the gbin files of the cross-match specific External Catalogues using the CU9 data models.

CU9-WP974-S-ALGO-020 16.0 MAN Draft

Detailed description and specifications of the procedure that calculates the cross-match of the Gaia Catalogue with the External Catalogues starting from the defined input and obtaining the defined output.

A cross-match software compliant with the specified algorithm must be available in ASDC for cross-match validation or directly to calculate the cross-match.

CU9-WP974-S-ALGO-020 16.0 MAN Draft

Definition of the validation process of the cross-match algorithm. Parent: CU9-ADV-T-FUN-080

The cross-match validation must be performed following the cross-match validation re-quirements.

Definition of the data set needed for the cross-match validation. Parent: CU9-ADV-T-FUN-080

CU9-WP974-T-DOCU-020 16.0 MAN Draft

Provide documentation on original External Catalogues and a detailed description of the corresponding data models.

Parent: CU9-DOC-S-FUN-040

Provide documentation on cross-match specific External Catalogues and generate a de-tailed description of the corresponding data models.

(14)

Provide documentation on cross-match tables (output of cross-match activities) and gener-ate a detailed description of the corresponding data models.

Parent: CU9-DOC-S-FUN-040

Provide documentation on cross-match algorithm and on the tests performed to develop it. Parent: CU9-DOC-S-FUN-040

Provide documentation on the validation of the cross-match algorithm. Parent: CU9-DOC-S-FUN-040

2.5 Science Alerts

CU9-WP975-T-COOR-020 1.1 MAN Draft

....TBD...

2.6 Documentation

...TBD...

Parent: CU9-DOC-S-FUN-040,CU9-DOC-S-PLN-060,??

This section may be deleted, generation of the documentation should be added as requirements in each sub-WP section, in addition a requirement on the publication of the documentation should be sent to WP920 for inclusion in their SRS document.

2.7 Help Desk

See elsewhere for the SRS of the relevant workpackage (WP953)

(15)

WP970 Requirement Missing Top level CU9 requirement

CU9-WP971-M-COO-020 No requirement on coordination between CU9 WPs ex-ists in the CU9 SRS.

(16)

References

[AB-026], Brown, A., Arenou, F., Hambly, N., et al., 2012,Gaia data access scenarios sum-mary,

GAIA-C9-TN-LEI-AB-026,

URLhttp://www.rssd.esa.int/cs/livelink/open/3125400

[WOM-033], O’Mullane, W., 2009, Gaia Catalogue and Archive Software Requirements and Specification,

GAIA-C9-SD-ESAC-WOM-033,

URLhttp://www.rssd.esa.int/cs/livelink/open/2907710

[WOM-086], O’Mullane, W., Luri, X., Gracia, G., 2014,CU9 Software Development Plan,

GAIA-C9-PL-ESAC-WOM-086,

(17)

A

Requirements traceability

The traceability between this SRS and parent requirements such as the SSS should be given here.

A scriptmakeRequirementsTraceParents.rbis provided in CU1/docs/common/scripts to create this from the higher level requirements optionally specified in thereqTeX macro i.e. the PARENT requirement. If the requirements contain tex ’labels’ which start with ’req:’ then they will become clickable links in the table. If you do not have labels you may use the script

addReqLabels.rbwhich attempts to add labels to all requirements.

The following table provides traceability for derived requirements within this requirements specification, and also to level 0 requirements in WOM-033.

Parent Requirement Requirements in this document CU9-ADV-T-FUN-020 CU9-WP973-T-DATA-020

CU9-ADV-T-FUN-040 CU9-WP973-T-DATA-020

CU9-ADV-T-FUN-080 CU9-WP974-S-DATA-020, CU9-WP974-S-DATA-040, CU9-WP974-T-DATA-020, CU9-WP974-T-DATA-040, CU9-WP974-S-ALGO-020, CU9-WP974-T-FUNC-020, CU9-WP974-S-ALGO-020, CU9-WP974-T-FUNC-020, CU9-WP974-T-DATA-020, CU9-WP975-T-COOR-020 CU9-ARC-M-PLN-020 CU9-WP971-M-PLAN-020 CU9-CAT-M-PLN-080 CU9-WP973-T-DATA-020 CU9-CIF-T-FUN-080 CU9-WP973-T-DATA-020

CU9-CIF-T-MAN-020 CU9-WP972-T-FUNC-200, CU9-WP973-T-DATA-020, CU9-WP973-T-DATA-020

CU9-CIF-T-MAN-040 CU9-WP973-T-DATA-020

CU9-CIF-T-MAN-140 CU9-WP973-T-DATA-020

CU9-DOC-S-FUN-040 CU9-WP974-T-DOCU-020, CU9-WP974-T-DOCU-040, CU9-WP974-T-DOCU-060, CU9-WP974-T-DOCU-080, CU9-WP974-T-DOCU-100, CU9-WP975-T-DOCU-020

CU9-DOC-S-PLN-060 CU9-WP975-T-DOCU-020

CU9-ITG-T-FUNC-200 CU9-WP973-T-DATA-020

CU9 Science Enabling Applications Development Work Package Software Requirements Specification (WP970)

Requirements Specification (WP970)

prepared by:

X. Luri, P.M. Marrese, F.Julbe, H. Enke, N. Walton,

G. Gracia, G. Comoretto

approved by: X. Luri, P.M. Marresse

reference:

GAIA-C9-SP-UB-XL-036

issue:

revision:

1

date:

2014-11-18

Abstract

Document History

Contents

Acronym List

1

Introduction

1.1

Objectives

1.2

Scope

1.3

Assumptions

1.4

Applicable Documents

1.5

Requirement Definition

2

List of requirements

2.1

General requirements

2.2

Advanced data access tools

2.3

Data Mining

2.4

Cross-Matching

2.5

Science Alerts

2.6

Documentation

2.7

Help Desk

References

A

Requirements traceability