Iterative Reduction Processing

As commented above, the complexity of the data reduction has implied its decomposition into different tasks or processing systems in charge of specific parts of the processing. The system break down approach is very common for large processing systems to allow a distributed development. The decomposition has been based on the identification of those major processing tasks which may operate in a relatively independent manner limiting the data interdependencies. Practically, all tasks are in fact in- terdependent from the point of view of the data but from a development point of view, a well defined ICD allows completely decoupled components to be developed and even operated in different locations. This approach was also taken due to the fact that Gaia processing system was developed in many countries and by teams of varying competence [O’Mullane et al., 2006b].

Gaia processing must be fully self–calibrating with very reduced external connections. It must solve together the attitude, instrument parameters and the object catalogue parameters (positions, proper motion, parallaxes, etc.) mainly from the observations and the housekeeping data coming from the spacecraft. Therefore, the same data that are ultimately forming the astrometric catalogue are also used to reconstruct the attitude and to determine the instrument calibration parameters. Similar considerations, but with less entanglement, apply to the spectroscopic and photometric data, where reference stars will be used to make the initial calibrations and

determine the photometric system. For the work presented in the thesis, the photometric and spectroscopic reduction operation is not relevant and therefore it will not be covered.

The main iterative steps are:

Step 0

Reconstruct observational and housekeeping measurements from the raw telemetry packets.

This is done only once by IDT as the data is received by MOC. Step 1

Treat raw observational measurements to calibrate the Bias and the Astrophysical Background.

Done for the first time in the daily pipeline at DPCE by IDT and improved as part of the data reduction by IDU as later described in Sections 4.4 and 4.5).

Step 2

Apply calibrations obtained from Step 1 to reconstruct the esti- mated photo counts of the observed objects from the raw obser- vational measurements. This step basically revert the CCD image signal digitisation, subtract the Bias and Astrophysical Background and correct the PEM-NU of the readout.

In this step external calibrations for the PEM-NU and CCD cosmet- ics are also applied. This operation is performed in IDT and IDU and it is a prerequisite for any further processing of the observed images. Step 3

From the reconstructed photo counts of the observed objects (see Step 2) the LSF/PSF response of the instrument is determined. The LSF/PSF calibration is one of the more complex tasks. It depends not only on the observations but also in the astrometric solution from Astrometic Global Iterative Solution (AGIS), the photometric solu- tion from Photometric Pipeline (PhotPipe) and the Cross-Match. AGIS and PhotPipe solutions provide essential parameters required

for the proper modelling of the image shape response of the Gaia instrument; such as the AC motion, magnitude and colour of the source, zero-point reference for the model, etc. The full set of pa- rameters and dependencies for the LSF/PSF task are provided in Section 4.6.

A preliminary calibration is done in FL (or by offline processes) but it is IDU the one in charge of computing and progressively improving the final LSF/PSF calibration in each DRC.

Step 4

Using the calibrated LSF/PSF library (see Step 3), the Image Pa- rameters are obtained.

This is also done for the first time in IDT for the newly received data but they are subsequently improved by IDU on a DRC basis. Step 5

Finally from the Image Parameters and the Cross-Match, the atti- tude, the Geometric Calibration and the main astrometric parameters of all objects are recomputed together providing a global update of the Gaia catalogue.

The system in charge of this global update is called AGIS and runs on a DRC basis.

The subsequent execution of all steps from Step 1 to Step 5 is what ulti- mately will make it possible to reach the final mission goals on astrometry. The time scale for the described iteration loop is much longer than that of the near-real-time processing, of the order of six months. Six months is approximately the time required to assure that new observations for almost the full sky are made available for each DRC.

Next sections describe the most relevant DRC systems involved in the core data reduction, namely AGIS, PhotPipe and IDU.

3.2.1 Astrometric Global Iterative Solution

The AGIS or astrometric core solution is the cornerstone of the data pro- cessing [Lindegren et al., 2012] and [O’Mullane, 2012]. It provides the focal plane calibration, the improved attitude solution and updates the main astrometric parameters of the Gaia catalogue. AGIS determines six astrometric parameters:

• Position on celestial sphere: α and δ. • Parallax (distance): $.

• Proper motion: µα∗ and µδ.

• Radial proper motion: µR (only computed for a small subset of

sources).

These parameters are determined using (in theory) no a priori knowledge of these quantities but deriving them from observation data alone in a self–consistent manner. The solution constitutes an internally consistent celestial reference frame, but does not coincide with International Celestial Reference System (ICRS). The solution is then transformed into ICRS using a uniform rotation computed using a subset of primary stars and quasars. Recent findings indicate that the BA is not as rigid as expected and that fluctuations observed may not be completely corrected Mora et al. [2014]. This issue may require the inclusion of some additional external references to get rid of the systematic bias on the parallax parameters [Hobbs and Lindegren, 2011].

Resolving the resulting system of equations combining the six astromet- ric parameters of all sources, the attitude and the geometric calibration parameters is intractable in a direct way with today’s computational capa- bilities [Bombrun et al., 2010]. Basically one would obtain a system with 1012 measurements for determining 5 × 109 unknowns in a globally and self-consistent manner. Instead of that, the adopted approach is to resolve each block of parameters separately and iterate globally until reaching a

Figure 3.3: Simple AGIS iteration diagram where A stands for At- titude, S for source parameters and C for the Focal Plane geometry

calibration parameters (Credit: Gaia/DPAC/AGIS)

convergence in the parameters updates. Figure 3.3 illustrates this very basic AGIS internal iteration. For more details on AGIS design and im- plementation details see O’Mullane et al. [2011]

The geometric calibration model developed for AGIS is in charge of pro- viding the transformation matrix between the focal plane coordinates and the celestial coordinates [Lindegren, 2008a]. This transformation works in both spatial (position in the field of view) and temporal scale (time of the observation). It also tries to disentangle the purely geometric response (GEO-CAL) with respect to the offsets introduced by the colour and mag- nitude response of the instrument and the other image distortion terms i.e. the CTI effects. The non-purely geometric terms (COMA-CAL) obtained in the AGIS calibration should eventually be removed (set to zero) being completely absorbed by the LSF/PSF model calibration.

AGIS is operated at DPCE.

3.2.2 Photometric Pipeline

Broad–band photometry will result from the fluxes measured in the AF CCDs during the star transits, while the dedicated BP/RP CCDs will pro- vide dispersed images in blue and red bands. The photometric calibrations will ultimately provide the fluxes (magnitudes) and basic spectral informa- tion for all observed stars. Combined with the parallax measurements this will provide stellar luminosities, the actual fluxes produced by stars, for stars in a wide range of temperatures, composition and stages of evolution.

The system in charge of the photometric calibrations is called PhotPipe [van Leeuwen et al., 2011]. The PhotPipe data processing has been de- signed splitting the process into three steps:

Pre-processing:

In this first phase, the raw observations are processed in order to ”clean” them from several effects. At this stage Bias and Astro- physical Background is subtracted and possible contamination and blending effects are considered. Also the effects of the CTI are also accounted for in this pre-processing stage.

Internal calibration

The internal calibration will combine all different transits of a given source to a common reference internal system producing a mean Gaia observation. This internal calibration accounts for the differential in- strumental effects (in CCD sensitivity, flux losses by aperture effect, variations of the LSF/PSF, spectral dispersion and geometry, etc.) and depend on the colour and type of the source. It is worth men- tioning that the selection of calibration sources ensuring a good rep- resentation of all kind of observed sources is fundamental [Carrasco et al., 2015].

External calibration

Once the mean Gaia observations are produced, a final step, the external calibration, transforms them to absolute fluxes and wave- lengths. In principle, few calibration sources are needed but they need to have accurate determinations of their absolute fluxes and their non-variability. For this purpose, a big international observa- tional effort has been done using on ground telescopes.

For the astrometric reduction, the photometric processing basically pro- vides the source colour information which is used for colour dependent calibration and for the chromaticity correction of the astrometry. The photometry result must, therefore, be available for the processing of the next astrometric solution in each DRC.

PhotPipe runs at DPCI in Cambridge. Within DPCI, the Source Envi- ronment Analysis (SEA) is also run. The SEA software is used to check for the presence of faint companions to sources, which can disturb, or even distort, the astrometric and photometric measurements. This software is to be implemented later during the mission, as SEA requires good data coverage to be successful and eventually could also be used in the IDU processing.

3.2.3 Intermediate Data Updating

Intermediate Data Updating (IDU) is the instrument calibration and data reduction system more demanding in data volume and processing power of DPAC. Intermediate Data Updating (IDU) aims to provide:

• An updated Cross-Match table using the latest attitude, geometric calibration and source catalogue available

• Updated calibrations for Bias, Astrophysical Background and instru- ment LSF/PSF model

• Updated Image Parameters; location and fluxes.

The design and development of the IDU system has been the main objective of this thesis and thus the next two chapters have been devoted to its detailed description and operation. This section, on the other hand, only covers the role of IDU in the iterative data reduction.

As presented in Section 3.2, the successive iterations between IDU, AGIS and PhotPipe (as shown in Figure 3.4) are what will make possible to achieve the high accuracies envisaged for the final Gaia catalogue.

Basically, IDU incorporates the astrometric solution from AGIS resulting in an improved Cross-Match but also incorporates the photometric solution from PhotPipe within the LSF/PSF calibration obtaining also improved Image Parameters. These improved results turns out to be the starting

Figure 3.4: Diagram of the IDU, AGIS and PhotPipe system operation and interdependency; from raw data until the updated source catalogue

updates

point for the next iterative reduction loop. Without IDU, Gaia would not be able to provide the envisaged accuracies and its presence is key to get the optimum convergence of the iterative process on which all the data processing of the spacecraft is based.

IDU software design and development is also managed by CU3-UB group and integrates developments from CU5-DU10 and CU3-Torino.