7.3 Development of a NIR Science Reduction Pipeline
7.3.2 NIR reduction steps
The implemented near-infrared data reduction scheme is illustrated in Fig. 7.4. Data prod- ucts for the different reductions steps from the raw image to the final science frame are provided in Fig. 7.5. The full data reduction procedure can be broken up into the two independent processing blocks (i) single image reduction and (ii) image summation. The pipeline is designed in a way to yield final image products for one or both of the processing blocks without further manual interaction. The user provides an input catalog with an ar- bitrary number of raw images belonging to the same telescope pointing and taken with the same filter and upon pipeline start all further image handling procedures are automated. Single image reduction
The first processing block handles the calibration steps of the individual raw frames (upper left panel of Fig. 7.5 and subpanel 1) and the subsequent sky modelling and subtraction as shown in Fig. 7.4.
As initial calibration step, the raw input image is flatfielded to correct the detector inhomogeneities. For NIR detector arrays this is particularly critical since (i) the quantum efficiency can vary by a factor of two over the FoV (global variations), and (ii) the electroni- cally independent NIR pixels exhibit significant pixel-to-pixel changes (local variations). A master flatfieldfor each filter can be created fromdome flatfieldstaken inside the telescope dome andsky flatfieldstaken during twilight in a region devoid of bright objects. Since the thermal dome emission imprints a significant amount of structure in the dome images, the master flatfield mainly relies on the detector variations determined from the sky calibra- tion data with the dome flatfields acting only as initial correction of the former. The final master flatfield contains the information on the global and local detector variations. It is normalized to a median value of 1 and placed in a calibration frame directory, where the reduction pipeline can access it. By dividing the raw input frame by this master flatfield (subpanel 2) the intermediate image product contains variations due to the received signal and not the detector.
Single Image Reduction raw images bad pixel mask master flatfield Flatfield Correction
Bad Pixel Correction
Object Masking
NIR Science Reduction Pipeline
Sky Modelling & Subtraction reduced
single images
Object Mask Creation Image Summation
Object Search & Characterization Cosmics Removal object mask first reduction second reduction st ar t second re duct io n lo op Image Alignment Stacking & Weighting
stacked sum image optimally weighted sum image
3. bad pixel mask 2. flatfield
1. raw 4. reduced single
6. object mask 7. masked image 8. unbiased sky
11. final sum
10. cosmics 12. weighted sum 5. first pass sum
9. final single
Figure 7.5: Near-infrared science reduction steps. The top panel shows the full 15.40×15.40 FoV for a raw H-band image (left) and the final reduced sum image containing 75 input frames. Note that about 3000are lost at each field edge due to the dithering (right). The blue squares indicate the 20×20 zoom field for the small panels centered on the galaxy cluster RX J0018.2+1617 at z= 0.55. The small cutouts with numbers 1-12 illustrate the reduction steps of Fig. 7.4 by showing intermediate data products.
As a second step, a bad pixel correction is applied. The OMEGA2000 detector array contains about 0.5% pixels which are either not functioning at all (dead pixels) or exhibit significant deviations from a time-linear signal response (hot or flickering pixels). These pixels can be identified through a linearity analysis and are flagged in the bad pixel mask (subpanel 3) of the calibration directory. The values of these flagged bad pixels in the flatfielded image are then replaced by the interpolated pixel value of the four closest good pixels.
Note that near-infrared measurements are usually obtained by reading out the detector array at the beginning and end of the integration time and subtracting the two to determine the signal. Hence this double-correlated readout scheme does not contain abiasoffset as in CCD data. The detector dark current, i.e. the measured pixel values without an external signal, is in principle an additional (small) contribution to be corrected. However, tests have shown (H.-J. R¨oser, private communication) that thedark currentis not stable enough to be accurately modelled and subtracted since it depends sensitively on the exact detector temperature and its thermal history. Instead of an explicit subtraction, thedark current is treated as part of the overall background and implicitly corrected with the sky background subtraction.
The most critical and important part in the near-infrared reduction process is the sky background modelling. The dominating background components are (i) atmospheric air- glow (see Fig. 8.1) and (ii) thermal emission of the ambient structures at long wavelengths. Smaller contributions arise from (iii) scattered moonlight, (iv) Zodiacal scattered light (Cox, 2000), and (v) the uncorrected instrumental dark current. As discussed in Sect. 7.1, the science objects of interest can be easily a factor of 100–1 000 fainter than the back- ground surface brightness. Consequently this implies that the local background around the astronomical objects has to be known with uncertainties of <10−3. The dominating
airglow emission exhibits significant temporal changes on time scales of several minutes and spatial variations on scales of a few arcminutes. Hence the sky modelling requires a temporally and spatially local background estimation derived directly from science data. The pre-requisites to an accurate sky modelling have been implemented in the observing strategy through (i) a sufficiently short telescope pointing time ttel that allows tracing the
temporal variations and (ii) well-chosen dither offsets that enable a reconstruction of the local background without being biased by the flux of the astronomical objects of interest. The following recipe yields an accurate but preliminary estimate of the sky background. (i) Take the image together with the three11 preceding and following frames,i.e. the back-
ground is determined from a total of seven frames taken within a period of ±3–5 minutes. (ii) For every detector pixel (image coordinates) select the corresponding pixel values of all seven frames. Due to the dither strategy, which places the real objects at different detector positions for each image, most pixel values will only contain a background signal, whereas a minority will additionally have an object signal component12. (iii) The median
11Optimized pipeline parameters for the XDCP NIR data reduction are cited for this discussion. How-
ever, all critical parameters of the software can be specified and adjusted to the actual science needs. The background for the first (last) images of an observation series are estimated from the first (last) 7 frames.
of these seven values is taken as a good initial guess of the actual background level for the specified pixel. However, any object flux in one or more of the seven values systematically biases the median to a slightly higher level, i.e.the background is overestimated due to the additional non-background component in some pixels. (iv) This background bias can be corrected in next order by applying aκσ-clipping algorithm to the data values. A Poisson estimate of the local background standard deviation σ is obtained from the initial back- ground estimate, the gain13 of the detector array, and the local quantum efficiency of the
detector pixel14. Any pixel values deviating more than three standard deviations from the
initial background estimate are clipped, and the median is recomputed from the remaining object-cleaned values. (v) The κσ-clipped median value is taken as the background esti- mate for the specified pixel. The procedure is repeated for all detector pixels resulting in a full sky frame for the input science image. (vi) The modelled background sky is subtracted from the science image and replaced by a constant representing the average sky value in order to preserve the proper counting statistics.
The flatfielded, bad pixel corrected, and sky subtracted frame is written to disk as the final result of the first single image reduction loop (subpanel 4). However, the applied κσ-clipping for the first order correction of the local sky value could only eliminate object flux which has been detected at more than three standard deviations above the normal background level in the single frame, i.e. for fairly bright sources. The bulk of fainter sources and the wings of brighter objects below this clipping threshold will still bias the local background model to slightly higher values. The subtracted sky might thus still be overestimated resulting in a systematically lower flux for an object at the given position, which can reach significant levels for faint galaxies (see Sect. 7.3.3). This lost flux can be largely recovered with a second iterated background modelling loop as discussed below. Image summation
The reduced single image contains the data of integration timettel,i.e.typically 60 seconds
or less. In order to reach the required depth, the single images have to be stacked to build up a deep sum image with the full on-target exposure time texp. To achieve this, the dither
offsets have to be reversed and the images aligned in world coordinates, i.e. the physically same objects in the individual frames have to end up on top of each other. During this stacking procedure cosmic ray events, or short cosmics, can be identified and removed.
The alignment and stacking procedure uses the following approach: (i) The first input image is declared as master frame onto which all others are positioned and co-added. (ii) The reduced single master image is searched for typically 50–100 reference stars, whose centers-of-mass, as derived from the stellar flux distribution, are recorded. (iii) From the
objects larger than the dither offsets of 2000–3000the assumption is not fulfilled, but the introduced initial
error is corrected during the second iteration.
13The gain of a detector is the conversion factor between digital count units and physically detected
electrons via the photoelectric effect.
14In practice, an additional normalization factor has to be considered which is necessary to standardize
image header of the next input frame, the expected offsets with respect to the reference master frame are computed. Using these approximate offsets, the catalogued reference stars are searched for in the vicinity of the expected detector positions. Once matched, the exact object centers are determined resulting in an improved offset determination. (iv) The final image offset is determined from the outlier-clipped average of all matched stars. Since the OMEGA2000 optics exhibits negligible image distortions over the whole FoV, the frames can be co-added at this stage by simple XY-coordinate shifting to the nearest integer pixel offsets.
The single images still contain the unwanted cosmics which are to be corrected for without modifying the flux of the real astronomical objects. Cosmic ray events, caused by charged particles going through the detector, can be identified from their characteristic signature of single or few isolated pixels with increased count levels without a correspond- ing counterpart in the next aligned image. In principle, the cosmics would be removed by a simple median process over a sufficient number of pixel-aligned images, similar to the procedure for the background determination. The complication arises that the median procedure efficiently removes everything that is not present in at least half of the images, including the outer wings of real astronomical objects in variable seeing conditions, and the central flux peak of objects. The peaks can exhibit large gradients towards the neigh- boring pixels and are thus likely damped when medianing over the different object flux distributions in each frame. This effect can decrease the total flux of stellar sources by about 20%, which is avoided by applying the following procedure: (v) The median image of eleven world-coordinate aligned input frames is determined and used as object reference image. This intermediate sum image is now devoid of cosmics and contains the real ob- jects at the proper locations but with biased flux levels. (vi) Each individual frame is now checked pixel-by-pixel whether the value is more than five standard deviations above the corresponding median reference value, where the standard deviation is determined in anal- ogy to the sky modelling procedure. (vii) If a cosmics candidate is detected, the reference median image is searched for an astronomical object within a radius of three pixels around the outlier position. In case a source is found, the initial pixel value is classified as object flux and co-added to the master image as is. If no source is matched, the outlier value is classified as cosmic and replaced by the median value of the reference image, which is in turn co-added to the master sum image. (viii) This procedure is repeated until all input images have been aligned, cosmics cleaned, and co-added to yield a final deep sum image (subpanel 5). The cosmic ray events, as shown in subpanel 10 of Fig. 7.5, have now been removed from the field without influencing the total flux of the astronomical sources.
The last two subsections have provided a summary of the full first data reduction loop, which is in principle equivalent to the quicklook reduction system available at the telescope. The next two subsections will discuss further improvements towards the goal of the best achievable science-grade data reduction quality.
Iterated background subtraction
The complete science-grade reduction process requires two additional main components. Firstly, the creation of an object mask with registered positions of all sources. Secondly, an additional iterated loop through the reduction procedure but now with removed source fluxes for an unbiased second order sky background correction.
The object mask is created from the final deep sum image of the first reduction pass (subpanel 5) which contains the best available information on the location and size of the astronomical objects at this processing stage. The objective of the object mask is to cover all sky areas in the FoV which contain a detectable source signal and use only the pixels with an unbiased background value for the reconstruction of the sky model. In practice the mask is created using an iterative κσ-clipping algorithm applied to the smoothed deep sum image. An initial smoothing is helpful for the identification of faint object halos, i.e. the PSF wings for point sources are extended halos for galaxies. Pixel values deviating more than about two standard deviations from the median background level are classified as object pixels and flagged in the mask frame. This procedure is iterated to push the depth of the mask to sufficiently faint levels. The final object maskis shown in subpanel 6, which now distinguishes the black regions with background pixel values and the masked white areas with object flux contributions.
With the object mask in hand, the second reduction loop can start from the very beginning (see Fig. 7.4). The initial calibration steps flatfielding and bad pixel correction are applied as before. The sky modelling procedure on the other hand is extended by two additional steps. (i) Theobject mask is projected onto each individual input frame. To do this projection properly, the exact image dithering offsets as determined during the first image summation loop are used to shift the object mask to the correct world-coordinate aligned location in each input image. (ii) The pixel values of the masked object regions are replaced15 by the unbiased median background level of the 100 nearest unmasked values.
This replacement scheme is designed to yield robust results for arbitrary large objects and to take into account local background variations across the FoV. The masked input image (subpanel 7) with all object flux replaced is then passed on to the sky modelling procedure. The final unbiased sky model (subpanel 8) is now the result of a second order corrected flux- removed and κσ-clipped background determination. Subtraction from the original input image yields the final iterated reduced single image (subpanel 9), and after the subsequent summation and cosmics removal (subpanel 10) of all frames the final co-added deep sum image is available after the second processing loop (subpanel 11).
Advanced optimizations
The final sum image is a well-reduced and science-ready data product. However, with respect to the objective of reaching the maximum possible depth, two additional optimiza- tion schemes can be applied. Firstly,fractional pixel offsetsinstead of integer values can be implemented for the stacking procedure, and secondly,optimal weighting of the individual input frames can improve the final signal-to-noise ratio.
The discussed image summation procedure determined the accurate XY-offsets of the frames with respect to the master frame, but then rounded the exact values to the nearest integer pixel offsets in order to match the underlying pixel grid. This way, systematic 15Two additional replacement schemes with lower performance expectations are implemented. 1) The
masked pixel is ignored for the sky determination, which leads to difficulties for very large objects. 2) The masked pixel is replaced by the global median, which does not take into account local variations.
truncation errors of up to half a pixel in each spatial direction are unavoidable. For the OMEGA2000 pixel scale of 0.4500/pixel this implies systematic XY-shifting errors of 0–0.22500 per direction, which can add up in quadrature to a maximum systematic dis- placement error of 0.3200 for a stacked frame. The integer pixel shifting has thus two consequences, both gaining importance in good seeing conditions. (i) The stacked object flux is not as compact as defined by the seeing limit and (ii) the 2-dimensional flux dis- tribution of point sources does not follow its natural radially symmetric Gaussian shape. This latter tendency of the object cores to become boxy or square-like is easily visible for the fainter objects in the left panels of Fig. 7.6. The shape improvements after applying the fractional pixel offsetscheme can be seen by eye in the right panels of the same figure. In order to implement fractional offsets onto the underlying fixed detector grid of 2 048×2 048 pixels, the flux values in each pixel have to be distributed over four cells of the master grid in a flux conserved way. The flux distribution matrix is calculated once for each frame from the exact offsets according to the fractional geometrical overlap of a pixel with the underlying master grid. The pixel values of the frame to be co-added are then distributed using this pre-determined matrix. In turn, each pixel of the master grid, receives fractional flux contributions from four adjacent pixel values of the single image.
The question whether a faint object will be detected depends primarily on the signal- to-noise ratio of the source (see Equ. 7.1), i.e. on its contrast with respect to the noise