How To Process Data From A Casu.Com Computer System

(1)

CASU

Processing: Overview and

Updates for the VVV Survey

Nicholas Walton

Eduardo Gonalez-Solares, Simon

Hodgkin, Mike Irwin

(2)

22 Mar 2012 Nic Walton - CASU Updates - VVV @ Vina del Mar 2

Pipeline Processing Summary

●

Data organization (check images, headers, rename

files, ingest into raw data archive)

●

Data processing (dark correction, linearity, flat field,

sky, destripe, stack jitters, cataloguing, astrometric and

photometric calibration)

●

Produces one stacked pawprint image for each OB (+

associated confidence map and catalogue)

●

Mosaic (tile) the stacked pawprint for each OB (+

associated confidence map and catalogue) [in

progress]

(3)

Processing Status: Mar 2012

(4)

VISTA Survey Progress

http://casu.ast.cam.ac.uk/vistasp/overview

(5)

(6)

VVV data volumes are significant

(7)

VISTA focal plane

Turning this …

(8)

… to this … e.g Orion

(9)

P

ip

el

in

e

O

ve

rv

ie

w

(10)

… generating Science Data Products

●

products consist of:

—

calibrated single exposure images

—

shifted ”average” stack frames (pawprints) + conf maps

—

calibrated stacked pawprint catalogues

—

filled area tile images + confidence maps

—

calibrated tile object catalogues

—

sky background images, flats, darks, bad pixel mask

●

science products are MEF

(multi-extension FITS)

files

(images Rice-compressed)

(11)

Image processing steps:

removing the instrumental signature

• Reset correction (debias - inline)

• Dark correction

• Linearity correction

• Flat field correction

• Sky background correction

• Destripe - controller level pickup

• Crosstalk, persistence and fringing corrections

are not necessary

(12)

12

VISTA data flow - I

●

raw data transfers on USB disk (Rice-compressed MEFs)

●

ingest & verification → raw data archive

●

create off-line tape backups

●

update calibration files monthly (flats, linearity, masks)

●

parallel nightly processing at OB-level (darks updated)

—

stacked pawprint images instrumental signature removed

—

catalogue generation from pawprint images & conf maps

—

astrometric & photometric calibration

●

header updates → pawprint OB-level science products

●

check derived QC info & sample of images

●

processing web page updates and progress tracking:

(13)

(14)

VISTA data flow - II

●

detector level monthly photometric zpt updates

—

illumination correction tables

●

mosaic OB-level tile image and create confidence map

●

tile image cataloguing (Tangent Plane projection)

—

nebulosity filter 6 component pawprints

—

mosaic and correct for sky levels and distortion

—

generate tile catalogue

—

grout tiles to fix PSF and detector zpt variations

●

check derived QC info & sample of images (cf. OB grade)

●

ingest to post-processing database enables checks:

—

FITS header contents, file size, provenance and calibration files,

exploration of long-term trends, survey progress, data access

http://casu.ast.cam.ac.uk/vistasp/imgquery/search

(15)

QC plots

summarise:

astrometry; seeing;

stellar ellipticity;

Survey progress

overview

(16)

Photometric ZP variation

Aluminised mirror

Silvered mirror

Mag ZP Monitoring

Cryostat window

cleaned

P85

P86

P87

P88

(17)

Tile Creation

1 tile = 6 stacked paws each with 16 detector level images

Tile

Confidence Map

Tiles and confdence maps

@ 1 GB ~ 100GB

(18)

Tile Generation & Corrections

http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/tiles

●

A number of subtleties in extracting the catalogues from the files

●

PSF variations across the 96 detectors in each time

●

PSF variations increased during poorer seeing periods

●

GROUTING

removes a number of these issues

●

For any object computes a photometric correction by reference to the

aperture correction at that object weighted by the confidence map

Example magnitude and colour-colour diagrams for

tiles before (left) and after (right) grouting

(19)

Astrometric Calibration

2MASS - VISTA

Linear solution

per detector

Tabulated systematics

from stacked residuals

WCS - ZPN projection

rms < 100 mas

(20)

Photometric Calibration

●

colour equations

—

convert 2MASS to instrumental system

●

2MASS s:n > 10 in J,H,Ks and

—

0 < (J-Ks) < 2 & 0 < (J-Ks)_o < 1 (extinction corr)

—

0 < (J-Ks) & (J-Ks)_o < 1 & (J-Ks) < 0 (update extcorr)

—

no restriction

●

NIR ~100-1000 “standards” per pointing

—

required to be stellar and unsaturated

●

Zpt + error per pointing; can compare with FS fields

●

monitor long term Zpt behaviour

—

average monthly detector Zpt offsets

—

illumination corrections

(21)

VIRCAM Stripes

(22)

Removing the (time varying)

background sky …

●

Tilesky – double pass combination of all observations in

tile(s)

●

Pawsky – single pass combination of all observations in a

pawprint with object masking iterated ‘dynamically’

●

Pawsky with object mask – as above, but the mask is defined

beforehand using e.g. deep stacked tiles

●

Offset sky – use a sky taken nearby (spatially & temporally)

●

Pawsky and “half” tilesky - minus ****

(23)

Before And After Background

Correction

(24)

(25)

(26)

Software modules

●

nebuliser

—

removes complex background variations

—

enhanced object detection & parameterisation

●

mosaicer

—

CASU tiling software developed for VISTA

●

grouter

—

applied to tile catalogues to remove the effect of PSF

variations and photometric throughput (+ MJD column)

●

psf’ers – under development …

—

automatically generates detector-level PSFs

—

and performs PSF photometry

(27)

Background Subtraction

(28)

(29)

Background Subtraction

(30)

Improved object detection

(31)

Issues with tiles

●

Imperfect sky subtraction pawprint matching : low

level discontinuous artifacts

●

Variable PSF across single pawprints : each detector

has different PSFs

●

Variable seeing conditions : each pawprint has

different PSFs

●

Variable extinction during the observations :

variation of ZP over tile

(32)

PSF in each detector

(33)

PSF fitting

(34)

(35)

Photometric Distortion over a Tile

(36)

CASU Software Release

(37)

CASU Software Release: imcore

(38)

Known Issues

http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/known-issues

●

Bad pixels in detector number 1

●

Point-like objects in flat and sky frames

●

Detector 16 variable QE

●

Bad channel in detector 6

●

Filter wheel out of position

●

Detector #4

●

Spurious detections around bright stars

●

Images with large ellipticities

●

Persistance

●

Tile images variable depth

(39)

CASU – new features under

development

●

Improved PSF Fitting

—

Tailored for high density fields, thus VVV

●

Improved handling of multi-epoch data

●

Improved data transfers over the EVALSO link

●

Implementation of CASU pipeline to run in 'quick

look' mode at Paranal

—

Generation of Reflex workflows for processing VIRCAM

data

(40)