2.11 Major concepts of the software package
2.11.2 Operating environment
In order to both have a portable and robust set of tools, one has to build a software package on the top of widely standardized and documented environment. The most widespread and approved standard is the “Portable Operating System Interface” or POSIX28, that intended
to standardize almost all layers of the operating system, from the system-level application program interfaces (APIs, such as file manipulation or network access) up to the highest level of programs such as shell environments, related scripting languages and other basic utilities.
The actual development of the package fi/fihat was done under GNU29/Linux30 sys- 28http://en.wikipedia.org/wiki/POSIX
29http://www.gnu.org/ 30http://www.kernel.org/
2.11. MAJOR CONCEPTS OF THE SOFTWARE PACKAGE
Table 2.4: An overview of data files used to store information required by the image reduction process or created during the reduction. Each file type is referred by its extension.
File size (→) O(1) ∝ Nframe ∝ Nobject
# of files(↓)
O(1) *.config: generic informa- tion about the whole reduc- tion and observational con- ditions (name and coordi- nates of the target field, in- volved reduction algorithms and their fine-tune parame- ters)
*.list: list of frames to be processed during the reduc- tion
*.stat: basic statistics for each frame (both image statistics such as number of detected objects and infor- mation on the observational circumstances, e.g. zenith distance, airmass, elevation of the Moon, stellar profile FWHM)
*.cat: list of objects and some catalogue information that is used during the re- duction
*.lcstat: light curve statistics (also known as “magnitude-rms” statis- tics)
∝ Nframe *.trans: astrometric so-
lution (the transformation that maps the reference cat- alogue to the coordinate system of the image) *.kernel: kernel solution (the convolution function used in the image subtrac- tion process)
*-psf.fits: best fit point- spread function for a given image
@ @
*.fits: calibrated images1
*-sub.fits: convolved and subtracted images1
*.stars: list of detected sources and their proper- ties (coordinates, shape pa- rameters, brightness esti- mation)
*.phot: instrumental pho- tometric measurements
∝ Nobject *.xmmc: best fit and Monte-
Carlo distribution of the pa- rameters of the light curve model function (if the ob- ject is turned out to be in- teresting)
*.info: summary informa- tion of the planetary, or- bital and stellar data for the actual object (if the object is indeed a planet-harboring star)
*.lc: light curves
*.epdlc: de-trended light curves involving only the External Parameter Decor- relation algorithm
*.tfalc: de-trended light curves involving the Trend Filter Algorithm
@ @
1Strictly speaking, the size of these files does not depend on the number of objects that are extracted from the image and/or
tems, that is one of the most frequently used POSIX compliant, UNIX-like31 free operating
system. The main code was written in ANSI C (featured with some GNU extensions) and intended to be compiled without any difficulties on various other UNIX systems such as SunOS/Sparc and Mac OSX. The compilation of the package does not require additional packages or libraries, only the GNU C Compiler (gcc32), its standard library (glibc33), the
associated standard header files and some related development utilities. (Such as make34 or
the ar35 object archived. In almost all of the systems these come with gcc as its depen-
dencies) Therefore, all of the requirements of the package include only free and open source software (F/OSS).
In practice, to have a complete data reduction environment the users of the package might have to use additional text processing utilities such as an implementation of the AWK programming language (for instance, gawk36, that is included in all of the free GNU/Linux
systems) and basic text processing utilities (such as paste, cat, sort, split, included in the textutils/coreutils37 GNU package). And finally, for visualization purposes, the
SAOImage/DS9 utility38 (Joye & Mandel, 2003) is highly recommended.
2.12
Implementation
In this subsection I summarize the standalone programs that are implemented as distinct binary executables. The programs can be divided into two well separated groups with respect to the main purposes. In the first group there are the programs that manipulate the (astronomical) images themselves, i.e. read an image, generate one or do a specific transformation on an image. In the second group, there are the programs that manipulate textual data, mostly numerical data presented in a tabulated form.
Generally, all of these programs are capable to the following.
• The codes give release and version information as well as the invocation can be logged on demand. The version information can be reported by a single call of the binary, moreover it is logged along with the invocation arguments in the form of special FITS keywords (if the main output of the actual code is a processed FITS image) and in the form of textual comments (if the main output of the code is text data). Preserving
31http://en.wikipedia.org/wiki/Unix-like 32http://gcc.gnu.org/ 33http://www.gnu.org/software/libc/ 34http://www.gnu.org/software/make/ 35http://www.gnu.org/software/binutils/ 36http://www.gnu.org/software/gawk/ 37http://www.gnu.org/software/coreutils/ 38http://hea-www.harvard.edu/RD/ds9/
2.12. IMPLEMENTATION
Table 2.5: An overview of the standalone binary programs included in the package, displaying their main purposes and the types of input and output data.
Program Main purpose Type of input Type of output
fiarith Evaluates arithmetic expressions on images as operands.
A set of FITS images. A single FITS image.
ficalib Performs various calibration steps on the input images.
A set of raw FITS images. A set of calibrated FITS image. ficombine Combines (most frequently averages) a
set of images.
A set of FITS images. A single FITS image.
ficonv Obtains an optimal convolution trans- formation between two images or use an existing convolution transformation to convolve an image.
Two FITS images or a single im- age and a transformation.
A convolution transformation or a single image.
fiheader Manipulates, i.e. reads, sets, alters or removes some FITS header keywords and/or their values.
A single FITS image (alterna- tion) or more FITS images (if header contents are just read).
A FITS image with altered header or a series of key- words/values from the headers. fiign Performs low-level manipulations on
masks associated to FITS images.
A single FITS image (with some optional mask).
A single FITS image (with an altered mask).
fiinfo Gives some information about the
FITS image in a human-readable form or creates image stamps in a conven- tional format.
A single FITS image. Basic information or PNM im-
ages.
fiphot Performs photometry on normal, con- volved or subtracted images.
A single FITS image (with addi- tional reference photometric in- formation if the image is a sub- tracted one).
Instrumental photometric data.
firandom Generates artificial object lists and/or artificial (astronomical) images.
List of sources to be drawn to the image or an arithmetic ex- pression that describes how the list of sources is to be created.
List of sources and/or a single FITS image.
fistar Detects and characterizes point-like sources from astronomical images.
A single FITS image. List of detected sources and an optional PSF image (in FITS format).
fitrans Performs generic geometric (spatial) transformations on the input image.
A single FITS image. A single, transformed FITS im-
age. fi[un]zip Compresses and decompresses primary
FITS images.
A single uncompressed or com- pressed FITS image file.
A single compressed or uncom- pressed FITS image file. grcollect Performs data transposition on the in-
put tabulated data or do some sort of statistics on the input data.
A set of files containing tabu- lated data.
A set of files containing the transposed tabulated data or a single file for the statistics, also in a tabulated form.
grmatch Matches lines read from two input files of tabulated data, using various crite- ria (point matching, coordinate match- ing or identifier matching).
Two files containing tabulated data (that must be two point sets in the case of point or co- ordinate matching).
One file containing the matched lines and in the case of point matching, an additional file that describes the best fit geometric transformation between the two point sets.
grselect Selects lines from tabulated data using various criteria.
A single file containing tabu- lated data.
The filtered rows from the input data.
grtrans Transforms a single coordinate list or derives a best-fit transformation be- tween two coordinate lists.
A single file containing a coordi- nate list and a file that describes the transformation or two files, each one is containing a coordi- nate list.
A file with the transformed co- ordinate list in tabulated from or a file that contains the best- fit transformation.
lfit General purpose arithmetic evalua-
tion, regression and data analysis tool.
Files containing data to be ana- lyzed in a tabulated form.
Regression parameters or results of the arithmetic evaluation.
the version information along with the invocation arguments makes any kind of output easily reproducible.
• All of the codes are capable to read their data to be processed from the standard input and write the output data to the standard output. Since many of these programs manipulate relatively large amount of data, the number of unnecessary hard disk op- erations should be reduced as small as possible. Moreover, in many cases the output of one of the programs is the input of the another one. Pipes, available in all of the modern UNIX-like operating systems, are basically designed to perform such bindings between the output and input of two programs. Therefore, such a capability of redi- recting the input/output data streams significantly reduce the overhead of background storage operations.
• The programs that deal with symbolic operations and functions, a general back-end library39 is provided to make a user-friendly interface to specify arithmetic expressions.
This kind of approach in software systems is barely used, since such a symbolic spec- ification of arithmetic expressions does not provide a standalone language. However, it allows an easy and transparent way for arbitrary operations, and turned out to be very efficient in higher level data reduction scripts.
• The programs that manipulate FITS images are capable to handle files with multiple extensions. The FITS standard allows the user to store multiple individual images, as well as (ASCII or binary) tabulated data in a single file. The control software of some detectors produces images that are stored in this extended format, for example, such detectors where the charges from the CCD chip are read out in multiple directions (therefore the camera electronics utilizes more than one amplifier and A/D converter, thus yield different bias and noise levels). Other kind of detectors (which acquire individual images with a very short exposure time) might store the data in the three dimensional format called “data cube”. The developed codes are also capable to handle such data, therefore it is possible to do reductions on images obtained by the Spitzer Space Telescope, that optionally uses such data structures for image storage.
The list of standalone binaries and their main purposes that come with the package are shown in Table 2.5.