The natures of dark matter and dark energy are still unknown to us today. New and up- coming surveys designed to study these phenomena will characterize very large numbers of objects; for instance, the Large Synoptic Survey Telescope (LSST;Ivezic et al.(2009)) plans to observe billions of galaxies over almost half the sky. We can use the redshift of an object as a proxy for its distance or lookback time; we determine redshifts by evaluating the light we receive from a galaxy.
One way to estimate the redshift of a galaxy is by analyzing a detailed spectrum, which in general yields very accurate results. Unfortunately, the acquisition of galaxy spectra of galaxies is not an easy task, since for faint objects such as those studied by LSST very long exposure times are required to obtain an adequate signal-to-noise ratio, while the number of objects that are targeted during a given observation is limited by technical challenges. As a result it will be practically impossible to obtain spectroscopic redshifts (also referred to as spec-z’s) for the great majority of objects studied by LSST and other deep imaging surveys. Alternatively, distant galaxies can be characterized using broad-band photometry, which provides measurements of the fluxes through a particular filter for all objects in the field of view of an instrument. With this method, many objects can be observed at the same time, and good signal-to-noise is achievable even for very faint objects, making it the only feasible
The downside to this is the much smaller amount of information provided by broad-band imaging. As a result, inferences about the redshifts of galaxies from their photometry – commonly known as photometric redshifts or photo-z’s – are considerably less precise than their spectroscopic counterparts.
It is common for photometric redshift algorithms to estimate probability density functions (PDFs) for the photo-z of an object, since PDFs provide considerably more information than a single “point” estimate of redshift or than a point estimate plus assumed-Gaussian errors. PDFs and point estimates are closely related; it is common to estimate point values directly from photo-z PDFs. For instance, one could use the redshift at which the PDF has its greatest value (zpeak) or the first moment of the PDF (i.e., the PDF-weighted mean
of redshift, zweight; in some cases this may be calculated only using the highest peak of
the probability distribution) as a single estimate of the photometric redshift. When either of these point values are compared to independent spectroscopic redshift samples, one generally finds substantial scatter, as well as a significant fraction of “outlier” redshifts which are far from the estimated photo-z. It is not uncommon for photo-z estimates to be biased on average when compared to spectroscopic z’s.
Just as point estimates can have imperfections, the PDFs provided by existing photo- metric redshift codes have proven in the past to be unreliable, not meeting the statistical definition of a probability density function (Fern´andez-Soto et al. (2002),Hildebrandt et al. (2008),Dahlen et al.(2013), etc.). One way this may be seen is by investigating uncertainty estimates derived from PDF measurements. If credible intervals (the Bayesian equivalent of confidence intervals) are properly constructed, then 68% of spectroscopic redshifts should lie within the 68% credible intervals of the corresponding photo-z PDFs, 95% of spectroscopic redshifts should lie within the 95% credible intervals, etc. The reality can be far from this scenario; in recent tests, the number of spectroscopic redshifts within a given credible region may be much higher or much lower than what would be expected if PDFs have been properly constructed, depending on the particular photo-z code and the size of the credible interval considered (Dahlen et al. (2013)).
In this paper, we present simple methods which can help to correct for low-order deficien- cies in the probability density functions output by photometric redshift codes, resulting in
better estimates of both point values and credible intervals. This is done using the Quantile- Quantile plot (also known as the Q-Q plot; Wilk & Gnanadesikan(1968)) constructed with a subsample of objects with spectroscopic redshift measurements as well as the corresponding photo-z PDFs of the same objects.
We take advantage of the fact that, if the photo-z PDFs fulfill the standard statistical definition of a probability distribution, then the values of the cumulative distribution func- tions (CDFs) constructed from the PDFs and evaluated at the actual spec-z’s of the objects should follow a uniform distribution from zero to one. When this is not the case, it is an indicator that the photo-z PDFs are imperfect. In this paper, we consider the impacts on the Q-Q plot of a bias in the PDFs (shifting them from the true PDF); an underestimate or overestimate of errors leading to PDFs that are too narrow or too broad; an inaccurate level of asymmetry in the PDFs (i.e., inaccurate skewness); or finally, cases where the tails of the PDFs are too large or too small (i.e., inaccurate kurtosis).
In a recent paper published while this paper was being written, Freeman et al. (2017) used Q-Q plots to investigate the impact of differences between the distribution of properties of objects with spectroscopic redshifts used to train photo-z algorithms and the properties of the objects to which the algorithms are applied, as well as to evaluate their methods for mitigating this effect. In this paper, we consider the utility of the Q-Q plot as a general tool for assessing photo-z PDF accuracy, as well as methods for optimizing PDFs using statistics based on these plots.
In Section 2.2 we explain in detail the methods we use to assess the quality of photo-z PDFs, and present Q-Q results for a variety of simple cases to illustrate the information available. In Section 2.3 we describe how information from Q-Q statistics can be used to calibrate photo-z PDFs to more closely fulfill the statistical definition, and illustrate our methods with mock data. Finally, in Section 5.2 we summarize and discuss the findings of this study.