2.2 Camera Response Function
2.2.4 Robertson’s Method
In Section2.2.3, the response function of the camera was estimated using an average of all input pixels to determine the high dynamic range light values. However, since observed pixel values arise from quantisation of the product of the exposure time and the irradiance present, the quantisation error for the light will be lower for higher exposure times. Robertson et al. improved radiometric self-calibration by assigning higher weights to pixel values taken at higher exposures [104].
Consider N aligned exposures of a static scene, with known exposure times, ti
(i = 1, . . . , N ). The set of known observations for the jth pixel of the ith exposed image is denoted yij (i.e. pixel values {0 − 255}). The objective is to determine the
underlying light values or irradiances, denoted byxj, that give rise to the observations
yij. Since only the exposure time is being varied, the amount of light contributing
to the output value yij will be tixj. To account for image capture noise, an image
values. The quantity tixj + Nijc is then mapped by the camera’s response function
f (·), to give the output values (Equation 2.18):
yij = f (tixj+ Nijc) (2.18) Sinceyijare digital numbers,f (·)maps the positive real numbers, <+ = [0, ∞), to an
interval of integers, O = 0, . . . , 255, for 8-bit data. The observations (or irradiances)
xj are in the range space O of the response function f (·), and the desired result
belongs to the domain of f (·) (which is in the range <+). If the function f (·) is known, a mapping from O to <+ can be defined by Equation 2.19.
f−1(yij) = tixj + Nijc + N q ij = tixj + Nij = Iyij (2.19) Nij = Nijc + N q
ij is the overall noise term consisting of image capture noise (Nc
ij) and image quantisation error (N q ij).
The noise quantisation error term (Nijq) is introduced from assigning f−1(yij) = I(yij). This introduces an error from mapping the interval of integers, O = 0, . . . , 255, to Iyij, which is now a discrete series of positive real numbers, <
+ = [0, ∞). The
Nij terms are modelled as zero-mean independent Gaussian random variables, with
variances, σij2. For convenience the variances are replaced with weights, wij = 1/σij2. Weights are chosen based on the confidence that the observed data is accurate. The response function of a camera will typically be steepest, or most sensitive, towards the middle of its output range (i.e. pixel values of 128 for 8-bit data). As the output levels approach the extremes (0 and 255), the sensitivity of the camera typically decreases. For this reason, the weighting function is chosen such that values near 128 are weighted more heavily than those near 0 and 255 (Equation 2.20).
wij = wij(yij) = exp −4(yij − 127.5) 2 127.52 (2.20) The function chosen in Equation2.20is a Gaussian function that has been scaled and shifted so that wij(0) = wij(255) = 0 and wij(127.5) = 1.0. This choice of weighting function implies very low confidence in the accuracy of pixel values near 0 and 255, and high confidence in pixel values near 128.
From Equation2.19,Iyij are independent Gaussian random variables, and the joint probability density function is given by Equation 2.21.
P (Iy) ∝ exp −X i,j wij(Iyij − tixj) 2 (2.21) Here, a maximum-likelihood approach is taken to find the high dynamic range image values. The maximum-likelihood solution finds the xj values which maximise the
probability in Equation 2.21. Maximising Equation 2.21 is equivalent to minimising the negative of its natural logarithm, which leads to the following objective function (Equation 2.22):
O(x) =X i,j
wij(Iyij − tixj)
2 (2.22)
Equation 2.22 is easily minimised by setting the gradient to zero (∇O(x) = 0). This yields the desired high dynamic range image estimate (Equation2.23).
ˆ xj = P i wijtiIyij P i wijt2i (2.23)
Images taken with longer exposure times are weighted more heavily, as indicated by the ti term in the numerator of Equation 2.23. However, Equation 2.23 requires
that the Iyij values (i.e. the response function) be known. In general, the response function is not known, but it can be recovered by an initial calibration step. If using the same camera in the future, the calibration step is unnecessary, and Equation2.23
may be applied directly.
Objective function (from Equation2.22) will be used to determine theIyij values re- quired to define the response function. To estimate theIyij values from Equation2.22, thexj values are required, but these are also unknown. Therefore both values must be
estimated simultaneously. The objective function for the case of an unknown response function is given by Equation2.24.
O(I, x) =X i,j
wij(Iyij − tixj)
2 (2.24)
An additional constraint on the response function (f (·)) is required when estimating
Iyij and xj together using Equation2.24. TheHDRimage estimates, ˆxj, are arbitrary in scale. To be viewed in 8-bit display ˆxj needs to be mapped to a usable range (0, . . . , 255). Since the scale of ˆxj is directly dependent on the scale of Iyij, the estimates for Iyij are constrained such that ˆI128 = 1.0. This is enforced by dividing each ˆI by ˆI128.
A form of Gauss-Seidel relaxation is used to determine the response function. Seidel relaxation minimises the objective function with respect to a single variable, then uses these new values when minimising with respect to subsequent variables. Here, Equa- tion 2.24 is minimised with respect to each Iyij first. Then the restriction mentioned previously (I128 = 1) is enforced. Finally, Equation 2.24 is minimised with respect to eachxj. This constitutes one iteration of the algorithm.
The initial estimate for ˆI in the first iteration is chosen as a linear function2, with
ˆ
I128= 1.0. The initial estimate of ˆx for the first iteration calculated by Equation2.23,
2pfscalibration also allows an initial log estimate of ˆI which will converge faster for most camera
using the initial linear estimate of ˆI. Again, to minimise the objective function, the partial derivative of Equation 2.24 with respect to Iyij is taken and set equal to zero (Equation 2.25). ˆ Im= 1 |Em| X (i,j)Em tixj (2.25)
Em = {(i, j); yij = m}, is the set of indices where index
m was observed for the input images. |Em| is the cardi- nality of Em (i.e. the number of times m was observed).
After scaling the response function such that ˆI128 = 1.0, minimisation is performed with respect to each ˆxj. This merely involves using Equation 2.23 and completes one iteration of the algorithm. The process is repeated until convergence, where the rate of decrease in the objective function falls below some minimum threshold, ε
(Equation 2.26).
O(I, x)k− O(I, x)k−1 < ε (2.26) Where ε is a very small number, set depending on the desired convergence; k
is an integer corresponding to the kth iteration of the Gauss-Siedel relaxation. Just like Mitsunaga and Nayar’s method (Section 2.2.3), the technique is performed for each colour channel (RGB) separately. Figure2.2shows theCRFobtained through Robertson’s Method for the Nikon Coolpix used in this research.