Minimizing Algorithm

Initialization

Choosing the right initialization for our algorithm is important. In Section 4.3.4 we described how trying to rerun our VSR algorithm from [54] failed as the rather smooth inputs where considered optimized and left us stuck in a local minimum. Bilinear interpolation produces very smooth results and using it as initialization would also leave us caught in a local minima. Thus we use the projection in (4.12)

R = mn

M NRh⊗ Rv

to initialize both flows and intensities to get unblurred (but also jagged) initial- izations. We have tried (with our non-simultaneous VSR algorithm from [54]) to initialize the intensities with bilinear interpolation instead, and as predicted we got stuck in a local minimum with outputs so smooth that they were hardly different from their inputs. The fact that our variational method is satisfied with blurred regions is why motion blur will also stay untouched doing VSR. Thus we do not change the artistic work of the film makers. Algorithms using structure tensor regularization also cannot de-blur already (too) blurred regions as it is pointed out by Roussos and Maragos in [86].

If we start with bilinear interpolation of the HR flows using the non-simulta- neous VSR algorithm from [54] it does not change the intensity output, which is a clear indication that an accurate and reliable HR flow is necessary to get optimal VSR results. The LR flow that we initialize with R is computed on the input LR sequence using a full multiresolution scheme with the same type of fixed point approach as described below for the intensities. It is the energy in

(4.9) that is minimized, but with the modification that u is replaced by the low resolution sequence u0 such that x only runs over the LR domain. To calculate

the backward LR and HR flows on the sequence, the energy in (4.9) is minimized on a time reversed ({1, 2, 3..., n} 7→ {n, n − 1, n − 2, ..., 1}) version of u0 and u

respectively.7

Core Algorithm

To efficiently minimize the energy in (4.10) we handle the nonlinearity of its Euler-Lagrange equation (4.11) by using a fixed point approach. First we define

A(u) = 2ψ0_(|∇u|2_{) and B(u) = 2ψ}0_(|£ ~

Vu|2), which represents the nonlinear

parts of (4.11). We only update A(u) and B(u) in a number of outer fixed point iterations in order to have a fully linear approximation of (4.11). For each outer fixed point iteration, we run a number of inner iterations on the now linearized system using a Gauss-Seidel relaxation scheme with the super resolution constraint incorporated: We back-project the energy update in each inner iteration modulated by a positive weight, α, in order to respect intensity bounds. A similar solver is used to minimize the energy of the flows in (4.9). The overall sketch of the algorithm is:

• Let u0_{be the initialization of the HR sequence, and} • let ~v0_{be the initialization of the HR flow.}

• For i = 0 until I − 1

1. Run L outer fixed point iterations each with M inner relaxation iterations of the flow solver.

2. Run J outer fixed point iterations each with K inner relaxation iter- ations of the intensity solver.

3. In each solver always use the latest updated version of the other component (u for ~v-solver and the other way around).

• Output the final HR sequence u, and if needed, the final flow ~v.

The intensity solver used above is given next. Denoting the null space of R by

T as in Sec. 4.3.5 and by PT the orthogonal projection onto T as in (4.14), we

have:

• Let u0_{be the initial guess for the HR sequence.} • For j = 0 until J − 1

1. Let Aj _{:= A(u}j_{), B}j_{:= B(u}j_{), u}j,0_{:= u}j_.

2. for k = 0 until K − 1

(a) From uj,k_{, compute ¯u}j,k+1_{by one Gauss Seidel sweep on G(A}j_{, B}j_).

(b) Form the update vector ¯δj,k_{:= u}j,k_{− ¯u}j,k+1_.

(c) Project it onto T , δj,k_{:= P} T(¯δj,k).

7_{The multiresolution scheme in computing the flow employs a down- and upsampling}

scheme described in [11] and is the same as using R to scale from level to level with rounding to the nearest integer frame size as typical scale factors give non-integer frame sizes.

(d) Update current estimate uj,k+1_{:= u}j,k_{− αδ}j,k_.

3. uj+1_{:= u}j,K_.

• Output the HR sequence uJ_.

Leaving out the projection, the flow solver runs similarly. Projection Details

As described in Sec. 4.3.5 the projection allows processing in blocks, e.g. in blocks of 5 × 16 = 80 pixels in the SD to 720p HD case mentioned, so for any

M × N block step 2(c-d) above is uj,k+1_l = uj,k_l − α Ã ¯ δ_lj,k+ M N_X i=1 PT_{(l, i)¯δ}j,k i ! , l = 1, ..., M N (4.15) due to the dependance of the update of all other pixels in the support area. A simple example: In the 2x2 SR case, if the suggested update for pixel 1, ¯δj,k₁ = 4 and ¯δ_lj,k= 0 for the remaining pixels in the current block (l = 2, 3, 4), equation (4.15) tells us to add one to all four pixels and subtract four from pixel 1. This is the normalization incorporated in PT _{as dictated by the super resolution}

constraint.

The variational (diffusion) part of our algorithm follows the minimum-maxi- mum principle (see for instance [80]) and keeps intensity values within bounds, but we need the weight α in (4.15) to ensure that the projection does not create values out of bounds as it does not follow the minimum-maximum principle. If a value in a block, B, is detected out of bounds, α is recursively lowered from 1 to 0 in steps of 0.1, and all values in the block B recalculated with the new α, stopping when all values in B are within bounds. This can potentially stop the evolution of the sequence, but it was found not to do so in practice. The problem of out of bound intensities was limited to very few regions around extremely high contrast edges and thus the computational overhead by enforcing bounds was also negligible.