Spatial upsampling

Spatial upsampling or upscaling is the process of taking a single static image and increasing it in resolution based on the data and metadata for that image.

Most scenes commonly rendered feature some degree of spatial coherence. That is, regions located close to each other are more likely to be the same colour. This has obvious real-world parallels, such as painted walls or coloured materials. Spatial coherence is thus an obvious pick for efficient rendering, as if we can guess the colour of a pixel reliably based on the pixels surrounding it we do not need to draw ever pixel ‘blind’.

Figure 2.1–k-nearest pixels (in this casek= 10) are within the read circle surrounding our blue pixel in the upscaled image.

Nearest-neighbour

Nearest-neighbour upsampling is a common, straightforward method of increasing an image’s resolution. Each pixel in the upsampled image is set according to the pixel it is closest to when the larger image is projected onto the smaller one. It gives a blocky result, as the aliasing in the smaller image is magnified.

k-means upscaling

Rather than taking the nearest estimate of the value of a pixel, alternative results can be gotten by instead taking a weighted average ofkpixels surrounding the one we wish to value, as shown in Figure 2.1. The method proposed by Shepard [34] weights the averages according to distance from the pixel being computed, as the accuracy of the pixel ought to decrease with distance.

Many other methods have been proposed of varying complexity, but they are often com- putationally intensive for the results they give.

Gaussian filtering

Gaussian filtering achieves a blur over the image by operating a gaussian kernel, an av- eraging of the value of pixels over a region per-pixel. A gaussian filter can be used to smooth an upsampled image, but it introduces a significant blur and does not preserve hard edges.

Bilateral filtering

smooth an image while maintaining detected edges. It does this by using a range filter as well as the spatial kernel to weight values which deviate less. For a positionpand a result

Jp: Jp = 1 kp ∑ q∈Ω Iqf(∥p−q∥)g(∥Ip−Iq∥) (2.16)

Wherefis the spatial filter on positionsp,qandgis the range filter on the corresponding image valuesIp,Iq.kpis a normalising value andΩis the set of values we cover with the

kernel.

High-dimensional Gaussian filtering

While directly stretching a low resolution image to a higher resolution is something of a blunt instrument, more subtle methods can be utilised to guide a higher resolution image created from low resolution information. Specifically, high resolution metadata buffers can guide edge-aware methods like bilateral filters. A high resolution depth buffer, geometry buffer or colour map can provide distinct edges to what would be an otherwise blocky upsampling.

Range

Figure 2.2– The splat-blur-slice pattern. The red line is the re-sampled points.

Adams et al. [36] detail a generalised high-dimensional Gaussian filter using the per- mutohedral lattice, a reimplementation of the splat-blur-slice model proposed in [37]. In that paper, the authors propose that the process of computing Gaussian-like filters can be achieved quickly by embedding the input signal in a high-dimensional space (splat- ting), performing an unguided gaussian blur on that space (blurring) and then retrieving the original signal as samples from the now-blurred space (slicing). The splat-blur-slice pattern is illustrated in Figure 2.2.

The permutohedral lattice places the signal points it receives into a high-dimensional lat- tice constructed from uniform simplices, permitting computationally simple barycentric interpolation in both the splatting stage as well as simple access to local neighbour sim- plices in the blurring stage.

With a high-dimensional filter in the manner of a bilateral filter, it is possible to use the guidance of one or more metadata buffers in such a lattice to guide the edges of an upsampled and filtered low-resolution full render. This includes depth buffers, geometry buffers, motion buffers and potentially more specific buffers such as reflection edges. Both [38] and [39] which we will cover shortly include guided high-dimension filtering as part of a larger caching strategy.

Joint bilateral filtering

Kopf et al. [40] adapted bilateral filtering in this fashion to describe Joint Bilateral Up- sampling, a method for combining a low-resolution set of data extracted from an image with the edge data in the original high-resolution in order to upsample the data.

˜ Sp = 1 kp ∑ q↓∈Ω Iqf(∥p↓ −q ↓∥)g(∥I˜p−I˜q ∥) (2.17)

Wherepandqare the image coordinates,I˜represents the original image,Srepresents the downsampled image,S˜represents the upsampled output, and the↓indicates coordinates which are mapped to the low-resolution image. f andgremain as filter weights, andkp

remains as a normalising factor. This method was adapted by Yang et al. [41] to rendering, demonstrating the use of a high-resolution geometry buffer as a set of edges to guide an upsampling of a low-resolution full render.