Processing

The raw data from the captured ILF and the tracked light sources need to be pro- cessed to obtain the appropriate representation ready to be used for the renderer.

6.2.4.1 Processing ILF light probes

The initial base ILF capture produces sets of HDR light probe images which are processed similar to the method described in Chapter 4. As there were no video HDR camera available for this work, bracketed still LDR images were taken with a fisheye lens mounted on a digital camera which were then merged into HDR images. The marker based detection described in Chapter 4 has been used again to detect the global position of the captured light probes in the environment. The image and position pairs are converted next into the ILF ray-database using the method described in Chapter 4.

As described previously in Chapter 4, the ILF ray-database is constructed with reference to the “ILF planes”. The dimensions of these ILF planes originally only needed to form a convex hull around the rendering space and they did not need to adhere to the actual scene boundary dimensions. However, as Temporal ILF uses the DCP technique, the base ILF needs to be mapped to the scene geometry. The

ILF planes in this case are constructed according to the scene boundaries acquired in the Temporal ILF capture. This way, there is an easy and direct mapping to the ILF data and the corresponding scene geometry. The illuminance data form the light sources are separately stored in Source Light Fields (SLF) as described in the Chapter 4 and mentioned later in the Chapter 5. The position and dimensions of these SLFs are also constructed according to the real-world position and dimensions of the corresponding light sources. The result is a base ILF ray-database that is editable by the DCP technique and ready to be used by a physically based renderer.

6.2.4.2 Processing of the Tracking Data

The Temporal ILF capture tracks the position and the intensity of the light sources that are changing in the scene in the given time period. The primary objective is to obtain the position and intensity of all the light sources at any given point of time. As mentioned earlier, there are two approaches to track the light sources. The first one is a proposed method with high accuracy, however expensive equipment has restricted its implementation in real-world implementations done in this PhD. The second method is the marker detection method where the position detection is automatic and but the intensity and colour detections are manual.

Both the tracking approaches acquire the position and the radiometric prop- erties of the light sources in each frames of the recorded video. Each of the frames are timestamped to avoid confusion. During the position detection, a custom software detects the position of each light sources and produces the data in the format shown in the diagram in Figure 6.6. For example, for the first frame, there are two pieces of data for the timestamp and the number of light sources (SLF in ray-database) detected in the frame. Each such source has three data entries for the “ID” of the light source as well as the colour and the position. The IDs are integers (ideally starting from zero) mapped to corresponding light sources. The cardinality of the detected light sources per frame may not be the same as it only records a light source if there is a change detected in it.

Figure 6.6: Tracking data format.

an input video. Each frame of the video produces the the tracking data in the mentioned format. A stream of such tracking data connected by the timestamps is obtained from the video which then is used by the renderer in the simulation part of the Temporal ILF pipeline.

6.3 Rendering

The objective of Temporal ILF is to produce a rendered video of the duration of the temporal ILF capture. The final part of the Temporal ILF pipeline is rendering where the changing light properties over time is used to simulate the change the position, intensity or colour of the concerned light sources synthetically in the base ILF. Each frame of the output video is rendered by asnapshot of the base ILF with the changed light sources. The tracking data includes the change information for all the frames in the intended video. An output video could be rendered with the following algorithm:

Algorithm 5Temporal ILF Pipeline: Simulation

1: procedureVideoRenderer(ILF handle, Scene, T rackingData) 2: td read(TrackingData);

3: i 0;

4: whilei < td.TotalFrames do . reading ith frame

5: j 0;

6: whilej < td[i].SLFcountdo . fromjth SLF of ith frame 7: updateSLFdatabase(td[i].SLF[j].ID, td[i].SLF[j].Colour,

8: td[i].SLF[j].P osition);

9: j++;

10: end while

11: updatedILF DCP.propagateChanges(ILF handle, Scene); 12: renderFrame(updatedILF, Scene);

13: saveFrame(td[i].T imeStamp);

14: i++;

15: end while 16: end procedure

Algorithm 5 describes the procedure VideoRenderer which renders a single frame of rendered image by iterating through all of the frames of the tracking data. for each f rame in the tracking data, it again iterates through all the SLF (light sources) recorded in that frame and updates each of these SLFs with the recorded colour and position by their IDs. After all of them are updated, the VideoRenderer calls the DCP technique to propagate the changes just made to the SLF to the

indirect light stored in the ILF database. the DCP.propagateChanges procedure produces theupdatedILF which is then used to render the scene for the frame and the image is subsequently saved to the disk with the Timestamp for the current f rame.