Capture Frame Plugin

The Frame was discussed from the user view in Section 4.3.2. It is a special image, shown in Figure 4.4, that takes a snapshot of the area beneath it. This section will describe the implementation, which is hidden from the user view by the Frame’s simple appearance.

In the first iteration of the tool, capture was achieved by copying the (orthogonally aligned) rectangular block of pixels within the bounding box of the Frame from the graphics frame buffer and loading it in as a new image. The new image slides in from the corner of the table to attract attention.

The implementation is a bit more complicated than the description above would imply. When the capture action is triggered (by holding a pen/finger stationary on the frame for

> 1 second – a dwell), a capture event is placed onto the event queue. When the scene is

next redrawn (usually within a few milliseconds), it is drawn without non-capture objects (e.g. image decorations, personal space polygons and menus). When the Frame itself would have been drawn, we instead calculate the bounding box of the Frame, copy pixels within the (partially drawn) framebuffer to system memory and enqueue two events – one (full) redraw to the event queue and a save-capture event, which is executed in a synchronised background thread.

The background thread receives the semaphore post and the event which points to the copied framebuffer data. We first perform the texture pre-processing steps on the framebuffer data – scaling it to have dimensions that are the closest powers of two to the original (width, height) and generating mipmaps – which are kept (for now) in system memory. The foreground thread is notified that the textures are ready, and the foreground thread transfers textures to the graphics card, similar to the techniques for regular photos (§3.5.5.1). An Animation object is created that moves the new Image into the centre of

the table.

After notifying the foreground thread, the background thread begins converting the captured pixels into a JPEGimage. First we flip it35, and then convert it using the open source libjpeg library. The image is saved to disk with a time stamp and an identifier for the user who initiated the frame capture. None of this requires access to the graphics card, and so while this occurs we continue to process input events and redraw the scene. Once the capture is saved, control of the background thread returns to the thread manager which waits for the next capture event.

The process above (using a bounding box and the display’s framebuffer) has some quality and accuracy problems. The framebuffer pixels do not contain all the detail that might be obtained from the image data, and we do not orient the pixels to reflect the orientation of the Frame. There are three techniques now used, in what has been called the SmartFrame.

5.5.8.1 Virtual Camera Reorient Technique

We improve on the old method, with minimal computation impact, by changing the

viewport to wholly encompass the Frame, before the first redraw. That is, Dwelling on

the SmartFrame repositions the 3D camera (thus rotation is preserved); captures; saves a composited, full-screen image to disk; and loads it (without actually redrawing the

display, so the camera repositioning is not seen by the user). Because the orientation is now important, the representation of the SmartFrame includes an arrow to indicate up.

Dwelling on the SmartFrame repositions the 3D camera (thus rotation is preserved); captures; saves a composited, full-screen image to disk; and loads it. This is accomplished without redrawing the display, so the camera repositioning is not seen by the user.

This solves the orientation problem and improves image quality when the Frame is small. It also requires very little additional computational overhead, and will produce images the same size as the screen (e.g. 1024 × 768) to save on disk. However, at this stage, we still do not use the full detail obtainable from the original file, which might be important for a “zoom-in” action. This takes longer (see below) whereas the capture from theframebuffer is quick and allows almost immediate feedback to be given for the user.

Improving on this is more complex. Detail beyond a 1024 × 1024 texture36 is discarded when the original file is loaded. Hence, for greater detail, we must return to the original file on disk. Furthermore, we can no longer use the graphics card for the clipping, scaling, rotating and compositing of the images that compose the area beneath the Frame object. Instead, we use the program convert from the open source ImageMagick toolkit to perform these tasks by manipulating the files on disk.

Listing A.5.2 shows the technique used to reposition the camera, given the position, scale, orientation and aspect ratio of the SmartFrame.

5.5.8.2 Texture Offsets Technique

Processing the original file is slow, so we offer an alternative that does not require a new image to be created. It functions similar to a zooming method on textures. Computation and memory-usage is improved over the camera-reorient technique, but it cannot be used for all captures. First we must determine this technique is possible for a given capture.

We detect whether the SmartFrame is wholly contained in another Image (i.e. we are performing a crop of a single image). If it is not, we keep the 3D-camera-reposition image created, load it into texture memory and continue with slow file processing in the background (if enabled). Otherwise, if the Frame is wholly contained, we still keep the framebuffer capture and save it to disk (as it is a low-quality version of the eventual result), but we perform alternative processing for the texture loaded onto the display.

To detect containment, we determine whether the same image is located at the centre, and all four corners of the Frame by selecting the object at those pixels, ignoring the Frame itself. At the same time, we determine where (what offsets) in the image the corners are, thus determining offsets into that image’s texture. Now, instead of loading the screen-grab (which is still saved to disk), we instead load a special Resource that shares the texture of the target image, but uses the texture offsets previously found. To display this new image, we create a display list that shares the texture with the image captured but, rather than binding points (0,0), (0,1), (1,1), (1,0) of the texture, we bind the points of the corners of the Capture Frame, translated onto the image that is being captured.

Detail Listing A.5.3 shows the technique used to determine the texture offsets required. In summary, the procedure returns a Boolean indicating whether the creator Frame is wholly within the Image source. If it is, the object coordinates for source that correspond to where the corners of the creator lie are returned to the caller in the first argument – these may then easily be translated into texture offsets for the source texture. First, we create object coordinates for the corners of the creator, relative to the creator – these are simply the four points (±a₂, ±1₂), where a is the aspect ratio of the creator (Frame). We convert these to world coordinates, using the cached transformation matrices used to position

5.5. Plugins CHAPTER 5. DESIGN

creator in the environment, then convert them to source-object coordinates using the inverse of the transformation matrices used to position source in the virtual environment.

If any of these transformed points is outside the bounds of the object coordinates for the

source (i.e. outside (±a₂, ±1₂), with a the aspect of the source) we know that the corners of

creator are not wholly contained within source.

Once we have these points, they are used to bind the corresponding points of the existing texture in source to the corners of a newly created referenced Image, that we load at the same position as the SmartFrame, with an alternative animation (to indicate a crop). This animation causes the cropped area to grow out of the source image.

5.5.8.3 File Manipulations for the “perfect” crop

If, in addition to the containment requirement above, we have access to the image file on disk, then after loading the referenced Image, we formulate a 3-step transformation for the

convert program from ImageMagick, as alluded to above. This has three stages: a crop

to the bounding box of the rotation (to speed up stage 2); a rotation; and a second crop. This results in a new file on disk, made from the original image – that is, the best data available. Once the operation completes, we reload the newly created image off disk, as it will either contain more pixel data than the original; or it will contain less pixel data but with equivalent quality, and thereby be more space-efficient.

Detail Listing A.5.4shows the technique used to determine the set of transformations required to carry out on the file on disk, using the convert program.

The stages are outlined in the listing. Notably, we crop twice – crop is a much faster operation to perform than rotate, and so we perform an initial crop to reduce to the minimum number of pixels that are required to perform the rotate. The result has waste where rotated parts now lie outside the final crop, which are subsequently removed by the second crop.

The values off[0].x, off[0].y ... off[3].x, off[3].y are the texture offsets (each in the range [0.0, 1.0]) that were found from the Texture Offsets calculation in Section5.5.8.2.

5.5.8.4 libframe Plugin Library

This section describes two frames: the original Frame, that takes orthogonal regions from the display framebuffer, and the multi-faceted SmartFrame. Clearly, some of the functionality can be shared. However, calling code between plugins is typically poor practice – there is no guarantee that the other plugin has been loaded – and some platforms37 have

a limited dynamic library implementation that does not support this (backreferences). To share the implementation, and potentially make capturing functionality available to other plugins, common parts of the frame implementations are reached using a plugin library – libframe. Then, separate plugins encapsulate specifics and allow each type of frame to coexist, or be readily disabled.