Real-Time Image Abstraction by Directed

Filtering

JAN ERIC KYPRIANIDIS AND JÜRGEN DÖLLNER

FIGURE 3.3.1 An image abstracted using our framework. Left: Original image (Copyright Anthony Santella). Right: Abstracted result.

I

In this chapter we present a framework of automatic image processing techniques that create high-quality, simplified, stylistic illustrations from color images, videos, and 3D renderings. Our method extends the approach of [Winnemöller06] to use iterated bilateral filtering for abstraction and Difference-of-Gaussians (DoG) for edge detection. We present enhancements to these techniques to improve the quality of the output by adapting them to the local orientation of the input.

To smooth low-contrast regions while preserving edges, we present a fast separable implementation of the bilateral filter. Our approach works by aligning the bilateral filter with the local structure of the image. The first pass filters in the direction of the gradient, and the second pass filters in the direction of the tangent. To extract salient edges, we present a fast two-pass algorithm. A 1D DoG filter is applied in the direction of the gradient, followed by a second pass that applies smoothing along flow curves that are derived from the local structure of the image. Both techniques require a smooth vector field that represents local orientation. We show how such a vector field can be computed from the eigenvectors of the smoothed structure tensor. Besides gradient calculation, only linear separable smoothing with a box or Gaussian filter is necessary.

A schematic overview of our framework is shown in Figure 3.3.2. Input is typically an image, a frame of a video, or the output of a 3D rendering. We start with the construction of the flow field by calculating the structure tensor. The structure tensor is then smoothed, and the eigenvector corresponding to the local tangent direction is stored in a texture map. Then the input is iteratively abstracted by using the orientation-aligned bilateral filter. We perform a total of n_α iterations. We apply color quantization to the result. After n_e n_aiterations, we extract edges from the intermediate result using the separable flow-based DoG filter. In our examples we use n_α and n_a = 4 Finally, the extracted edges are superimposed on the output of the color quantization.

FIGURE 3.3.2 Overview of our abstraction framework.

Our framework uses different color spaces. The flow field construction is performed in RGB color space. This has the advantage that the computation is not affected by noise that may be introduced through color space conversion.

Applying the bilateral filter in RGB color space may result in unwanted color bleeding artifacts [Tomasi98]. Therefore, a color space that separates luminance and chrominance should be used. In our previous work [Kyprianidis08] as well as in [Winnemöller06], the CIE-Lab color space is used. For simplicity, we use the Y’CbCr color space here, which is a popular standard in the television industry. Conversion from RGB to Y’CbCr can be performed using the following matrix whose coefficients are based on the ITUR BT.601 specification:

The inverse transform from RGB to Y’CbCr is given by

Color space conversion is only necessary before bilateral filtering and during the final composition step, where the detected edges are superimposed on the output of the color quantization.

FLOW FIELD CONSTRUCTION

The basic idea behind flow field construction is to find a local direction in which the rate of change is minimized. Our approach is based on the eigenvalues of the structure tensor [van Vliet95]. To make full use of RGB color information during computation, we combine it with Di Zenzo’s multi-image gradient method [Cumani89]. Let f(x,y) denote the RGB color value of the input image at pixel (x,y). We begin by calculating an approximation to the directional derivatives in the x- and y-directions using the Sobel filter:

f_x(x₀,y₀) = − f(x₀ − 1,y₀ − 1) − 2f(x₀ −1,y₀) − f(x₀ − 1,y₀ + 1) + f(x₀ + 1,y₀ − 1) + 2f(x₀+ 1,y₀) + f(x₀ + 1,y₀ + 1)

f_y(x₀,y₀) = −f(x₀ − 1,y₀ − 1) − 2f(x₀,y₀) −1) − f(x₀ + 1,y₀ − 1) + f(x₀ − 1,y₀ + 1) + 2f(x₀,y₀ + 1) + f(x₀ + 1,y₀ + 1) The structure tensor at (x₀,y₀) is then given by

The induced quadratic form measures the squared rate of change of f in a certain direction. We are interested in the extremal values and the corresponding directions. From linear algebra we know that the extremal values are given by the square roots of the eigenvalues:

The corresponding eigenvectors are

Since the structure tensor is a symmetric positive semidefinite matrix, the eigenvalues are non-negative real numbers and are orthogonal to each other. The eigenvector points in the direction of maximum change. The eigenvector points in the direction of the minimum rate of change.

By calculating the eigenvectors v2 for every point of an image, we get a vector field that is aligned to local edge orientation, as

shown in Figure 3.3.3a. Unfortunately, this vector field has discontinuities, and, accordingly, tracing its flow curves does not give good results. In order to smooth the vector field, we apply smoothing to the structure tensor. The smoothing of the structure tensor is a linear operation, but the effect on the eigenvector field is highly non-linear. Vectors of higher magnitude and vectors that point in similar directions get more weight. Also, vectors pointing in the opposite direction do not cancel out. The effect on the vector field is similar to the non-linear edge tangent flow filter (ETF) from [Kang07]. The advantage of our approach is that it only requires linear separable smoothing. That means the filter can be implemented as two passes, reducing the computational complexity from O(r2) to

O(r) per pixel. Furthermore, our approach creates good results without requiring multiple iterations. The result of applying a 5×5 Gaussian filter is shown in Figure 3.3.3b. Even better results can be achieved using a 7×7 or 9×9 Gaussian filter.

FIGURE 3.3.3 Tangent field induced by the structure tensor. (a) Before smoothing. (b) After smoothing.

Our implementation uses three rendering passes to construct the flow field. In the first pass we calculate the gradients and the structure tensor. The structure is stored as a triple (E,F,G) in a temporary texture (Listing 3.3.1). The second pass takes this temporary texture as input and applies a 1D Gaussian filter with kernel [14641] in the horizontal direction (Listing 3.3.2). The third pass applies the Gaussian filter in the vertical direction and then calculates the eigenvector v₂ (Listing 3.3.3). Our algorithms require a normalized direction. We therefore store the normalized eigenvector in the first and second component and the length of the eigenvector in the third component of the texture map. We refer to this texture as the tangent flow map (tfm).

Listing 3.3.1 Fragment shader for calculating the structure tensor

uniform sampler2D img;

v oid main (v oid) {

iv ec2 uv = iv ec2(gl_FragCoord.xy); v ec4 c = texelFetch2D(img, uv , 0); v ec4 u = ( -1.0 * texelFetch2D(img, uv + iv ec2(-1, -1), 0) + -2.0 * texelFetch2D(img, uv + iv ec2(-1, 0), 0) + -1.0 * texelFetch2D(img, uv + iv ec2(-1, 1), 0) + +1.0 * texelFetch2D(img, uv + iv ec2( 1, -1), 0) + +2.0 * texelFetch2D(img, uv + iv ec2( 1, 0), 0) + +1.0 * texelFetch2D(img, uv + iv ec2( 1, 1), 0) ) / 4.0; v ec4 v = ( -1.0 * texelFetch2D(img, uv + iv ec2(-1, -1), 0) + -2.0 * texelFetch2D(img, uv + iv ec2( 0, -1), 0) + -1.0 * texelFetch2D(img, uv + iv ec2( 1, -1), 0) + +1.0 * texelFetch2D(img, uv + iv ec2(-1, 1), 0) + +2.0 * texelFetch2D(img, uv + iv ec2( 0, 1), 0) + +1.0 * texelFetch2D(img, uv + iv ec2( 1, 1), 0) ) / 4.0;

gl_FragColor = v ec4(v ec3(dot(u.xyz , u.xyz ), dot(v .xyz , v .xyz ), dot(u.xyz , v .xyz )), 1.0); }

Listing 3.3.2 Fragment shader for smoothing the structure in the horizontal direction

uniform sampler2D img; v oid main (v oid) {

iv ec2 uv = iv ec2(gl_FragCoord.xy); gl_FragColor = v ec4(v ec3(

1.0/16.0 * texelFetch2D(img, uv + iv ec2(0, -2), 0).xyz + 4.0/16.0 * texelFetch2D(img, uv + iv ec2(0, -1), 0).xyz + 6.0/16.0 * texelFetch2D(img, uv + iv ec2(0, 0), 0).xyz + 4.0/16.0 * texelFetch2D(img, uv + iv ec2(0, 1), 0).xyz + 1.0/16.0 * texelFetch2D(img, uv + iv ec2(0, 2), 0).xyz ), 1.0);

}

Listing 3.3.3 Fragment shader for smoothing the structure in the vertical direction and calculation of the flow field

uniform sampler2D img;

v oid main (v oid) {

iv ec2 uv = iv ec2(gl_FragCoord.xy); v ec3 g = v ec3(

1.0/16.0 * texelFetch2D(img, uv + iv ec2(-2, 0), 0).xyz + 4.0/16.0 * texelFetch2D(img, uv + iv ec2(-1, 0), 0).xyz + 6.0/16.0 * texelFetch2D(img, uv + iv ec2( 0, 0), 0).xyz + 4.0/16.0 * texelFetch2D(img, uv + iv ec2( 1, 0), 0).xyz + 1.0/16.0 * texelFetch2D(img, uv + iv ec2( 2, 0), 0).xyz );

float lambda1 = 0.5 * (g.y + g.x +

sqrt(g.y*g.y - 2.0*g.x*g.y + g.x*g.x + 4.0*g.z *g.z )); v ec2 d = v ec2(g.x - lambda1, g.z );

gl_FragColor = (length(d) > 0.0)? v ec4(normaliz e(d), sqrt(lambda1), 1.0) : v ec4(0.0, 1.0, 0.0, 1.0); }