Spatial saliency detection

In this part, Spatial saliency detection (SD), the virtual border-based distance transform in spatial domain, is designed.

Virtual border building

Instead of using the frame border pixels as the seed set, we propose to add virtual borders around the original frame to obtain with-virtual-border frame. The virtual bor-

Chapter 2 – Traditional techniques for salient object detection in videos

der, calculated using original frame border pixel values, is used to get the new seed set. Specifically, the virtual border is built in four steps (as shown in Fig.2.14): Frame Border Selection, Frame Border Division, Representative Pixel Selection and Virtual Border Padding.

a) Frame Border Selection: it may suppose that the salient object could be con- nected with two or more borders. However, from the existing video datasets we ob- serve that: in usual cases if the salient object appears in the frame border, it is often connected with only one border. Here for the sake of simplicity of the presentation, we select one original frame border to build the virtual border by two steps.

In the first step, Fast iterative Minimum barrier distance transform algorithm (FastMBD) [103] is applied to frame α to obtain the map M as Eq (2.1).

M = 1 3(M1

0

+ M20+ M30). (2.1)

where M10, M20 and M30 are obtained respectively from three color channels of frame

α in the CIELab color space. For each color channel I with the size of h1 × w1, M0 is

generated as follows: if the pixel x ∈ r1 (r1 being the border of the frame α), its value

in M0 is set to 0. If pixel x ∈ r2 (r2 being the non-border of the frame), its value in

M0 is initialized as ∞. Two auxiliary maps τ and ψ are initialized by the pixel values in each channel of the original image. Let the 4-adjacent pixels around a pixel x in the region r2 be xup (up pixel), xleft (left pixel), xdown (down pixel) and xright (right pixel).

Using the update function, M0 and the auxiliary maps are firstly updated in raster scan order, secondly updated in inverse raster scan order with y ∈nxdown, xright

o

, and thirdly updated in raster scan order again. The update function is shown as follows: if M0(x) > Oy(x) (y ∈

n

xup, xleft

o

), M0(x), τy and ψy are updated to Oy(x), maxnτy, I(x) o

and minnψy, I(x)

o

respectively, where Oy(x) = maxnτy, I(x) o

− minnψy, I(x) o

.

In the second step, the frame border nearest to the non-zero region in the map M is selected to build the virtual border. Here, the threshold δ is used to determine the non-zero region.

b) Frame Border Division: after one original border selected, the corresponding di- vided border is obtained from the original frame border (with width u). The DUB, the DDB, the DLB and the DRB are shown in the middle left part in Fig.2.14. The reason lying behind this division is that: the region in the frame corner is often connected with two borders and its feature is also related to these two borders. Thus, the irregular

2.3. Virtual Border and Guided Filter-based (VBGF) algorithm

Virtual border building

Frame Border Selection Frame Border Division Representative Pixel Selection Virtual Border Padding

Mean

Minimum SAD Ground Truth

Minimum SAD Median Ground Truth Map M

Map M

(1) (3)

(2) (4)

Figure 2.14: Virtual border building: (1): two examples of map M obtained by applying FastMBD on the frame; and then for each frame, the closest border to the salient region is selected to build the virtual border; (2): generating the divided border from the high- lighted frame border (with width u), h1is the frame height, w1 is the frame width and l is

the ratio of the corresponding border length, four divided borders: the DUB, the DDB, the DLB and the DRB are shown; (3): two examples of the representative pixel selec- tion, where “Mean” means the representative pixel is chosen using the mean value of the border’s intensities and “Median” means choosing the median value of the border’s intensities as the representative pixel, the red dotted line denotes the virtual border padded with the selected representative pixel; (4): building and padding the virtual bor- der (with size v) with representative pixel value, four virtual borders: VUB, the VDB, the VLB and the VRB, are shown in four different textures.

Chapter 2 – Traditional techniques for salient object detection in videos

shape connecting three borders is used to calculate the virtual border. The parameters u and l are selected empirically. In this chapter, u is set to 5 and l is set to 18%. Pre- liminary experiments showed that these values make the algorithm robust to various background complexities.

c) Representative Pixel Selection: for the generated divided border, sum of absolute differences (SAD) is computed for each pixel by summing all the absolute differences between this pixel and other pixels in the divided border:

SAD(x) = X

x0∈DB

|I(x) − I(x0)| (2.2)

where DB ∈ nDUB, DDB, DLB and DRBo, I is the feature channel. The pixel having the minimum SAD is selected to represent the divided border. For color images, the SAD is computed by summing the three color channels:

colorSAD(x) = X x0∈DB X i∈{r,g,b}   I i (x) − Ii(x0)  (2.3)

We have also considered using the mean or median value of the border’s intensities as the representative pixel value. Various experiments conducted on different frames have shown that the minimum SAD choice performs better than the mean and the median values in most of the cases (cf. the 1st example image in Fig.2.14 where the representative pixel is chosen from the salient object instead of the background when using the mean value of the border’s intensities). The same way, choosing the median value of the border’s intensities as the representative pixel value fails, which can be seen on the 2nd example image in Fig.2.14. As the minimum SAD performs better in most cases and in order to be more robust in all situations, we adopt the minimum SAD in the proposed method.

d) Virtual Border Padding: around the selected original frame border, we build the corresponding virtual border with the above representative pixel to get the with-virtual- border frame D. The VUB, the VDB, the VLB and the VRB are shown in the middle right part in Fig.2.14. Existing methods usually regard the border (with width 1) to be background and seed sizes are set to be 1. Here we set the virtual border size v to 5, which helps the proposed “virtual border building” to be applied to other distance transform based saliency detection methods.

2.3. Virtual Border and Guided Filter-based (VBGF) algorithm

Virtual border building Frame

Spatial saliency map Ground truth

Figure 2.15: (Better viewed in color) An example of the spatial saliency detection. The red dotted line denotes the virtual border.

Saliency computation

After the “virtual border building”, the spatial saliency map SSM is obtained by apply the FastMBD [103] to the with-virtual-border frame D and then remove the virtual border from the resulted map to obtain the spatial saliency map SSM. One example is given to show the process of spatial saliency detection in Fig.2.15.