Procedia Computer Science 56 ( 2015 ) 104 – 110
1877-0509 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the Conference Program Chairs doi: 10.1016/j.procs.2015.07.175
ScienceDirect
The 12th International Conference on Mobile Systems and Pervasive Computing
(MobiSPC 2015)
Content-aware image retargeting for image display on foldable
mobile devices
Li-Wei Kang
a,b,*, Ming-Fang Weng
c, Chao-Long Jheng
b, Ching-Yu Tseng
b, Sudhish
Kasaba Ramesh
d, Ankit Gureja
d, Huan-Cheng Hsu
e, Chia-Hung Yeh
faGraduate School of Engineering Science and Technology, National Yunlin University of Science and Technology, Yunlin 640, Taiwan bDepartment of Computer Science and Information Engineering, National Yunlin University of Science and Technology, Yunlin 640, Taiwan
cSmart Network System Institute, Institute for Information Industry, Taipei 106, Taiwan
dSchool of Electrical & Computer Engineering, University of California Santa Barbara, Santa Barbara, California, USA enstitute of Information Science, Academia Sinica, Taipei 115, Taiwan
fDepartment of Electrical Engineering, National Sun Yat-sen University, Kaohsiung 804, Taiwan
Abstract
With growing mobile data consumption, foldable displays have enabled further enlargement for the display of a mobile device without increasing the size and weight of the device. This paper addresses a possible problem of image distortion for image displays on foldable mobile devices. When a mobile device is folded with a certain degree, the part of a displayed image in the folded area (or bending zone) of the device may be squeezed. To provide better visual quality for a foldable display, we formulate the problem as a piecewise content-aware image retargeting problem. To compensate the squeezed region of an image in the bending zone of a device, we propose to expand this region toward the left and right sides, respectively, based on the image content. In the meantime, the original left and right regions of the image should be also adjusted accordingly. In this paper, we investigate how to apply an image retargeting technique to solve the problem for better identifying visually significant contents of the image and produce more pleasing display result.
© 2015 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the Conference Program Chairs.
Keywords: Foldable devices; bendable devices; mobile devices; image retargeting; saliency detection.
* Corresponding author. Tel.: +886-5-534-2601 ext. 4518; fax: +886-5-531-2170.
E-mail address: lwkang@yuntech.edu.tw
© 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
1. Introduction
Mobile data consumption has grown enormously over the past several years. This growth was enabled not only by the increasing availability of faster 3G/4G networks1, but also by the evolution of smart/mobile-phone hardware
based on higher battery capacity, powerful processing and graphics capabilities as well as the increasing availability of multimedia data or digital contents distributed over the Internet. A critical issue of this evolution is the display size, which has been dramatically increased with the presentation of the iPhone in 2007, relying on an on-screen touch interface replacing all of the buttons. This display style has now dominated the interface of most mobile phones2.
Therefore, with faster networks continuously developed (e.g., 5G networks3) and increased multimedia data to be
communicated, the pressure on display sizes is increasing, while the form factors of current mobile phones are limited. For example2, the HTC Titan is with a 4.7-in. display and the Samsung Galaxy Note 4 is with a 5.7-in.
display, where the limits of size for mobile devices have been reached and even exceeded for most consumers. On the other hand, tablets have emerged as the portable alternative for laptops/netbooks. Tablets offer a display size of up to about 13-in. (e.g., MacBook Air 13 (13.3-in.), Microsoft Surface Pro 3 (12.5-in.)) and can be carried around. Nevertheless, they are about 2~3 times the size and weight of smartphones and do not fit into a pocket, which limits the current sales volume of tablets to be much lower than that of smartphones.
A solution for enabling larger displays in mobile devices would be to replace the glass display with a larger, lightweight display that can be folded to a shape fitting a pocket. Currently, foldable displays are the only one solution to the problem of having a small device with a large perceived display area by retaining the advantages of current mobile phones, such as small form factor, low weight, and long battery life, while extending the display size2. The first report on foldable displays dates back to 20054, and then a number of companies demonstrated
foldable-display prototypes that enable thin and light-weight products with larger foldable-displays. For example2, Polymer Vision has
demonstrated foldable displays that are only 0.1 mm thick and can be bent more than 100,000 times to a radius as small as 5 mm. In addition, Sony demonstrated a foldable display both by using electrophoretic material and OLED (organic light-emitting diode) material in 2010 and 2011. Moreover, Samsung demonstrated a very different type of foldable display construction in 2009 and 2010, where only the bending zone is non-rigid. Furthermore, a report from the Business Korea hints that a fully foldable smartphone could hit the market as early as 2016 from Samsung5
. On the other hand, while the foldable display technology was not completely ready to market, several related prototypes and frameworks have been also presented from academic communities. For example, Khalilbeigi et al.6
present a novel device concept that features double-sided displays which can be folded using predefined hinges, which enables users to dynamically alter both size and shape of the display and also to access the backside using fold gestures. In addition, Lee, Hudson, and Tse7 implemented four interactive foldable display designs using image
projection with low-cost tracking and explore display behaviors using orientation sensitivity. Moreover, Tan et al8.
showed how a flexible mobile device with unrestricted folding can create more realistic and augmented experiences by leveraging simple 2D origami form to give applications new affordances: enabling rapid customization of the device shape. Furthermore, Chen et al.9 addressed and solved the problem that on the flexible touchscreens of a
foldable display, some touch area around the folded line is not touchable in users’ operation.
So far, most related works in foldable devices focused on the hardware design (including the selection of materials and the mechanism design for a foldable device)2,4-8, while only a few ones consider the user interaction on
the folded area9. However, to the best of our knowledge, none of them considers the problem of image adaptation for
image display in the folded area. In this paper, we address the problem of image distortion for image displays on foldable mobile devices. When a device is folded with a certain degree, the part of a displayed image in the folded area of the device may be squeezed. To provide better visual quality for a foldable display, we formulate the problem as a piecewise content-aware image retargeting problem. To compensate the squeezed region of an image in the folded area, we propose to expand this region toward the left and right sides, respectively, based on the image content. In the meantime, the original left and right regions of the image should be also adjusted accordingly.
The major contribution is three-fold: (1) to the best of our knowledge, we are among the first to address the problem of image distortion in the folded area of a foldable mobile device; (2) we formulate the problem as an image retargeting problem; and (3) we solve the problem using a piecewise content-aware image retargeting technique. The rest of this paper is organized as follows. Sec. 2 presents the proposed piecewise content-aware image retargeting
technique for image adaptation in the folded area of a foldable device. In Sec. 3, experimental results are demonstrated. Finally, Sec. 4 concludes this paper.
2. Proposed method for image display on foldable mobile devices
In this section, we first present the problem formulation for image adaptation on the folded area of a foldable mobile device, followed by presenting the proposed algorithm to solve the problem.
2.1. Problem formulation
As a conceptual example illustrated in Fig. 1(a), we consider the problem that some part of a displayed image on the folded area of a foldable device may be squeezed. To adapt the squeezed region of the image and provide better visual experience to a viewer, we propose to expand the squeezed region toward the left and right sides, respectively, while adjusting the original contents of the two-side regions accordingly, as illustrated in Fig. 1(b). To accomplish this, we first simply partition the image into three parts, namely, the left, middle, and right parts. We then enlarge the width of the middle part and accordingly shrink the widths of the left and right parts, respectively. However, straightforward image resizing operators, such as scaling, usually do not produce satisfactory results, since they are oblivious to image content10-11. Hence, in our method, we apply content-aware image retargeting, which resizes an
image while taking its content into account to preserve visually significant regions and minimize distortions. Moreover, different from current retargeting approaches12-16, our goal here is not only to resize an image into
different sizes. In our framework, we, respectively, apply image retargeting to enlarge the middle part of the image and shrink the left and right parts of the image accordingly, and the overall size of the image is kept the same. Therefore, in this paper, we formulate the image adaptation problem as a piecewise content-aware image retargeting problem.
Fig. 1. Illustration for the considered problem of image adaptation on the foldable area of a foldable mobile device: (a) a conceptual example showing that some part of a displayed image on the folded area may be squeezed; (b) the concept of our solution for this problem.
2.2. Proposed method
As shown in Fig. 2(a), to adapt an image I of size m×n displayed on a foldable mobile device, we first partition the image I into the three sub-images, namely, the middle, left, and right parts, denoted by I1, I2, and I3 of sizes m×n1,
m×n2, and m×n3, respectively, where n = n1 + n2 + n3. To expand the middle part (I1), and shrink the left (I2) and
right (I3) parts of the image I, we apply a content-aware image retargeting algorithm10-16 to, respectively, enlarge I1
into I’1 of size m×(n1 + 2×k), while shrinking I2 into I’2 of size m×(n2 - k) and shrinking I3 into I’3 of size m×(n3 - k),
where 0 ≤ k ≤ n2, n3, and the overall width of the image I is still kept to be n, as illustrated in Fig. 2(b).
The selection of the parameter k depends on the “folding degree ( )” of the device (90o ≤ ≤ 180o). That is, the smaller the folding degree is, the larger the parameter k will be, and vice versa. When = 180o, no folding is
performed and k = 0. Based on our experience, the parameter k is determined as:
where is the minimum percentage of image resizing ( is empirically set to 5% in our experiments). The proposed image adaptation algorithm for foldable mobile devices is summarized in Table 1, where the employed content-aware image retargeting algorithm is denoted by the function called ImageRetargeting(), described in the next subsection.
Fig. 2. (a) Illustration of partitioning the image I into the three parts; (b) illustration of enlarging the middle part while shrinking the left and right parts, respectively, of I.
Table 1. Summarization of the proposed image adaptation algorithm for foldable mobile devices.
Function ImageAdaptation(I, m, n, n1, n2, n3, k)
Input: an image I of size m×n, where I is partitioned into I1 of size m×n1, I2 of size m×n2, I3
of size m×n3, n = n1 + n2 + n3, and the target sizes of I1, I2, and I3 are m×(n1 + 2×k), m×(n2 -
k), and m×(n3 - k), respectively (as shown in Fig. 2). Output: Retargeted image I’ of size m×n.
Step 1: Apply the content-aware image retargeting algorithm for image enlargement to
enlarge I1 into I1’ of size m×(n1 + 2×k).
Step 2: Shrink I2 into I2’ of size m×(n2 - k) via ImageRetargeting(I2, m, n2, n2 - k). Step 3: Shrink I3 into I3’ of size m×(n3 - k) via ImageRetargeting(I3, m, n3, n3 - k). Step 4: Integrate I1’, I2’, and I3’ to obtain the final retargeted image I’ and return I’.
2.3. Content-aware image retargeting
In the proposed method, any content-aware image retargeting algorithm10-16 can be employed. The main goal of
content-aware image retargeting is to arbitrarily resize an image while taking its content into account to preserve visually significant regions and minimize distortions. There have been several related approaches proposed10,
roughly including content-aware cropping-based, segmentation-based, seam carving-based12-13,16, warping-based14-15, patch-based14-15, and multi-operator-based11 approaches. A common key step of most retargeting approaches is first
to perform saliency detection17-18 to an image to identify the visually significant regions of the image by obtaining a
saliency map for this image.
By considering the tradeoff between retargeting performance and computational complexity, in the stage, we simply employ the seam carving algorithm12-13 in our algorithm as the content-aware image resizing function called ImageRetargeting() used in Table 1. In the seam carving algorithm, the saliency map (or energy map/importance
map) for an image is first calculated by computing the gradient value for each pixel in the image. The gradient value of a pixel is then treated as the energy for this pixel. Moreover, a seam is defined as a 1-D connected path with the lowest energy in an image, where the sum of the pixel energy values in this path is the minimum. That is, the pixels in a seam (vertical (from top to bottom), or horizontal (from left to right)) are with visually least significant in an image. To shrink the width of an image by 1 pixel, the seam carving algorithm simply removes the vertical seam with the lowest energy. This operation can be iteratively performed until the target width is achieved. The seam carving function called ImageRetargeting() for shrinking the width of an image is summarized in Table 2. Similar
process can be used to shrink the height of an image by iteratively removing the horizontal seam with the lowest energy.
Table 2. Summarization of the seam carving algorithm for shrinking the width of an image.
Function ImageRetargeting(I, m, n, n’)
Input: an image I of size m×n and the target size of m×n’ (n’ < n). Output: retargeted image of size m×n’.
Step 1: Compute the saliency map of I via
(gradient calculation).
Step 2: Find the seam S* with the least energy via
.
Step 3: Remove S* from I and update I.
Step 4: Repeat Step 1 to Step 3 until the size of I reaches m×n’. Step 5: Return I.
On the other hand, to enlarge an image using seam carving, the basic idea is to approximate an inversion of the seam removal process and insert new artificial seams to the image. Therefore, to enlarge the width of an image by k pixels, it can be achieved by finding the first k seams (with least energies) for removal, and duplicate them to achieve image enlargement. It should be noted that the content-aware image retargeting process (for both image enlargement and shrinkage) can be replaced by any advanced retargeting methods10-11,14-16, while the step for the
determination of saliency map can be replaced by any further saliency detection techniques17-18.
3. Experimental results
To evaluate the performance of the proposed image adaptation algorithm for foldable mobile devices, we implemented the proposed method both on the PC with Matlab and on the Android platform with the Java SDK, and perform the simulation on the HTC One mobile phone. In fact, the proposed framework was developed based on the request from the Industrial Technology Research Institute (ITRI) of Taiwan, where a foldable mobile device prototype is still under construction, as illustrated in Fig. 319. Hence, the proposed framework can be viewed as a
“pre-research,” and once the prototype is ready, our method will be tested online.
Fig. 3. Examples of the foldable mobile device prototype developed by ITRI, Taiwan (under construction)19.
To perform our experiments, we empirically set n1 = 0.2×n, n2 = 0.4×n, n3 = 0.4×n, and = 0.05. The size of each
image to be tested is set to m = 320 and n = 240, while the tested folding degrees are set from 180o (without folding)
to 90o. Fig. 4 shows examples of partitioning an image into the three parts, and applying the seam carving algorithm to enlarge/shrink them, respectively. In addition, Fig. 5 shows the results by simulating the folding degrees from 180o to 90o. Based on Figs. 4-5, it is expected that the proposed algorithm should be workable when actually being
Fig. 4. Examples of partitioning (a) an image into (b) the three parts, and applying the seam carving algorithm to enlarge/shrink them, respectively; (c) the found seams with least energies for duplicate (image enlargement) or removal (image shrinkage); (d) the retargeted results.
Fig. 5. Retargeted images with folding degrees of (a) 180o (without folding); (b) 170 o; (c) 160 o; (d) 150 o; (e) 140 o; (f) 130 o; (g) 120 o; (h) 110 o;
4. Conclusions
In this paper, we have addressed the problem of image distortion for image displays on foldable mobile devices. When a mobile device is folded with a certain degree, the part of a displayed image in the folded area of the device may be squeezed. To provide better visual quality for a foldable display, we have formulated the problem as a piecewise content-aware image retargeting problem and proposed an image adaptation algorithm with content-aware image retargeting algorithms to solve the problem. For future works, we are currently investigating better saliency detection algorithm to more precisely identify the visually significant region of an image, where we consider learning-based fusion of multiple saliency maps obtained by different approaches for an image. In addition, quality assessment techniques for image retargeting20 may be also integrated into our method for iteratively improving the display results. On the other hand, the proposed algorithm will be evaluated on a really foldable mobile device in the near future.
Acknowledgements
This work was supported in part by the Ministry of Science and Technology (MOST), Taiwan, under Grants MOST 103-2221-E-224-034-MY2.
References
1. Salkintzis AK. Interworking techniques and architectures for WLAN/3G integration toward 4G mobile data networks. IEEE Wireless
Communications 2004;11:50-61.
2. Huitema E. Frontline Technology: the future of displays is foldable. SID Information Display 2012;28:6-10.
3. Agyapong P, Iwamura M, Staehle D, Kiess W, Benjebbour A. Design considerations for a 5G network architecture. IEEE Communications
Magazine 2014;52:65-75.
4. Huitema E, Gelinck G, Lieshout PV, Veenendaal EV, Touwslager F. Flexible electronic-paper active-matrix displays. Journal of the Society
for Information Display 2005;13:181–185.
5. http://www.digitaltrends.com/mobile/foldable-smartphone-screens-possible-in-2016/.
6. Khalilbeigi M, Lissermann R, Kleine W, Steimle J. FoldMe: interacting with double-sided foldable displays. Proc. Int. Conf. Tangible,
Embedded and Embodied Interaction 2012.
7. Lee JC, Hudson SE, Tse E. Foldable interactive displays. Proc. ACM sym. User Interface Software and Technology 2008.
8. Tan D, Kumorek M, Garcia AA, Mooney A, Bekoe D. Projectagami: A foldable mobile device with shape interactive applications. Proc.
ACM Conf. Human Factors in Computing Systems 2015:1555-1560.
9. Chen TY, Chen SH, Chen HY, Shih WK. An enhanced user interface design with auto-adjusting icon placement on foldable devices. Proc.
IEEE Int. Conf. Systems, Man, and Cybernetics 2014:1797-1800.
10. Vaquero D, Turk M, Pulli K, Tico M, Gelfand N. A survey of image retargeting techniques. Proc. SPIE 7798, Applications of Digital Image
Processing XXXIII 2010.
11. Rubinstein M, Gutierrez D, Sorkine O, Shamir A. A comparative study of image retargeting. ACM Trans. Graph. 2010;29. 12. Shamir A, Avidan S. Seam carving for media retargeting. Commun. ACM 2009;52:77-85.
13. Avidan S, Shamir A. Seam carving for content-aware image resizing. ACM Trans. Graph. 2007;26.
14. Wang YS, Tai CL, Sorkine O, Lee TY. Optimized scale-and-stretch for image resizing. ACM Trans. Graph. 2008;27.
15. Lin SS, Yeh IC, Lin CH, Lee TY. Patch-based image warping for content-aware retargeting. IEEE Trans. Multimedia 2013;15:359-368. 16. Sung YH, Tseng WY, Lin PH, Kang LW, Lin CY, Yeh CH. Significance-preserving-guided content-aware image retargeting.
Springer-Verlag in series of Smart Innovation, Systems and Technologies, Advances in Intelligent Systems & Applications 2012.
17. Fang Y, Lin W, Lee BS, Lau CT, Chen Z, Lin CW. Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Trans. Multimedia 2012;14:187-198.
18. Achanta R, Hemami SS, Estrada FJ, Süsstrunk S. Frequency-tuned salient region detection. Proc. IEEE Computer Society Conf. Computer
Vision and Pattern Recognition 2009;1597-1604.
19. https://www.itri.org.tw/chi/Content/Publications/contents.aspx?SiteID=1&MmmID=2000&MSid=621024032135465165
20. Hsu CC, Lin CW, Fang Y, Lin W. Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss. IEEE J. Selected Topics in Signal Processing 2014;8:377-389.
