Implementation of ultrasonic touchless interactive panel using the polymer-based CMUT array

Implementation of ultrasonic touchless interactive

panel using the polymer-based CMUT array

Te-I Chiu, Hsu-Cheng Deng, Shu-Yuang Chang, Shi-Bing, Luo

Industrial Technology Research Institute, ITRI Hsin-Chu, Taiwan, R.O.C.

Email: [email protected] Abstract—An innovative touchless interactive panel based on the

ultrasound echo-location principle is proposed. The fabrication process of the polymer-based capacitive micro-arrayed ultrasonic transducer (P-CMUT) array is described. A method is then proposed for determining the position of the user’s finger relative to the screen based upon the time of arrival (TOA) of the acoustic echo signals reflected from the fingertip and the geometrical relationship between the transmitter and the receivers. Various signal processing techniques and supporting algorithms are proposed for obtaining reliable and precise fingertip position estimation. The feasibility of the interactive panel is verified by constructing a prototype model and performing a series of representative pointing operations using a commercial computer application.

I. INTRODUCTION

Touch-screen technology, which was introduced by the Apple iPhone in 2007 [1], enables a far more intuitive and convenient human-machine interface (HMI) than that made using a mouse or a physical keyboard. As shown in Fig. 1, the iPhone uses virtual buttons and controls that appear on its screen. As someone touches the screen, the featureless rectangle becomes an interactive surface. One can place his fingertip on an on-screen arrow and slide it from left to right. When his fingertip moves, the arrow moves with it. With the touch-screen technology, the effective view area is enlarged. For some people the interaction between the human finger and an on-screen image, and its effect on the iPhone’s behavior was more amazing than all of its other features combined. Therefore, touch-screen technology has been selected as the standard input device for many computer systems (Laptop, smart phones, LCD TV, self-checkout kiosk, and etc.). However, touch-screen technology is not without drawbacks. For example, because the user’s finger (or pointed stylus) must touch the screen to effect a response, only sliding or clicking actions can be performed. Furthermore, the screen is liable to become scratched or otherwise damaged following prolonged use. As a result, innovative touchless control interfaces that can track hand movements in three-dimensional (3D) space and even recognize 3D hand gestures have been avidly studied.

Figure 1. The Apple iPhone with the touch panel. [1]

In Steven Spielberg’s futuristic Minority’s Report movie, Tom Cruise turns on a wall-sized digital display simply by raising his hands. Like an orchestra conductor, he gestures into empty space to pause, play, magnify and pull apart videos and photos using sweeping hand motions. Minority’s Report takes place in the year 2054. The touchless technology that it demonstrates may arrive many decades sooner. In fact, to go beyond touch and create touchless control interface technology, devices that employ the principles of background light, infrared, image recognition and ultrasound have already been proposed for implementation. The Sharp Company developed a new LCD panel with photo sensors built into each pixel of the panel (Fig. 2(a)) [2]. The photo sensors indicate the position of the finger in proximity to the panel by detecting the variance in background light intensity covered by the finger. However, the photo sensor cannot accurately measure the distance between the finger and the panel. Israel’s 3DV systems® has developed a 3D video imaging technology (Fig. 2(b)) [3] that enables users to control computer applications using any intuitive hand or head gestures, making it ideal for PC-based gaming and web-conferencing. 3DV systems® have been purchased by Microsoft for Project Natal for the Xbox 360. The 3D imaging technique employs the ZCamTM _{web camera with an} infrared laser source providing accurate depth information with high depth resolution capability. However, the ZCamTM

must be appended to an existing computer system because of its dimension scale. Infrared light is very sensitive to background light and the ambient environment. In 2008, a laptop released from Toshiba’s Qosmio G55 (Fig. 2(c)) [4], offered a 3D input method powered by a camera-based visual gesture control system. However, the video analytical method could not provide accurate depth information with high resolution. The complexity of the background image may reduce the recognition efficiency.

(a) (b) (c)

(d) (e) (f)

Figure 2. The touchless human machine interface using different kind of technology [2-7].

Figure 3. HMI system comprising touchless interactive panel with ultrasonic transducer device.

Technology that could detect 3D hand gestures using ultrasound waves was first patented in the US by NOKIA (Fig. 2(d)) [5]. The ultrasound sensors on the handsets could pick-up the user’s movements and transform them into various actions. A prototype for this new touchless Nokia handset has not yet been realized. Elliptic Laboratories (Norway) has developed a suite of touchless software and hardware HMI for 3D pointing applications that have been installed and in use at hospitals in Oslo (Fig. 2(e)) [6]. Navisense (USA) also developed a miniature touchless pointing device that has been installed with cell phones and PC application software. (Fig. 2(f)) [7]. Based on ultrasound echo-location technology, the position of a finger is recognized in the air from up to 1-2 meters away from the ultrasonic transducers. Bulky sensors are placed on, next to or separate from a monitor. Accordingly, an innovative idea involving a HMI system comprised of a touchless interactive panel based on the P-CMUT technology is proposed. The

fabricated ultrasonic transducers are embedded in the monitor. A scenario using the HMI system is schematically shown in Fig. 3. It is expected that this interactive panel enables the user to execute all of the functions of a commercial software application using different finger gestures without the need for additional peripheral hardware, such as a mouse or keyboard.

II. FABRICATION OF THE P-CMUTDEVICE The P-CMUT has exists for decades and has been wildly used for the excitation and detection of acoustic waves [8,9]. A typical example of the P-CMUT array and its operating scheme is shown in Fig. 4. The constituent components includes a membrane, a passivasion layer (optional), upper and ground electrodes, an insulation layer (optional), a edge post, and a supported substrate. Each membrane is fixed peripherally and is suspended at a tiny distance from a bottom electrode with an air cavity between the electrodes. The passivasion layer is designed to protect the upper electrode from ambient corrosion during operation. The insulation layer prevents short-circuiting when the membrane collapses. The membrane generates ultrasonic waves by applying an alternative signal superimposed onto a DC bias on the top electrode. Reflected wave reception is detected by measuring the capacitance difference as the membrane is vibrated.

Figure 4. P-CMUT operating scheme and constituent components. The P-CMUT was fabricated using a highly reliable micro-machining surface process at a temperature of 350°C or less. This process is compatible with fabricating transmitting/sensing electronic circuits on the same substrate. To render the P-CMUT totally transparent, the membrane, electrodes and substrate are fabricated using polyimide (PI), Indium Tin Oxide (ITO), and glass substrates, respectively. The relevant fabrication process steps are depicted in Fig. 5. The process is illustrated as follows:

(A) PVD sputtering thin ITO layer as the bottom electrode on the glass substrate;

(B) Define the cavity area;

(C) Electroplate the copper sacrificial layer; (D) Define the rails using wet-etching;

(E) Spin coat liquid-polyimide as the membrane and open the etching holes;

(F) Metallizing the thin ITO layer as the top electrode; (G) Spin coat liquid-polyimide as the passivasion layer; (H) Pattern the top electrode area;

(I) Remove the sacrificial layer using wet-etching and supercritical fluid replacement techniques;

(J) Plant Ni bulk pads onto the electrodes and make Al wire-bonding connections to the PCB.

(A) (I) (H) (G) (F) (E) (D) (C) (B) Substrate (Glass) Electrode (ITO) Membrane (Polyimide) Sacrificial layer (Cu) Bulk pad (Ni) Passivasion (Polyimide) PCB supporter

(J)

Figure 5. Surface micromachining process of the P-CMUT array. An appropriate high aspect-ratio wet-etching method with the supercritical fluid replacement technique was adopted in this study to remove the sacrificial layer so the membrane and cavity can be precisely defined to minimize the sticking probability during membrane release. Fig. 6(a) shows the fabricated M×N arrays of 0.5mm x 0.5mm square element composed of 1020 cells on the transparent 4-inch glass substrate. The scanning electron microscope (SEM) images of the P-CMUT cell with 350X and 1200X of magnification views are shown in Figs 6(b) and 6(c), respectively. The shape of a single cell is a hexagon that is tangential to the periphery of a circle with ~142μm in diameter. The thickness of the membrane is 1.9μm. The ITO top and bottom electrodes measure 110μm in diameter and 1000Å in thickness. The membrane is separated from the substrate by a 2.5μm high cavity. The thickness of the glass substrate is 1000um. Table 1 outlines dimensional parameters of the P-CMUT cell. P-CM UT array on Glass substrate 350X 1200X (a) (b) (c)

Figure 6. Photographs of the P-CMUT array. (a) Arrays of the P-CMUT element on a glass substrate. (b) SEM photo of membrane cells with 350X magnification. (c) SEM photo of a membrane cell with 1200X magnification.

TABLE I. DIMENSIONAL PARAMETERS OF THE P-CMUT CELL.

III. 3DPOSITIONING USING ULTRASOUND ECHO -LOCATION PRINCIPLE

Echo-location, also called biosonar, is the biological sonar with ultrasound produced by several echo-locating animals such as dolphins, most bats, and most whales. Echo-locating animals emit ultrasound waves out into the environment and listen to the echoes from the waves that return from various objects in the environment. Unlike some sonar that relies on an extremely narrow beam to localize a target, animal echolocation relies on multiple receivers. Thus, echo-locating animals have two ears positioned slightly apart. The echoes returning to the two ears arrive at different times and at different loudness levels, depending on the position of the object generating the echoes. These animals employ echoes to locate, range, and identify nearby objects for navigation and foraging. A touchless interactive panel is developed in this research based on the ultrasound echo-location principle.

Fig. 7 schematically shows the operating scheme for the touchless interactive panel implemented using groups of single ultrasonic transmitters (element E) and three ultrasonic receivers (elements A, B and C) on P-CMUT array. The three receivers are located at different apogees of an equilateral triangle ΔABC. The transmitter E is positioned at the

Components Parameter value (μm)

Membrane diameter 142

Membrane thickness 1.9

Top electrode diameter 110

Top electrode thickness 0.1

Cavity gap height 2.5

Bottom electrode 0.1

Post thickness 20

geometric center of the triangle _{ΔABC. The coordinates for} the transmitter E, receivers A, B, and C are defined as: E:( L0, 3,0); A:( L− ,0,0); B:(L,0,0); C:(0, 3L,0)[10]. The unknown coordinates for the fingertip P are to be determined as P (x,y,z). To validate our original concept in advance, the 40-kHz air-coupled PZT transducers are used in terms of the P-CMUT array.

Figure 7. Schematic illustration of ultrasonic pulse-echo transmitting/receiving scheme based on P-CMUT array.

Time (sec) A m p lit u te (m V ) 0 0.0005 0.001 0.0015 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 Input Signal by 1-Tx 40kHz Tone burst (10-cycles)

TB

TA

TC

Received Echo Signal by 3-Rx(s)

Figure 8. Time-based variation of transmitted tone-burst signal and reflected echo signals.

Operating with the PZT transducers, the transmitter emits a 10-cycle square-wave tone-burst of 40-kHz. The echo signals are scattered from the fingertip (P) and detected by receivers A, B and C. Fig. 8 shows the maximum amplitude correlation between the transmitted pulse signal and the reflected echo signals over the signal fly time. Accurate estimates of the TOA at receivers TA, TB and TC can be obtained using the sliding-window averaging method and the matched filtering method. The matched filtering method processes a known piece of signal that is imbedded in noise. The filter will maximize the signal to noise ratio (SNR) of the signal being detected so that the maximum can be taken out clearly.

Based upon the geometric relationship between the transmitter and the receivers, the ultrasound flight paths from the transmitter to the three receivers can be expressed as

1. Flying path DEPA of E to P to A

) ( ) . 3 ( 2 2 2 2 2 2 _y L _{z x L y z} x DEPA= + − + + + + + (1) 2. Flying path DEPB of E to P to B

) ( ) . 3 ( 2 2 2 2 2 2 _y L _{z x L y z} x DEPB= + − + + − + + (2) 3. Flying path DEPC of E to P to C

) ( 3 ) . 3 ( 2 2 2 2 2 2 _y L _{z x y L z} x D_EPC= + − + + + − + (3)

Having experimentally obtained the TOA at each receiver, the flight paths from the transmitter to the individual receivers are obtained by multiplying the respective TOA by the velocity of sound in air. The three flight path terms: DEPA, DEPB, and DEPC can be regarded as known parameters. Rearranging eqs. (1), (2), and (3), a set of algebraic equations are obtained, namely,

( , , ) ( 3) ( ) , 2 2 2 2 2 2 1 z x L y z DEPA L y x z y x f _{= + − + + + + + −} (4) ( , , ) ( 3) ( ) , 2 2 2 2 2 2 2 z x L y z DEPB L y x z y x f _{= + − + + − + + −} (5) ( , , ) ( 3) ( 3 ) . 2 2 2 2 2 2 3 z x y L z DEPC L y x z y x f = + − + + + − + − (6) To determine the actual coordinates P (x,y,z), the Newton’s method (also known as the Newton-Raphson iterative method) is applied to successively find better approximations to the zeroes (or roots) of the nonlinear real-value functions. The idea behind the method is as follows: when one starts with an initial guess root which is reasonably close to the true root, the function is approximated using its tangent line (which can be computed using calculus), and one computes the x-intercept of this tangent line. This x-intercept will typically be a better approximation of the function's root than the original guess. This method is then iterated. The iterative search process is continued until calculated root convergence has been obtained. To solve the roots of the set of algebraic equations (4), (5), and (6) using the Newton-Raphson method, the iterative algebraic equation is expressed as . , , ( , , ( ) , , ( 3 2 1 3 3 3 2 2 2 1 1 1 ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ − = ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ Δ Δ Δ ⋅ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ i i i i i i i i i i i i z y x f z y x f z y x f z y x z f y f x f z f y f x f z f y f x f (7)

where i=1, 2, 3…, n which denotes the number of iterations. To obtain the sub-elements of the Jacobian matrix in Eq. (7), the partial differential derivatives of functions (Eqs. (4), (5), (6)) are derived as , ) ( ) 3 ( 2 2 2 2 2 2 1 z y L x L x z L y x x x f + + + + + + − + = ∂ ∂ (8) , ) ( ) 3 ( 3 2 2 2 2 2 2 1 z y L x y z L y x L y y f + + + + + − + − = ∂ ∂ ₍₉₎ , ) ( ) 3 ( 2 2 2 2 2 2 1 z y L x z z L y x z z f + + + + + − + = ∂ ∂ (10) , ) ( ) 3 ( 2 2 2 2 2 2 2 z y L x L x z L y x x x f + + − − + + − + = ∂ ∂ ₍₁₁₎ , ) ( ) 3 ( 3 2 2 2 2 2 2 2 z y L x y z L y x L y y f + + − + + − + − = ∂ ∂ ₍₁₂₎ , ) ( ) 3 ( 2 2 2 2 2 2 2 z y L x z z L y x z z f + + − + + − + = ∂ ∂ ₍₁₃₎ , ) 3 ( ) 3 ( 2 2 2 2 2 2 3 z L y x x z L y x x x f + − + + + − + = ∂ ∂ ₍₁₄₎ , ) 3 ( 3 ) 3 ( 3 2 2 2 2 2 2 3 z L y x L y z L y x L y y f + − + − + + − + − = ∂ ∂ ₍₁₅₎ . ) 3 ( ) 3 ( 2 2 2 2 2 2 3 z L y x z z L y x z z f + − + + + − + = ∂ ∂ ₍₁₆₎

With an initial fingertip coordinate guess xi, yi, and zi, we can obtain the Δxi, Δyi, and Δzi from Eq. (7) and the corrected coordinates xi+1, yi+1, and zi+1 for the next iteration input can be calculated as

xi+1=xi +Δxi, (17) yi+1=yi+Δyi, (18) zi+1=zi+Δzi, (19)

Within only a few iterations (5~10 iterations), an accurate solution for the 3-D coordinates P (x,y,z) of the user’s fingertip relative to the screen can be determined. The desired accuracy criterion ε is satisfied with the following equation,

. 10 % 100 ) ( ) ( ) ( 3 ) 1 ( 2 3 ) 1 ( 2 2 ) 1 ( 2 1 ) ( 2 3 ) ( 2 2 ) ( 2 1 ) 1 ( 2 3 ) 1 ( 2 2 ) 1 ( 2 1 − + + + + + + ≤ × + + + + − + + = f i f i f i f i f i f i f i f i f i ε (20) IV. IMPLEMENTATION OF THE ULTRASONIC TOUCHLESS

INTERACTIVE PANEL

Figure 9. The systematic block diagram of the ultrasonic touchless interactive panel.

Fig. 9 illustrates a systematic block diagram of the touchless interactive panel. The system consists of eight subsystems that are categorized and described briefly as follows.

1. Power supply system: This is used to supply high DC bias voltage for the P-CMUT operation and driving voltage for the circuit components.

2. Transducer system: The P-CMUT is designated as the core component for generating and detecting ultrasound wave signals.

3. Transmitting circuit system: The transmitting circuit is used to generate a small AC signal. A function generator and a RF switch ICs are required to produce the tone-bursts and the signals are amplified using a power amplifier IC. 4. Propagation system: The propagation medium for the

ultrasonic wave is air. The transmission loss resulting from the spreading and absorption should be considered.

5. Target scattering system: The user’s finger-tip is the scattering object. The scattered acoustic field is depends on the dimension, shape, and material properties of the object. 6. Receiving circuits system: The receiving circuit is used to

detect echo signals. The echo signals are magnified using the pre-amplifier. By switching the three signals using the analog multiplexer and filtering 110V/60Hz noise using the filter. Then after amplifying using the low-noise amplifier, the analog signals are converted to digital ones using the A/D converter.

7. Signal processing system: The echo-location positioning and the signal processing algorithms necessary for

computing the 3-D coordinates of the point P are conducted in FPGA/DSP MCU module.

8. Output and display system: A Java interface program is used to correlate the fingertip position with the requisite cursor movement or action commands in various computer applications.

Figure 10. Prototype of the ultrasonic touchless HMI system. Except for electronics, computers, and mobile phones, the ultrasonic touchless interactive panel is able to applied to several engineering applications. (1) For medical applications, it can be used as a pointing and control system for surgeons when using computers and instruments in a sterile area. (2) For entertainment applications, the user can play and interact with a TV game using his hands. (3) For factory environments, a workman can operate computers or machines in a dirt and pollutant environment without using gloves. (4) For electronic products, Fig. 10 presents an ultrasonic touchless HMI prototype system coupled with the Google Earth program. The experimental results successfully confirm the ability of the user to perform a variety of actions such as zoom, scroll/pan and tilt/rotate images simply by moving the finger in an appropriate manner. The reaction time per one input command is within 100 msec.

V. CONCLUSIONS

This study developed a touchless interactive panel that allows the user simply use his/her fingertip in the air to carry out tasks without need for a mouse, keyboard or touchscreen.

2D P-CMUT arrays are fabricated onto the panel with a transparent substrate using a surface micro-machining process. The ultrasonic echolocation principle is applied based on a one-transmitter and three-receiver operating mode. With the semi-analytical Newton-Raphson iterative algorithms, the 3D coordinate of the fingertip can be accurately and rapidly obtained. In the current implementation, a prototype device with 40-kHz air-coupled PZT ultrasonic transducers was successfully used to prove the touchless interaction concept. A demo manipulating Google Earth demonstrated that the ultrasonic positioning method could practical by measuring the time of arrival. More effort is needed to improve and realize the P-CMUT in terms of PZT transducers in the near future. Another topic of future study is to miniaturize the P-CMUT transmitting and receiving circuits onto a standalone PCB. The P-CMUT will be designed toward higher bandwidth (Δf>80%) and long-term reliability for generating stable signals. The current system is being extended in these directions.

ACKNOWLEDGMENTS

The authors would like to thank the Industrial Technology Research Institute (ITRI) and the Ministry of Economic Affairs (MOEA), R.O.C. for financially supported of the advanced research project, No. 8301XS3D10.

REFERENCES [1] Website: http://www.apple.com/iphone/ [2] Website: http://www.sharp.co.jp/ [3] Website: http://www.3dvsystems.com/

[4] Website: http://blog.laptopmag.com/hands-on-with-the-qosmio-g55 [5] USPTO Application Number: 20080204426.

[6] Website: http://www.ellipticlabs.com/confluence/display/elliptic/Home [7] Website: http://www.navisense.com/

[8] I. Ladabaum, X. C. Jin, H. T. Soh, A. Atalar, and B. T. Khuri-Yakub, “Surface Micromachined Capacitive Ultrasonic Transducers,” IEEE Trans. Ultrason. Ferroelectrics. Freq. Contr., vol. 52, pp. 678-690, 1998.

[9] D. Y. Chiou, M. Y. Chen, M. W. Chang, and H. C. Deng, “Finite element modeling, characterization, and optimization design for the polymer-typed capacitive micro-arrayed ultrasonic transducer,” Microsyst. Technolo., vol. 14, pp. 787–797, 2008.

[10] S. T. Lu, “Stragety for designing an innovative ultrasonic positioning system,” J. Sci. Eng. Technolo., vol. 3, no. 4, pp. 41–47, 2007.