IDENTIFICATION AND POSITIONING BASED ON MOTION SENSORS AND
A VIDEO CAMERA
Jun KAWAI
Information System Laboratory Graduate School of Doshisha University
Kyotanabe, Kyoto, Japan email: [email protected]
Kimio SHINTANI Early Childhood Education
Tokiwakai College Hirano-ku, Osaka, Japan
Hirohide HAGA and Shigeo KANEDA Information System Labolatory Graduate School of Doshisha University
Kyotanabe, Kyoto, Japan
ABSTRACT
In recent years, many papers have demonstrated “ubiqui-tous” services for indoor human location. Under the con-ventional method of position detection, the major approach is to generate electromagnetic waves or light rays as in
Ac-tive Badge or RFID tags. However, in the case of practical
applications for infant education, parents are wary of meth-ods that employ radiated electromagnetic waves. In this article we propose a high-precision positioning system for children in a room, using only a combination of image data and passive motion sensors. We evaluate some alternative implementations experimentally. These evaluations probe whether the combination of a pedometer approach to ana-lyze motion sensor data and moving-object detection from video camera images is the best combination with respect to ease of operation and accuracy. The proposed method cannot yet analyze data in real time. It may, however, be possible to apply the technique to various fields with little need for expansion, based on location information.
KEY WORDS
Position detection, image processing, sensor, activity track-ing, infant education, ubiquitous computing
1
Introduction
Ubiquitous application systems based on position informa-tion are currently under investigainforma-tion. One example of re-search in progress relates to appliances based on indoor user positioning[1]. The mainstream approach in exist-ing systems includes detectexist-ing one’s position by generat-ing electromagnetic waves as in Active Badge[2] or RFID tags[3]. In addition to networked appliances, there is the growing field of applying information about a person’s door position. It would appear that the application to in-fant education in nursery schools or child-care centers is one of them. Actually, research into a system that speci-fies the location of children by RFID tags is already well established[4].
However, there are some issues regarding the appli-cation of ubiquitous technology for infant eduappli-cation. First, the positioning accuracy achieved by applying electromag-netic waves is under dispute. In the case of using infrared (IR) waves, as with Active Badge, a person’s position is not
Figure 1. Image of Our Proposal System
always detectable; depending on pose, especially for unsta-ble children. Also, tags using weak radio signals may come under the influence of noise from other electronic devices. Moreover, the location of tags is sometimes mistaken due to the diffraction and reflection of electromagnetic waves. Therefore, positioning systems that depend on electromag-netic waves are not necessarily accurate.
The other issue is of parents’ consent. It is proved that these weak electromagnetic waves cause no health prob-lems. However, parents and school teachers often do not accept the usage of IRs and electromagnetic appliances be-cause of their emotional reason.
We use a combination of motion sensors[9] and video cameras, which are totally passive1, to detect users’ loca-tions, and propose a method of identifying and position-ing usposition-ing the data from these devices. Figure 1 shows a schematic diagram of our proposed system. To evaluate the combinations of camera and sensors, we have built some prototypes and compared their results. The evaluation re-sult indicates that the combination of the pedometer ap-proach for motion sensor data and the moving object detec-tion technique for video camera image is the best selecdetec-tion with respect to operation and accuracy. There are no elec-tromagnetic waves emitted under our proposed method.
1In general, passivity means the object does not have built-in
batter-ies, but in this paper passivity means it is a radio-free device, similar to “passive radar.”
Our prototype system is able to identify and detect the position of an object, especially a child in a room or school-yard, with a margin of error under 10 cm. Such highly detailed position data can be used to analyze children’s ac-tivities and mutual association. This makes it a powerful tool for investigating infant education. Furthermore, the daily collection of data about children’s positions can help teachers review their teaching. Moreover, the system offers guidance for teachers in preparing teaching plans.
2
Positioning System in the Field of Infant
Education
2.1
Expected Information Services for Infant
Education
Obtaining position data is one of the major themes of ubiq-uitous computing. Unlike the case of outdoors, where GPS is available, other measures are required to report position in case of being indoors. Many services that depend on po-sition information are currently under development, espe-cially in the field of information-gathering appliances, and some context-sensitive service applications will be devel-oped based on position detection services.
Recently the technique of position detection was ap-plied to a kindergarten, with the first commercial trial be-ing a service that locates children usbe-ing RFID tags. This service provides sequential information on a child’s loca-tion. For example, the information answers questions like, “how many times did he/she go to the restroom today?” or “which room is he playing in now?” This system was ap-plied to Yoshizuka Kindergarten in Fukuoka Pref., Japan.
A system for acquiring children’s positions has the potential to be applied in several fields. For example, this system is a useful tool for conducting research for infant education. Because such large human resources are re-quired to conduct such research, few experiments have been done. However, by applying sensor technology, we expect to make this task easier using location information.
We started investigating the application of ubiquitous technology to the field of infant education. Our target is the automatic detection of group action and the automatic generation of teaching records.
2.2
Conventional Positioning Systems
Existing positioning systems are listed in Table 1, along with evaluations of each method’s advantages and disad-vantages for nursery schools. We take particular note of the passivity of sensing devices. At the time of writing, there have been no clearly demonstrated negative effects of electromagnetic waves on humans. Some countries de-clare that electromagnetic waves are safe within accept-able safety levels. However, high-intensity radio waves are not emitted near kindergartens or primary schools; in fact, in Sweden, kindergartens and primary schools are moved
Table 1. The Drawbacks and Advantages of Location Sens-ing Technologies
Technology Advantage Drawback Active
Badge
Easy operation Low accuracy, emitting radio waves
PHS Reliable tracking Low accuracy, emitting radio waves Marker Tracking Passive method, high accuracy Identification im-possible, depend on lighting condi-tion Barcode Recognition
Passive method Low recognition rate, hiding bar-code Active Marker Identification possible, clear lighting condition LED output, complicated equipment Motion Sen-sor
Passive method Low accuracy, drifting offset, data transfer
from areas where levels of electromagnetic radiation are high. On the other hand, an accuracy within 10 to 20 cm is required in position detection to actualize the services in a kindergarten.
For the reasons stated above, there is promises in the use of “acceleration sensors” and “marker tracking.” The idea of our proposed method is identification and position-detection with the combination of two proceedings. We aim to realize a passive, highly accurate and identifiable positioning system, combining sensor data analysis with image processing.
3
A New Indoor Positioning without Energy
Radiation
3.1
Overview of the Proposed Method
We proposed an information integration-type positioning method, using camera images and sensor data. This oper-ates according to the following steps.STEP1 Collect tri-axial acceleration data from a motion
sensor. These data include a sensor ID, and we apply these data to identification in STEP 3.
STEP2 Take the position data from camera images. These
data have accurate position data, but it is not possible to identify the moving object.
STEP3 Integrate the two sets of information, thus giving
Figure 2. Flowchart of Server Process
Table 2. Alternatives of Processing Process
Sensor data processing
Acceleration integration Pedometer method Image processing Marker tracking
Moving object detection
The information from the motion sensor will be able to send by using radio waves in real time. To evaluate un-der the condition of emitting no electromagnetic waves, we conduct experiments using accumulated data in the mem-ory, even though our proposal is not dependent on the data acquisition method.
Next we describe the details of the system. Figure 2 shows the process flowchart for the server computer. At first, the system collects the acceleration data with object IDs from sensors attached to each child’s waist. These data will be processed to obtain positioning data. There are many ways to do this. One is numerical integration to trans-late acceleration data into position data. The other one, like a pedometer does, notices up-and-down movements such as walking, and calculates the position from the pedometer’s output from and the object’s stride. At the same time, the system receives position data on moving objects by pro-cessing camera images. We apply and compare two meth-ods – marker tracking and moving-object detection – in this evaluation.
Finally, the system collates data from sensors and camera images, and concurrently locates the objects’ po-sitions using the sensor IDs. Table 2 shows the substances of the subroutine in Fig. 2. The system is constructed by combining these methods. In the next section we describe the details of processes used here.
Figure 3. Image of Marker LED (Light Adjusted)
3.2
Data Acquisition
3.2.1
Sensor Data Processing
The following two approaches were selected to process the motion sensor data.
Acceleration Integration
The axis of acceleration sensor swings with the ob-ject’s action. To calibrate the axis with images taken using a fixed camera, the system corrects the axes using the ro-tation data from the sensor’s initial location. The sensor’s coordinates are inputted to the Euler angle (Z-Y-X coordi-nates). The system translates Euler coordinates into camera coordinates. It then rotates the acceleration vectors along camera axes, and integrates the acceleration data to posi-tioning data.
Pedometer Method
The acceleration integration method comes under the influence of a drifting offset. Consequently, data verifica-tion would fail due to the accumulaverifica-tion of errors in a long experiment. To solve this problem, we have adopted the pedometer method to count the number of steps resulting from up-and-down actions such as walking. In addition, the system detects an object’s position by direction of move-ment and moving distance per unit of time.
3.2.2
Image Processing
The following two approaches were selected to calculate the position of moving objects from the TV camera image.
Marker Tracking
In our evaluation experiment for adults, we used red LED on headsets as markers, and recorded from above. Figure 3 shows an image from the experiment. The marker-tracking method has higher accuracy than the moving-object detection that will be shown in the following, be-cause the marker always lies in the center of an object. This method does, however, have some problems: the marker is an obstacle for children, and it is difficult to record from
Figure 4. Image before Processing
Figure 5. Image after Process 2
“above.” This forms the basis of comparison with moving-object detection.
Moving Object Detection
Although the above marker-tracking method has high accuracy and reliability, children have to wear the marker device. “Hands free” is desirable from the viewpoint of easy maintenance, but what we really need is a high-accuracy position detection method without markers. Thus, we implement the following processes2.
Process 1 Obtaining the difference between every two
ad-jacent pictures.
Process 2 Binarization, smoothing, and interference
rejec-tion.
Process 3 Labeling the consecutive area.
Process 4 Calculation of the coordinates for each label. Process 5 Integration of the neighboring area.
This process allows us to obtain the centrobaric posi-tion of objects that are moving from the background image. Figure 4 is an image recorded before these processes, and Fig. 5 is an image taken after Process 2, with the white
2If we can employ far-infrared cameras, the positions of humans can
be easily acquired. However, far-infrared cameras are far too expensive for the childhood education domain.
Table 3. Experiments
No. Sensor data processing Image processing 1-1 Acceleration integration Marker tracking 1-2 Pedometer method Marker tracking 2-1 Acceleration integration Moving-object detection 2-2 Pedometer method Moving-object detection
areas showing moving objects. The system abstracts not only people, but also columns and partitions due to changes in the light conditions. On the other hand, integration of the neighboring area is a measure to split one object into a number of objects. This method outputs the centrobaric position of moving objects; However, output data also in-cludes other objects like columns. For this reason, some artifices are required for data mining in order to apply this method effectively.
3.3
Data Collation
Two methods of data collation are required. One is for marker-tracking results, and another is for moving-object detection results. Because the marker-tracking result is highly accurate with respect to position, the aim of data col-lation is the identification. We adopt the coefficient of cor-relation to compare the sensor data and the marker-tracking results[7]. The system computes the coefficient of correla-tion in all assortments, and the highest value of the coeffi-cient of correlation is deemed the correct combination.
The moving-object detection result contains the error data, as shown in Fig. 5, and the result is discrete data because the object vanishes when it is not moving. Con-sequently, we adopt the summation of coefficient of partial correlation for each continued data in order to join the dis-crete data. The result of data collation is shown in Section 4.2.
4
Results of Experimental Evaluation
4.1
Experimental Conditions
We conducted four separate experiments and compare their results. We attempted different combinations of the pro-cesses described in preceding section, with the experiment types shown in Table 3. Five subjects (students) partici-pated in the experiments, and one of them had a sensor unit connected to a laptop PC. The PC recorded sensor data with time stamps. Each subject put on headgear with an LED for marker tracking. We took images with an overhead camera from a height of 7 m. The field of activity was limited to the range of the camera, and the subjects moved freely within that range. Each experiment lasted 5 min.
Figure 6. Sensor Holder’s Position (x-coordinate) by Ac-celeration Integration
Figure 7. Sensor Holder’s Position (x-coordinate) by Marker Tracking
4.2
Experimental Results
Experiment 1-1 (Acceleration Integration and Marker Tracking)
We show the positioning results by performing dou-ble integration of acceleration data in Fig. 6, and present the position of the sensor holder by marker tracking in Fig. 7. The result of marker tracking does not contain the identification information. Therefore, to identify the sensor holder, we compare the coefficients of correlation in Table 4. In general, the coefficient of correlation values are com-pared with their absolute values. However, negative val-ues are expressed as inverse correlation, which means that the two experimental subjects go just in the opposite di-rection, so we ignore negative values. According to Table 4, marker-tracking ID 1 is the best value on x-coordinate and y-coordinate; ID 1 is the sensor holder. Although this
Table 4. Result of Experiment 1-1 ID Correlation X Correlation Y Result
1 0.259 0.265 °
2 0.100 -0.003 ×
3 -0.220 0.035 ×
4 -0.080 -0.041 ×
5 0.076 -0.081 ×
Figure 8. Sensor Holder’s Position (x-coordinate) by Pe-dometer Method
experiment was successful, the coefficient of correlation value was not high because of the sensor’s drifting offset.
Experiment 1-2 (Pedometer Method and Marker Tracking)
This experiment is an improvement on the sensor pro-cessing to pedometer method. Figure 8 shows the position data of the sensor holder, while Table 5 presents the coeffi-cient of correlation values. In comparison with Experiment 1-1, this experiment was not affected by the sensor’s drift-ing offset; therefore, the result is acceptable with respect to the x-coordinate. However, there was little difference for the y-coordinate value in regard to ID 2. This is thought to be an aftereffect of holding the length of stride constant. The experimental subjects were often stepping on the same spot or sidling to avoid collisions, which was a problem that seriously affected the system as it attempted to target the children. To solve this problem, we need to find a way for the system to automatically determine the length of stride.
Experiment 2-1 (Acceleration Integration and Moving Object Detection)
This experiment employed moving-object detection for image processing. In contrast with marker tracking, output data are discontinuous, as shown in Fig. 9. To joint and compensate for these discontinuous data, we joined
Table 5. Result of Experiment 1-2 ID Correlation X Correlation Y result
1 0.871 0.296 °
2 0.159 0.226 ×
3 -0.060 -0.248 ×
4 0.208 -0.088 ×
5 0.167 -0.000 ×
Figure 9. Centrobaric Position (x-coordinate) of Moving-Object Detection
them with continuous data and removed the error data with the preparation. Figure 10 shows the result, which includes the acceleration integration result and data selected by col-lating discontinuous data (see Section 3.3). The positive area of the y-axis is about the y-coordinate, while the nega-tive area of the y-axis is about the x-coordinate. According to the figure, there are correct results between 20 and 30 sec., and between 40 and 50 sec., but as for the rest, the sys-tem chooses the wrong data during the collation stage. We assume that the reason for the mistake is the inconsistency in the sampling rate. The sampling rate of video camera is about 33 ms (30 fps), and the sampling rate of the mo-tion sensor is about 10 ms. Because of this inconsistency, data collation is badly managed with little time lag. The other reason is incomplete discontinuous data joining. To solve these problems, we examine how the time stamp will be implemented to the sampling-rate inconsistency, and the object chasing, which labels an object in the same way, to the problem of discontinuous data.
Experiment 2-2 (Pedometer Method and Moving Ob-ject Detection)
In this experiment we combined the pedometer method and moving-object detection. The results are shown in Fig. 11. Experiment 2-2 results were more ac-curate than those of Experiment 2-1 because the results for sensor data and image processing were approximately
Figure 10. Result of Experiment 2-1
Figure 11. Result of Experiment 2-2
equal. However, this combined method disrupted the y-coordinate data at around 37 sec. We suppose that the cen-trobaric position does not always signify the subject be-cause the system does not detect the unmoving body part.
5
Discussion
In this section, we compare each process, sensor data pro-cessing and image propro-cessing.
Comparing Experiments 1-1 and 1-2, the results indi-cate that the pedometer method is superior to acceleration integration, since that is not under the influence of sensor’s drifting offset. However, that does depend on the subject’s stride length. To obtain the position information only from the motion sensor, we will refine the pedometer method, for example, by automatically determining the stride length.
On the other hand, marker tracking can map the exact location of subjects and collate data directly by the coeffi-cient of correlation. This image-processing is better than others from the view of data-processing simplicity. How-ever, wearing the marker causes some discomfort to the subject. Consequently, we take particular note of a moving-object detection method that can obtain position informa-tion without resorting to extraordinary devices. In the case
of using marker tracking in the image-processing stage, the results are highly accurate. However, the moving-object detection process outputs discontinuous data that are diffi-cult to use for practical purposes. To solve this problem, we have updated the process of moving-object detection, where the system converts the data into consecutives to approximate the marker-tracking results as far as possible. Additionally, we use the coefficient of correlation for data collation. This assessment is good for continuous data, but for discontinuous data, we will discuss a better indicator of collation. In accordance with the results of all experiments and future progress, we consider that the combination of the pedometer method and moving-object detection is the best.
In the above implementation, the motion sensor was connected to a note PC, but this was too large for infant to carry around. Therefore, we must develop a sensor unit to be worn on the child’s waist. Such a device calls fordura-bility, enough data capacity for one ?day’s activity, and a small size that does not limit the wearer’s activity.
Next we discuss a unit that employs a computer board for embedded systems[8] and motion sensors. In the case where some subjects carry a sensor, every sensor collects the data in each clock. This makes it difficult to set the time of the data. One solutions is to adopt time-stamp signals, although there is a trade-off between time accu-racy and emission of the electromagnetic waves. Our pro-posed method aims to obtain location information without the emission of electromagnetic waves. If we do use elec-tromagnetic waves as time stamps, we should further ex-amine its sending cycle and power.
In this work, we set one camera above the activity area, but in a real field, it would not always be possible to set the camera directly above. Additionally, the activ-ity area would not be as limited as it was in the kinder-garten. Moreover, only one camera causes limitations in practical use. Consequently, in future work we plan to use multiple cameras, though there is some difficulty in-volved, like parameters passing between cameras. Future work will involve using stereo cameras for shooting in the near-horizontal plane, and, moreover, we are eying the pos-sibility of using a far-infrared cameras for moving-object detection. This type of camera ignores background noise such as outside light.
6
Conclusions
We focused on position detection using large sensors, and proposed a position-detecting to realize many services for kindergartens and nursery schools. The proposed method could accurately detect each child’s position and identify him/her by collating sensor data and image processing. The most notable features of that are as indicated below.
• No emission of electromagnetic waves.
• Identification by collating sensor data image data.
• Marker free (hands free) for image processing.
To implement the proposal, we experimented with combinations of sensor data processing and image pro-cessing. The acceleration integration and pedometer ap-proaches were selected for sensor data processing, while marker tracking and moving object detection were em-ployed for image processing. The experimental evaluation results from the combination of the pedometer method and moving-object detection indicate that this is the most suit-able combination for kindergartens and nursery schools.
References
[1] Y. Tajika, T. Saito, K. Teramoto, N. Oosaka, M. Isshiki M, Networked home appliance system us-ing Bluetooth technology integratus-ing appliance con-trol/monitoring with Internet service, IEEE Trans.
Consumer. Electronics., 49(4), 2003, 1043-1048.
[2] R. Want, A. Hopper, V. Falcao, J. Gibbons, The Ac-tive Badge location system, ACM Transactions on
In-formation Systems, 10, 1992, 91-102.
[3] H. Hayashi, T. Tsubaki, T. Ogawa, M. Shimizu, Asset Tracking System Using Long-life Active RFID Tags,
NTT Technical Review, 1(9), 2003, 19-26.
[4] H. Takasugi, Recording children’s activities us-ing a radio tag system (In Japanese), Work-shop by Kyushu Private Kindergarten Federation,
http://www.yoiko.ed.jp/tokusyu/0180-10.htm, 2003. [5] S. Nakagawa, K. Soh, S. Mine, H. Saito, Image
Sys-tems Using RFID-Tag Positioning Information, NTT
Technical Review, 1(7), 2003, 79-83.
[6] J. Hightower, G. Borriello, Location systems for ubiq-uitous computing, IEEE Computer Mag., 34(8), 2001, 57-66.
[7] E.W. Weisstein, MathWorld, Wolfram Research, Inc., http://mathworld.wolfram.com/
[8] Intrinsyc Software International, Inc., http://www.intrinsyc.com/products/cerfcube/
[9] NEC TOKIN Corp., http://www.nec-tokin.com/english/product/pdf dl/3DMotionDK e.pdf
