1-4244-1144-0/07/$25.00 ©2007 IEEE.
EVALUATION OF A CONTEXT-AWARE MOBILE PEER-TO-PEER
NAVIGATION APPLICATION
Juuso Ohtonen Timo Koskela Nonna Kostamo Mika Ylianttila
University of Oulu University of Oulu University of Oulu University of Oulu
Oulu, Finland Oulu, Finland Oulu, Finland Oulu, Finland
ABSTRACT
In mobile networking, the use of context information in a versatile manner is emerging. Context information, which characterizes the situation of an entity, provides ease of use in mobile devices with limited input and output capabilities. A mobile peer-to-peer navigation application uses context information to verify this statement. It acts as a graphical user interface for group-based context-aware communication and as an enabler of inter-application cooperation. It uses an intelligent application platform for distributing context information. The analytical results consist of quantitative evaluation, involving functional testing and delay analysis, and a qualitative evaluation, involving user testing. The concepts of group-based context distribution and inter-application cooperation were found functional and useful.
I. INTRODUCTION
Mobile devices are becoming more capable to sense their environment, e.g. GPS chips are integrated to mobile devices. Context information available should be processed in a meaningful way and provided for the applications.
In [1], four different reasons are enumerated why usage of context information is difficult. First, context is acquired from unconventional sensors. Second, context must be abstracted to make sense for the application. Third, it may be acquired from multiple distributed and heterogeneous sources. Fourth, it is dynamic. To overcome these difficulties, a software framework – Context Toolkit − is proposed to make building context-aware applications easier.
Context information can be used to enhance the usage of mobile devices with limited input and output capabilities. First, information and services may be directly presented to the user based on his context information and preferences. Second, context information can be used to semiautomatic or automatic configuration of services, reducing user input. This truly brings forth the benefits of using rich context information in improving the user experience. [1]
In the following, the concepts of group-based context-aware navigation are introduced. The results from the quantitative and qualitative evaluations are presented. This is followed up by the analysis of the results and conclusions.
II. CONTEXT-AWARE MOBILE PEER-TO-PEER NAVIGATION
APPLICATION
A. Description of Application
NaviP2P has been presented in [2]. In short, NaviP2P is a context-aware and especially location-aware application, which runs in Symbian Series 60 mobile devices. The application utilizes location information received from a GPS device to present peer-to-peer (P2P) community members on
the map. The users can track and interact with community members in various ways ranging from mobile P2P file sharing to Internet telephony. By enabling application interaction, NaviP2P realizes the concept of application supernetworking, which means collaboration of applications that utilize a set of common functionalities for sharing contextual information as well as managing sessions and connectivities. Thus, NaviP2P practically acts as a graphical user interface for P2P community networking.
Fig. 1 exemplifies the concept of application supernetworking. The user has selected another user and wants to interact with him. There are three interactions available: Send message, Mobile P2P and VoIP. The question mark denotes that the application needed for interaction is installed on the local device, but not on the other user’s (remote) device. The information about supported interactions is one of the context items transferred between the users.
Figure 1: Configuration of user interface according to the context information about supported interactions. NaviP2P requires a platform to deliver both local and remote context information. For this purpose, Plug-and-Play Application Platform (PnPAP) is used. PnPAP is a modular middleware component that provides seamless use of multiple protocols and connectivities, as well as management of several types of context information. PnPAP manages the distribution and delivery of context information to requesting applications. The architecture overview of context distribution is presented in Fig. 2. [2]
Applications act as views to context information, representing the information to users along with the application-specific data, e.g. maps in NaviP2P. Context information is transferred between PnPAP nodes using SIP. SIP packet contains payload that is in interoperable XML format, Presence Information Data Format (PIDF) [3]. PIDF has been extended by us to cover presence, location, heart rate, altitude, speed and supported types of interaction. B. Context Distribution Sequence Description
Fig. 3 describes the sequence of context distribution. The initiating user (A) has NaviP2P running, and the other user (B) has any application running on top of PnPAP. User A selects Track user, NaviP2P initiates a request for visibility change. The request is forwarded from user A’s NaviP2P to user A’s PnPAP, and over the network to user B’s PnPAP. Next, user B’s PnPAP forwards the request to the PnPAP-compatible application. After receiving the acceptance from user B, user A’s PnPAP requests for context information from user B’s PnPAP. (Acceptance is only given once between peers, but can be cancelled later. Denial of service attacks are prevented by accepting requests only within the trusted peer group.) Finally, user B’s PnPAP starts sending context information to user A’s PnPAP.
Figure 3: Sequence description of context delivery. III. QUANTITATIVE EVALUATION:DELAY ANALYSIS
A. Previous Work
A preliminary NaviP2P delay analysis was provided in [2]. The delay of the following sequence is analyzed: a user starts the application, enters her user name and password, joins a peer group and selects the map view. The mobile phone and GPS device were assumed to be up and running and the software installed before the sequence. As a result, technical delay is approximately 17.4 seconds, which mainly consists of GPRS latencies in joining the peer group and downloading the map. User interaction delay is approximately 11 seconds, consisting mainly of entering user name and password in the login screen. A conclusion was drawn that faster connections (e.g. WLAN), reduced map size, or locally stored maps would reduce the technical delay and improve user experience. Also, automatically entered user name and password would reduce the user interaction delay dramatically after the initial login. The work in [2] does not encompass the delays of context distribution; this is corrected by the authors in the following.
B. Test Execution
The test sequence begins from the state where the peer group has been joined and a map has been displayed. The test measures the delays of the sequence described in Fig. 3 using two Symbian S60 phones (Nokia 6630) with PnPAP and NaviP2P installed. The SIP messages are routed via SIP Express Router in cellular network (GPRS).
There were 20 tests conducted during one test period by the leading author, who was experienced in using the application. The total delay between the different test runs including both user interaction (application waiting for the user to interact) and technical delays (system executing the requested operations) varied from 7.93 seconds to 10.69 seconds, mainly caused by the variation in the user interaction delay. The results, average delays, are summarized in Table 1.
The OK messages were utilised to calculate round trip times (RTT). Thus, the values in the table present the round trip times. However, from the total delay, the delays of OK packets can be subtracted, getting the actual technical delay marked on the last line of the table. Therefore, the sum of technical delays with RTTs is not equal to the total technical delay. Furthermore, the total delay describes the time between user A selecting the Request for context command and user B’s context information being displayed on the screen of user A. The grand total, including both user interaction and technical delays, is slightly over nine seconds (9.09 seconds). The delay of context packet transfer is continuous, since it occurs every time a context packet is transferred from B to A.
Table 1: Results of delay measurements.
Action User interaction
delay (s)
Technical delay (s) A selects Request for context, the
request is processed sent to B
~1 0.11 Visibility change is transferred
from A to B
2.44 B processed the packet and asks
user to accept the request ~1.3 0.17
Accept command is processed
and packet is sent back to A 0.13
Accept is transferred from B to A 1.80
A processes packet and sends request context packet
0.27 Request context is transferred
from A to B 1.59
Request is processed and a
context packet is sent back 0.31
The context packet is transferred
from B to A 2.36
A processes the context packet
and displays data for the user 1.33
Total ~2.3 6.79
C. Discussion
The delays are divided to user interaction and technical delay. The user interaction delay is approximately 2.3 seconds. Dissecting the technical delay, there can be seen two subsets:
network delay and processing delay. Network delay is 8.20 seconds subtracted with the delay of OK packets, yielding 4.48 seconds. The processing delay is 2.31 seconds consisting of processing the packets and enabling user interaction. The network delay constitutes half of the total, and the other half is divided equally between the processing delay and the user interaction delay. A bit surprisingly, the portion of network delay caused by the slow GPRS connection is not more than a half of the total as seen in Fig. 4. Due to the high experience level of the user, the measured interaction delay is near the best-case scenario.
Figure 4: Division of delays in context distribution. The high delay on the first packet, Visibility change (2.44 seconds RTT), may be caused by the packet configuring the routers. Thus, the consequent packets can be transferred faster. Transferring a context packet is slightly more stressing on the network (2.36 seconds RTT) than transferring Accept (1.80 seconds RTT), or Request context (1.59 seconds RTT) messages, because context packets are larger. Using XML in context packets does not imply a significant increase in delay: context packets’ sizes are about 1.5 kilobytes and all other packets’ sizes are 20 to 100 bytes, but the difference in the delays is proportionally much less.
The work in [2] showed that displaying the map takes 28.4 seconds, i.e. 17.4 seconds of technical delay and about 11 seconds of user interaction delay. Combined with the results above, it takes almost 40 seconds before the user can see the location and other context information of the selected user. Moreover, real-life user interaction delays would probably be greater. For instance, if the user did not immediately notice the request for context, the delay of accepting the request could be minutes or even hours. Possibly context distribution within a group could be always allowed, removing the initial request for visibility change. At least in this limited test setup, improving the speed of user interaction is almost as important as improving the network connection.
Most network traffic in a context distribution system is transferring the context packets. Before using PIDF, we used a binary data format, in which location, presence, heart, rate, altitude, speed, time, and supported types of interaction were compressed in 335 bytes. The format was somewhat extensible but had zero interoperability with other systems. Comparison with the binary format reveals that the PIDF implementation requires a substantially greater amount of bandwidth; packet sizes are presented in Table 2. The size of a PIDF packet, including IPv4, UDP, and SIP headers, is 1929 bytes, whereas a packet with the message body in binary format is only 680 bytes. The difference is almost three-fold. The division between the overhead of IPv4, UDP, and SIP headers, and the actual message bodies is illustrated in Fig. 5.
Table 2: Context packet sizes comparison (bytes).
Payload SIP UDP IPv4 Total
Binary format 335 317 8 20 680
PIDF 1583 318 8 20 1929
0 500 1000 1500 2000
Binary
PIDF Message Body
SIP UDP IPv4
Figure 5: Message format overhead comparison (bytes). When the binary format is used, the overhead of different headers is about 51 % (345 bytes per 680 bytes) of the total packet size. Using PIDF format, the portion of the overhead is about 18 % (346 bytes per 1929 bytes). In both cases, SIP headers constitute about 92 % (317 bytes per 345 bytes) of the overhead. Thus, a similar approach as in VoIP could be taken: SIP negotiates starting up a session, but a much lighter protocol is used to transfer the actual data packets. In VoIP, this lighter protocol for audio is Real-time Transport Protocol, whereas in context distribution, the actual context information could be transferred in plain UDP packets. In conclusion, in mobile environment, slow connections may require other solutions for distributing context information at the expense of interoperability.
When context information is directly exchanged between peers (in pure P2P fashion) using GPRS connection, distributing context information chokes the network as the number of users grows. This could be alleviated by using faster connections such as WLAN. Since processing data, however, drains the precious battery life of a mobile device, a better alternative is to reduce the amount of data being transferred.
In [4], the techniques for optimizing SIMPLE-based presence traffic are discussed. Methods of having the presence server reduce the amount of messages, using conditional and partial presence, and compressing the messages are presented. Common notify for multiple watchers (delivering the updates for multiple watchers in one notify message between domains), aggregation of notify messages (collecting multiple presence updates into one notify message), and timed presence (presence given for a long period) are suggested. Heuristics, including on-demand presence (e.g. at the time setting up a call) or adapting notification rates (e.g. rarely contacted friends are notified infrequently) are proposed. [4] Our architecture could be improved with more centralized context management scheme and by employing the previously discussed techniques. This scheme can be implemented with dedicated context servers or by adding some hierarchy to the P2P network delegating the management of context information to super-peers.
As for the economical aspect, the pricing models with traffic-based fees do not suit context data distribution. A more realistic pricing model for data traffic would be a flat-rate price with a fixed monthly fee. However, this might put the network under stress if context updates were continuous and the service gained popularity.
IV. QUALITATIVE EVALUATION:USER TESTING
A. Testing Environment
The small-scale field trial took place from July to October of 2006 in the area of the city of Oulu. Testing concentrated on evaluating NaviP2P application in a real usage environment. The users were provided with a Symbian S60 phone (Nokia 6630 or 6680) to use the application and a sports computer (FRWD F-500) for retrieving GPS location and biometric data. Biometric data was used in Wellness application – an electronic exercise book – that was included in the testing to demonstrate the concept of application supernetworking. B. Description of Survey Data and Methods
The testing was realized in groups of three people due to the restricted amount of equipment. There were 15 respondents in total, 3 female and 12 male ones. The testing was started with a kick-off meeting, where the equipment was introduced to the group and the group was given tasks to familiarize with the applications. The kick-off meeting was held for each group separately, which was followed by a test period of 6 to 8 days of unlimited usage of the applications.
In designing the questions, Nielsen’s model of the attributes of system acceptability was considered. The questions aimed at covering especially the usefulness. Usefulness is polarized into utility (can the system in principle do what is needed) and usability (can the system in reality do what is needed). [5]
The data was collected using questionnaires. The users were inquired about the perceptions of NaviP2P application and experiences of the test use. The users assessed the given statements on the scale of four (1-4), maximum four points standing for total agreement.
C. Results
All the test users had a quite strong technological background, which might contort the results. The possessed technical experience, however, eases the use, but also enables more comparative evaluation of the tested applications. It should also be noted that the majority of users were young adults who are usually keener on new technology. The test users were divided into two groups: first based on their earlier usage of smart phones and second based on their earlier usage of navigation services. A smart phone is here defined as a phone in which applications can be installed by the user and is capable of running multiple applications simultaneously.
1) Grouping Based on Smart Phone Usage
The reason for dividing users according to their experience of smart phones was based on the assumption that the familiarity with the environment affects the user perceptions. Thus, experienced smart phone users were predicted to have clearer and more realistic expectations towards the applications.
Table 3 reveals that with experienced test users the interest lies with the sharing of context information, where especially presence information (mean 4.00) was seen very important. Both user groups considered application usability rather good, after all, the means are more than 2.5. However, it is worth
noticing that the map functions were found much easier to use among the experienced users (mean 3.71) than the inexperienced ones (mean 2.83). We also noticed the high standard deviation with the performance of map retrieval. Especially inexperienced users (s.d. 1.38) did not presumably have a clear conception of what is sufficiently fast map retrieval, since they did not possess earlier usage experiences.
Table 3: Grouping based on previous usage of smart phones. High usage Low usage
Usability mean s.d. mean s.d.
Map functions were easy to
use 3.71 0.49 2.83 0.75
Key commands in the map view were logic
3.25 0.46 3.17 0.41 Map retrieval from server was
sufficiently fast
3.13 0.99 2.71 1.38 Wellness start-up worked
expectedly
2.43 1.13 0.77 3.00 App. was sufficiently fast 2.75 0.50 2.57 0.53 Application was easy to learn 3.00 0.82 2.86 0.69 Needed additional guidance 2.25 0.96 2.43 0.98 Application was stable 2.25 0.50 2.71 1.25
Utility
Seeing location of other user was useful
3.43 0.53 3.29 0.95 Seeing presence of other user
was useful 4.00 0.00 2.86 0.38
Wellness start-up was a useful
function 2.71 0.49 0.69 2.86
Need for session start-up with
other applications 2.43 0.79 0.97 2.71 Happy with the functionality
of the application
2.25 0.50 2.57 0.79 Other
Reluctant to let other users see
my location 1.50 0.58 1.43 0.79
Use for a fully developed
version of application 1.50 1.00 2.86 0.90 Willing to pay for a fully
developed version 1.25 0.50 2.14 1.07
Application supernetworking functionalities, i.e. Wellness application start-up from the map view, were highly appreciated among the experienced users; we assume that they adopted the concept better. The inexperienced users possibly saw the concept somewhat irrelevant or confusing. The most promising result was, nonetheless, that the both experienced (mean 1.50) and inexperienced (mean 1.43) user groups did not mind sharing their location information to other users. It also became evident that the service functionality and user guidance require some improvements.
2) Grouping Based on Navigation Service Usage
As a second criterion for user grouping, we used the earlier experience of navigation services. However, none of the users was actively using navigation services, and only one used them sometimes. Thus, the first group consisted of users with no earlier experience, and the second group consisted of users
with at least some experience. As for the results, the earlier experience was expected to affect especially NaviP2P’s usability. Experienced users may adapt more easily to the new environment; on the other hand, they may expect NaviP2P to offer the same functionality as its commercial counter-parts.
According to the test results, the more experienced test users found map functions again notably easier to use (mean 3.75) than the inexperienced ones (mean 2.60). There was once more a great standard deviation in how the map retrieval latency was perceived among both the more experienced (s.d. 1.20) and inexperienced users (s.d. 1.14). In conclusion, there were no major differentiating factors in user perceptions between the groups; therefore, the result table of navigation service usage was excluded from this paper.
3) Analysis of Overall User Experience
Majority of the users had rather or very positive impression of the application before (93%) and after (80%) the testing. It seems, nonetheless, that the high expectations were not completely fulfilled – a common problem in commercial service development. In addition, the first impression was much poorer with both groups with high experience, where it was easier for users to compare NaviP2P with their earlier experiences on smart phones and navigation services. A greater part (79%) also had fun using the application, and again, both groups with inexperienced users enjoyed it more. It can be assumed that the charm of novelty influenced the inexperienced user groups during the test period.
A large part (64%) of the test users thought that the service idea was useful. Again, both groups with inexperienced users found it more appealing; standard deviation, however, was relatively high in both inexperienced and experienced groups. Even though the majority of the test users saw the service idea as useful, only a minor part (40%) thought that they would actually have use for the commercial service; of them, a greater part (60%) was ready to pay for the use. To sum up, almost every fourth (24%) test user was both willing to use and pay for the service. Both groups with high experience from either navigation services or smart phones were more willing to pay for the service. This was somewhat surprising, since their experience of the service was worse than in the low-usage groups. The more experienced users might have been more critical towards the service, but on the other hand, they are early adopters of new mobile technology and, thus, willing to pay for it. It is also worth noticing that only a small minority (13%) was ready to invest into a smart phone to be able to use the service. This might indicate the users’ real intentions to use the service, when real money is involved.
V. DISCUSSION AND CONCLUSIONS
This paper presented implementation and evaluation of the mobile group-based P2P navigation application and intelligent middleware for distributing context information.
Measurements revealed that the slow GPRS connection is the main cause of delays in our implementation. Solutions to achieve scalable context distribution were presented. By introducing a centralized context management, the amount of data transferred could be reduced by filtering traffic.
In user testing, the users familiarized themselves with the concepts of group-based context utilization and application supernetworking. Valuable information was gained on the utility and usability of these concepts. The concepts were found utilizable, although usability was hindered by the fact that the applications were in a prototype stage. Moreover, revealing context information to other users did not face reluctance, at least in this test setup.
Similar studies, which test the usability of mobile navigation services in the field, are non-existent. However, some related research can be found [6]. The main problem of this kind of research is the lack of methodology in the research settings of field studies [7]. The questions: “How, where and whom to measure?” are still waiting sound answers. Still, the research community is starting to justify this reality based testing method as part of academic research. A good example of this is the extensive literature on comparative research of testing usability in laboratory or in realistic environments [8].
Previous research on navigation services has shown that people with no or little experience of navigation services have difficulty to assess the utility or usability of the service without real experience [9]. This led us to use realistic testing environment in our research.
Mobile devices are becoming increasingly sensor-rich, and their networking and processing capabilities improve. This enables the development of commercial applications with group-based context distribution and seamless cooperation. The presented delay and scalability analysis provided technical guidelines, whereas the user tests elicited people’s opinions of such services and their willingness to use them. These experiences can be utilized in the development of commercial group-based navigation services.
REFERENCES
[1] D. Salber, A. Dey, G. Abowd, “The Context Toolkit: Aiding the Development of Context-Enabled Applications,” in the 1999
Conference on Human Factors in Computing Systems (CHI'99),
Pittsburgh, PA, USA, pp. 434-441.
[2] J. Ohtonen, O. Kassinen, M. Ylianttila, “Feasibility Study of a Mobile Peer-to-Peer Navigation Application,” in Proceedings of the 17th
Annual International IEEE Symposium on Personal, Indoor and Mobile Radio Communications, September 2006.
[3] H. Sugano, S. Fujimoto, G. Klyne, A. Bateman, W. Carr, J. Peterson,
Presence Information Data Format (PIDF), 2004, RFC 3863.
[4] V. K. Singh, H. Schulzrinne, M. Isomäki, Boni, Presence Traffic
Optimization Techniques, Technical Report, Department of Computer
Science, Columbia University, NY, USA, October 2006, 16 p. [5] J. Nielsen, Usability Engineering. Academic Press, 1993, 358 p. [6] M. Perttunen, J. Riekki, K. Koskinen, M. Tähti, ”Experiments on
Mobile Context-Aware Instant Messaging,” Collaborative Technologies and Systems. Proceedings of the 2005 International Symposium, May 2005, pp. 305-312.
[7] N. McNamara, J. Kirakowski, “Functionality, Usability, and User Experience: Three Areas of Concern,” ACM Interactions, vol. 13, issue 6, pp. 26-28, November and December 2006.
[8] C. M. Nielsen, M. Overgaard, M. B. Pedersen, J. Stage, S. Stenild, ”It's Worth the Hassle!: the Added Value of Evaluating the Usability of Mobile Systems in the Field,” Proceedings of the 4th Nordic conference
on Human-Computer Interaction: Changing Roles (NordiCHI '06),
October 2006, vol. 189, pp. 272-280.
[9] E. Kaasinen, “User Needs for Location-Aware Mobile Services,”
Personal and Ubiquitous Computing, vol. 7, issue 1, pp. 70-79, May
