An Evaluation Framework for Cross-Platform Mobile Application
Development Tools
by Sanjeet Dhillon A Thesis Presented to The University of GuelphIn partial fulfillment of requirements for the degree of
Master of Science in
Computer Science
Guelph, Ontario, Canada Sanjeet Dhillon, December, 2012
ABSTRACT
AN EVALUATION FRAMEWORK FOR CROSS-PLATFORM DEVELOPMENT TOOLS
Sanjeet Dhillon Advisor:
University of Guelph, 2012 Professor Qusay H. Mahmoud The mobile application market is becoming increasingly fragmented with the availability of multiple mobile platforms that differ in development procedures. Developers are forced to choose to support only some platforms and specific devices due to limited development resources. To address these challenges, numerous tools have been created to aid developers in building cross-platform applications, however, there is no metric to evaluate the quality of these tools. This thesis introduces a framework for evaluating the features, performance and discuss development experience of existing and future cross-platform development tools. The framework is implemented by benchmarking several tools and the results identify a disparity in the features and performance. This research is carried out in collaboration with industrial partner Desire2Learn, through an NSERC Engage Grant.
Acknowledgements
I would like to thank all those individuals that assisted me in the process of completing this thesis. There are many staff, faculty and others who have contributed to bringing me to the point of completing this degree.
I would like to first thank those who provided me personal support, my friends and family. In particular, I would like to thank my friends Nicholas and Samaneh who have been a constant force of encouragement and motivation. The staff at the Centre for Students with Disabilities office has gone above and beyond providing guidance in completing my degree and I am most appreciative of them.
Development contributions for some applications described in this thesis were provided by Undergraduate Research Assistants at the Centre for Mobile Education and Research (CMER) who I would like to thank for their hard work, in particular Sacha Bagasan, Domenico Commisso and Justin Carvalho.
Dr. Fangju Wang has been very receptive and helpful in providing guidance in my seminar and in finalizing this thesis.
I would like to thank my advisor, Dr. Qusay H. Mahmoud for the assistance and support during the duration of this research and for the funding I received in the form of Research Assistantships through CMER which is partially funded by Research In Motion. Additional funding was provided through an NSERC Engage grant with industry partner Desire2Learn. The staff from Desire2Learn have contributed ideas and provided real-world questions to be answered by this research. Their development queries have proven invaluable in shaping the framework to what is needed by the market.
TABLE OF CONTENTS Chapter 1 Introduction ... 1 1.1 Motivation ... 2 1.2 Thesis Statement ... 4 1.3 Research Approach ... 5 1.4 Organization of Thesis ... 6
Chapter 2 Background and Related Work ... 8
2.1 Current State of the Mobile Platform Landscape ... 8
2.2 History of Cross-Platform Development ... 9
2.3 CPDTs for Mobile Development ... 10
2.3.1 Mobile Web ... 12 2.3.2 Adobe PhoneGap ... 13 2.3.3 Appcelerator Titanium ... 15 2.3.4 Rhomobile Rhodes ... 16 2.3.5 Adobe Air ... 17 2.3.6 MoSync ... 18
2.3.7 Tool Analysis and Comparison ... 19
2.4 Related Work ... 20
2.5 Summary ... 23
Chapter 3 An Evaluation Framework for CPDTs ... 25
3.1 Framework ... 25 3.2 Phase I: CPDT Capabilities ... 26 3.2.1 CPDT Basic Elements ... 27 3.2.2 Development Environment ... 27 3.2.3 User Experience ... 27 3.2.4 Device Access ... 28 3.2.5 Sensors ... 28 3.2.6 Geolocation ... 28 3.2.7 Notifications ... 28 iv
3.2.8 Monetization ... 29
3.2.9 Security ... 29
3.3 Phase II: Performance Benchmarks ... 29
3.3.1 Processor Intensive Benchmarks ... 30
3.3.2 Data Driven Benchmarks ... 31
3.3.3 Device Access Benchmarks ... 31
3.3.4 User Experience Benchmarks ... 32
3.3.5 Test Procedure ... 32
3.4 Phase III: Development Experience Discussion ... 33
3.4.1 Tool Related Discussion ... 34
3.4.2 Development Experience Discussion ... 34
3.5 Summary ... 34
Chapter 4 Implementation and Experiments ... 36
4.1 Experiment Parameters ... 36
4.1.1 CPDTs and Native Development Kits ... 36
4.1.2 Devices ... 37
4.1.3 Assumptions... 38
4.2 Phase I: CPDT Capabilities ... 39
4.3 Phase II: Performance Benchmarks ... 39
4.3.1 Application Skeleton ... 40
4.3.1.1 Interface ... 40
4.3.1.2 Test Modules ... 41
4.3.1.3 Log ... 41
4.3.1.4 Iterations ... 43
4.3.2 Processor Intensive Benchmark Tests ... 43
4.3.2.1 AES Encryption and Decryption ... 43
4.3.2.2 Input Validation... 44
4.3.3 Data Intensive Benchmark Tests ... 45
4.3.3.1 Local PIM Access ... 45
4.3.3.2 Remote Service Access ... 46
4.3.3.3 Sorting ... 47
4.3.4 Device Access Benchmark Test ... 48
4.3.4.1 Microphone Usage ... 48
4.3.5 User Experience Benchmark Tests ... 49
4.3.5.1 UI Elements ... 49
4.3.5.2 Screen Transition ... 51
4.3.6 Test Implementation ... 52
4.4 Phase III: Development Experience Discussion ... 53
4.4.1 Tool Related Discussion ... 54
4.4.2 Development Experience Discussion ... 54
4.5 Summary ... 56
Chapter 5 Results and Evaluation ... 57
5.1 Phase I: CPDT Capabilities ... 57
5.2 Phase II: Performance Benchmarks ... 58
5.2.1 Processor Intensive: AES Encryption ... 59
5.2.2 Processor Intensive: Input Validation ... 59
5.2.3 Data Driven: Local PIM Access ... 64
5.2.4 Data Driven: Remote Service Access ... 64
5.2.5 Data Driven: Sorting ... 65
5.2.6 Device Access: Microphone Usage ... 68
5.2.7 User Experience: UI Elements ... 69
5.2.8 User Experience: Screen Transition ... 70
5.2.9 Overall Result ... 71
5.3 Phase III: Development Experience Discussion ... 72
5.3.1 PhoneGap ... 72 5.3.2 Appcelerator Titanium ... 73 5.3.3 Adobe Air ... 74 5.3.4 MoSync ... 74 5.3.5 Native Tools ... 75 5.4 Analysis ... 76 5.5 Summary ... 78
Chapter 6 Conclusion and Future Work ... 79
6.1 Conclusion ... 79
6.2 Future Work ... 80
Bibliography ... 82
Appendix A: CPDT Features to be Evaluated ... 90
Appendix B: Benchmark Application Skeleton ... 92
Appendix C: Benchmark Application UI Elements Test ... 99
Appendix D: AES Encryption Data ... 105
Appendix E: Contact Data ... 106
Appendix F: Remote Data Script ... 109
Appendix G: Benchmark Application Sorting Algorithm ... 110
Appendix H: Images for UI Testing ... 112
Appendix I: Results of Phase I Evaluation ... 114
Acronyms and Abbreviations
API Application Programming Interface
CMER Centre for Mobile Education and Research
CSV Comma Separated Value
CPDT Cross-Platform Development Tool
CSS Cascading Style Sheet
GPS Global Positioning System
GUI Graphical User Interface
HTML Hypertext Markup Language
IDE Integrated Development Environment
JVM Java Virtual Machine
MFLOPS Millions of Floating Point Operations Per Second
MVC Model View Controller
NDK Native Development Kit
PIM Personal Information Manager
SDK Software Development Kit
UI User Interface
UX User Experience
List of Figures
Figure 1.1: BioChem Euchre Deck Application on BlackBerry, Android and iOS ... 3
Figure 2.1: Developer Interests in Mobile Platforms ... 9
Figure 2.2: IBM Worklight Architecture ... 14
Figure 2.3: Performance Characteristics of Various CPDTs ... 21
Figure 3.1: CPDT Evaluation Framework ... 26
Figure 3.2: Benchmarking Procedure ... 33
Figure 4.1: Native Implementation on iOS, Titanium implementation on iOS ... 41
Figure 4.2: Input Validation Test Procedure ... 45
Figure 4.3: Remote Access Test Procedure ... 47
Figure 4.4: Sorting Test Procedure ... 48
Figure 4.5: Screens for UI Elements Test ... 49
Figure 4.6: Screen A, Screen B and Screen C for Transition Test ... 51
Figure 4.7: Control Structure for Transition Test ... 52
Figure 5.1: Input Validation Test Results on Android Platform ... 61
Figure H.1: image1.png ... 112
Figure H.2: image2.png ... 112
Figure H.3: Icons for Menu ... 113
List of Tables
Table 2.1: Programming Environments for Mobile Platforms ... 8
Table 2.2: Tools for Cross-Platform Application Development ... 10
Table 2.3: CPDT Features ... 11
Table 4.1: Compatibility Matrix for Tools Included in Study ... 37
Table 4.2: Devices Used for Testing ... 38
Table 4.3: Tests Implemented in the Study ... 53
Table 5.1: Number of Supported Features ... 57
Table 5.2: AES Encryption Test Result Matrix ... 59
Table 5.3: Input Validation Test Result Matrix ... 60
Table 5.4: Test Variability Statistics for Input Validation on Android ... 61
Table 5.5: T-test Calculations for Android Native and PhoneGap Mean on Android 62 Table 5.6: T-test Calculations for iOS Native and Titanium Means on iOS ... 62
Table 5.7: T-test Calculations for BlackBerry 7 WebWorks and PhoneGap Means on BB7 ... 63
Table 5.8: T-test Calculations for BB10 Native and WebWorks Means on BB10 ... 63
Table 5.9: T-test Calculations for PhoneGap and Titanium Means on Android ... 63
Table 5.10: Local PIM Access Test Result Matrix ... 64
Table 5.11: Remote Service Access Test Result Matrix ... 65
Table 5.12: Sorting Test Result Matrix ... 66
Table 5.13: Summary Statistics for ANOVA Calculations on Android ... 67
Table 5.14: ANOVA Calculations for Android using data from Table 5.13 ... 67
Table 5.15: Summary Statistics for ANOVA Calculations on iOS ... 68
Table 5.16: ANOVA Calculations for iOS using data from Table 5.15 ... 68
Table 5.17: Microphone Usage Test Result Matrix ... 68
Table 5.18: UI Elements Test Result Matrix ... 70
Table 5.19: Screen Transition Test Result Matrix ... 71
Table 5.20: Overall Results of Performance Evaluation ... 71
Table A.1: CPDT Features in Evaluation ... 91
Table I.1: Phase I Evaluation of CPDT Features ... 115
Chapter 1
Introduction
The world of mobile operating systems has been becoming increasingly fragmented in recent years. Each company or consortium has developed their own non-standard method of developing applications for their devices using a variety of programming languages and Software Development Kits (SDKs). This means that an application built for Google’s Android operating system will not run on the rival Apple iOS platform. There is further fragmentation inside platforms with some applications not being able to be run on different versions or devices of the same platform making development far more challenging.
Fragmentation of the market is caused by many factors. Aside from software platform diversity, hardware variations create difficulty porting a User Experience (UX), including interaction method and User Interface (UI) from one device to the next [3]. With over 20 million tablet devices sold in 2011, it has been difficult to enable applications to maximize the larger screens of these devices [64]. Many developers are forced to choose to support only some platforms and versions due to limited financial resources or knowledge of coding techniques for each platform. Without wide developer support, platforms are seen as weak and have difficulties in marketing products. This is true for new operating systems and new releases that break compatibility with the legacy software. An example of this is Research in Motion moving to the BB10 platform breaking compatibility with natively developed BlackBerry OS applications [57].
With the number of languages used, developers must be very versatile and businesses need to expend significant resources to have their software available on more than one platform. Even with the added expense, an August 2012 Appcelerator and IDC survey shows companies continue to be very interested in multiple platforms
despite the difficulties [7]. In 2011, developers have shown interest in running on twice as many platforms as a similar survey in the previous year and now averaging an incredible four operating systems [4] [6]. This trend has continued to increase in 2012 [7].
With so much interest in availability of applications for many platforms, there is an apparent problem facing developers in requiring the ability to have their applications available to the widest possible audience, but lack resources to build natively for each platform. In order to further the reach of applications, without expending the considerable time needed to learn the quirks of each platform, using Cross-Platform Development Tools (CPDTs) may be the answer. The CEO of InRuntime has said using CPDTs has reduced the time to market by 70% [68]. This shows that using cross-platform tools can be very helpful in many ways. However, there is no adequate measure to ensure these CPDTs are as capable as native development tools. This leaves developers unsure if they can make use of these tools to create strong applications with native like functionality and performance.
This chapter will present a discussion of the motivation for research in this area. It then will go on to introduce the thesis statement and outline the structure of this thesis.
1.1
Motivation
Prior to the start of the research in this thesis, as part of the work at CMER, we developed the BioChem Euchre Deck application for the BlackBerry platform [19]. This application provides a learning interface for students using flash cards created by Professor John Dawson of the University of Guelph. Following initial development, we were left wondering how to best provide the application to students on all platforms. While choosing a cross-platform solution and developing the application, several shortcomings were noted. The lack of adequate comparison of these tools
made the task of finding which tool would best meet our requirements very difficult and highlighted the need for an evaluation method.
PhoneGap was seen in our unstructured evaluation to be an ideal candidate for simple, widely deployed applications. The application shown in Figure 1.1, was built and deployed. The wider reach from using a CPDT allowed for over 6000 downloads on Android and 2000 on each of iOS and BlackBerry showing the benefits of going cross-platform.
Figure 1.1: BioChem Euchre Deck Application on BlackBerry (left), Android (center) and iOS (right)
Unfortunately, with the lack of evaluation available, we did not know if the tool met the needs of our application prior to the start of development. Some limitations were apparent with the PhoneGap Build service such as the inability to use focused view on the BlackBerry platform, thus creating applications that have a mouse cursor. This is expected to be resolved in a subsequent release of PhoneGap but left us with the need to compile the BlackBerry version of the application using the standard WebWorks SDK and a different config.xml file which provides information about the application. The application itself did not need to be modified, however. Had we had an evaluation of the PhoneGap tool available to us prior to developing
this application, we would have known that this limitation made it not ideally suited to our purpose.
Some CPDTs have been developed and numerous features are implemented to work across many devices but it is difficult to ascertain which features are missing. Much of the information available for these tools comes from the vendors and there is little independent analysis available to developers. Additionally, there is a lack of comparison between these frameworks and their native development counterparts. How well do they perform? Do you lose functionality? Which are the best?
With no current benchmarks available, a process for testing the frameworks against one another and their native counterparts is required in order to provide guidance and answer the central question of finding a cross-platform development solution without trade-offs. Benchmarking protocols and tools will need to be developed in order to test the many frameworks as well as a study to be conducted into features and shortcomings of each.
1.2
Thesis Statement
This thesis focuses on exploring the viability of using CPDTs for mobile smartphone development in detail. Although cross-platform tools on mobile devices have come far from early incarnations, perceived trade-offs and inefficiency still hampers their use. The goal of this thesis is to create a framework to provide thorough comparison of CPDTs to one another and native development tools to determine if they can be used as an efficient development method in place of native tools. Through development of a feature and benchmark intensive evaluation framework for CPDTs, developers will gain the knowledge needed to determine which tool to use for their application.
In order to determine if CPDTs can be as capable as native development tools, a set of evaluation criteria for development tools has been created. A CPDT should be evaluated on the basis of features, performance and developer experience.
By creating an evaluation framework and applying it to select CPDTs, developers will gain useful knowledge of which tool best suits their purpose and if a cross-platform solution is a viable, cost-effective alternative to traditional native development.
1.3
Research Approach
The main focus of this research was the development of the evaluation framework. This includes the benchmarking tools and the discussion of the obtained quantitative results for several CPDTs in the marketplace.
In order to determine whether the central thesis has sound basis, numerous steps were taken:
1. Survey the tools available for both native and cross-platform development, 2. Discover prior research into evaluation of such development tools,
3. Create a high level evaluation framework using real-world development questions,
4. Develop benchmarks to achieve quantitative results and apply it to several CPDTs and native tools,
5. Analyze results, and make conclusions.
Although CPDTs do currently exist, it is still unknown whether or not these tools meet the performance and feature requirements to be comparable with their native counterparts.
Very little research has been conducted on this subject as these CPDTs are all quite new with most performance analysis being anecdotal, not scientific. It is believed that mobile web-based CPDTs may suffer from performance issues but it is unknown if it is true with the recent advancements in JavaScript performance [4]. A benchmarking framework must be developed in order to test these CPDTs against one another and native code. The development questions will be obtained by surveying current research, developer surveys, discussions with our industry partner and by discovering the capabilities of native tools.
During the development of this evaluation framework, certain assumptions had to be taken into account. The tools are expected to have the ability to run similarly developed benchmark tests while remaining cross-platform. The tests are developed using technologies that while tablet and smartphone friendly, only focus on smartphone testing. Testing for tablets is left for future work. It is understood that certain background processes cannot be stopped and may impact testing so care should be taken to reduce their influence on the results.
This research is conducted as part of an NSERC Engage project with the industry partner, Desire2Learn, a leader in development of e-learning software. Desire2Learn has provided real-world questions and feedback on how this framework can best benefit mobile developers. The development expertise and knowledge of development issues of the staff at Desire2Learn has been incorporated into development of each step of this research.
1.4
Organization of Thesis
In Chapter 1, we provided an introduction to the subject area, discussed the motivation behind conducting this research and described the thesis statement. The following chapters will describe an evaluation approach for mobile development tools. The thesis is structured as follows:
Chapter 2: Discusses the topic in more detail to provide necessary background information and details on some select CPDTs. It then discusses limited the prior work in this area.
Chapter 3: Describes the proposed solution, an evaluation framework that can be used for any CPDT.
Chapter 4: Introduces an implementation of the framework in more detail and the experiments used to test its viability.
Chapter 5: Presents the results of the testing of various CPDTs and native development tools using our evaluation framework.
Chapter 6: Provides concluding remarks and future work that can be used to extend this framework.
Chapter 2
Background and Related Work
It is important to understand each of the mobile platforms and development languages before being able to understand the scope of the problem solved by CPDTs. This chapter will provide background on the mobile landscape and an overview of various CPDTs. This overview is necessary to understand which aspects are most important and should be addressed in any evaluation framework. This will be followed by discussion of related work for evaluating CPDTs.
2.1
Current State of the Mobile Platform Landscape
The current landscape has eight major smartphone platforms listed in Table 2.1. However, many of these are losing importance with iOS, Android, BlackBerry 10 and Windows Phone becoming the major remaining players for the foreseeable future [7].
Table 2.1: Programming Environments for Mobile Platforms
Operating Environment Preferred Programming Language
RIM BlackBerry OS Java ME, HTML, JavaScript RIM BlackBerry 10 C/C++, HTML, JavaScript, Java
Apple iOS Objective-C
Google Android Java (Harmony Flavored), C and C++
HP WebOS HTML, JavaScript
Windows Phone C#, .NET
Symbian C,C++
Samsung Bada C++, HTML, JavaScript
With the number of languages listed, developers must be very versatile and businesses need to expend significant resources to have their software available on more than one platform. Comscore, a leader in measuring digital market share, estimates that in the United States, Android has over 52% market share with the rest being split between Apple, RIM, Microsoft and Symbian [23]. Figure 2.1 shows the
level of developer interests in mobile platforms. This figure includes mention of both tablet and smartphone editions of each platform.
Figure 2.1: Developer Interests in Mobile Platforms [7]
With so much interest in availability for many platforms, there is an apparent problem facing developers in requiring the ability to have their application available to the widest possible audience, but lack resources to build natively for each platform. In order to further the reach of applications, without expending the considerable time needed to master each platform; using a CPDT may be the answer.
2.2
History of Cross-Platform Development
Java was believed to be marvelous for PC development for allowing a program to be written once and translated into an intermediary language before being run on a Java Virtual Machine (JVM) on a user’s system. This allowed a single program to be run on Windows, Mac and Linux with no platform dependency. Java Micro Edition (Java ME) came to mobile devices but never went mainstream often thought to be due to its limited capabilities, perceived performance issues, and device fragmentation [16] [24]. Java was soon abandoned by some handset makers or forked into customized
versions with added functionality. Cross-platform tools that extended the Java Mobile architecture had gained some ground with Bedrock, Celsius, NeoMAD and alcheMo entering the market [54]. These tools allowed applications to run on a significant number of devices that supported Java, but the tools were outpaced by significant leaps in technology for native platform SDKs. These tools seemed to lose relevance with the advent of the new era of highly versatile smartphones.
Since 2011, a new set of CPDTs has been coming to market with new features being added rapidly over time. This latest generation of tools, will be discussed in the next section.
2.3
CPDTs for Mobile Development
There are several tools for cross-platform mobile application development available on the market today with various levels of functionality and compatibility. This new generation of CPDT allows more control, functionality and diversity than the previous generation of Java based tools [24] [37] [54]. The Cabana tool in [27], helps bridge the gap between the older and newer generation in terms of features. We have researched five tools as shown in Table 2.2. These tools were chosen due to their flexibility, feature support and popularity amongst developers. The chosen tools provide samples of different approaches to platform development with cross-compiled, runtime and web wrapper styles. They were also required to support only a minimum of two distinct platforms allowing for a wider array of analysis of emerging CPDTs. Mobile web browser based applications are included for comparative purposes.
Table 2.2: Tools for Cross-Platform Application Development
CPDT BlackBerry OS BlackBerry 10 iOS Android Windows Phone 7 Bada
Mobile Web Adobe PhoneGap
Appcelerator Titanium Beta
Rhomobile Rhodes 3.0+ 1.6+ Adobe Air
MoSync
The CPDTs described in Table 2.2 vary greatly in capabilities, language and platform support. They can be split into categories based on the method used to compile and run the applications built using the tools. While some CPDTs such as MoSync can cross-compile the application into native code, others use a web wrapper, essentially running the application in an HTML5 compatible browser object. Traditional runtimes and extensions such as those used in Adobe Flash are widespread on desktops and notebooks but have limited support on mobile platforms and are being phased out. The alternative is using a similar approach to bridge the divide which is found in Adobe Air [72]. Each of these methods has distinct advantages over purely web-based applications adding possible performance, user-interface and notification enhancements not available to web developers. These added capabilities can quickly become an interesting proposition for a developer wishing to have more fully featured applications.
Some of these tools are already in use by the likes of NBC Universal, eBay and The New York State Senate allowing them to provide the application to as many customers as possible at a low cost [8].
Table 2.3: CPDT Features CPDT Development Language Compilation Type Native UI Elements Access to Sensors * File System Access User & System Notifications App store Support Mobile Web HTML5 + JavaScript Runs in Browser Limited Limited Adobe PhoneGap HTML5 + JavaScript Native Web App Appcelerator Titanium HTML5 + JavaScript / Python / Ruby / PHP Native Code and Runtime Rhomobile Rhodes Ruby Runtime Adobe Air ActionScript/
HTML/AJAX Runtime MoSync C / C++ / HTML5 + JavaScript Native Code *Camera, GPS, Accelerometer 11
The tools presented in Table 2.2 use different scripting languages and provide various features that are not compatible across all mobile platforms. In Table 2.3 we show many features provided by each tool, including the development language of choice.
2.3.1 Mobile Web
The promise of the mobile web as a standard based architecture is that it will display applications similarly despite being on different platforms and devices. Rich web applications can be written using HTML5 and JavaScript. These provide a high degree of functionality like in Google’s GMail web application. This idea is not a new approach and although it does solve many issues for developers it does add a few new concerns. Mobile web applications that run through a phone’s browser often do not match the phones native UI which can be quite confusing. However, these applications can leverage JavaScript and CSS (Cascading Style Sheets) frameworks such as JQuery Mobile [65], Zepto [30] and Sencha Touch [60] to provide much needed UI enhancement with little work on the developer’s part. Many such frameworks exist to enhance the web development experience and these are only three of them that have been optimized for mobile handsets.
Browser security historically has not allowed storage of local data or access to device sensors such as an accelerometer. The webinos framework proposed in [42], provides an alternative to allow more functionality but does not have widespread adoption. Additionally, the browsers themselves have been fragmented with different implementations of JavaScript and rendering engines which can make functionality and interface behave inconsistently on different devices [59]. The largest concern is often offline access where there cannot be any access to the application when there is no connectivity.
Are these problems a deal breaker in using this as a development environment? The answer is quite subjective but the downsides are growing to be less and less as
the technology progresses. HTML5 has allowed for deeper integration with the handset and with WebKit being nearly ubiquitous across handsets, the rendering issues of the past are becoming less of a major consideration [20]. Unfortunately, living in the browser still blocks these programs from the application markets which are useful in helping end users find the developers work.
While mobile widgets add some features, it is still largely limited to the browser [40]. An approach taken by many CPDT developers is taking the concept and ideals of widgets and the mobile web and enhancing it with the features that were missing. The next sections will discuss tools that take an integrated approach to cross-platform development.
2.3.2 Adobe PhoneGap
PhoneGap [1] is an implementation of the recently renamed Apache Cordova open source project. Cordova and PhoneGap are tightly linked and will be discussed as one tool. PhoneGap is one of the most popular mobile web based approaches to cross-platform development. The tool takes advantage of the standardized web technologies used for mobile web development and brings them to native web applications. Much like the W3C’s Widget standard; PhoneGap uses HTML, CSS and JavaScript to write applications that are then encapsulated in a browser object to look like applications built using native code. With this approach, the developer can submit the application to the application store to enable purchase and high visibility. Along with deep integration with the phone operating system and hardware, applications can make use of local storage and are fully available when there is no network connectivity. Many of these advantages are not present in standard mobile web applications.
In Table 2.2 and Table 2.3 we see the wide support for devices and features with PhoneGap. Through an API, developers can make use of various JavaScript functions use underlying native device capabilities. Most notably missing is native
UI elements, however this trade-off was made in order for the ability of having it available on so many platforms.
Many 3rd party tools are available to be used with PhoneGap, along with the already mentioned JavaScript libraries; the appMobi XDK for PhoneGap [12] provides an IDE and testing environment to aid developers significantly. This comes with tools to emulate the look of the mobile application on a variety of devices. appMobi additionally provides an analytics platform and their own build service similar to that provided by PhoneGap directly [11]. This service allows compilation of applications without the need to install an SDK for each mobile platform. With the addition of this XDK, the PhoneGap development lifecycle begins to resemble that of native applications much more closely.
Another tool that enhances PhoneGap is IBM Worklight [36]. It provides additional APIs and server side integration frameworks for enterprise level applications. The architecture of Worklight, seen in Figure 2.2, makes heavy use of PhoneGap to bridge the JavaScript code to the native device. It adds the Worklight API to enhance security, add analytics and access to their middleware, Worklight Server. This server provides the connection to the enterprise backend systems for the mobile application [37].
Figure 2.2: IBM Worklight Architecture [35]
Rather than using the Chrome browser frame based approach as in the appMobi XDK, Worklight has an IDE called Worklight Studio that is based on Eclipse. Once applications are deployed, all analytics and push notifications are handled through the Worklight console [37].
Whether using the PhoneGap directly, appMobi XDK or the suite of Worklight tools, PhoneGap provides an incredibly flexible approach for mobile application development. For small scale applications that need to be completed quickly to large enterprise level applications, there is a variant of this tool suitable for many use cases.
2.3.3 Appcelerator Titanium
Appcelerator Titanium [8] shares many traits with PhoneGap and the mobile web. Development for all three use standard web development languages, however Titanium differs in some important areas.
Running on an open source core, Appcelerator products allow for applications to run natively on devices without embedding a browser object as we have seen in PhoneGap. Instead, their approach uses a runtime object and compilation method to optimize and compile code. Options are provided to compile and run using a runtime; build into a web application, or a hybrid application similar to PhoneGap. The claim is that performance levels are increased to match those of native applications when using the runtime based approach [10] [68]. This assertion by the vendor requires independent analysis through a standard benchmarking architecture to determine accuracy. The UI approach also differs greatly with the use of native UI elements rather than focusing on a purely web-like interface. This is seen as a major advantage by many people as the application will not cause confusion by having a different look and feel than those built using native tools. Others, especially those transitioning mobile web applications, may find this as a negative and find this handcuffs the customizability of the interface [6].
Like the IBM Worklight extension to the PhoneGap core, Titanium provides a host of services for analytics and server side hosting. Similar to Worklight, server side features such as data storage and push notifications are available. The Appcelerator team does not provide the application the same level of end to end security guarantees found with Worklight and is more of a consumer, rather than enterprise focused architecture [10] [37]. Titanium includes Titanium Studio, an Eclipse based IDE and a set of testing tools and emulators.
Appcelerator Titanium lacks in platform support with only iOS and Android being fully supported with BlackBerry OS support being put into beta release one year ago [9]. This beta was discontinued and replaced with an upcoming BlackBerry 10 OS beta. Supporting only two of the major mobile platforms provides only the minimum required for being called cross-platform and large segments of the market are left out when developing applications using this tool.
2.3.4 Rhomobile Rhodes
Rather than using web standards, a different approach has been taken by Motorola Solutions in Rhodes [45]. Rhodes is developed by Rhomobile which has been acquired by Motorola Solutions, the enterprise and government focused division of Motorola. This part of the company was not acquired by Google and is independent of the Google Android project [68].
This method is based on the Ruby programming language which allows current Ruby developers to have an easy transition. Not all Ruby gems are translated to work inside Rhodes; however, more can be added as needed [50]. The tool provides strong UI tools including access to native UI elements for Android and iOS. Unlike many other tools, Rhodes has full Model View Controller (MVC) support, negating the need to have business logic in the view as JavaScript may have in other CPDTs
[63]. This separation can have major positive impact within the development process for various screen dimensions.
Rhodes uses a runtime based approach that has a runtime built using the C++ NDK for Android, which interprets the Ruby bytecode for the platform. The Rhodes vendor considers this to be more efficient than using Java on Android [68]. Rhodes offers the RhoStudio IDE with build tools, debuggers and simulators. Application hosting and enterprise data connection is a strong push for Rhodes with RhoConnect and RhoSync server. This positions RhoSync to compete for the enterprise market and Worklight. The final piece, RhoHub puts the services together into an interface with Git-powered source control and an online build service similar to PhoneGap Build. While open sourced under an MIT Licence, much of the functionality is only available through paid subscription for services from RhoMobile [68].
2.3.5 Adobe Air
Adobe Air is aimed at building rich internet applications that can be run on many platforms. It uses Adobe Flash, Adobe Flex, HTML, ActionScript and Ajax for development scripting. Flash skills are easily found in the industry and that is perceived as a major advantage for Adobe Air. Existing Flash applications can often be translated to run as native applications using Air instead of them being run through a browser extension. Air applications, unlike Flash, require installation of the runtime as well as each application. Mobile support on Android and iOS is present with an independent runtime available on Android and necessary elements being packaged with the application in iOS. As of version 2.6, most features are on par for the Android and iOS versions of the platform. It is suggested that runtimes like these can be an effective way of battling the cross-platform question however; there has been a mixed reception from device manufacturers [18] [20].
Air developer tools are some of the most robust and supported in the industry. Since the tool stems from the popular Flash tool, the same tools used like Flash
Builder and Dreamweaver can be used for development. When combined with Adobe Flex technologies, enterprise applications are possible with rich UI capabilities that may surpass those of other platforms. Adobe does not provide the integrated experience and the end to end functionality, tools and services provided by some CPDTs previously discussed.
These runtime based approaches found in Appcelerator Titanium, Rhomobile Rhodes and Adobe Air do have significant drawbacks in some areas. The main one is the dependence on the runtime environment itself. The mobile OS makers may at some point block these from their stores and the included runtime can cause larger file size. The features may also lag behind what is released natively as it is integrated into the runtime. As execution often uses just in time compilation, performance may be impacted when using any of these runtimes [43].
2.3.6 MoSync
MoSync is much like Air, however, the major differentiator with MoSync to other CPDTs is that it uses cross-compilation where the code is translated into native Objective C and Java code depending on the platform. The C++ compiler used in MoSync outputs an intermediary language that is then optimized and outputted as a binary for the desired platform. This is a different approach that may lead to increased performance of applications. MoSync also offers MoSync Wormhole a PhoneGap like CPDT that uses a browser object to display the application. C and C++ code can be translated using their wormhole technology to be used in these web based applications.
MoSync offers compatibility with OpenGL ES, which allows 3D graphics for game development therefore unlike many other CPDTs, this is suitable for cross-platform gaming [68]. It additionally is quite extensible allowing the addition of C and C++ libraries to your applications. Cross-compilation can be a very useful
technique allowing the ability to deal with compiled native code in the end for increased optimization.
2.3.7 Tool Analysis and Comparison
Choosing one CPDT over another can be a daunting task, available skills and project requirements will often dictate which to use when comparing the capabilities of the CPDTs. The choices are significantly narrowed when special features such as native UI’s and the ability for 3D graphics are needed in a CPDT.
The runtime based CPDTs, must include the runtime with the application and in many instances create larger file sizes [34]. Cross compilation may be an effective way of completing the task; however, these approaches discussed still have limited APIs. Web based CPDTs like Appcelerator Titanium and PhoneGap offer many features however have limited platform support in some cases. The lack of native UI elements can be problematic but also provides freedom for custom designed layouts. With all of the tools studied, compatibility to use device functions such as the accelerometer, camera and notifications are present. These provide a much more integrated experience as opposed to mobile web apps where there is still little in the way of integration, particularly with notifications to users.
These CPDTs come with a variety of costs and licenses involved that are rapidly being adjusted. Some such as PhoneGap are free and open source and others like Appcelerator have free and paid tiers. The tiered model with support options in higher tiers is typical and also used by PhoneGap. As the tools have evolved, so have these pricing models which are a moving target.
With each of the cross-platform methods, adequate debugging functionality and documentation was still seen to be lacking or outdated [49]. Currently, many resources are restricted to documentation provided by the tool makers themselves so
full objective comparison is difficult to establish in these early days and requires personal experience with each CPDT.
2.4
Related Work
Little research is available comparing CPDTs. Some comparison of features has been done in [32] and [68] although they lack some depth. The CPDT evaluators used a 13 item chart in [32] to allow comparison of tool features. Many important items like storage and camera access are covered in the survey. However, neither of these reports includes performance evaluation or discussion of development practices and detailed costs.
In [47], the authors discuss many CPDTs and provide partial comparison. Their work includes discussion of native versus web-based user interface elements and the importance of well performing applications. However, the authors state that they are not concerned with the internal workings of the tools and only if the applications will be approved for the application stores as the main development need. The article discusses the lack of debugging tools in many current CPDTs and provides an 8 point feature comparison. The authors develop a simple application that creates a screen with a text label and measured the RAM usage and start time for nine CPDTs. The results are found in Figure 2.3.
The results show the quick launch time for the application built using the native Android SDK but a much slower start for other tools such as Titanium and PhoneGap. Runtime based CPDTs seem to fare the worst and have large RAM footprints. This information provides useful understanding of the possible performance differences with developing applications on the Android platform using CPDTs but does not include other platforms or more extensive tests beyond initialization. It is possible with further testing that an application may perform poorly for initialization but run remarkably well afterwards but this cannot be determined by this test.
Figure 2.3: Performance Characteristics of Various CPDTs [47]
Mobile devices have been benchmarked before using a variety of approaches. Benchmarking has been around for a long time and can be applied to many forms of computational tasks where the overall capability of a processor or system is calculated and compared using complex benchmarking tools. These consist of a set of intensive tasks that measure the time it takes to complete. Currently, the PCMark suite is the most prominent desktop PC benchmarking software and uses several open source and commercial applications [61]. Using the contained test suites allow CPU, memory, graphics and hard disk performance analysis. There is no equivalent gold standard of these test suites in the mobile arena [66], although work is being completed on native benchmarks for various platforms. Some currently available are Quadrant [13] and AnTuTu [5]. Current published work in mobile benchmarking is extremely limited. This is non-existent in comparing these mobile CPDTs to their native counterparts outside of a simple test on Android.
Rather than running through pre-selected algorithms as with the aforementioned tools, the TMAPP project running on a Meebo OS based netbook used applications such as Firefox and Open Office to run through a script of tasks. The authors suggest that this method is superior to preselected algorithms as these are real world use cases [38]. However, this may be difficult to replicate with the security models on some mobile platforms. Many benchmarks available for PCs cannot be translated over to mobile devices due to restrictions from the operating system, such as running a script of opening and closing multiple applications as done in TMAPP [38].
The Rodinia Benchmark Suite [21], outlines the creation of a benchmark suite for heterogeneous systems. Included in their testing are neural networks and breadth-first search in trees. Rodinia makes use of GPU accelerators using OpenMP, OpenCL and Cuda as design elements. While not applicable to mobile uses, this is an interesting use of an application of a comparable benchmark across many devices that was shown to be successful.
A similar benchmark approach is used to compare algorithms written using various programming languages in [17], [25], [46], and [51]. Each builds a benchmark using standard algorithms that are implemented independently in each language. The resulting tests show language and compiler efficiencies much in the same way that we seek to find CPDT application performance. In [17] the author uses the Linpack benchmark suite as a guideline for the algorithms due to their long study over decades. Various sorting and complex scientific calculations are used in each of these papers to test each language and provide a common line of comparison.
A positive UX is important for all applications. Creating a consistent UX when using platform development can be difficult. In [67], metrics for strong cross-platform UX are discussed. The authors find that the common themes of usability, performance, social integration and context-aware services were important for users when they are using a similar service on both mobile and desktop terminals. A CPDT
that wishes to work across device categories should ensure they have this functionality built in.
The Swerve Studio and X-forge CPDTs are discussed in [73]. These tools are focussed on cross-platform game development which is shown to have its own set of requirements that differ from those discussed in previous papers.
Current research in this area has particular limitations. When comparing CPDTs, research has focussed on narrow scales that only incorporate very few features and will not provide the required breadth necessary for decision making. Furthermore, performance benchmarks are limited to opening an application and do not provide further processing. These evaluation criteria are inadequate to draw conclusions on if there are differences in performance depending on which tool is used. More advanced testing and a larger picture of development tools is necessary to answer the greater questions of if these tools can replace native development for certain tasks.
2.5
Summary
In this chapter we have discussed the many mobile platforms that currently exist in the marketplace and the confusion and difficulties this may bring. A new generation of CPDTs have been outlined with some comparison provided.
Benchmarking procedures for mobile devices and applications are still in their infancy, however, some work has been done in [38] as well as some generally available applications. Investigation of several programming languages and how to adequately compare them are found in literature in [17] and [33].
It can be seen from an overview of previous work in this area that only surface evaluation of comparing CPDTs is available. As these tools are fairly new, benchmarks comparing them are not available at this time; however, comparable
benchmarking procedures used for other purposes have similarity and can be adapted for this purpose.
In the next chapter we will present an evaluation framework for CPDTs. The phases of evaluation and components of the framework will be outlined at a high level.
Chapter 3
An Evaluation Framework for CPDTs
The emergence of the many CPDTs in the market leave an important question unanswered; does using a CPDT to build and deploy your application impact the quality of the application? Furthermore, do the costs for development decrease with usage of a CPDT instead of building an application natively multiple times? The ability for a CPDT to have all of the features and performance, as well as decreasing costs in both monetary amounts and time for development when compared to native development are currently still open questions. This chapter presents a framework for evaluating CPDTs in order to provide recommendations to developers and determine if any trade-offs must be made when using a CPDT.
3.1
Framework
Evaluating a CPDT must be done using a variety of methods. Since there is currently little research available comparing the capabilities of CPDTs to one another and against native platforms, this framework uses standard practices adapted from other frameworks in similar domains to solve this problem.
The aim of this framework is to ensure a scientific and equitable basis can be used to determine the current state of performance for CPDTs. The scope of the framework is limited to applications that are not graphically intensive therefore it should not be applied to game development. Game development is a special case that requires different tools that are dissimilar to those used in standard application development and therefore would require different evaluation methods. Gaming uses different engines and 3D graphics where frame rates are an important factor rather than the items discussed in this framework. Similarly, this research focuses on the smartphone but is extensible to tablets in future work.
Figure 3.1: CPDT Evaluation Framework
The evaluation framework consists of three phases as shown in Figure 3.1. The first phase focuses on determining the capabilities of the framework using a large set of features some tools offer. The second phase provides a set of benchmarks that must be implemented using the CPDT and tested for completion time. This is followed by the final phase where more subjective development experience items are discussed in order to provide context and additional information to understand the result. Completing each of these phases will allow for a full view of the capabilities of the studied tool and the results should be summarized in an evaluation report.
3.2
Phase I: CPDT Capabilities
Phase I, provides an evaluation of CPDT capabilities and features, when tested against a large set of features that development platforms can potentially allow. The evaluation is checklist-based and will require the evaluator to go through the documentation and SDK of the CPDT to see if the features are supported and create a chart showing compatibility. The columns in this chart are grouped into various overarching categories of development such as access to sensors and security. Future iterations of native platforms and CPDTs may have new features included that are not currently listed. The framework is designed such that overarching categories will remain consistent while the sub-elements may need to be updated.
Some features can be considered as supported, not supported or partially supported. Within an evaluation, an overview of the features that comprise this should be included to summarize the result.
3.2.1 CPDT Basic Elements
All CPDT evaluation will bring some common basic questions such as which mobile platforms are supported and what license CPDT is under. This first section should outline these basics that must be known to understand a CPDT and include features found in current native SDKs that simply cannot be otherwise categorized. This section also includes initial and ongoing costs for using the CPDT which may be substantial in some cases.
3.2.2 Development Environment
Learning about the development environment is important when choosing which CPDT best suits one’s needs. This category will include the robustness of the IDE and the type, development languages available to be used with the CPDT and debug environment. These make it easier to gauge the time required to learn new tools. Also important is finding the type of compilation used by the CPDT, many varieties exist and may yield different performance.
From a development environment, the developers need the tools to build applications that are of the same quality of native development tools. They should allow for quick debugging and may provide simple integrated methods of compiling applications. The basic characteristics of this environment must be listed and made available and any integration with specialized APIs documented.
3.2.3 User Experience
In order to create a compelling application, an appealing user interface must be created. Within this section, we will discuss the ability to have user interfaces that
meets interface guidelines provided by each platform vendor. Since these guidelines often change, the framework will be focused on confirming the available flexibility of the CPDT to be able to meet such guidelines.
3.2.4 Device Access
Devices have many internal OS information storage locations that contain device files and application data or user information. In order for some applications to best function, access to information repositories must be an available option in CPDTs. Additionally, having hooks into the lower level device functions and hardware features may enhance performance in specific networking and graphically intense applications.
3.2.5 Sensors
Mobile devices have a multitude of sensors that can gather data to feed into applications for an unlimited number of purposes. New sensors are quickly being added which native SDKs quickly adopt, however CPDTs may not have access to each of these. This section should outline what sensors are available.
3.2.6 Geolocation
Some specific sensors can be used in combination with the mobile OS to enable geolocation of the device. Even though GPS may seem like the only feature needed for adequate positioning capabilities, there are many other possibilities for finding locations. Some tools may not allow usage of each of these methods or may not have the ability to choose which method is used, this should be outlined here.
3.2.7 Notifications
Important information often must be sent to the device from external services in order to notify the application or the user of some information or to perform an action. The availability of a method to perform these actions with reliability is necessary for many
applications. Platform vendors may provide their own method of sending device notifications and integration with these or the ability to use a 3rd party solution should be specified.
3.2.8 Monetization
In order to enable monetization of an application, the CPDT must support a variety of features to give developers the freedom to make use of the pricing scheme they choose. Many free applications may want to use an advertisement platform, which can be provided by the platform vendors or a 3rd party, where paid application developers may be most interested in the ability for their users to make an in-app purchase with ease. Other monetization models may become common and could be further included. The ability to get the application in users’ hands with high visibility through various outlets such as application stores should be included.
3.2.9 Security
Security is of concern for most service providers to ensure the user data is not mishandled and the application source code is not made available to those who should not have it. Methods to securely store information and retrieve it through network connections must be included in CPDTs. Additionally, the CPDT should ensure that when deployed, users do not have the ability to read application source code.
3.3
Phase II: Performance Benchmarks
Performance metrics are an important part of testing the ability of these CPDTs to perform tasks as well as natively built applications. For this phase, the benchmark tests will be developed as a suite and deployed for each CPDT by the evaluator and compared to similarly developed native tests. The difference in the result on the same device from one implementation to another is the important factor being investigated. When a new CPDT becomes available, the framework would require development of
the testing suite using the new tool, while adhering to the requirements set out in the framework.
The performance of the underlying technology and compilation optimizations of each CPDT will be tested by benchmarking an application developed using that tool. The results of this benchmark will show if the CPDT is able to create high performing applications.
As much as possible, tests will follow well-evaluated algorithms that have previously been included in other benchmarking suites for mobile devices or PCs mentioned in 2.4. Some tests may not be able to be implemented on every CPDT or on every platform due to restrictions within the mobile operating system or in the tool itself.
3.3.1 Processor Intensive Benchmarks
First we will evaluate processor intensive applications and look to the SunSpider JavaScript Benchmark version 0.9.1 for AES encryption and input validation algorithms [71]. These are two often used items on mobile devices and will be completed constantly by mobile applications. Using the SunSpider JavaScript code as a guideline for the algorithm, both the AES and input validation benchmarks will produce an output in milliseconds. The tag cloud test can also be used from SunSpider in order to process a large amount of text and determine the frequency of word usage. This can be useful when adding indexed search functionality to an application.
Another testing suite to be used for benchmarks is the Linpack for Android application based on Linpack Java [55]. This is an open source Java application that provides many tests studied over the past 40 years and used in comparison in [17]. Linpack benchmarks measure how fast the system can solve linear equations with Gaussian elimination. This will provide a score rated in Millions of Floating-point Operations Per Second (MFLOPS). This test has been shown to improve in later
versions of Java on the same hardware due to new efficiencies in the virtual machine and will provide useful insight into how the other tools perform [55].
The aim of using any of these well-studied tests is to stress test the CPU to see if code from cross-platform tools is as efficient as lower level native code.
3.3.2 Data Driven Benchmarks
Mobile applications must make use of many data sources, and often combine them into a digestible form for users. In order to simulate testing for this, local and remote databases using the CMER APIs will be accessed, data extracted and sorting algorithms used to sort large arrays of data. Potential tests can be found in [17] and [46] using the heap sort algorithm. Some tests should also use the internal sorting functions provided by the CPDT API.
The CMER APIs are provided to allow for a set of data to be pulled from our CMER server using a local connection. This is important for any remote testing to remove the irregularities based on communication through the cellular network. The implementation of these APIs can vary based on the tests that are implemented in the specific evaluation. This can be very limited or broad based on the framework implementation.
3.3.3 Device Access Benchmarks
Mobile applications have the ability to capture information from multiple sensors and save the information or use it within the application. Tests will be conducted polling the microphone and recording audio clips of random length. The time taken for recording initialization will be documented. A similar test can be conducted for the initialization and shot to shot time for the camera. Since the tests using both native SDKs and CPDTs are conducted on the same device, the characteristics of the device will not affect the result.
3.3.4 User Experience Benchmarks
One of the most important aspects for user experience with an application is responsiveness. These tests will add and remove UI components in quick succession and render new application screens and items. This will be done to perform evaluation of the ability of the CPDT to render interface components and transition quickly from one screen to the next. If a CPDT suffers in UI rendering, it will provide a poor experience for users of applications built using it.
3.3.5 Test Procedure
When ready to move into the testing phase, a set of actions should be completed. Before testing on any device, certain actions should be taken to ensure the tests are as accurate as possible. First update the device system software to the latest version including any bug fixes and patches that are available. Then a factory wipe should be performed to remove any user applications and customizations that may take CPU cycles away from the test.
After installing the benchmark application, perform a full device reset to clear any programs in memory. Furthermore, close any running background tasks that are able to be closed. Run the benchmark with the full 3 external iterations with averaged results showing on the screen and sent to the logging server. A visual description of the test procedure can be found in Figure 3.2.
When testing with multiple CPDTs, each application should be run separately and the results compiled for Phase II of the evaluation report. Once testing is complete, the evaluation report can be compiled using the development experience from developing the benchmarks, to answer the questions posed in Phase III.
Figure 3.2: Benchmarking Procedure
Following these test procedures should ensure outliers are removed from testing for unforeseen delays when completing the test. It is important in all benchmarking to complete all tests on an equitable basis and to repeat them to find average results [21].
3.4
Phase III: Development Experience Discussion
In Phase III, the evaluator will investigate criteria that you are unable to put a measurement on in any simple way. Due to the nature of these CPDTs, not everything can be put into a measure or checklist, they instead need to use the experience gathered while completing Phase I and II to discuss parts of the tool in more depth. This section will lead discussion of some features to enable the reader to make meaningful judgments of the level of functionality provided. The discussion will be led in such a way to limit possible bias from the evaluator and should be used to provide the context to understand the results in Phase I and Phase II.
3.4.1 Tool Related Discussion
Some characteristics of CPDTs can be difficult to speak of in a chart form and require lengthy explanation. Characteristics of the tools discussing how well maintained the CPDT is for updates, new features and bug fixes should be discussed here. Additionally, further discussion of costs associated with using the tool and any ongoing costs can be included. This would include costs associated and features available for any cloud or analytics services offered.
The discussion should also revolve around the development IDE and features supported. Answer if the features and debugging methods included provide the expected level of functionality.
3.4.2 Development Experience Discussion
This section aims to use the development experience gained by completing the other phases of this evaluation framework to discuss items such as relative application size and the strength of the UI toolkit. Flexibility of interface construction can be important and should be noted.
The evaluator should attempt to discuss whether the tool allows for write once, run anywhere development or the level of customization required for different platforms. Additionally discuss the overall robustness of the development experience and provide specific examples of where using the tool is helpful and any drawbacks. Lastly, speak about the learning curve for using this CPDT in this section of the report.
3.5
Summary
Choice of which development tools to use for mobile application creation has never been greater. While building natively for each platform may provide the quickest access to new features, CPDTs have their place in the market. Through
implementation of each phase of this framework, CPDTs available today or new tools released in the future can be evaluated based on the criteria discussed in this chapter. This will provide a balanced look at the features, performance and development experience for any CPDT. Armed with this information, developers are provided with a detailed and unbiased view of tool capabilities and will be able to best decide which meets their criteria for their applications.
This framework requires one to take the overarching themes and apply them in more detail to create adequate tests for not only today’s CPDTs, but future tools. The capabilities, performance metrics and experience discussion in this chapter provide the subject matter applicable to further testing.
In order to test the framework for its suitability it was applied to various CPDTs. This will not only show the strength and usefulness of the framework, it allowed for evaluation of current top tier tools in order to provide immediate guidance on the suitability of each for different applications.
In the next chapter, we will implement this framework to create specific tests that are relevant to the CPDTs on the market today. The experiments that were conducted will be outlined and discussed.
Chapter 4
Implementation and Experiments
Using the framework developed in Chapter 3, we will discuss how this framework has been implemented, in order to evaluate today’s CPDTs. As the framework is designed to outline only the overarching principles, the first step is to break down each phase in detail in order to create a fair set of tests that are relevant to the development tools being chosen for evaluation.
This chapter outlines the tools and devices used for this evaluation. This provides an overview to readers of the evaluation results of the scope of this study. Following this, a detailed outline and specifications for each phase of evaluation is provided.
The implementation presented in this chapter, should satisfy the criteria outlined in Chapter 3, while implementing it and focussing on today’s environment of CPDTs. The scope of the evaluation need not encompass all aspects but should provide a thorough basis of comparison for the CPDTs included in the study.
4.1
Experiment Parameters
In order to provide structure for the experiment, the following subsections will outline the tools included as part of the study, the devices used for testing and any assumptions factored into the results.
4.1.1 CPDTs and Native Development Kits
This study has been conducted using many development tools. Both cross-platform and native development kits will be included to provide a basis for comparison. The tools included and platforms supported can be found in Table 4.1. Entries marked with a dash are incompatible.
Table 4.1: Compatibility Matrix for Tools Included in Study
Android iOS BlackBerry 10 BlackBerry 7
WebWorks - - Compatible Compatible
BB10 Native
SDK - - Compatible -
Android Java
SDK Compatible - - -
iOS Native - Compatible - -
PhoneGap Compatible Compatible - Compatible
Appcelerator
Titanium Compatible Compatible - -
Adobe Air Compatible Compatible Compatible -
MoSync Compatible Compatible - -
While Adobe Air is compatible with BlackBerry 10, it is excluded from our testing due to many issues with the beta software. This can be included in later testing once the platform has stabilized. Appcelerator Titanium’s BlackBerry 7 Beta software was unavailable for testing as its development has now been discontinued. The performance benchmarks in Phase II will be developed for each of the compatible tools and platform combinations. This selection of tools, allows for a wide variety of comparisons to take place with multiple tools running on each platform.
Some CPDTs offer a variety of methods for development. The Titanium development focused on application built and run using their runtime technology while MoSync applications focus on cross-compilation only. Adobe Air applications use the ActionScript, cross-platform development approach.
4.1.2 Devices
Each platform has all tests conducted on a single device using both native and cross-platform tools. The devices to be used are found in Table 4.2.
Table 4.2: Devices Used for Testing
Platform Device Model Software Version
Android Samsung Galaxy S II (i9100)
4.0.3
iOS Apple iPhone 4s 5.1.1
BlackBerry 7 BlackBerry Bold 9900 7.0.0.579 Blackberry 10 BlackBerry 10 Dev Alpha 10.0.4.197
4.1.3 Assumptions
In order to test the CPDTs, certain assumptions must be taken into account. It has been deemed adequat
