EVALUATING THE MOBILE DEVELOPMENT FRAMEWORKS APACHE CORDOVA AND FLUTTER AND THEIR IMPACT
ON THE DEVELOPMENT PROCESS AND APPLICATION CHARACTERISTICS
A Thesis Presented to the Faculty of
California State University, Chico
In Partial Fulfillment
of the Requirements for the Degree Master of Science
in
Computer Science
by
© Michael Gonsalves 2018 Fall 2018
EVALUATING THE MOBILE DEVELOPMENT FRAMEWORKS APACHE CORDOVA AND FLUTTER AND THEIR IMPACT
ON THE DEVELOPMENT PROCESS AND APPLICATION CHARACTERISTICS
A Thesis by
Michael Gonsalves Fall 2018
APPROVED BY THE INTERIM DEAN OF GRADUATE STUDIES:
Sharon Barrios, Ph.D.
APPROVED BY THE GRADUATE ADVISORY COMMITTEE:
Tyson Henry, Ph.D. Bryan Dixon, Ph.D., Chair
Graduate Coordinator
Kevin Buffardi, Ph.D.
PUBLICATION RIGHTS
No portion of this thesis may be reprinted or reproduced in any manner unacceptable to the usual copyright restrictions without the written permission of the author.
TABLE OF CONTENTS
PAGE
Publication Rights ... iii
List of Tables ... vi
List of Figures ... vii
Abstract ... viii
CHAPTER I. Introduction ... 1
Background ... 1
Statement of the Problem ... 2
Purpose of the Study ... 4
Organization of the Study ... 4
II. Review of Related Works ... 6
Background ... 6
Cross-Platform Development Approaches ... 7
Evaluating Cross-Platform Development Frameworks ... 10
III. Methodology ... 13
Overview ... 13
Task Station ... 14
Development Hardware Devices ... 22
Development Framework Environments ... 23
Source Code Evaluation ... 24
Application Characteristics and Performance Profiling ... 25
Development Process Qualitative Differences ... 31
Summary ... 31
CHAPTER PAGE
IV. Results and Discussion ... 33
Overview ... 33
Source Code Evaluation ... 34
Application Characteristics and Performance Profiling ... 39
Development Process Qualitative Differences ... 47
Summary ... 70
V. Summary and Recommendations... 72
Summary ... 72
Limitations and Recommendations for Future Research ... 73
References ... 78
LIST OF TABLES
TABLE PAGE
1. Source Code Lines of Code, Number of Files and Dependencies ... 35
2. Source Code Comparison of Native vs. Cross-Platform ... 35
3. Android Application Package Size and Installed Size ... 41
4. iOS Application Package Size and Installed Size ... 41
5. Android Application Menu RAM Usage ... 42
6. Android Application View Tasks RAM Usage ... 42
7. iOS Application Menu RAM Usage ... 43
8. iOS Application View Tasks RAM Usage ... 43
9. Android Application Startup Times ... 45
10. Android Application View Transition Times ... 45
11. iOS Application Startup Times ... 46
12. iOS Application View Transition Times ... 46
13. Languages and Testing Tools ... 60
LIST OF FIGURES
FIGURE PAGE
1. Transition Diagram of Task Station ... 15
2. Menu Page of The iOS Flutter Application ... 16
3. View Tasks Page of the iOS Flutter Application ... 17
4. Task Details Page of The iOS Flutter Application ... 19
5. Add/Edit Task Page of the iOS Flutter Application ... 21
6. Android Native Screenshots ... 49
7. iOS Native Screenshots... 50
8. Android Flutter Screenshots ... 51
9. iOS Flutter Screenshots... 52
10. Android Cordova/Ionic Screenshots ... 54
11. iOS Cordova/Ionic Screenshots ... 55
12. Flutter widget tree example ... 57
13. Ionic Framework HTML example ... 58
ABSTRACT
EVALUATING THE MOBILE DEVELOPMENT FRAMEWORKS APACHE CORDOVA AND FLUTTER AND THEIR IMPACT
ON THE DEVELOPMENT PROCESS AND APPLICATION CHARACTERISTICS
by
© Michael Gonsalves 2018 Master of Science in Computer Science
California State University, Chico Fall 2018
Mobile application development is an area of software engineering with its own unique development approaches and challenges. The market for mobile applications is highly fragmented, not only across different mobile operating system platforms but also across devices and operating system versions within the same platform. The goal of any application developer is to reach the largest possible audience while minimizing
development time, cost, and effort. The current market consists of two major platforms, Android and iOS. Without the use of cross-platform tools it would be necessary to produce the same application twice to reach both marketplaces.
A number of cross-platform development frameworks are available to developers and each one has its own benefits and drawbacks. This paper details a
comparison between two cross-platform development frameworks, Google’s Flutter and Apache Cordova with the Ionic framework. In order to compare the two frameworks against each other and against the experience of developing native applications for each platform, an application prototype was developed using four different frameworks. A task management application was developed as a native Android application, a native iOS application, a Flutter cross-platform application, and an Apache Cordova cross-platform application using the Ionic user interface framework. A series of source code and
application performance profiling evaluations were performed in order to determine the impact of choosing a cross-platform development framework on both the development experience and the performance and perceived quality of the deployed applications. Several areas of qualitative differences between the development experiences of Flutter and
Cordova are also detailed.
CHAPTER I
INTRODUCTION
Background
Mobile application development is a relatively newer field in software engineering with a potential target audience of billions of users. Statista estimates that around 1.5 billion Android or iOS smartphones were sold globally in 2016 [1]. Such a large possible market makes the development of mobile applications a desirable focus for
software developers. A total of 197 billion applications are estimated to have been downloaded in 2017 alone [2].
Unfortunately, the worldwide market for mobile applications is extremely fragmented. Over the past ten years, the market has moved from several competing mobile operating systems (BlackBerry, Windows Mobile, Symbian, Android and iOS) down to predominantly two platforms, Android and iOS [3]. Even if a developer chooses to focus on just Android and iOS, fragmentation is still a major issue. Joorabchi, Mesbah and
Kruchten [4] describe two different types of fragmentation in the mobile development field, fragmentation across platforms and fragmentation within the same platform. Android and iOS each have their own development environments, software development kits (SDKs), programming languages, and user experience standards. Even within just one platform a large array of possible device types can be found including phones, tablets, watches, TV devices and more. Each device has its own hardware configuration, screen size and
operating system support. This level of fragmentation makes developing mobile software especially challenging if the goal is to reach the entire mobile market.
For well over a decade, researchers have been trying to determine how mobile application developers can reach the largest possible audience with the least amount of development effort. In 2017, El-Kassas, Abdullah, Yousef and Wahba [5] described six broad categories of cross-platform development approaches and listed around 20 different cross-platform tools and frameworks. Some of those tools are available for developers now (MoSync, Apache Cordova, Xamarian, etc.) and some are research or experimental
prototypes (MobDSL, MD2 and ICPMD). With such a large number of options, it can be difficult for a developer to know where to start when choosing an approach to support multiple mobile operating systems [6]. While this problem is not exclusive to the field of mobile application development, it remains an open important research area due to the unique nature of mobile development and the continuous evolution of the market and development frameworks.
Statement of the Problem
The framework chosen for the development of a mobile application for multiple target platforms can have an impact on both the development effort required to create the application and on the perceived quality and performance of the generated application. The goal when choosing a framework is to maximize the reusability of the project’s code base and to minimize the total effort required to support multiple platforms while still producing a final product that is responsive, attractive, easy to use, and consistent across platforms.
Dhillon and Mahmoud [7] describe the ultimate goal of write once, run
anywhere (WORA) software development and how while development tools have come a long way, each approach still has its limitations that prevent it from fully meeting that goal. As the world of mobile application development continues to evolve, there is a need to examine the available cross-platform development approaches and frameworks and determine if the goal of WORA is achievable. This study evaluates two cross-platform development frameworks in order to determine if they allow developers to produce WORA software while maintaining application quality similar to native platform applications.
In order to narrow the focus of this thesis, two cross-platform frameworks were chosen for evaluation. The first, Apache Cordova [8] (often referred to as PhoneGap), is one of the oldest and most mature frameworks available and it has been featured in numerous research articles [9], [10], [11], [12]. Cordova allows developers to use web technologies such as HTML, CSS and JavaScript to build mobile applications that can run on multiple target mobile operating systems. The web applications are packaged into WebViews that are embedded into native platform applications. Cordova by itself does not provide any custom user interface elements for mobile applications. The Ionic framework [13] was used in conjunction with Cordova in this study in order to develop web
applications that look and feel like mobile applications.
The other framework, Google’s Flutter [14], is very young and has not yet been widely included in mobile application development research. Flutter is especially intriguing because of Google’s support, considering that Apple’s iOS is their largest competitor in the mobile operating system market place. Developers using Flutter write their applications using the language Dart and the Flutter engine compiles the application to native ARM
code packaged with the Flutter runtime engine that renders the user interface to allow the application to run on both Android and iOS. In-depth details of the frameworks evaluated in this thesis will be discussed in Chapter IV.
Purpose of the Study
This thesis explores two research questions through the process of evaluating Flutter and Apache Cordova with the Ionic framework.
1. What impact does the choice of a development framework have on the application development process and the long term maintainability of the application?
2. What impact does the choice of a development framework have on the performance and perceived quality of the generated applications?
The main focus of this paper involves the development of four application prototypes: a native Android application, a native iOS application, a Flutter application and an Apache Cordova (using the Ionic framework) application. The development of these applications and the evaluation of the generated applications will be detailed and the results will provide insight into the research questions of this investigation.
Organization of the Study
This thesis is organized into five chapters. Chapter II consists of reviewing related works in the areas of mobile application development and cross-platform
application development approaches and frameworks. In Chapter III, the methodology of
application prototypes and the profiling and evaluation of the prototypes. Chapter IV will explore the characteristics and evaluated metrics of each prototype as well as a review of the key differences between writing applications using the cross-platform frameworks Flutter and Apache Cordova. Finally, Chapter V will include a summary of the
investigation, the limitations of the research, and recommendations for future areas of research.
CHAPTER II
REVIEW OF RELATED WORKS
Background
Mobile application development differs from traditional software development in a number for ways. Application developers have to consider a variety of factors
including the battery life of the host device, the unique lifecycle of mobile applications including frequent interruptions, and the relatively limited hardware capability of mobile devices [15]. Mobile applications also tend to be much smaller than traditional desktop applications and more reliant on third-party libraries and application programming
interfaces (APIs) [16]. The combination of these unique aspects of mobile development and the large potential audience of mobile applications make the field a rich environment for research.
One of the largest challenges of mobile application development is overcoming the fragmentation of the global market. The market place has largely narrowed down to two competing mobile platforms, Android and iOS. Both platforms have their own platform specific characteristics such as different user interface styles and guidelines, different development environments and SDKs, and different app store policies regarding minimum expectations [4], [15], [17]. This makes development of native applications that run on both major platforms effort intensive and potentially unfeasible for small development teams. In
order to reach the largest possible audience, and therefore achieve the highest return on investment for the development effort, this challenge has to be overcome.
There are two approaches to this challenge: developing separate native applications from scratch for each platform or utilizing a cross-platform development framework. Francese, Gravino, Risi, Scanniello and Tortora [12] surveyed mobile development engineers and found that a majority of teams developed separate native applications rather than trying to utilize a cross-platform solution. This approach is potentially the most expensive in terms of effort as there is little opportunity to reuse business logic or user interface code between the two platforms. As Joorabchi, Ali and Mesbah [18] found, creating separate applications can also result in a variety of
inconsistencies appearing in the deployed applications. Smaller development teams lacking large budgets would want to minimize their efforts and costs and ideally a cross-platform application development framework would give them the ability to meet those goals.
Cross-Platform Development Approaches
A wide variety of cross-platform development approaches and tools are currently available for use. An extensive amount of research has been done to try and evaluate the available frameworks and provide best practices for developing applications for multiple mobile operating systems. Xanthopoulos and Xinogalos [19] categorized cross-platform development approaches into four broad approaches: web applications, hybrid applications, interpreted applications and generated applications. These categories will be examined in more detail later in this section. Each type of approach has its own benefits and drawbacks and choosing the best approach for a particular application depends
on a number of factors. Those factors include the ease of deploying applications to multiple platforms, the hardware and data access requirements of the target application, the
preferred programming language and environment of the development team [17], and the importance of native “look and feel” of the application [19]. Ciman, Gaggi and Gonzo [20] add that the licensing of the different tools and the community support available are also important factors.
Web Applications
Creating a mobile friendly web application is probably the most easily portable approach to cross-platform development. Most mobile devices now have web browsers installed that support newer web standards such as HTML5 and modern CSS and
JavaScript. This approach allows developers to quickly be able to support different mobile applications and even potentially desktop browsers as well. There are a number of
drawbacks, however, including the fact that mobile web sites cannot be installed on a device or sold through app stores and they lack access to many of the hardware features of the device [10].
A newer version of the web application approach has been developing over the past few years. Majchrzak, Biorn-Hansen and Gronli [21] described Progressive Web Applications (PWAs) and their advantage over traditional mobile websites. PWAs have the potential to be cached on a device for off-line use and they can access hardware features through the use of bridges. However, this newer approach still lacks consistent browser and platform support and it remains to be seen if the platforms will embrace them due to the fact that the applications are not sold through their marketplaces.
Hybrid Applications
The hybrid cross-platform development approach is a combination of the web application approach and developing an integrated application. The most largely known hybrid cross-platform tool is Apache Cordova which allows application engineers to use web technologies to create applications and host them in a native WebView widget. Using the hybrid approach allows a similar level of portability across platforms as web
applications with the added ability for an application to be sold through app stores and installed on devices. Hybrid applications also have greater access to the features of a device through JavaScript bridges. The reduction of development effort potentially comes with a performance penalty and a lack of native user interface look and feel. Malavolta, Ruberto, Soru and Terragni [22] examined the user reviews of applications created through the hybrid approach and found that Android users perceived their user experience and performance to be poorer compared to natively developed applications. A similar study done by Mercado, Munaiah and Meneely [23] found that hybrid applications were more prone to user complaints regarding usability, reliability and performance. Two big reasons for the perceived difference in overall quality are the potential added overhead from the use of JavaScript bridges and the fact that the user interfaces of the applications do not use native user interface elements. If the performance needs and complexity of an application are not large factors, and meeting a native look and feel user interface standard is not a requirement, this approach can still be a good option even with its limitations [11].
Interpreted Applications
The Interpreted and Generated cross-platform development approaches both generate applications that perform and behave closer to natively developed applications
than either mobile web applications or hybrid applications. The interpreted approach
involves translating source code to platform-specific instructions in real time [5]. A runtime interpreter or virtual machine is packaged with the source code to handle the translation and generate native user interfaces. Appcelerator Titanium, Xamarian and Flutter fall into this category. Interpreted applications tend to have larger memory footprints and overhead due to the included interpretation frameworks but perform and look similar to native
applications [6].
Generated Applications
The generated approach involves compiling source code to native byte code that can be run without a virtual machine or interpreter. Examples of this approach include XMLVM [24] and ICPMD [25]. The advantage of this approach is that the generated applications are fully native applications and therefore do not suffer from the user interface and performance issues that the other cross-platform approaches introduce. This approach is still largely experimental however as cross-compiling code from one platform to another is incredibly complex and APIs and features need to be mapped precisely in order to be supported.
Evaluating Cross-Platform Development Frameworks
A number of cross-platform evaluation frameworks have been developed in order to help choose the approach and tool best suited for a particular application. Ahti, Hyrynsalmi and Nevalanien [9] describe a framework that looks at both subjective
measures (user experience, application appearance and easiness of development) as well as
requirements. Dhillon and Mahmoud [7] break their framework into three phases: the capabilities of a cross-platform development tool, the performance metrics of the tool, and the development experience that tool provides (including available support and the learning curve of the tool). Heitkötter, Hanschke, and Majchrzak [10] created two sets of criteria, infrastructure and development. The infrastructure criteria include the licensing and cost of the tools, the platforms that are supported, the long-term support of the tool, and the ability of the tool to provide the look and feel users expect. Among the development criteria are the development environment provided, the maintainability of the generated applications and the ability to move to another framework in the future with minimal effort.
The type of application being developed may be the biggest factor in choosing the appropriate framework. Applications requiring a lot of graphical support and
animations will likely require a framework that provides better performance and lower overhead [20]. Graphic intensive games should realistically be developed using the native SDKs instead of using a cross-platform framework [21]. Light-weight data driven
applications may be a good choice for the hybrid approach. The hybrid approach may also be ideal for applications that are intended solely for intra-enterprise use and need to be developed quickly [9]. Ohrt and Turau [26] argue that the most important considerations are whether a framework satisfies the developer’s needs and whether the deployed application meets user expectations.
There are a number of limitations when it comes to evaluating the available cross-platform development frameworks and approaches. According to Majchrzak and Gronli [27], any research done can only be a snapshot of the current field as the number of available tools keeps expanding and the existing tools continue to evolve. The
overwhelming number of development approaches makes it an impossible task to evaluate all of the tools at the same time. These factors make it likely that this area of software engineering will continue to be a rich source for future research.
CHAPTER III
METHODOLOGY
Overview
The goals of this investigation include determining the impact a cross-platform development framework has on both the development effort required to create an
application and on the performance and perceived quality of the generated applications. In order explore the research questions of this thesis, four prototype applications were
developed and analysis was conducted in several different areas. The first area included analyzing the source code and files needed to develop the applications and the second area involved the profiling and evaluation of the generated applications to gauge their relative performance and user experience. In addition to the quantitative analysis performed, the experience of producing the different applications also provided insight and information about a number of qualitative differences between the development experiences of the different approaches.
As stated in the introduction of this thesis, there are many different cross- platform development approaches available to mobile developers. In order to make the research manageable, two frameworks were chosen for comparison not only against each other but against the use of the native platform SDKs as well. The two frameworks chosen were Flutter and Apache Cordova (using the Ionic UI Framework). This chapter will detail the application prototype developed for this investigation and the development process for
each framework’s prototype. Native Android and iOS application prototypes were also developed to serve as points of comparison for the cross-platform prototypes.
Task Station
The application prototype developed for this research is a simple task management application called Task Station. The application consists of several pages including a menu page, a page listing all existing tasks in the database, a page for creating new tasks or editing existing tasks, and a page to see the full details of a single task. A transition diagram of the application is shown in Figure 1. This application also includes the following features: the ability to access the camera of the target device, the ability to access the networking feature of the device in order to retrieve data from a remote source, the ability to write to the local storage on the device, and the ability to use Google’s Firebase suite to store and retrieve data and images from the cloud. These feature
requirements provided the ability to investigate a variety of areas to get a comprehensive look at the performance characteristics of the application and any limitations of each development approach. The development of an application more complex than a standard
“hello world” application also provided a realistic codebase to examine.
Task Station uses Google’s Cloud Firestore to store the task information for users. Cloud Firestore is a NoSQL online database that also features offline device data persistence and automatic cloud syncing. Google’s Firebase Storage is used as a storage bucket to allow users to upload and download photos attached to tasks. The Firebase suite is available for a variety of systems including Android, iOS, web (JavaScript) and Flutter
(through first party plugins). The use of Firebase made it possible to have a consistent simple data and file storage system across the four development platforms.
Fig. 1. Transition Diagram of Task Station
Menu Page
The menu page of Task Station is the first screen that users see upon opening the application. Figure 2 shows the user interface of the menu page. The page consists of the application’s logo, a button to navigate to the view all tasks page, a button to navigate to the add task page and button to sign in to the application or sign out if a user is already signed in. A small text field is located under the authentication button to display the current user’s name if a user is currently authenticated through Firebase. If there is not a current authenticated user, the navigation buttons are disabled and the username text field is hidden.
Fig. 2. Menu Page of The iOS Flutter Application
View All Tasks Page
The view all tasks page of the application allows the user to view a list of the tasks they have saved in the Firestore cloud database. The user interface for the view tasks page is shown in Figure 3. Tasks that have due dates in the future are displayed with a sky blue background. Overdue tasks, with due dates before or equal to the current date, are displayed with a red background. Tasks that have been marked complete and not deleted are given a grey background. By default, tasks that have been marked as completed are not displayed in the list.
Fig. 3. View Tasks Page of the iOS Flutter Application
There are a few different ways that a user can filter the tasks that are displayed in the list. The first filtering widget on the page is a select widget that contains a list of task categories (work, school, home, pets, etc.). The default value of this widget is to display tasks from every category. By setting a category filter, users can narrow down the list of displayed tasks. The second method of filtering tasks is setting a date range. The categories for the date range select widget are “all”, “today”, “this week”, and “this month”. The default value is set to view tasks from any date range. Selecting a category other than “all” will only display tasks satisfying the criteria of the selected range. The third and last filtering widget is a checkbox or a switch that allows the user to show or hide tasks that have been marked completed. The default behavior is to hide completed tasks. When a user changes the value of any of the three filtering widgets, the application
updates the visible list of tasks on the page. The filtering options can all be used together. For example, a user can choose to view tasks that are marked with the “school” category that are due on the current date while hiding completed tasks.
The tasks in the list are sorted first by due date and then by priority. Tasks with a higher priority value are displayed before tasks with lower priority values that are due on the same day. Each row in the list displays the name of the task, the task’s
assigned category and the task’s due date. In order to view more details of a task, the user can tap a row on the list to navigate to the task details page for the task in that row.
View Task Details Page
Users can view all of the information associated with a task on the task details page. The selected task’s name, category, due date, priority value and recurrence
information is displayed on the page. If a photo has been taken and saved with the task, the photo is displayed to the user. The user interface for this page can be found in Figure 4.
Task Station has a very simple photo syncing system in the case that a user is logged into the same account on multiple devices. If a task has a saved photo associated with it but the photo is not located in the local storage of the device, the application attempts to download the photo from Firebase Storage. If a task has a photo that is located on the device, the application checks to see if the photo is saved to the cloud. If the photo is not located in the cloud storage, the application attempts to upload the photo. One limitation of this simple system is that if a photo is deleted from the cloud it is not
deleted from the local storage of any other devices that have already downloaded the photo.
Fig. 4. Task Details Page of The iOS Flutter Application
There are two buttons located towards the bottom of the task details page. The edit task button will navigate the application to the add/edit task page in edit mode to allow the user to edit the selected task. The second button is displayed with the titles
“complete task” or “uncomplete task”. If a task is not marked as complete, the user can tap the complete task button to initiate the appropriate completion behavior for the task. If the user selects a task that is marked as complete, the “uncomplete task” button will set the completed value of the task to false. After taking the appropriate action, the
application will navigate back to the view tasks page.
Tasks can be set to recur if the complete button is tapped on the task details page. A task can have its recurrence status set to “none” or it can be given a time frame (days, weeks, months or years) and a number. For example, a task can be set to recur in 3 days or 2 weeks. This feature allows a user to track tasks that need to be done repeatedly without having to re-enter them. If a task has the recurrence category of “none”, it will simply be set as complete and it will remain in the database but by default it will not be displayed in the task list. If a task is set to recur, its due date will be adjusted according to its recurrence category and number.
Add/Edit Task Page
The fourth and final page of Task Station is the add/edit task page. This page is used for two different situations. Users can either navigate directly to this page from the menu page to add a new task to the database or they can navigate from the task details page and edit an existing task. The user interface of the page will be slightly different depending on if the page is in viewed in add mode or edit mode. Figure 5 shows the user interface for the add/edit task page in add mode.
In add mode, the page will be displayed with its widgets set to default values. The first widget is the task name text entry field. This field is used to provide a title or name for the task. The next widget is a select widget used to assign a category for the task. A date selection widget is provided to prompt the user to select a task due date. Next, a text area allows the user to provide a description of arbitrary length. The take photo button will launch the device’s camera screen to capture and save a photo that is paired with the task. Once a user takes a photo, the take photo button changes to a delete
photo button. If the button is tapped after a photo has been taken, the application will delete the photo from memory and the take photo button will be displayed again.
Fig. 5. Add/Edit Task Page of the iOS Flutter Application
Under the photo button, a slider widget controls the priority value of the task. Each task has a priority value between one and ten. Tasks with higher priority values are displayed first in the list on the view all tasks page. The last data field on the page controls the recurrence behavior of the task. By default, the recurrence value is set to none. If the checkbox is checked, a new section appears with a text entry field for a number and a select widget for the recurrence category. At the bottom of the page is the add task button. Tapping this button will cause the application to validate the information
provided for the task and save the task to the database. The user will then be navigated back to the view all tasks page.
If this page is viewed in edit mode, the values of each of the data entry
widgets will be set according to the data of the selected task. In addition to the difference in default values, the user interface of the page also has some behavioral differences. If a photo has been taken previously for the selected task, the photo will be displayed and the photo button will be set to “delete photo”. Deleting a photo in edit mode will remove the photo from local storage and cloud storage if necessary. The user is first prompted by an alert dialog that the photo will be deleted before any action is taken.
At the bottom of the page two buttons will be visible, the edit task button and the delete task button. When the user is finished making changes to the task they can tap the edit task button to save the changes to the database and navigate back to the view tasks page. If a user taps the delete task button, an alert dialog will warn them that they are attempting to delete data. If the user confirms their request, the task’s data will be deleted from the database and the application will navigate to the view all tasks page.
Development Hardware Devices
Xcode is required in order to develop applications for iOS devices and Xcode is only available for Apple computers running the OSX operating system. In order to meet this requirement, a Mac was chosen as the development machine. The computer used for developing the applications in this experiment was a 2012 series MacBook Pro running macOS High Sierra 10.13.6. In order to get a realistic sense of the performance
instead of relying on software emulators. A 2016 Samsung Galaxy S7 running Android 8.0.0 was used for the Android device and a 2013 iPhone 5s running iOS 11.4.1 was used for the iOS device.
Development Framework Environments Native Android Development
The native Android version of Task Station was developed with the Android Studio integrated development environment (IDE). Android Studio version 3.1.4 was used for both the writing and the compilation/building of the application. Version 26.1.1 of the Android SDK, version 28.0.0 of the Android SDK platform tools and version 1.8 of the Java Development Kit were used in the build process. The primary programming language for native Android development is Java, XML is used to define the user interface layouts and configuration information.
Native iOS Development
Xcode 9.4.1 was used as the IDE for the development of the native iOS application. Swift 4.1 was used as the primary programming language. User interface development was managed through the use of Xcode’s visual layout editing tools called storyboards.
Flutter
Android Studio 3.1.4 was used as the IDE for the development of the Flutter applications. The primary programming language was Dart 2.0. Version 0.5.1 of Flutter was used along with version 173.4700 of the Dart Android Studio plugin and version
27.1 of the Flutter Android Studio plugin. The same Android platform tools that were used for the native Android application were used for the compilation of the Android Flutter Runner application. Xcode 9.4.1 was used for the compilation of the iOS Flutter Runner application.
Apache Cordova and the Ionic Framework
The IDE used for developing the Apace Cordova with the Ionic framework application was Visual Studio Code 1.25.1. Cordova 8.0 was used for compiling the Cordova applications along with version 7.0.0 of the Cordova Android Platform tools and version 4.5.5 of the Cordova iOS Platform tools. Version 3.9.2 of the Ionic framework was used with version 4.1.0 of the Ionic command line interface tools. The programming and markup languages used included Typescript, Angular 5.2.11, HTML and CSS. The same Android Studio, JDK, Android SDK and platform tools that were used for the native Android application were used to compile the Android Cordova application. The same Xcode tools that were used for the native iOS application were used for the iOS Cordova application.
Source Code Evaluation
Analyzing the source code of a mobile application can provide a lot of information about the amount of development effort it takes to write the application and how much effort it will take to maintain the application over time. Syer, Adams, Zou and Hassan [28] examined the source code of Android and BlackBerry native applications and focused on two key dimensions, the characteristics of the source code and the number and
source code of the applications they used the metrics of the number of files containing source code, the number of classes in the application and the total lines of code in the application. In theory, applications that have larger codebases and more files will require more effort to develop and maintain. The number of library dependencies that an
application has can make it less portable to other frameworks and it could also add performance overhead and instability over time as the libraries change.
This investigation’s analysis of the source code of the different generated application prototypes includes an examination of the number of lines of code, the number of user managed files and the number of dependencies of the source code. The platform portability, how much of the source code can be reused in order to develop applications for both Android and iOS, was also examined. The source code metrics tool Tokei was used to count the lines of code in the user managed source files and main configuration files for each prototype. In order to determine the number of dependencies each prototype required, the configuration files (build.gradle files for Android, podfiles for iOS, pubspec.yaml files for Flutter and package.json files for Apache Cordova and Ionic) were examined and the required libraries were counted in each file.
Application Characteristics and Performance Profiling
Substantially more research has been done on comparing the benchmarking results of applications developed using cross-platform frameworks. Ahti, Hyrynsalmi and Nevalainen [9] presented an evaluation framework that used the quantitative measures of application starting time, device random-access memory (RAM) usage and installed application size. Another study by Willocx, Vossaert and Naessens [6] examined the
response time of an application as well its CPU usage, RAM usage and required disk space (both application package size and installed application size). Majchrzak and Gronli [27] took their analysis a little deeper and studied the comparative performance of applications when they accessed hardware elements of the device such as database manipulation or using the device’s camera.
After the completion of the four application prototypes, a series of application profiling tests were done all of the applications on both an Android device and an iOS device. The goal of these tests was to collect information in the following metrics: the size of the application package before installation, the memory footprint of the
application when installed, the application’s use of the device’s RAM while running, the startup time for each application from the user’s point of view and the response time for view transitions. This data provides answers to the question of whether a performance penalty exists when a cross-platform development approach is used when compared with a natively developed application.
Application Package Size and Installed Footprint
Determining the size of the Android application packages was done by examining the output folders in the project folders of each project after the latest build was generated. After a project is built, Android studio produces an .apk file that is used to install the application onto a target device. For the purposes of this experiment, all three of the application packages examined were from debug builds. This means that they are not necessarily as compressed as the final application release builds and they contain debugging information that allows a user to debug or profile the application running on a
device or emulator. After locating each .apk file, the file size was gathered by examining the file’s properties.
The memory footprint of the application after it was installed on an Android device was measured by reviewing the storage settings of the target device. Only the actual size of the installed application was measured, any document or cache data associated with the application was ignored.
The method of collecting the package file size for the iOS applications is not as precise as the one used for the Android applications. Apple only allows developers to upload applications if they have a paid developer’s account and the final packaging of an application happens during the upload process. Because it was not possible to access a final .ipa package file for the iOS applications, an estimation was used instead. This estimation was determined by archiving each project in Xcode using a profiling build type. After each project was archived, the contents of the archive were examined and the payload application file inside the archive was moved to a separate folder and
compressed. The compressed file is not necessarily the same size of a final App Store package but it provides an estimation of the formally packaged file.
The method for determining the final installed size of an application on an iOS device was similar to the method used for the Android applications. The storage settings of the iPhone were examined and only the size of the application was measured. All associated data and cache information was ignored.
Memory Usage Profiling
Due to the fact that this research involved producing applications for two different operating systems, a strategy for profiling the RAM usage of each application had to be developed separately for each platform. The profiling of the Android
applications was completed using the Android Profiler tool packaged with Android Studio. Apple’s Xcode Instruments profiling tool was used for the testing of the iOS applications.
For each application two measurements were taken, the RAM usage of the application after the application was launched and the menu page was active and the memory usage of the application after the application navigated to the view tasks page. Both profiling tools displayed the RAM usage of the application as a chart that ran for the length of the application’s active lifetime. The measurements were taken at the point that the RAM usage leveled off after initial shifts during either the launch of the application or the transition to a new page. For the purposes of this experiment, all of the profiled applications were signed into the same account and used the same dataset of tasks. The profiling process was repeated a total of three times for each application. The purpose of the series of tests was to determine the comparative memory requirements of each application and to determine if there was a performance penalty due to any overhead involved in the non-native applications.
Application Start-up Time and View Transition Time
There are a variety of different strategies that could be used to measure the time it takes for an application to launch and be responsive and the time it takes for an
series of console logging statements that print out a time value that is compared to a previous measurement taken before launching the application or changing views. This strategy could potentially give researchers an accurate measurement of time but it can be difficult to standardize the implementation of the strategy across multiple different development frameworks. Including the console logging statements is also potentially risky because their inclusion changes the application itself and the test is not necessarily measuring the original application’s code. This value would still be useful for comparison but it would not be an absolute measurement of time.
Another potential strategy is to measure startup or transition time according to the user’s perspective by capturing video of the process of launching the application and navigating between views. For the purposes of this experiment, a video was made for each of the six applications (native Android, Flutter Android, Cordova Android, native iOS, Flutter iOS and Cordova iOS). The videos were produced by filming the screen of the test devices with an external camera. Each video begins with footage of a user’s finger tapping the appropriate application’s icon on the device’s screen. After the application icon is tapped, the application starts to launch and first displays a splash screen. After the splash screen fades away, the initial menu page is shown followed by the user’s finger tapping the add task button and navigating to the add/edit task page in add mode. After the add/edit task page loads, the user navigates back to the menu page by either tapping the hardware back button on the device or tapping the backward navigation button in the application’s navigation bar.
After all of the videos were filmed, the footage was edited into clips using Apple’s iMovie application. Each video was edited into three clips as follows:
1. Starting with the point of contact of the user’s finger with the application’s icon up until the application’s splash screen fades and the menu page is fully loaded and visible.
2. Starting with the point of contact of the user’s finger with the add task button on the menu page up until the add/edit task page is fully loaded and visible.
3. Starting with the point of contact of the user’s finger with the appropriate backward navigation button up until the menu page is fully visible again.
After the editing process was completed, the length of each clip was examined in iMovie and recorded. The complete process was repeated so that each application was filmed and the length of the edited clips was recorded a total of three times. In order to make the device’s environment as consistent as possible, no other applications were running in the foreground while each application was launched and each application was removed from active memory after each filming period was complete.
There are a couple of limitations to this strategy. The first limitation is that it is difficult to pinpoint the exact moment in the footage when an event occurs so the editing of the clips can be subjective. The second limitation is iMovie does not provide precise time measurements for the length of a clip, it only displays the length in terms of seconds (ex. 1.2 seconds, 1.5 seconds, etc.). These limitations prevent the strategy used in this experiment from resulting in precise time values for each activity measured but it does give provide a useful comparison point from the perspective of the user’s experience with the application.
Development Process Qualitative Differences
There are a number of differences in the development experiences of the approaches and frameworks explored in this paper. In addition to the source code analysis and application profiling described in this chapter, the following aspects of mobile
software development will be described thoroughly in Chapter IV:
• User interface elements and styling
• User interface design
• Learning curve
• Debugging strategies and tools
• Live reload behavior
• Device filesystem and hardware access
• Google Firebase support
• Licensing and cost
• Application splash screens and icons
• Application compilation and deployment
The process of developing the four prototype applications provided details for the different qualitative categories listed. These categories are important areas to explore in order to come to conclusions about the research questions of this thesis.
Summary
The combination of these areas of analysis provided useful information and helped to shed light on the research questions of this thesis. The analysis of the source code
and development process helped to determine the impact a development approach has on the effort required to develop and maintain an application for multiple platforms. The application performance profiling portion of this experiment helped to answer the question of whether the generated applications have any noticeable performance or quality
characteristics based on the approach used to develop them. The qualitative comparisons of the different development approaches rounds out the larger picture of the development experience for each approach. The results of these analyses will be reviewed in the next chapter.
CHAPTER IV
RESULTS AND DISCUSSION
Overview
With so many cross-platform development framework options available, it is important to thoroughly research development tools before choosing the best option for a project. Do the available tools provide the device hardware access that the application requires? Do the available tools help save on the development time and effort in order to produce applications for multiple platforms versus producing native applications for each platform? Are any measureable performance or memory footprint penalties introduced into applications developed using cross-platform frameworks? What other impact does the choice of development framework have on both the application development process and the characteristics of the deployed application? This experiment was intended to find answers to these questions through the process of evaluating two of the cross-platform frameworks currently available, Flutter and Apache Cordova with the Ionic framework.
After completing the development of the four Task Station prototypes (native Android, native iOS, Flutter and Apache Cordova with the Ionic framework), a series of source code, memory footprint and performance profiling evaluations were performed. The source code evaluations helped to determine the impact that each strategy had on the development experience in terms of initial effort and long term maintainability. The application characteristics and performance profiling evaluations were used to examine
the impact that each strategy had on the performance and perceived quality of the deployed applications. The ideal goal of a developer is to minimize the effort it takes to create and deploy software without sacrificing the quality of their products.
There are also a number of key differences between the development experiences of using Flutter or Apache Cordova and developing native applications for Android and iOS devices that cannot be easily measured through tests. Reviewing the available documentation of each framework and using the framework tools to develop the Task Station prototypes provided insight into these differences and the impact they have on the developer. Some of these qualities may be of a subjective nature due to the preferred programming language and development environment of the developer. Other areas such as the documentation available for the frameworks and the integration of existing software libraries such as Google Firebase are important things to consider to determine if the tools are sufficient for the project being designed. In this chapter the results of the quantitative evaluations will be detailed along with the qualitative features and characteristics of each development strategy.
Source Code Evaluation
The static source code evaluation portion of this thesis was focused on the number of lines of code written for each prototype, the number of user managed files for each prototype and the number of external dependencies for each prototype. By
examining these measurements it is possible to draw some conclusions about how the choice of a development framework or strategy impacted the effort required to deploy
number of user managed files and external dependencies for each of the four
development frameworks. Table 2 combines the data for the native Android and iOS prototypes to give a better look at the differences in strategies between using cross- platform development frameworks and writing native applications for both platforms.
Table 1 – Source Code Lines of Code, Number of Files and Dependencies
Application LOC Files Dependencies
Native Android 1,870 24 4
Native iOS 1,534 12 6
Flutter 1,267 16 7
Cordova/Ionic 1,123 27 36
Table 2 – Source Code Comparison of Native vs. Cross-Platform
Application LOC Files Dependencies
Native Combined 3,404 36 10
Flutter 1,267 16 7
Cordova/Ionic 1,123 27 36
Lines of Code
The number of lines of code written for an application provides a metric to evaluate the comparable effort it takes to write a piece of software. The data can also be used to determine how easy it would be to maintain the software throughout its lifespan. Lines of code is not a perfect measurement as some of the differences in count between programs could be due to the code style of the framework. Generally speaking, if one codebase has more lines of code than another it could be said that it took more developer effort to write and because there is more code to maintain it could require more effort to revise or update the code in the long term [29].
According to the data in Table 1, the native Android application required the most lines of code written with 1,870 lines. In part, this could be attributed to Android requiring more lines of code to connect elements from the user interface to the
application code and the fact that Android separates the layout functionality into separate XML files for each page of the application. Both the Flutter and Cordova prototypes required fewer lines of code than their native platform counterparts. This is especially impressive when the numbers for the native prototypes are combined. According to Table 2, the strategy of developing two native applications required a total of 3,404 lines of code while the cross-platform framework strategies required close to 1,200. This means that by using Flutter or Cordova the number of lines of code to write or maintain is only a little more than a third of the lines required for producing separate native applications for both platforms. The Flutter application codebase was a little more verbose than the Cordova codebase but not by a significant margin. Both cross-platform frameworks had codebases that were shared 100% between the Android and iOS applications with the exception of the platform specific configuration/manifest files. This seems to suggest that the ideal of write once run anywhere code is possible when using Cordova or Flutter as the development framework.
User Managed Files
Each development framework has its own project structure. Applications developed using the native Android SDK are broken down into Java class files, and XML layout and resource files. Native iOS applications are made up of Swift class files and Storyboard layout files. Flutter applications combine the functionality and layout of the
class files, HTML files for layouts and CSS files for styling details. If a framework requires developers to maintain a larger number of files it may lead to a cleaner
separation of concerns but it could also mean that the project is more complex to manage. For the purposes of this thesis, a user managed file is a file that a typical developer would either create or edit in a project. All of the development frameworks examined had a number of files that were generated or modified automatically during the build process.
Table 1 includes a count of user managed files for each of the four application prototypes. The Cordova project contained a total of 27 files, this increase is due to the fact that the code for each page of the application is broken down into separate
Typescript, HTML and CSS files. By comparison, the Flutter project required 16 files. The Cordova/Ionic project structure has the advantage of better separating the concerns of the layout and behavior of an application page but it can also result in a larger number of files to manage as projects grow.
Table 2 provides a comparison of the number of user generated files for Flutter or Cordova against developing separate native applications for Android and iOS. Both of the cross-platform frameworks provided a significant reduction in the number of project files that a developer would have to manage. After examining the lines of code and the number for files for each project it is clear that a cross-platform approach can reduce the overall effort required to deploy software to both platforms.
Dependencies
Applications that rely on a large number of external dependencies can potentially be more difficult to maintain in the long term [30]. As software libraries
continue to evolve, changes can be introduced that result in bugs, errors or
incompatibilities in applications that use them. There is also the potential that external libraries could introduce security vulnerabilities that the developer may not be aware about [31]. This experiment included an evaluation of the number of required
dependencies of the codebases of each Task Station prototype.
The number of dependencies for each version of Task Station can be found in Table 1. The native Android and iOS applications mostly only required the Google Firebase libraries to be integrated into their codebases. Flutter required the Firebase libraries and a few plug-ins including the “image_picker” plug-in for device camera access, the “path_provider” plug-in for device filesystem access and the “intl” plug-in for working with dates. The Cordova/Ionic project required the inclusion of 36 different libraries. One reason for the large number of dependencies is that Apache Cordova does not provide a user interface framework by itself and all device hardware interactions require separate plug-ins. The Ionic framework was included in order to provide a user interface suited for mobile applications. The Cordova project also required the inclusion of a variety of Angular libraries, Cordova device libraries, Ionic libraries to wrap the Cordova libraries, the Firebase libraries and the AngularFire2 library to provide Firebase support using Angular. The Cordova/Ionic codebase was by far more dependent on external libraries than any of the other codebases. This could mean that projects built with Cordova and Ionic could be difficult to maintain over time as changes to the required libraries could result in “application breaking” behavior.
The source code evaluations provided information to help answer the question
maintenance experience of a mobile application. Both the Flutter and Cordova/Ionic applications required fewer lines of code than writing separate native applications. The cross-platform applications also required the management of fewer files. The Flutter application required several additional dependencies than the native applications but the Cordova/Ionic application required an extensive list of dependencies. It appears that with the exception of a large number of external dependencies of the Cordova framework, the cross-platform frameworks simplified the development experience by requiring only a single smaller codebase when compared to developing multiple native platform
applications. This research question will be discussed further later in this chapter in the qualitative evaluation portion.
Application Characteristics and Performance Profiling
The purpose of the application performance profiling evaluations was to help answer the question of what impact the choice of a development framework has on the performance and perceived quality of the generated application. The source code evaluation results suggest that the use of a cross-platform development framework can significantly reduce the effort required to create and maintain software for multiple platforms. Does that reduced effort come with a tradeoff in the quality of the deployed application? There are a number of benchmarking tests that can be performed to judge the performance of software. The tests performed in this experiment include a look at the compiled package size of an application, the memory footprint of an installed application, the RAM usage at two different points in the life of a running application and the time it took for an application to launch and to transition between views. The results of these
tests will help to determine if there is a trade off in terms of development effort and application quality.
Application Package Size and Installed Footprint
Flutter and Cordova/Ionic reduced the amount of code it took to write applications for both Android and iOS devices. The user managed codebases may be smaller, but how much additional overhead is included in the deployed applications? Flutter applications include the Flutter engine in order to render the user interfaces of the applications. Cordova applications package the Typescript, HTML and CSS application code into a WebView that is embedded into a native application. Both frameworks require the addition of libraries or plugins in order to manage interaction with the filesystem or hardware of devices. All of this additional overhead could potentially impact the size of the compiled applications. In general, users prefer applications that have smaller packages as they take less time to download and require a smaller footprint of device memory when installed [26].
Table 3 and Table 4 display package size and installed size of the deployed Android and iOS versions of Task Station. A review of the data suggests that the Cordova/Ionic applications had the smallest package sizes and the smallest installed memory footprints. The web based applications were even significantly smaller in size than the native platform applications. The Flutter applications had larger packages and required more memory to install than both the Cordova/Ionic and native platform applications. Cordova’s overhead does not seem to be apparent in the results of this particular evaluation but the Flutter SDK and the Flutter engine do have a notifiable
the large size of the project’s included Firebase pod files. The default build settings for Xcode do not appear to streamline the size of project archives for the type of build used in this experiment.
Table 3 – Android Application Package Size and Installed Size Application Package (MB) Installed (MB)
Android Native 5.70 5.69
Android Flutter 16.9 41.29
Android Cordova/Ionic 1.40 2.79
Table 4 – iOS Application Package Size and Installed Size Application Package (MB) Installed (MB)
iOS Native 78.90 66.20
iOS Flutter 22.00 74.00
iOS Cordova/Ionic 3.80 7.90
Memory Usage Profiling
The package size and installation memory footprint evaluations provided some information about the impact of a development framework on the characteristics of deployed applications. Some measurement of the memory overhead of a cross-platform framework could potentially be made through those evaluations but it could also be measured in the amount of memory required during the life of a running application. In order to attempt to measure the RAM usage of each type of application, measurements were taken using platform specific profiling tools at two points during the life of the application. The first point was after the application finished launching and the menu page was available for user interaction. The second point was after the application
transitioned to the view tasks page and the list of tasks was loaded. The process of taking measurements was repeated a total of three times. These measurements provided more detail regarding the performance overhead of the Cordova/Ionic and Flutter applications.
The RAM usage results for the Android applications can be found in Table 5 and Table 6. The most obvious observation is that the Flutter and Cordova/Ionic
applications had higher RAM usage than the native Android application. It is difficult to draw conclusions about a comparison between the RAM usage of the Flutter and
Cordova/Ionic applications themselves as the values varied each time the measurements were taken and no clear trend was found. The results for the menu page showed
Cordova/Ionic requiring slightly more memory while the view tasks page results showed the opposite.
Table 5 – Android Application Menu RAM Usage
Application Trial #1 (MB) Trial #2 (MB) Trial #3 (MB)
Android Native 97 99 88
Android Flutter 150 143 139
Android Cordova/Ionic 165 147 147
Table 6 – Android Application View Tasks RAM Usage
Application Trial #1 (MB) Trial #2 (MB) Trial #3 (MB)
Android Native 115 101 103
Android Flutter 176 174 180
Android Cordova/Ionic 164 175 155
The results of the iOS application RAM usage measurements are contained in Table 7 and Table 8. The iOS results look very different from the Android results. The Flutter prototype had the highest RAM usage for both the menu page and view tasks page measurements. During the time the menu page was active, the native iOS prototype had a lower RAM usage than the Cordova/Ionic prototype but the results were flipped when the view tasks page was active. It may be that the Cordova/Ionic prototype requires a higher baseline RAM level but then stays fairly consistent throughout the application’s lifetime while the memory usage of the other prototypes varied according to the workload.
Table 7 – iOS Application Menu RAM Usage
Application Trial #1 (MB) Trial #2 (MB) Trial #3 (MB)
iOS Native 64 64 64
iOS Flutter 101 102 102
iOS Cordova/Ionic 78 89 89
Table 8 – iOS Application View Tasks RAM Usage
Application Trial #1 (MB) Trial #2 (MB) Trial #3 (MB)
iOS Native 93 94 93
iOS Flutter 125 128 128
iOS Cordova/Ionic 71 77 75
After reviewing the RAM usage data for the Android and iOS applications it appears that the overhead of the cross-platform frameworks has a noticeable impact on memory requirements. The results of this evaluation were consistent with the findings of other previous studies in that the cross-platform applications had higher memory usage while active [11], [6], [26]. The Flutter prototypes for both Android and iOS required the
highest amount of memory and the Cordova/Ionic required more memory than the native applications with the exception of the view tasks page measurement for the iOS device. Overall it appears that the memory requirements for the Cordova/Ionic application vary slightly but are fairly consistent while the Flutter application requirements increase with higher workloads (such as more widgets to render or data to process). The Task Station prototypes are not data intensive applications so it is possible that other types of
applications may behave differently if the same type of evaluations were performed.
Application Start-up Time and View Transition Time
One of the most important aspects of an application in terms of perceived quality from a user’s perspective is the responsiveness of the application. Does the
application load quickly? When the user interacts with the application’s interface does the application respond right away? Mobile application users expect that an application will respond to interaction immediately and perceive applications that do not meet that expectation as poorer in quality [6].
Three different evaluations were performed in order to attempt to measure the responsiveness of each version of Task Station. The first evaluation was a measurement of the amount of time it took between the point at which that the user tapped the icon of the application on their device and when the menu page of the application was fully rendered. The second evaluation measured the time that elapsed between the user tapping the add task button on the menu page and the add task page rendering fully. The last test measured the time between the point where the user requested that the application navigate back to the menu page and the point where the menu page was fully visible
