DATA SYNCHRONIZATION ON MOBILE DEVICES

(1)

FINAL BACHELOR PROJECT AT

UNIVERSITY COLLEGE OF NORTHERN DENMARK

S

OFTWARE

D

EVELOPMENT

DATA SYNCHRONIZATION ON MOBILE

DEVICES

UNDER THE SUPERVISION OF MOGENS HOLM IVERSEN

ZLATIMIR ZAHARIEV AND GEORGI DAMYANOV

JUNE 16, 2015

(2)

Abstract

This report describes techniques and strategies for data synchronization on iOS and Android devices. Hence, the research is based on problems that may occur in future stages of the Skonto project. Currently, the Android and iOS apps download and store data from the server and present it to the final user. With the increase of data and a demand for an upload from client to server, the current implementation may not be sufficient. Thus, the solution presented in this report provides a set of relevant techniques for optimal data transfer, data storage and

(3)

Introduction

During our internship at Webs&Apps we were trusted with the development of the Android and iOS versions of the Skonto mobile app. Its purpose is to enlist shops, which provide exclusive discounts to registered users. The app users have a digital membership card and get notified when they are close to a shop with a discount. While approaching the first release of the product, a concern was raised regarding how the system would scale with significant data growth and additional requirements.

Architecture

The current architecture of the system, Figure 1, was developed before the start of this project. Influenced by the initial requirements and budget, the design considerations were not backed by a solid research.

The system consists of a backend, Android, iOS mobile clients and a small user-friendly web client for the administrator/owner. The backend is storing all the data in a database and exposes it for the mobile clients through a web service. When the mobile apps are open, they check the time of the last update and download any new or updated data from the server.

Considering the initial expectation that the data in the system will not change often, a local database was implemented on the mobile clients. As a result the data is available offline and the user can quickly browse through it.

(6)

Problem statement

With the increase of data and the current implementation of the architecture, a negative impact on performance may be experienced. Additionally, it does not support upload from mobile device to server and offline changes, which may be required for future releases.

Considering those problems a new solution has to be designed. This requires a research regarding data synchronization and storage. Therefore, the following final project topic was found as most suitable:

How to optimally synchronize data between mobile applications and server?

Research method

For the purpose of this project a secondary research also known as desk research method will be used. It consists of finding data from previous researches and reaching a conclusion. This method is the most viable for the project, since data synchronization is not something we have previously worked on, but it as a researched topic. The approach consists of exploring the available

strategies for data synchronization and storage on mobile devices, while elaborating on their advantages and disadvantages. Additionally, the current Skonto data synchronization strategy will be analyzed and possibly improved.

(7)

Data synchronization

With the recent development and improvement of network communications, data

synchronization has become a fundamental part of almost every mobile application. As a result, nowadays an instant access to real-time data is available; information is accessible and editable offline, while providing the user only with the most relevant data. However this comes at a price. The implementation of the synchronization adds complexity to every project and variety of challenges like: data consistency, bandwidth limitation and data-freshness.

Synchronization patterns

In order to handle the challenges, involved with data synchronization, there are a number of strategies, which are adaptable in different context, inspired by the experience of other people, personal knowledge and research.

In order to describe every pattern, format similar to the one used in Design Patterns-Elements of Reusable Object-Oriented Software will be adopted. The patterns are divided into three

categories [1]:

 Synchronization mechanisms

 Data storage and availability

 Data transfer

Synchronization Mechanisms

The data synchronization mechanisms can be considered from two perspectives: uploading and downloading. The downloading is the transfer of data from the remote server to the mobile application, while uploading is the transfer from the mobile application to the remote system. Depending on the context and factors such as application requirements, fault isolation and user experience, different mechanisms can be implemented.

Asynchronous Synchronization

Synchronize data asynchronously without making the user interface irresponsive.

Problem

Blocking the interface while the data is being synchronized will result in a very poor user experience. The user may not be aware whether the app is doing something or it just blocked, since there is not visual indication about it. In fact Google went as far as not allowing networking calls, on the main thread in Android, thus preventing the UI from freezing on events like data synchronization [2]. Apple also recommends avoiding synchronous networking calls on the main thread [3].

(8)

Applicability

The applicability considerations for asynchronous data synchronization can be viewed from downloading and uploading perspectives:

 Downloading new data asynchronously has the advantage of not preventing the user interaction with the app. Applications, which are functional with stale data, can benefit from this. However, data freshness is a crucial factor for other applications like stock trading or betting, where asynchronous synchronization is not applicable.

 Uploading can be done without the user waiting for the transfer to complete. Once the synchronization is triggered he/she can continue using the app and when the process is completed an appropriate alert can be shown.

Solution

As shown on Figure 2, the synchronization is triggered by an event in the app and it is done without blocking the main thread. Once the task is done the result is returned to the code on the main thread and the app handles it.

Visual Representation

(9)

Consequences

Advantages

 Not blocking the main thread and the UI

o By using Asynchronous Data Synchronization, interface freezing is avoided while the information is being synchronized with the remote server.

Disadvantages

 Increased implementation complexity

o While it provides a much better user experience, implementation of Asynchronous Data Synchronization requires handling multiple threads and different execution paths, which results in more development time and more potential for bugs to hide.

Examples

A possible example is the post feature in social medias like Facebook. When the user uploads a post or a photo, after pressing the postbutton, the app continues to function properly and allows normal interaction. When the upload is finished the user gets visual indication whether it was successful or not.

Synchronous Data Synchronization

Manage a data synchronization event synchronously. A typical synchronous mechanism will return the result to the final user/client as soon as the execution is finished. While it occurs, the user interface is blocked.

Problem

This is an era of real time systems. "Customers want to see what the status of their order in e-shop is; the status of a parcel delivery - a real time parcel tracking; what the current balance on their account is." This shows the need of systems, which are constantly up to date and enable smooth manufacturing process in real-time. Additionally, many mobile applications are

connected to artificial intelligence systems and business analytics systems, which are dependent on the newest information. A different scenario may be that the mobile application system relies on a response for an action from a remote system before performing another action.

To prevent an application from entering unknown, non-functional states, a developer must ensure an application blocks until data synchronization is completed (successfully or unsuccessfully) [1].

(10)

Applicability

When synchronous synchronization is applied, the user has available the most recent

information, required for the functionality of the system. If asynchronous mechanism is adapted, the user runs the risk of working with stale data. Additionally, the synchronous data

synchronization may be crucial in mobile applications offering online services. If the user has the option of placing orders, It would be reasonable to confirm the success or failure of the purchase synchronously and make the user wait, rather than letting him/her do orders asynchronously, allowing the user to submit more than one order without acknowledgment of the previous ones [1].

Solution

Initially, the data transfer can be triggered vie event (such as user action). The process has to be performed synchronously. The application blocks and proceeds after the result is received. This process is visualized on Figure 3.

(11)

Consequences

Advantages

 The state of the mobile application is easily managed

o The synchronous mechanism adds simplicity, because of the instructions being executed in order. Compared to the synchronous approach, the asynchronous implementations significantly increase the complexity. It requires multi-threading and proper handling of many paths of executions and states. By using synchronous data synchronization, those extra paths and states can be eliminated.

Disadvantages

 Blocking user interface thread

o The use of synchronous data synchronization results in user frustration. When such an event occurs, the application may become unresponsive. Usually the data transfers are performed asynchronously on the

background, while showing a loading dialog. This does not block the user interface; it provides feedback regarding the synchronization and gives the opportunity to be canceled.

Examples

 A user opens an application. A loading dialog is shown while the newest data is downloaded. The application continues only after the result is received.

 User open banking applications and enters username and password. The credentials are validated, before letting the user further.

 A user purchases desired item from a mobile application. The order is processed synchronously. While such an event takes a place, loading dialog is shown. Finally, result for successful or unsuccessful order is presented to the user.

(12)

Data Storage and Availability

In many cased the data set is too large to be stored locally or has to be accessed offline. Data storage and availability patterns present a solution to these issues by managing how much data should be stored on the mobile device.

Partial Storage

Download and store data on demand in order to reduce unnecessary storage use.

Problem

The storage capacity on mobile devices is scarce; users have a lot of their own media and resources stored on the phone. As result they are often unwilling to use applications, which take too much of their valuable storage. Additionally, downloading all the data for the app at once while connected to a mobile network can be expensive.

Applicability

The app does not need the entire data set at the same time; therefore it can load most of the data on demand. For example an email app may pre-download only the recent emails and not store the whole communication archive on the device, since it will rarely need it.

Solution

Instead pre-fetching the whole data, the app synchronizes the information by using triggers only when it’s needed. A popular approach is to use a Virtual Proxy, which ‘creates expensive objects

on demand’ [4]. The proxy objects provides interface for accessing and manipulating the business object, however it only loads the data if it is needed. This is done with a Data Access Object, whichdetermines if the data is available locally and if not it loads it from the data source [5].Figure 4, visualizes the solution.

(13)

Consequences

Advantages

 Less storage is required

o One of the main benefits of using Partial Storage is the reduction of space usage. Instead of storing the whole dataset on the phone, it is loaded only when needed. As a result the app can use amounts of data much larger than what could be stored on the phone.

Disadvantages

 Dependent on network connection

o A drawback of this approach is the dependence on network connectivity. The app relies on loading the data on demand; therefore some of the functionality may fail in the lack of internet connection. Moreover, a slow bandwidth may significantly increase the required time for loading the data and it can cause a poor user experience.

Examples

Many of the email applications use partial storage, for example opening the Gmail app without internet connection shows the latest emails on the screen, however trying to search in the archive or load more emails brings a connectivity error. Therefore, it is safe to assume that Google is using the pattern for pre-downloading the recent email conversations, but if the user wants to search in his inbox they are fetching the results on demand from their server.

(14)

COMPLETE STORAGE

All the data is synchronized and stored on the mobile device. If the user needs it, the applications response or loading time is much faster.

Problem

The consumer electronics industry has spent the past 20 years making everything connect wirelessly to the Internet. An average smartphone connects wirelessly in three ways: via mobile broadband, Wi-Fi and Bluetooth. Despite the fact that wireless connectivity is widespread and available on many places, there are times when a network connection is not available, but the application must still function. In cases where the user has limited connectivity and mobile data, loading data on request may be an issue. Additionally, performing often synchronizations can reflect on the application performance.

Applicability

The whole data must be synchronized between the mobile device and the remote system. The complete storage pattern can be used in situations where the data set is not too large and it is not frequently changed. It can be used in cases where the information is required when wireless connectivity is not available.

Solution

Store the entire data set on the mobile device. It must be synchronized in cases the wireless connectivity is optimal.

(15)

Consequences

Advantages

 Not entirely dependent on network connectivity

o If wireless connectivity not available, the data can’t be synchronized, by the application provides certain functionality.

 Performance

o Storing the entire data set on the mobile device makes the applications more responsive and improves the user experience, since the time for requesting information from the remote system is eliminated.

Disadvantages

 Increased data storage

o The storage of the mobile device is highly valuable for the user; therefore occupying a significant part of it may be frustrating.

 Increased bandwidth usage

o The synchronization action transfers all the data, which may include redundant or duplicated information. This reflects on the mobile bandwidth usage.

Examples

 Most of the weather apps are a good example of applications, which are usable offline. The apps store the latest weather information locally and present it to the user even if there is no internet connection.

(16)

Data Transfer

Minimizing the resource usage for data synchronization on mobile application is crucial since network bandwidth is often a concern. The data transfer patterns provide transfer strategies for optimal resource utilization depending on the context.

Full Transfer

All the data is transferred from the server to the mobile device.

Problem

In some cases downloading the entire data set is the most straightforward implementation and it is not practical to spend time and resources in developing a more complex solution.

Applicability

Full transfer is usable when the data set is small and easily transferable from the server to the app. It is useful for apps that primarily download data from the server, which if corrupted can easily be re-downloaded.

Solution

Download or upload all the data when a synchronization event is triggered, Figure 6.

(17)

Consequences

Advantages

 Simplicity

o This solution is easy to implement, hence it saves on development resources and it is less prone to failure.

 Handling corrupted data

o In cases of data inconsistency it can be easily fixed by transferring the information again.

Disadvantages

 Redundant data transfer

o Downloading/Uploading all the data on synchronization can be wasteful if it has only partial changes, thus increasing the bandwidth usage

Examples

Simple apps like RSS feed readers or Word of the Day apps can benefit from full data transfer, since the data size is not too big and in cases of data corruption it can be easily re-downloaded.

Timestamp transfer

During synchronization event between the mobile device and remote system, only new and changed elements of the data set are transferred. This is achieved by using a timestamp, which shows when a record has been created or modified, compared with the time of the last

synchronization event.

Problem

With the limitation of the wireless connection speed and bandwidth, sending the entire data set, which contains redundant information, may not be efficient. The waste of resources has to be eliminated, and the load of information has to be minimized.

Applicability

Using this approach the data set of an application can be synchronized in specific pieces based on timestamp. The app can download or upload those parts based on comparison between the timestamps on the mobile device and on the remote system. For this purpose, every data element must have a timestamp field, showing the last time this particular record has been modified.

(18)

Solution

For a download, visualized on Figure 7, a timestamp from the last successful transfer is sent to the remote system with the request for new or changed data. The remote system compares the timestamps and returns only the data that has been added or modified after the timestamp provided by the mobile client application.

For an upload of data from the mobile device to the remote system, only the data that has been added or changed since the last successful upload is submitted.

Figure 7-Timestamp Transfer

Consequences

Advantages

 Minimized data set while transferring

o By using Timestamp Transfer, only the new and changed data is

synchronized. Decreasing the size of information has positive impact on the bandwidth usage.

(19)

Disadvantages

 Timestamp consistency

o The correct timestamps have to be used in order to keep the

synchronization and data consistent. It is a common practice to use the timestamp from the remote server for downloaded data, and use the device timestamp for uploaded data.

 Status

o Timestamp synchronization works well to keep track of changes in

general. A disadvantage is that the types of changes are not recorded. That makes the deletion operation complex.

Examples

 Well-known examples are cross platform applications for keeping a list of notes. Once a new note is added, it is synchronized through all devices based on timestamp without re-downloading older data.

Mathematical Transfer

Since the last synchronization, only changed parts of the data are transferred. This process is based on mathematical algorithm.

Problem

Considering the disadvantages of using full and timestamp transfer, a more optimal approach has to be used. The mathematical synchronization significantly reduces the use of resources and determines changes more efficiently.

Applicability

The dataset of an application can be mathematically synchronized with remote system. In this case the mathematical properties of the data are used for determining what part of the data has to be synchronized. This strategy is used in situations where the use of bandwidth has to be

significantly decreased or is restricted. However, this comes at a price and even though the mathematical approach minimizes the usage of resources, it is complex and it can be computationally intensive.

(20)

Solution

Mathematical algorithm is used to determine the difference of data between the mobile application and remote system, Figure 8.

Figure 8-Mathematical transfer

Consequences

Advantages

 Optimal bandwidth usage

o This method uses the least bandwidth compared to full and timestamp transfers.

(21)

Disadvantages

 Time consuming

o Depending on the data and the mathematical algorithm used for

determining a change, the process can be very computational intensive and time consuming.

 Based on mathematical properties of data

o Since this approach is based on unique mathematical properties, the algorithm has to be adapted to the specific data type. Hence there is no one-size-fit-all solution.

 Complex

o The implementation of this pattern is expensive from development perspective, therefore it may not be cost effective in particular cases

(22)

Data Synchronization in Android

This chapter will focus on how data synchronization is currently implemented between the Android mobile app and the server.

Skonto Android Architecture

Currently every time the application is opened, the Sync Adapter tigers a synchronization, which downloads the newest data through the REST Service Adapter and saves it into a SQLite

database by the use of the Content Provider. More detailed information for every component is provided after Figure 9.

(23)

 Sync Adapter – In general, this component encapsulates the code that calls tasks, which transfer data between mobile device and server. The execution of this code can be based on various options [6].

o Server data changes-The sync adapter runs based on response indicating server data change. This approach refreshes the data from the server without draining the battery with unnecessary network requests.

o Device data changes – In this case, the sync adapter runs when the information on the mobile device is modified. This ensures that the server has the latest data.

o Network connection is available- Scheduling the sync adapter to run every time when the network connection is available, ensured that the data is regularly updated.

o Interval – In this case the sync adapter can run either after a certain period of time or a particular time of the day.

o On demand – This option is triggered by user action. Running the sync adapter on demand and requesting an immediate sync is a rather inefficient, because it bypasses all the network and battery usage optimizations. This approach must be used in situations where the demand for immediate sync is necessary [7].

The sync adapter, implemented into the Skonto application, runs on demand every time the user opens the app, but it does not automatically connect and transfer data to the server.

 REST Service Adapter – This service adapter, part of the Skonto architecture, encapsulate the code for processing the data between the mobile device and a server. It is a component which:

o Prepares the HTTP URL and HTTP request body

o Executes the HTTP transaction

o Parses the HTTP response

o Inserts parsed data into the Content Provider

 Content Provider – The content provider manages the access to a structured set of data. It presents the information as tables similar to the one used in relational databases, consisting of rows and columns. Although data can be directly accessed from the database, the content provider is used because of its mechanisms. Its main purpose is to provide interface that connects data in one process with code running in another, but it also provides an abstraction from the underlying data source (SQLite Database). The information from the content provider is accessed by using Content Resolver. It consists of basic methods as insert, update, query, and delete, typical for persistence storage. When the data is modified, the content provider notifies the Content Resolver. On the other side, the Cursor Loader handles the notification by implementing a Content Observer.

(24)

The Cursor Loader is used for asynchronous communication with the content provider. It can query in the background, which avoids the UI thread being un-responsive. When the result is retrieved, the loader populates different views of the activities by using cursor adapters.

 Activities and Cursor Adapter – The activity is component of the application, which provides difference user interface elements [8]. The essential views in the Skonto applications are lists, which displays shops stored into the local database. When the activity and its views are created, a cursor loader is called. It runs asynchronously and query against the content provider in the background. When the result is retrieved, a cursor adapter displays the data from the query into a list view.

Synchronization mechanism in Android

The Sync Adapter responsible for triggering a synchronization event runs on demand

asynchronously. It runs every time the user opens the application, forcing it to execute methods for downloading the newest shop data.

Advantages

 Responsive user interface

o The sync adapter runs the synchronization tasks on a background thread. This does not block the interface letting the Skonto users enjoy a fully functional app, even when data is synchronizing.

Disadvantages

 The on demand synchronization is not completely practical in this case

o This is not a particular disadvantage related to the asynchronous mechanism, but it is a downside of the Sync Adapter, as it is specifically designed to conserve battery power when it runs sync according to a schedule. Synchronizing on demand force it to run, which is inefficient use of network and power resources [7].

o At this stage of the Skonto project, the data on the server is rarely changed.

Providing synchronization on demand leads to unneeded requests without the data being changed, which is degrading performance and wasting battery.

 Does not support upload operations

o The current implementation of Android Skonto application does not support uploading data from a mobile device to a server. It only downloads information and stores it into a local database without letting the user modify it. With the growth of the size of the project and the expectation of new requirements, the upload of data from the mobile device to the server may become essential to the Skonto application functionality.

(25)

Data storage and availability in Android

As a requirement of the Skonto application, the information about the shops should be accessed offline. For the purpose of this, the data on a mobile device must be synchronized with the Skonto remote system, whenever the application is started and network connectivity is available. When the Sync Adapter runs and the data transfer is performed, the information stored via Content Provider into a SQLite database, should be identical with the data on the Skonto server. This follows and implements the complete storage pattern described in the previous chapters. Therefore, some of the pros and cons of the pattern overlap with the implementation of the Skonto app.

Advantages

 Performance

o All the data is stored on the Android mobile device. By taking advantage of

Cursor Loader, which asynchronously loads information from the Content Provider and presents it to the user by using Cursor Adapter, the Skonto application responsiveness and user experience are improved.

 Efficiency

o The information is accessed from the local SQLite database. Thus unnecessary network calls are eliminated, which prevents decreased performance and waste of battery life.

 Offline functionality

o Storing all the data on the Android mobile device provides the full functionality of the Skonto application, even if there is not any network connectivity. It gives the user ability to search information about the shop and get notified whenever he or she is close to it.

Disadvantages

 Scalability

o Because of the limited storage that the Android mobile devices have and the probability of increasing the data, storing all of the information may not be practical.

(26)

Data transfer in Android

The strategy used for data transfer, when synchronizing between the Skonto Android application and the remote system, is based on timestamp. The Skonto Android app stores the time of the last modification in the Shared Preferences. The Shared Preferences provides a general framework that allows storing and fetching key-value pairs of primitive data type. This data persists even if the application is killed by the system or the user [9].

When a synchronization event occurs, the time of the last successful update is sent as a request to the Skonto server. The Skonto remote system receives the time and compares it with all records of the data set containing a field, which holds a timestamp showing the last time this element has been modified. If changes are identified, a response containing the newly modified records is build and sent to the mobile client. This implementation follows the timestamp transfer pattern, since it was the most reasonable and practical solution at the time.

Advantages

 Resource efficiency

o By taking an advantage of using timestamp transfer, which drastically decrease the amount of data transferred between the Android mobile device and Skonto server, it is also a reasonable strategy considering time and cost, essential constrains of the project.

Disadvantages

 Timestamp storage

o When implementing a timestamp transfer for Skonto, an extra care had to be taken on keeping the correct timestamp and its format. A minor difference could lead to a data inconsistency.

 Deletion

o When an item gets deleted, it cannot be simply removed from the server. There might be still mobile clients, whose devices are offline and have to be synchronized. When a data transfer is performed, the mobile client must be notified of the deletion of this particular item.

(27)

Android implementation challenges

The components responsible for synchronization in the Skonto Android applications are Sync Adapter and Rest Service Adapter. By following the guides, provided by Google developer website, the Sync Adapter was implemented in a straightforward manner. While implementing

Rest Service Adapter, responsible for sending request to the Skonto server and handling properly the given response, an extra time and effort had to be taken on keeping the correct time stamp. TIMESTAMP PRESERVATION

Every time a data transfer is executed, the Android mobile client sends a timestamp of the last synchronization. But what happens if it was not successful? This may be caused by problems with the network connectivity, issues while parsing the response from the server or SQLite database error, refusing an insertion or update of a shop. Those situations had to be handled properly and the timestamp must be updated only in case where no problems are encountered.

Data Synchronization in iOS

The focus of this part of the project is how data synchronization and storage were implemented in iOS, including a discussion of the architecture specific for the iOS app and which patterns are currently followed.

Apple’s Guidelines

Apple’s software and hardware ecosystem is often referred to as a walled garden. [10]

Developing for iOS requires careful examination of what Apple considers best practices, since the review process for the App Store is fairly strict and it is common for apps to be rejected on ground of not following the preset guidelines.

Therefore, when implementing the synchronization mechanism for Skonto, it was crucial to consider all the recommendations and requirements enforced by Apple, in order to ensure the successful release of the app.

(28)

Skonto iOS Architecture

The architecture of the app follows the well-integrated in iOS development MVC model.

Elaborating on the MVC related implementation details is out of the scope of this project; hence the focus will be on the data synchronization and storage. The following, Figure 10, is a diagram of the architecture from that perspective.

Figure 10-Skonto i0S architecture

 UIViewController is a fundamental class of the controller layer in every iOS application. The controller is responsible for updating the view according to the underlying data. Usually this class is subclassed and adapted according to the specific needs. The view controller coordinates with other controller objects, thus the app presents a consistent and coherent user experience. [11]. In the Skonto case, every time a

controller is loaded a pre-configured fetch is executed and the data is loaded from the

(29)

 Core Datais a framework that provides solutions to tasks related to object life cycle and object graph management, including persistence. Even though Core Data with underlying SQLite database is widely used for persistent storage on iOS, the framework differs from traditional databases in variety of ways. To begin with it focuses on loading the object graph in the main memory from where the objects can be accessed. It is in fact possible to use Core Data without any persistence, but in most cases it is reasonable to store the data. Additionally all Core Data objects are proper Objective C/Swift objects with their own methods, attributes and relationships. [12] Therefore passing them from one controller to another is as easy as passing a reference and all the related objects can be accessed on demand.

 Sync Engine is a custom class implemented with the Singleton pattern, therefore only one instance of the class is created through the app. Unfortunately iOS does not provide a

Sync Adapter like the one available for Android. The Sync Engine triggers an event every time the app is brought to the foreground. A slightly different approach could be to start the synchronization engine only when the app launches. However, iOS has the nature of keeping apps on the background as long as a user is often accessing them and only kill them if they are not used for a significant amount of time. This may lead to serious update delay for the most frequent users. Hence, the current synchronization timing was applied.

 The Networking Classesarebuilding the communication layer of the app. They are

primarily responsible for making request to the web service and parsing the results. Having them encapsulated in a different layer ensures that in future the app can easily adapt different method of synchronization with the server without having to change any other layers. Currently the networking classes are making HTTTP POST requests to the server, which return JSONS responses to be parsed from the app.

iOS Synchronization mechanism

Synchronization can be time consuming, since it is network dependent, therefore the engine starts on a background thread provided by the Grand Central Dispatch iOS mechanism and does not disturb the user interaction with the app. Grand Central Dispatch (GCD) is a set of language features and runtime libraries that provide easy access to concurrent code execution on iOS and OSX devices, hence it was a perfect way of implementing the asynchronous aspect of the data update. Similarly to the Android implementation, the iOS app shares some of the general pros and cons of the asynchronous synchronization mechanism.

(30)

Advantages

 Instant access to data and responsive interface

o Starting the synchronization when the app is launched ensures that the user will have the most recent data, however it may not happen instantly. As a result it is better to have old data then no data at all.

Disadvantages

 No data when the app opens for the first time

o Once the Skonto application has been downloaded for the first time from the app store, it may take considerable amount of time to get all the data from the server, depending on the amount of data and internet speed. As a result the user will not have any data to see, which can be quite unpleasant and if the synchronization takes too long he or she may even loose interest in the app

 Concurrency handling

o Even though Grand Central Dispatch significantly reduces the complication of concurrent programming, it is still harder to think and debug code parts that run simultaneously

iOS Data Storage and Availability

In the Skonto iOS implementation a need for persistent local storage was identified in the

beginning of the project, therefore after reflecting on the options Core Data was chosen to handle it. One of the considerations was to use a mature and reliable technology, since Apple

recommends using Core Data when persistence is needed and also considering the continuous improvements of the framework, it was safe to assume that Core Data will be alive and well maintained in the near future. Moreover, as previously mentioned Apple is quite keen with it’s application review process, hence minimizing the usage of third party frameworks is often welcomed.

Advantages

 Storage flexibility

o In addition to offline functionality and improved performance, positive effects of adapting the complete storage strategy, it also allows storing specific user settings for every shop in the app. For example, when a user marks a shop as favorite it is straightforward to just flag it in the local copy of the database. Presenting favorite shops happens by just querying the Core Data implementation for all the marked entries. Similarly all kind of other user dependent settings can be added to the shops, as a result of using local database.

(31)

Disadvantages

 Data migration

o As the app grows the increase of attributes and requirements is inevitable. In order to follow the complete storage strategy, new fields will have to be introduced to the local database. As a result the migration complexity will increase

exponentially.

Data Transfer in iOS

The current data synchronization is adapting the timestamp data transfer mechanism. The iOS app stores the time of the last update as NSDate object in the NSUserDefaults. The

NSUserDefaults is simply a dictionary accessible throughout the whole app, which is storing key-value pairs used for representing user settings. Similarly to the Android implementation, when building the HTTP POST request in the networking layer of the app, the timestamp is added as a parameter to the request, which is send to the server.

Advantages

 Cost effectiveness

o Since the data synchronization is triggered every time the iOS app is brought to the foreground and not only when it is launched, it is crucial to optimize the data transfer as much as possible. As in Android, timestamp transfer is the most suitable solution considering the available resources.

Disadvantages

 Extra coupling between client and server

o In order for the timestamp data synchronization to work properly, both mobile client and server have to agree on a date time format. This adds additional degree of coupling and if a new format has to be adapted for some reason it may require changes through the whole system.

(32)

iOS Implementation Challenges

While working on the implementation of the synchronization a few problems were encountered. Most of them were related to handling concurrency with the data insertion and data visualization, as well as data migration when the database structure has to be changed.

Core Data Concurrency

In order to not block the main thread while all the shops are inserted in the local database it was crucial to run the update and insertion of records on a separate thread. However, simultaneously the user has to be able to view the information, which was already inserted, hence some

concurrency issues were encountered.

In order to understand the concurrency issue with core data, a brief introduction to how the framework works is necessary. The following, Figure 11, shows an overview of it. Everything is stored as a database file and exposed to the framework as a Persistent store object, which is essentially the connection to the file. In order to access and query the data a Managed object context has to be created, Apple describes the Managed object context as a scratch pad for manipulating objects in the database [13]; it is usually shared through the app. Once a context is available it can be used to load or insert managed objects (records) in the database.

The main pitfall when working with concurrency and core data is to continue using the same shared managed object context from different threads. The context object is not thread safe, therefore using one object for the main and background thread can lead to unpredictable results. In the Skonto project, after encountering this challenge it was solved by introducing another context object, connected the same persistent store. This way it was ensured that data could be inserted in the local database via one managed context and viewed from the main thread through another managed context

Figure 11-Core Data concurrency

(33)

Data Migration

Another challenge in the data synchronization was changing and improving the database scheme between different versions of the app. Once the data model has changed the app has to know how to map the already existing data in the scheme to the new one. There are three major migration techniques available with Core Data: lightweight (automatic), manual and custom. Lightweight migration relies on Core Data inferring mapping model from the differences between the two data models. For this technique to work the changes in the model must fit an obvious migration pattern like [14]:

 Simple addition of a new attribute

 Removal of an attribute

 A non-optional attribute becoming optional

 An optional attribute becoming non-optional, and defining a default value

 Renaming an entity or property

Fortunately, through the Skonto project all of the data model changes fit into one of this

categories, therefore the data migration was not too complex. However, with every new version of the app the data model is likely to grow, thus significantly increasing the migration

(34)

Skonto Synchronization problems

Solved Problems - Deletion

One of the disadvantages of using Timestamp Transfer approach is the inability of keeping track of the status of a specific record. As described, the challenge for implementing the

Timestamp Transfer strategy on both platforms, iOS and Android, is to manage properly the delete operation.

To determine that an item has been deleted, a Boolean flag is used. For this purpose a new field is added, as shown on Figure 12, which indicates whether the item has been deleted or not.

Figure 12-Deletion

When synchronization occurs, the mobile client receives a response from the server with the newly modified records. The deleted field is used to mark a record so if a deletion is indicated, the mobile client would remove it from the local repository.

Scalability and future problems

With the growth of the system, and an analysis ofthe current Skonto data synchronization a few problems are possible to occur in the near future. First and foremost, the system currently does not support upload from a mobile device to server. This was not an issue in the early stages of the project, but as the apps mature it may become unavoidable. Furthermore, a large part of the data that is synchronized from the server to the app is not actively used. This combined with the expected growth of shop’s attributes to be stored may lead to a high level of redundancy.

Additionally, the present synchronization scheduling is not optimal, which generates unnecessary requests to the server.

(35)

Solution

Considering the problems described in the previous chapter, only the issues most valuable from stakeholder perspective will be analyzed. In this part of the project the focus will be on

optimizing the locally stored data and introducing data upload from mobile client to server. The solution will be designed, but not implemented. The main reason is the fact that the system is in very early stages and it is not known how it will advance in future. The risk is too high and possible implementation will not be practical, considering time and resources. Additionally, if the project evolves, the requirements and priorities may drastically change, which could eliminate the need for this particular implementation.

Problem Analysis

Initially, Skonto started primarily with discounts at fast food joints only. However, the owner quickly diversified and introduced other businesses. As a result, many new shops and categories were added to the app, which highly increased the amount of inessential information locally stored on the mobile device. Moreover, extra attributes such as comments and photo albums are expected to appear in future versions. Hence, it will not be practical to store them on the phone. Additionally, if the system proves to be successful extended functionality such as comments and ratings may have to be included.

After reflecting on those problems, the following requirements were identified as critical for Skonto’s near future:

 Data upload - include support for upload from mobile client to server in order to allow ratings and comments

 Reduce locally stored data – eliminate unnecessary information stored in the local storage of the mobile device

(36)

Design

This part focuses on building solutions for the analyzed problems. It is logically divided into two parts, each one with emphasis on a particular requirement from the previous chapter.

Data Upload

Sending data from the apps to the server has to be implemented in order to support a rating system and user comments for every shop. The user selected rating will be uploaded to the server through a web service. In cases of no internet connection the chosen rating will have to be stored in the local database until synchronization with the server is possible. When the user wants to leave a comment a similar approach will be adapted.

In order to implement this new functionality a few things have to be changed in the system at general. To begin with the backend has to be extended to store the user ratings and comments. Afterwards, a service has to be exposed in order to allow submitting ratings and comments from the mobile clients. The rating calculation and storage on the server side are outside the scope of this project. However, the solutions for iOS and Android will be discussed in detail in the following two parts.

(37)

DATA UPLOAD IN ANDROID

In order to support upload functionality, the Android architecture has to be modified, as shown on Figure 13. After a field, which indicates that a record is updated, has been added to the local SQLite database, the Content Provider must expose a method, which queries the dataset, selects and returns all modified records since the last successful synchronization. Additionally, the web service that connects the Skonto Android application with the remote system has to be modified so it requires a timestamp of the last successful data transfer, along with a list of updated shops. The rest of the components are unaffected.

Figure 13-Android Data Upload architecture

The process of synchronization, consisting of upload and download operation, between a Skonto Android application and Skonto server can be visualized as a sequence diagram as shown on Figure 14. It is initialized every time the user opens the application. When the activity is started, an event, which runs the Sync Adapter, is triggered. Before sending a network request to the Skonto server, the timestamp of the last successful synchronization, stored in the Shared Preferences, is retrieved and used for comparison for getting all newly modified records. Then the body of the POST Request is constructed, and the fetched timestamp and items are sent to the Skonto server. The server processes the information and returns a response, which contains new or modified shops for the mobile device. The shops are parsed, stored into the database via the

(38)

Figure 14-Android Data Upload SD

Data Upload in iOS

Similarly to the Android implementation the iOS architecture will have to be changed, as visualized on Figure 15, in order to support uploading. A modified attribute will have to be included in the local database in order to indicate when the last time the particular record has changed. When the Sync Engine is triggered it will get all the records that have changed since the last synchronization and it will send them as parameters to the Networking Layer. The changed data is included in the request to the web service and the response is handled without any difference.

(39)

Figure 15-i0S Data Upload architecture

The following sequence diagram, Figure 16, represents a detailed visualization of how the logic will flow when synchronization is triggered. The SyncEngine is still responsible for controlling the synchronization process; the first thing it does is taking the timestamp of the last

synchronization from NSUserDefaults and loading all the shops that have local changes waiting for upload. Once the shops are loaded from the database, through the Core Data framework, the

SyncEngine calls the networking class responsible for making POST Requests and parsing the server response. The networking class sends the updated data to the web service and in return receives new or updated records from the server. The records are being sent back to the

SyncEngine, which updates the local database and stores a new timestamp in the

(40)

(41)

Local data reduction

The following diagram, Figure 17, presents all the shops’ attributes currently stored on the mobile devices.

Figure 17

Most of the attributes are self-explanatory, however there are a few which should be described in more detail.

Isfavourite is an integer representation of Boolean, which shows whether the shop is marked as favorite on this particular device or not.

(42)

Notificationsareenabled is an integer representation of Boolean, which indicates if notifications related to this shop are allowed on the particular phone. The user can activate and deactivate the notifications for every shop.

Modified is a timestamp, which represents the time of last data change. This is used for uploading only the data that has changed since the last synchronization.

Notification_sent is a timestamp, which indicates when the last time a notification related to that shop was sent. A requirement for the Skonto app was to not send more than one notification per shop in 24 hours, as a result this field was introduced to keep track on the notification timing.

Rating and selected_rating are used to store the general rating of the shop and the one selected by the user. The selected _rating field is important because if the user picks a rating while he or she is offline it will be locally stored until synchronization occurs.

With the inclusion of new attributes, such as comments and photo albums, it will not be practical to store everything locally since it may result in a high usage of storage and data transfer

resources. Downloading and accessing the required information from the server on demand could avoid this. Similarly, the current set of attributes could be decreased.

After careful consideration and discussion with the owner it was decided, that if data has to be excluded and not available offline the local database could look the following way, Figure 18:

(43)

One of the main changes compared to the previous diagram is the exclusion of opening and closing hours, which were represented in many attributes since a part of the requirements was to show in the view of every shop if it is open or closed at the moment. Additionally, also the shop description was removed, as it usually consists of a long text, which occupies larger space than any other piece of information. A disadvantage of this reduction is the inability of providing working hours information and shop description when the phone is offline.

In extreme cases an even more drastic data reduction could be needed, hence another exclusion of locally stored attributes was considered. The following diagram, Figure 19, is the shop information that is regarded as absolutely essential.

Figure 19

Each one of these attributes is necessary for the proper functionality of the app. The flags for notifications, favorites and rating selection are used for storing user preferences; therefore they have to be stored locally. The title, category, city and region are essential for sorting the shops in the app. The coordinates are used for delivering notifications based on geofencing, even when no internet connection is available.

With the attributes reduction, a part of the information will be accessed on demand. This leads to many network request to the Skonto server which can be reduced with the help of caching. Nevertheless, caching is a complex subject and it is outside the scope of this project. Therefore it will not be elaborated upon.

Another data reduction strategy is to reduce the amount of shops synchronized on the mobile device. Since the Skonto application has discount at shops through whole Denmark, it is unreasonable for a user located in Aalborg or other city to have the data for the whole country saved on his/her device. Accordingly, the data set of synchronized shops could be based on user location, region or city. This way the locally stored information will be reduced significantly. However, handling frequent travelers and users living in between two major areas will be a challenge. Considering the desire for simplicity and the currently limited budget of the owner, such a solution will not be rational, hence the focus will be on attribute exclusion strategy.

(44)

LOCAL DATA REDUCTION IN ANDROID

After a careful consideration, the set of attributes, stored locally, has to be defined, so that some of them will be kept on the mobile device and the rest received via network call to the remote system. This is visualized on Figure 20. The data stored on the Android device will be accessed through a Content Provider, where an AsyncTask, which will be added to the

RestServiceAdapter,willhave the responsibility of sending a request to the Skonto server and visualizing the response. It will perform the network call asynchronously and publish the result to the interface only after the task is finished.

Figure 20-Android Local data reduction Architecture

With this solution, there are two ways of accessing the information, both of which are visualized and discussed accordingly:

 Data stored locally – When a information is saved on the mobile devices,the Cursor Loader communicate with the Content Provider,and desired data is returned.The process is visualized on Figure 21.

(45)

Figure 21-Data stored locally

 Data not stored locally – When data is accessed via network call, a new background thread, which does not block the UI thread, is started. A request is sent to the Skonto service and the response is published as a result on the user interface. The technique can be seen on Figure 22.

Figure 22-Data not stored locally

LOCAL DATA REDUCTION IN IOS

Similarly to the Android implementation, the iOS architecture has to be modified in order to support loading a set of shop attributes on demand. The bold parts of Figure 23 visualize which components of the current implementation will have to be changed. The UIViewController will have to start an asynchronous synchronization event on a separate thread with the help of the

Grand Central Dispatch (GCD). Afterwards the Networking Layer will load the additional attributes from the server and the process will return the data to the controller, which will update the view.

(46)

Figure 23-iOS Local data reduction

The following sequence diagrams visualize how the app will operate, depending on the data availability. The locally available attributes will be loaded from Core Data, Figure 24, while the additional shop attributes will be asynchronously downloaded from the server, Figure 25.

(47)

Conclusion

Even though the Skonto project is in its early stages, it was essential to consider how the system will scale in future. At the beginning of this paper, some potential problems were indicated. However, additional analysis was made to compare the current Skonto solution with other available synchronization strategies, in order to reveal any present and future problems resulting from the Skonto implementation. A few potential issues were identified, the most notable of which were, how the system would manage a higher number of attributes per shop and how an upload functionality could be integrated.

With the proposed solution design and some minor modifications of the Skonto system, it is possible to add upload functionality with offline support and reduce the data attributes stored locally on the mobile device. In situation of successful business development of the project, the design consideration will be implemented. Furthermore, once more resources are available, improvements such as caching and location based synchronization can be considered, analyzed and applied.

This project strengthened the knowledge we have obtained while working on the Skonto system and increased our understanding of data synchronization with the theory behind it. We believe that this experience will be more than valuable for our eventual future work with Skonto or any other mobile development project.

(48)

Bibliography

[1

]

Zach McCormick and Douglas C.Shmidt. (2012, October) Data Synchronization Patterns in Mobile Applicaiton Design, Accessed 2015 April 25. [Online].

https://www.dre.vanderbilt.edu/~schmidt/PDF/PatternPaperv11.pdf [2

]

Android. Networking, Accessed 2015 April 27. [Online].

http://developer.android.com/training/basics/network-ops/connecting.html [3

]

Apple Developer. Synchronous Networking, Accessed 2015 April 27. [Online]. https://developer.apple.com/library/ios/qa/qa1693/_index.html

[4 ]

Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, Design Patterns: Elements of Reusable Object-Oriented Software, 1st ed. USA: Addison-Wesley Professional;, 1994, Proxy Pattern.

[5 ]

Oracle. (2001) Data Access Object, Accessed 2015 May 1. [Online]. http://www.oracle.com/technetwork/java/dataaccessobject-138824.html [6

]

Android. Sync Adapter, Accessed 2015 April 29. [Online].

https://developer.android.com/training/sync-adapters/creating-sync-adapter.html [7

]

Android. Running Sync Adapter, Accessed 2015 April 29. [Online].

https://developer.android.com/training/sync-adapters/running-sync-adapter.html [8

]

Android. Activities, Accessed 2014 April 30. [Online].

http://developer.android.com/guide/components/activities.html [9

]