MOBILE METRICS REPORT

(1)

MOBILE METRICS REPORT

iOS vs. Android Development in 2015

A Ship.io Study for Mobile App Developers, Testers,

and Product Managers

(2)

Mobile developers understand the rising importance of continuous integration (CI). The promise of CI is to speed up the overall velocity of the app team by building and testing every single code change. This environment of rapid change means that code integration issues are caught early, and testers and beta users can provide feedback on these changes as quickly as possible. As CI has become commonplace for app development teams, and many are moving their CI infrastructure to the cloud, the importance of build and regression test automation has grown.

To date, there has been very little data available on the state of CI for mobile apps. Here at Ship.io, we decided to initiate a study with our users to shed light on several key issues:

Which developers are moving most rapidly to CI and most speciﬁcally, to CI in the cloud?

How frequently are CI developers committing code changes?

What are the most popular cloud revision control systems? Which ones are gaining in popularity?

What are the most common causes for build breaks in CI? How can they be prevented?

With build times becoming a critical path in the velocity of CI teams, how long does it take to build an app? Is iOS or Android faster?

What follows is the result of survey data from 500 developers, plus anonymized build data from Ship.io users.

(3)

PART I:

THE CLOUD CI ADOPTION LANDSCAPE

(4)

Cloud CI Adoption: iOS vs Android

Many app teams have adopted Continuous Integration as part of their engineering processes. Some teams have set up CI as an on-premise service, but many are moving this to the cloud for reasons such as cost savings, focus, and reliability of the CI system. An oft-cited reason for moving to cloud CI systems managed by others is the complexity and cost of keeping up with all the changes in the iOS and Android build and test ecosystem. 

DISTRIBUTION OF IOS VS ANDROID APPS WITH INTEREST IN MOVING TO CLOUD CI

Of all developers surveyed with an interest in Cloud CI, but not necessarily using CI systems today, the majority (50%) are only focused on iOS—ignoring Android altogether.

Other developers are focused on both platforms, with a small minority focusing only on Android.

PUBLIC vs PRIVATE APPS

Developers with an interest in CI are overwhelmingly building apps for Google Play and the App Store. Only a small fraction of these developers are building apps for internal use, or with a limited number of customers. Many of these internal-facing app teams have smaller app teams and longer release cycles, presumably diminishing their need and interest in continuous integration systems.

iOS: 50%

Android: 10%

iOS and Android: 40%

Public: 87% Internal: 13%

(5)

COMMIT FREQUENCY OF IOS VS. ANDROID CLOUD CI TEAMS 

iOS developers commit code more frequently.

Of the top 40 apps with the most frequent commits in the study data, 67% of them were iOS Apps. It is worth noting that the study data had builds from app teams that built both Android and iOS versions of their apps. iOS developers seem to be committing code more frequently and adopting the CI model of small, continuous commits.

COMMIT FREQUENCIES: IOS VS. ANDROID

Commit Frequency

0%

17.5%

35%

52.5%

70%

Number of builds per week

1 2 3 4 5 6

7 1

5 1 13

57

0 5

5 10

15 63

iOS Android

(6)

iOS developers make far more revisions than Android developers.

Across the spectrum of commit frequency, iOS developers, on average, are making twice as many commits as Android developers. This chart shows the frequency, in builds per week, for iOS and Android Developers. The majority of apps have one commit per week, or fewer. Of those that are very active, deﬁned as committing code more than 3 times per week, about two thirds of those developers are working on iOS. The aberration seen where Android developers are building 6 times a week, is due to a single app vendor with many apps, all sharing a common code base with many Android apps (with commit frequencies at 6 times per week. In general, iOS developers are about twice as likely to make more than 3 commits per week versus their Android counterparts. 

iOS developers, on average, are making twice as many commits as Android developers.

“ ”

(7)

BUILD AND REVISION FREQUENCIES OF SURVEYED DEVELOPERS 

Build frequency of Surveyed Teams

The build frequencies of developers with an interest in cloud CI is surprisingly higher than that of developers actually practicing CI in the cloud. Cloud CI systems have only recently appeared on the market. Those teams with legacy in-house CI systems are interested in migrating to a Cloud CI, but are migrating at a slower pace than newer and smaller app teams that are readily adopting Cloud CI.

OVERALL CI ADOPTION 

CI and Test Automation Adoption

Of app teams surveyed, the majority are leveraging continuous integration. Those not leveraging CI are often building only on-demand when they want to test a build, or are building with a simple regularly scheduled nightly or weekly build scheduler. Of teams that are leveraging CI processes, the majority of them are running some form of test automation on each build.

between 5 and 25 per week: 33%

between 25-50 per week: 33%

more than 50 per week: 19%

No, not doing CI right now: 37%

Yes, doing CI builds *and* tests: 35%

Yes, doing CI builds: 28%

(8)

WHY NOT CI?

The reasons why teams haven’t adopted CI vary widely. 49% are interested and plan to move to CI eventually but haven’t because of prioritization. Some teams believe they are too small to beneﬁt from the value of CI as their team only has one or two people working on the builds, or they have so few code revisions that they prefer to deal with builds manually. A minority of app teams are

concerned with the cost in money or time to move to a CI system.

CLOUD VS ON-PREMISE CI

Cloud vs On-Premise CI

Of all app teams surveyed that are leveraging CI processes, the majority were leveraging Cloud CI, with a only minority managing on premise, often legacy, CI systems.

Not a top priority, but deﬁnitely want to adopt CI soon: 35%

Not enough developers on team to need CI: 16%

Planning to move, just haven't gotten around to it yet: 14%

Money: 11%

Security: 10%

Time: 14%

Other: 6%

(9)

PART II:

THE SOURCE CONTROL LANDSCAPE

(10)

The landscape of revision control : iOS vs. Android & Bitbucket vs. GitHub

Many app teams have moved their revision control systems to the cloud. Mobile developers have overwhelmingly chosen to adopt either GitHub or Bitbucket to manage their code in the cloud.

Overall, we see that Android developers prefer Bitbucket, with a slight majority of iOS developers preferring GitHub. Most interesting is that both Android and iOS developers have been quickly migrating to Bitbucket in the past 18 months, with Bitbucket usage doubling in just the past 4 months.

THE TREND FOR BOTH IOS AND ANDROID IS A FAST MIGRATION TO BITBUCKET. 

IOS DEVELOPERS STILL PREFER GITHUB, BUT NOT FOR LONG.

ANDROID DEVELOPERS PREFER BITBUCKET

Bitbucket: 41%

GitHub: 59%

Bitbucket: 56%

GitHub: 44%

Relative % of tbucket Usage 15%

30%

45%

60% 52

33 39

38 41

13

iOS Android

(11)

Over the past 18 months, there has been a dramatic increase in the number of both iOS and

Android projects that are using Bitbucket. The relative percentage of projects that use Bitbucket have doubled in the past four months. Android developers have moved even more quickly than iOS developers to Bitbucket. The growth in Bitbucket adoption is expected to continue and dominate in the mobile cloud development world. 

The relative percentage of projects that use Bitbucket have doubled in the past four months.

“ ”

(12)

WHY BUILDS BREAK AND BUILD

SPEED METRICS

(13)

As many app teams have moved to a continuous integration (CI) model for development, they produce many more builds. This increase in build frequency means that the time spent building is far more signiﬁcant, and broken builds can randomize and delay product progress. The data shows that the majority of build breaks are avoidable, and the time spent building apps is low, but iOS builds are about twice as fast on average versus Android.

Top Sources of Broken Builds

Analyzing the source of build breaks across both iOS and Android reveals that there is a very long tail of build breaks as many engineers would suspect. Builds often break due to of various developer code syntax issues, but several build breaks account for the vast majority (74%) of all breaks in cloud CI builds:

• Missing File or Directory: Developers often forget to check dependencies and new ﬁles into the code repository.

• Lint Errors: When ‘lint’ is enabled on a build, it often catches common programming errors.

• Unit Tests: Many apps have regression test automation at the unit level. These tests fail during the build process, catching either changes in the app, or regressions.

• Linking: Builds often fail at link time because a referenced 3rd party library wasn’t added to the dependencies list, or project dependencies were not updated.

• Code Signing: The process of signing apps is still error prone. From missing signature files, or differences in debug versus release build configurations, code signing causes many CI headaches.

Here is % breakdown of build errors by issue type:

” ”

(14)

The remaining 26% of build breaks are often unique to the product and often that speciﬁc build build attempt. Basic code-syntax errors, and intermittent HTTP request failures while the build dynamically pulls in source from other dependencies and projects round up the remainder of build failures.

The good news is that most all of these build breaks could be caught by developers having a separate ‘clean’ enlistment and building and running lint locally. Even more breaks would likely be prevented if developers simply executed unit tests before submitting new code. Barring that, cloud CI systems make it easy to catch all these issues, and mark the build as failed until developers resolve the break.

Build Times: iOS vs Android

When app builds break, that means the team is often blocked and waiting for a new build. Build times are now a critical path for app teams. Analysis of iOS versus Android build times reveals that although iOS app build times are about twice as fast on average, the build times are still reasonably short at 2 minutes and 4 minutes on average, respectively.

0%

12.5%

25%

37.5%

50%

Missing File: 46% Lint: 9% Unit Test Failures: 9% Linker Code Signing: 4%

4 9 4

9 46

iOS

iOS: 2 minutes

(15)

A deeper look into the build time data shows that very few app build times, regardless of platform, exceed 30 minutes. The majority of app build times are less than one minute.

An interesting aspect of build times is the fact that more Android than iOS app builds complete in less than 2 minutes. But of all large/complex app builds (longer than 2 minutes), iOS app build times are generally faster by a factor of 2. 

TOP PRIORITIES INCLUDE FASTER RELEASES

Cloud vs On-Premise CI

Of all the things app teams were looking to prioritize in the coming year, faster release and update frequencies were a close second only to improving overall app performance.

Teams are quickly migrating to Cloud CI systems as a way to accelerate their overall velocity. Cloud CI enables teams to focus their time and money on other priorities such as performance and security. 

Builds

0 20 40 60 80

0-2 mins 3-5 min 6-8 mins 9-11 mins 12-14 mins 15-17 mins 18-20 mins 21-23 mins 24-26 mins 27+ mins 0.23 0.45

0.31 0.23 0.6

0.81 0.6 2.5

20 74

0.24 1.6 0.93 0.32

1.6 1 9.8 3.6

32 48.2

iOS Android

Improve app performance: 48%

Faster release/update frequency: 41%

Increase security/vulnerability testing: 11%

(16)

CONCLUSION

(17)

These Ship.io study results show that the world of CI for app teams is changing quickly and varies between iOS and Android app teams. Knowing the macro-level trends and learning from them, can greatly impact your app teams eﬃciency.

As app teams accelerate due to CI adoption and agile practices, teams are often spending an inordinate amount of time focusing on their local and custom build and test infrastructure. Those app teams that have moved to the cloud are saving money and time and focusing more on their app design and implementation.

• CI Adoption: Cloud CI adoption is on the rise, especially in the iOS community. iOS developers are more aggressively adopting cloud CI than Android developers. The number of teams adopting Cloud CI is doubling every 6 months.

• Commit Frequency: There is a wide range of commit frequencies across iOS and Android. iOS app developers generally have twice as many commits as Android developers. The commit frequency, and thus the number of builds, range from a high of around 16 per day, to a low of once every other week.

• Source Control Popularity: Two cloud source control repositories represent almost all cloud CI systems: GitHub and Bitbucket. The momentum is strong with Bitbucket which saw it’s share of usage double in just the past 4 months. Android users seem are more aggressively moving to Bitbucket, but iOS developers aren’t far behind.

• Build Breaks: Several common issues account for the majority of build breaks, but there is a very long tail of build break reasons. The good news is that many of these are avoidable.

• Build Time: Build times on iOS and Android are similar and range from mere seconds to a high of 40+minutes for apps with larger code bases and dependencies. 

(18)

Key Deﬁnitions:

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each commit is then veriﬁed by an automated build, allowing teams to detect problems early. (source: Thoughtworks)

Source Control: Revision control, also known as version control and source control (and an aspect of software conﬁguration management), is the management of changes to documents, computer programs, large web sites, and other collections of information. (source: wikipedia.com)

Commit. A commit, or “revision”, is an individual change to a file (or set of files). It’s like when you save a file, except with Git, every time you save it creates a unique ID (a.k.a. the “SHA” or “hash”) that allows you to keep record of what changes were made when and by who. Commits usually contain a commit message which is a brief description of what changes were made. (source: GitHub)

Agile. Relating to or denoting a method of project management, used especially for software development, that is characterized by the division of tasks into short phases of work and frequent reassessment and adaptation of plans. (source: dictionary.com)

Build The term build may refer to the process by which source code is converted into a stand-alone form that can be run on a computer or to the form itself. One of the most important steps of a software build is the compilation process, where source code ﬁles are converted into executable code. The process of building software is usually managed by a build tool. Builds are created when a certain point in development has been reached or the code has been deemed ready for implementation, either for testing or outright release. (source:

technopedia.com)

Code signing is the process of digitally signing executables and scripts to conﬁrm the software author and guarantee that the code has not been altered or corrupted since it was signed by use of a cryptographic hash.

Unit Test: A unit test is an automated piece of code that invokes a unit of work in the system and then checks a single assumption about the behavior of that unit of work. A unit of work is a single logical functional use case in the system that can be invoked by some public interface (in most cases). (source: artofunittesting.com) Linking: In computer science, a linker or link editor is a computer program that takes one or more object files generated by a compiler and combines them into a single executable file, library file, or another object file.

(source: wikipedia.com)

Lint. In computer programming, lint was the name originally given to a particular program that ﬂagged some suspicious and non-portable constructs (likely to be bugs) in C language source code. The term is now applied generically to tools that ﬂag suspicious usage in software written in any computer language. (source:

wikipedia.com)

(19)

About Ship.io

Ship.io is a powerful SaaS platform that development teams use to power Continuous Delivery (CI) of native iOS and Android mobile apps. Purpose-built to support and streamline today’s mobile development workﬂows, Ship.io automates your build-test-deploy cycles so you can ship quality

apps faster.

To learn more about Ship.io, visit us at:

http://ship.io