Developers typically face a specific set of challenges when building apps for one or more mobile platforms. A study performed by Joorabchi et al. [55] includes among the most relevant difficulties in the mobile development practices: the selection of the proper nature of the app to be developed (i.e., Native vs. Web or Hybrid applications); the limited capabilities of the typical device for a given platform; the choice between reusing others’ code or writing from scratch; the multiplication of time, effort and budget due to the multitude of devices and platforms the apps must be able to run on (i.e. Fragmentation); the rapid changes of requirements and typically rapid life cycle of an average Mobile app.
Those difficulties experienced when developing mobile (and, specifically, An- droid) apps are reflected, if not magnified, in the practice of testing. Moreover, Mobile app testing also has to take into account several aspects that can be com- pletely overlooked when testing traditional desktop applications. Muccini et al. [83], Kirubakaran et al. [56] and Kaur et al. [16] identified a set of characteristics of Mobile apps leading to specific forms of non-functional testing: mobile connectivity scenarios (i.e, coping with unreliable Wi-Fi or 3G connections) and rapid changes of
16http://firebase.google.com/docs/test-lab/robo-ux-test
17http://docs.aws.amazon.com/devicefarm/latest/developerguide/
2.5 Challenges in Mobile app testing 21
connectivity type; limited resources of devices; data intensivity of the applications; constant interruptions caused by system; very short time to market; very high amount of multi-tasking and communication with other apps.
Due to the listed difficulties, there is a substantial unanimity in the literature about a general tendency of Android developers to neglect automated testing, and to rely instead on manual testing only. As it emerges from sets of interviews to contributors to open-source projects performed by Linares-Vasquez et al. [67] and by Kochhar et al. [57], the time constraints, lack of properly documented testing tools, and high costs for developing and managing test artifacts are the main reasons for such preference towards manual testing procedures.
2.5.1
Fragmentation
The Fragmentation concept (also known as Device Diversity) encompasses, for the Android ecosystem, two different concerns [51]. Hardware-based fragmentation refers to the fact that devices based on the same Android operating system run on different processors, graphic cards, screen sizes, pixel densities. According to a report of August 201518, more than 24 thousand different devices, built by more than 12 hundred vendors, were existing at the time, and many display sizes, ratios and
pixel densities could be found (see figure 2.5).
Software-based fragmentation refers to the fact that several versions of the Android OS exist in parallel and that, at the same time, vendors and carriers may offer customizations for apps and OS GUIs. Device fragmentation is a specific issue of the Android ecosystem and not of Mobile development in general, being the number of available devices and maintained versions of the OS for iOS apps very limited.
Device fragmentation has relevant impacts on many aspects of Mobile develop- ment. First of all, for coping with software fragmentation of the OS, developers must cope with deprecated or even removed methods of the framework they use; hence, apps may need to be developed differently for different versions of the OS [49][99]. Software-based fragmentation, especially for what concerns the customized versions of the OS by different vendors, also creates concerns about the security of the apps [104].
Fig. 2.5 Relative screen sizes of Android devices available at August 2015 (source:https:
//www.xda-developers.com)
Fig. 2.6 Different layouts inflated for the same Activity on different devices (source:https: //developer.android.com/training/multiscreen/screensizes)
2.5 Challenges in Mobile app testing 23
From the point of view of system and GUI testing of Android apps, fragmentation translates specifically to the need of verifying the correct inflation of the layouts, which can be different to comply with the screen size, density and orientation of the device (since the same Activity can adopt different layouts according to the current orientation of the device: see figure 2.6). Layout-based, or 2nd generation, testing techniques must hence consider the possible different widgets of the layouts that are used to populate the Activities. Visual, or 3rd generation, testing techniques, must cope with the elements of the interface possibly not appearing on the screen, or appearing at different resolutions and hence invalidating the correct recognition of the screen captures used as visual identifiers.
2.5.2
Testing Hybrid and Web-Based applications
The development of hybrid applications comes in handy when small teams are not capable of building and maintaining several code bases for apps that are intended to work on different platforms. Several frameworks are available for building hybrid apps, e.g. React, Ionic, and Google’s Flutter.
While less prone to the fragmentation problem, Web apps, even when already tested for regular browsers, should be specifically tested in mobile environments for what concerns battery usage, connectivity issues, and performance [2]. GUI testing of web applications is also fundamental, in order to verify that all the elements of the user interface are visible, aligned and rendered in the same way as in the browser interface. A limitation for testing Web applications in the Android environment is the partial inapplicability of GUI Automation Frameworks specific to Android, and in general of Layout-based testing tools, with many of the most widespread testing tools (like Espresso or Robotium) providing limited or no support to what is loaded inside WebViews.
As reported by Ahmad et al. [1], lack of access to platform features, changes in contextual or environmental factors of the apps, integration and conformance issues, and diversity in user interface add other layers of complexity and fragmentation when testing hybrid Apps. United approaches, like Squish19, are typically at the level of abstraction of the GUI only (3rdgenerationtools), and leverage Capture & Replay approaches for test case generations.