Lecture 09 Data Storage

(1)

HIT3328 / HIT8328 Software Development for Mobile Devices Dr. Rajesh Vasa, 2011 Twitter: @rvasa Blog: http://rvasa.blogspot.com

Lecture 09

Data Storage

1 R. Vasa, 2011

Lecture Overview

•

Recap

•

Data Storage Options

•

Detour into File System Security

•

Saving Data to Files

•

Working with SQL Data Store (sqlite)

•

Android has a Monkey

2

R. Vasa, 2011

Mobile devices have a specialization

3

Context:: Semantic // Spacial // Social // Temporal

Web is good at

semantic

and

social

contexts

Mobile devices are good for

spacial

and

temporal

contexts

R. Vasa, 2011

Mobile phones are a fashion statement

4

Fashion

Culture

Time-Bound

User View: Ugly apps. ruin the “look” of a pretty (expensive) gadget

Concept Overload on the Web?

Paper, Kindle, Multiple Prices

More Buying Options

Used Book - New Book

What others bought

Cart, Wish list, 1-Click, Prime

Review, Like, Rate

Frequently Bought

Read Inside

Read Reviews

This what a first time online book buyer

will experience if they go to Amazon

Heuristics to Check for Usability

6

Usability/Design

Check List

Reflect on Answers

Make Changes

Sketch Paper Prototype Interactive Prototype Early Release Beta Release Final Release

(2)

R. Vasa, 2011

Key Aspects of the Check list

7

Logical UI Physical UI Graphical UI

Data Model Wording (Labels) Soft Keyboard Functional Organisation Navigation/ Exploration Ergonomics

Physical and Temporal ContextIcons and ImagesFont Type/Size Style/Colours Appropriate use of components

Tasks are verified within the context of a User Model

Tasks are generally derived from scenarios

(they can even be a 1:1 mapping)

R. Vasa, 2011

Lecture Roadmap - Where are we?

•

Recap

•

8

R. Vasa, 2011

•

Android offers the following options for data storage:

•

Shared Preferences (Key-Value pairs)

•

Internal Storage (Store on Flash memory)

•

External Storage (Store on SD Card)

•

SQLite Databases (RDBMS)

•

Network connection to store on server

9 R. Vasa, 2011

Storage (Android Design Choices)

•

Static data should be put into Resources (“res” folder)

•

Place any text file or custom data files in the “raw” folder (as a resource)

•

User modifiable data can be stored to Internal (or) External Storage

•

Every Application has the option to store “private” data

10

Internal Storage (via Raw Folder)

Store any raw data

in this folder

This data has to be provided at development time

Working with “Raw” Resource Files

InputStream inputStream =

getResources().openRawResource(R.raw.dfile);

BufferedReader br =

new BufferedReader(inputStream);

You can work with the BufferedReader (or) the InputStream directly as required for your application

(3)

R. Vasa, 2011

Resources are Read Only in Android

13

Data stored in the ‘res’

folder is read-only

Database file that needs to be updated cannot be stored in ‘res’

R. Vasa, 2011

Resources Stored in Compressed Form

14

Applications are deployed as APK files (compressed archives)

Resources are also

compressed as part of the

package creation process

If you have to access any resources,

they need to be

uncompressed

first

R. Vasa, 2011

Resource Compression Issues

•

On the surface the compress decision by Android designers seems reasonable given small persistent storage size.

•

But, decompression is a slow process

•

Additionally, prior to Android 2.3.:

•

If the uncompressed size of a file was greater than 1 Mb -- it did not decompress

15

Log Cat message when things fail in Android 2.2 due to large size

!"#$%&'(()*+,-./01!'*'1)23))4(1$56789:";;<!=>=<8=?1 +@AA-B,A1C(1@DEF.,A/

R. Vasa, 2011

So, where can an application write to?

16

•

Android offers options for data storage:

•

Shared Preferences (Key-Value pairs)

•

Internal Storage (Store on Flash memory)

•

External Storage (Store on SD Card)

•

SQLite Databases (RDBMS)

•

Network connection to store on server

Writing to File System has Security Implications

(

a number of choices were made during the design of Android

)

•

Recap

•

Writing to File System (Typical Model)

•

Application-A can create and update a file

•

E.g. !"#$%&%#'()*+,-./%0)$%&%1&2&

•

Application-B can read/update this file

stored in C:\data (if Application-A is closed)

•

Default option on desktop O/S:

•

Applications have full access to the file system (typically)

•

Security is achieved via access permissions bound to user

(4)

R. Vasa, 2011

Traditional Security Model (File Access)

•

Assumptions,

•

User is smart & computer savvy (_hmm..)

•

User has knowledge of file system (_hmm..^2?)

•

User is (can be) responsible for access

permissions

19

These assumptions were reasonable in the

early days of computing

But in mobile ....

R. Vasa, 2011

Security on Mobile Devices

•

File system is completely hidden from the user -- it is a (smart)phone to most people.

•

Phone == Consumer device (like a TV)

•

Data security is achieved via Sand-Boxing

20

User

has

UID (User ID)

In most modern O/S: File Access Permissions

are restricted based on a user id

R. Vasa, 2011

Security -- Android Devices

21 Android

Application

User

has

UID (User ID)

also has

The UID is generated at install time based on

the

signature

and package name in the APK

File System Access Permissions

determines

R. Vasa, 2011

Security Architecture

•

Android runs on top of Linux

•

Linux offers access restrictions by user id

•

Linux also treat external devices as files (in

general)

22

•

Applications (by default) do not have permission to,

•

Read/Write user data [e.g. contacts] / Intent Filters

•

Read/Write another apps. data [File permissions]

•

Access the network (which is mapped as a file)

•

Accessing/Talking to sensors etc. (also mapped as files)

Security in Practice (on Android)

•

Application-A cannot access data privately stored by Application-B

•

Each application runs in its own Sandbox

•

The user can get to data by Remote shell

access (shell access phone)

•

By default, the phone provider has ‘root’ access to the device

•

You can by-pass this by installing a custom kernel (or) by modifying the kernel

•

Once you have root access -- you can view data

Viewing the File System

Macintosh-4:platform-tools rvasa$ ./adb -d shell $ ls dbdata config cache efs sdcard acct mnt d etc data init.smdkc110.rc ... Folder where Applications store data

(typically) Shell access to phone

(5)

R. Vasa, 2011

File System is Locked (by default)

25

$ cd data $ ls

opendir failed, Permission denied $

Many areas of file system are restricted by default (i.e. phone is locked)

Note: You have full access to entire file system on emulator

Android/Apple/Windows Mobile have this restriction

R. Vasa, 2011

Emulator does not have any restrictions

26 # cd data # ls property misc local app-private backup data anr dontpanic app dalvik-cache system lost+found # 345$4%** 34*6$ 7$%&%7%** 340--68/$,.8$1-9/&8'+:;1%*< -68/$,.8$1%--8=/;:;1%*< -68/10+5:>1%*< (.61%/$,.8$:;1%*< -68/1+2%'*0+-:>1%*< 34

No permission problem here

R. Vasa, 2011

Viewing Data on Emulator

27 # cd data # ls property misc local app-private backup data anr dontpanic app dalvik-cache system lost+found 345$4$%&% 340-5.'1%/$,.8$1?985<-+%,[email protected] 5.'1%/$,.8$1*,.B8$+,-1$.6/0.%$- 5.'1%/$,.8$1*,.&8*- 5.'1%/$,.8$1-*%,+)*%,&--68/$,.8$1-9/&8'+ 345$4-68/$,.8$1-9/&8'+ 340-08A C80+- 345$4C80+- 340-

9-+,)=+.0.5%&8./-/data/data

This is the folder where

apps. store data

R. Vasa, 2011

Some Areas are Public (like SD Card)

28 $ cd sdcard $ ls Samsung sd Android DCIM layar media AussieWeatherRadar DiskCacheIndex41214.tmp Sounds download ScreenCapture swiftkey Music $ SD Card on a locked Galaxy S

Some applications

use SD Card for

data storage

$ cd AussieWeatherRadar $ ls IDR022background_img IDR022topo_img IDR022location_img IDR022range_img IDR024background_img IDR024topo_img IDR024location_img IDR024range_img

We can pull public files from phone

$ ./adb -d pull /sdcard/AussieWeatherRadar ./AWR

pull: building file list...

pull: /sdcard/AussieWeatherRadar/IDR023range_img -> ./AWR/IDR023range_img pull: /sdcard/AussieWeatherRadar/IDR023topo_img -> ./AWR/IDR023topo_img pull: /sdcard/AussieWeatherRadar/IDR021range_img -> ./AWR/IDR021range_img ....

....

IDR023topo_img

The ADB command is helpful for debugging & if you are curious to

see how an application works

You can also push files to phone

Lecture Roadmap - Where are we?

•

Recap

•

(6)

R. Vasa, 2011

Saving Data into a File (locally)

31 345$4$%&% 340- 5.'1%/$,.8$1-*%,+)*%,&--68/$,.8$1-9/&8'+ 345$4-68/$,.8$1-9/&8'+ 340-08A C80+- 345$4C80+- 340- 9-+,)=+.0.5%&8./-/data/data/files $ cd AussieWeatherRadar $ ls IDR022background_img IDR022topo_img IDR022location_img IDR022range_img IDR024background_img IDR024topo_img IDR024location_img IDR024range_img Internal Storage

(built-in flash memory)

External Storage (SD Card)

Two options typical available

R. Vasa, 2011

Saving Data into a File

•

Internal Storage (built-in flash memory)

32 Stored in /data/data/files

/** Sample code to help get started with saving location */

private void saveGeoLocation() throws Exception {

String FILENAME = "user_geolocations"; String string = "Melbourne, -40.23, 143.23"; FileOutputStream fos = openFileOutput(FILENAME, Context.MODE_PRIVATE); fos.write(string.getBytes()); fos.close(); } 345$4-68/$,.8$1-9/&8'+ 340-08A C80+- 345$4C80+- 340- 9-+,)=+.0.5%&8./-R. Vasa, 2011

Saving Data into a File

•

Internal Storage (built-in flash memory)

33

FileOutputStream fos = openFileOutput(FILENAME, Context.MODE_PRIVATE);

Modes available:

Private,

Append,

World_Readable,

World_Writable

R. Vasa, 2011

•

Recap

•

34

Assumptions (before we jump into sqlite)

•

You are able to read/write SQL statements

•

You know the basic concepts that underpin a

relational database (Tables, Primary Key, Indexes, Data types etc.)

•

You know the difference between a database (a file) and a DBMS (e.g. Oracle/sqlite/SQL Server etc.)

•

You have a general understanding of how to programmatically access a database (via the DBMS)

Android (and iOS) provide sqlite

•

Android and iOS both provide a light weight (in-process) database engine called sqlite in their phones

•

What is SQLite?

•

http://en.wikipedia.org/wiki/SQLite

•

Interesting (sensible) design choices:

•

Locks entire file when updating

•

Weakly typed (does not guarantee domain type integrity)

(7)

R. Vasa, 2011

Another explanation of sqlite

•

SQLite is a library (like java.io.*) you use to write/read a file -- your database

•

It is a library, so it is part of your application

•

You do not need to “ship/bundle the library” in

your apps. because sqlite is part of the Android (and iOS) libraries

37

ClassX uses java.io.* read_write locations.txt

ClassZ uses sqlite read_write locations.db

R. Vasa, 2011

sqlite is an embedded/in-process DBMS

38

ClassX and library are in same process

ClassX uses java.io.* read_write locations.txt ClassZ and sqlite are in same process

ClassZ uses sqlite read_write locations.db

similar to custom File I/O

R. Vasa, 2011

Getting Started - First Steps

•

Typical tasks when working with databases:

•

Open or Create the database (a file)

•

Create/Read/Update/Delete tables in DB

•

If query returns more than one row -- iterate

over this set of rows (using a Cursor)

•

Display/Use data in columns(of each row)

39 Database Table(s) Row(s) Column(s) R. Vasa, 2011

Open/Create Database - Concerns

•

API offers a helper class for opening/creating a database

•

In practice, we also encounter the following situations and have to design for:

•

Upgrading the DB from old to new version

•

Downgrading the DB (sadly .. also needed)

•

Handing errors during the opening/upgrade

•

Upgrade (or) Initialisation that takes a long

time

40

Open/Create DB - Android API

•

Android provides DEF8&+G*+/H+0*+, in API

•

Simplifies DB access via conventions (for

instance it stores db files is a standard location)

•

DEF8&+G*+/H+0*+, designed to simplify:

•

Opening/Creating/Upgrading databases

•

Handling errors

•

Managing long running tasks by using asynchronous background processes

Key DB Classes/Interfaces (in Android)

SQLiteOpenHelper SQLiteDatabase SQLiteStatement

Open/Create DB

Interact with DB

Encapsulates SQL

Statement

(8)

R. Vasa, 2011

What happens to a query?

43

SQL Query on execution returns a Cursor

Cursor (Interface)

Cursor offers

random

Read/Write

access to the result set of a query

Warning: Cursor implementations are not required to be synchronized

R. Vasa, 2011

What happens to a query?

44

When a Query is executed on

SQLiteDatabase, it returns a SQLiteCursor

Cursor (Interface)

Warning: SQLiteCursor implementations is not synchronized SQLiteCursor

implements

R. Vasa, 2011

Result Set & Cursors in DB Access

•

When you execute a select query on the DB,

•

It will return a set of matching rows -- the

cursor allows access to these rows

•

The result set is a copy of the information in the DB -- hence the cursor will not reflect any changes made after query (or) insert

•

Implications:

•

Close the cursor as soon as possible to avoid keeping large volumes of data in memory

•

Repopulate any views if data changes

45 R. Vasa, 2011

Lets get our feet wet

46

Database Access - Code Walkthrough

•

BirthDayDB Project

•

Demo of the

functionality

•

Code walk through in the lecture

DB Access can be complex

•

Consider a table with 30,000 rows

•

Query all will return a cursor for this entire set of matching rows

•

Android O/S has a primary UI thread

•

If actions on this primary UI thread take too long (more than a few milliseconds) -- the OS will shut down the thread

•

What will happen to long queries?

•

How can we handle this situation?

(9)

R. Vasa, 2011

•

Recap

•

49 R. Vasa, 2011

Android SDK has a Monkey

•

Android SDK provides a ‘dumb’ stress testing tool called Monkey

•

It will (pseudo) randomly press buttons on the phone -- generate events and does a whole lot of silly things

•

Ideally, your app. should survive this beating

50