Five Strategies for Performance Testing Mobile Applications

Five Strategies for

Performance Testing Mobile Applications

Mobile  applications  are  now  mainstream  business.    Businesses  ranging  from  health-­

care  providers  to  retailers  and  airlines  are  creating  applications  for  mobile  devices   including  iPhones,  iPads  and  Android  phones.    In  fact,  many  new  businesses  are   starting  with  a  mobile  application  first,  and  then  adding  web  applications  later.    

Few  businesses  can  afford  to  ignore  mobile  users.  In  April  2010,  Mary  Meeker  at   Morgan  Stanley  estimated  that  the  number  of  mobile  users  would  surpass  the  number   of  desktops  by  the  beginning  of  2014.

 That  tipping  point  may  in  fact  come  earlier,  as   mobile  adoption  has  continued  accelerating  since  she  made  that  prediction.

Mobile  technology  is  simply  ramping  up  faster  than  other  technologies  have  in  the   past  –  it  may  well  be  the  fastest  technology  adoption  curve  in  history.    And  this  has   important  implications  for  your  business  model.      You've  got  to  be  fast  to  market,  but   also  prepared  for  rapid  adoption.  If  your  application  performs  poorly,  or  fails  under   load,  multiple  competitors  are  ready  to  take  your  place  –  the  barriers  to  entry  are  low.  

Performance  testing  is  essential

Mobile  users  have  high  performance  expectations,  whether  using  a  mobile  application   or  accessing  a  website  from  a  mobile  browser.  According  to  Equation  Research,  58%  

of  mobile  users  expect  sites  to  load  roughly  as  quickly  on  their  mobile  phones  as  on   their  desktops  at  home.    

Sluggish  performance,  errors  or  crashes  will  send  many  users  to  a  competitor,  and   may  affect  their  willingness  to  do  business  with  you  at  all,  even  from  fixed  worksta-­

tions.    The  mobile  app  marketplaces  are  filled  with  competitive  offerings;;  if  you  want  a   spot  on  a  customer's  smart  phone,  you  have  to  earn  it.

At  the  same  time,  you  cannot  afford  to  spend  months,  and  millions  of  dollars,  testing   mobile  apps  before  you  deploy  them.    There  are  too  many  competitors,  and  the   market  is  growing  too  quickly,  for  you  to  sit  on  the  sidelines.

So  you  need  to  carefully  test  your  mobile  apps,  without  slowing  down  your  deployment   cycle.    And  merely  testing  for  functionality,  while  essential,  is  not  enough.  You've  got  to   confirm  how  your  apps  will  perform  in  the  real  world,  ensuring  that  your  infrastructure   delivers  the  performance  that  mobile  users  expect,  even  during  peak  loads.

In  Q4  2010  the  number  of   smartphones  shipped   exceeded  the  number  of   PCs,  for  the  first  time.

In  January  2011,  Apple   reached  10  billion  down-­

loads  from  the  AppStore.

According  to  the  Mozilla  

Foundation,  shaving  2.2  

seconds  off  load  time  

increased  download  

conversion  by  15.4%,  or  

60M  downloads  a  year.

Five  performance  testing  strategies  for  mobile

If  you  want  to  make  sure  that  your  mobile  app  will  function  reliably  and  consistently   even  under  heavy  load,  then  use  the  following  strategies  when  designing  and  imple-­

menting  performance  testing:

1.Test  in  the  production  environment 2.Test  at  2-­3  times  anticipated  peak  load 3.Test  accurately  for  the  mobile  environment 4.Test  across  geographies

5.Test  for  real  time  results

1.  Test  in  the  production  environment

Performance  testing  used  to  be  fairly  straightforward.    Set  up  a  lab  environment,  spin   up  a  few  load  generators,  watch  how  it  performs  and  extrapolate  upwards  to  your   expected  load.    

With  the  complexity  of  today's  web  infrastructure  and  mobile  users  scattered  around   the  globe,  extrapolating  performance  based  on  a  limited  lab  test  is  dangerous.  Testing   in  the  lab  or  a  staging  environment  is  helpful,  but  is  unlikely  to  reveal  the  performance   and  reliability  problems  that  appear  in  production,  at  scale.    

To  really  test  the  performance  of  your  mobile  application,  you  have  to  extend  your   testing  efforts  beyond  the  application  itself  to  the  broader  back-­end  infrastructure  and   application  ecosystem.  This  means  testing  in  the  actual  production  environment.    

Production  testing  does  not  necessarily  mean  putting  your  production  application  at   risk.  There  are  ways  to  test  in  the  production  environment  that  limit  your  exposure.  For   example:

Test  during  maintenance  windows  

Test  while  the  site  is  live,  but  during  a  low-­usage  time,  so  the  potential  impact  of   performance  testing  is  limited,  and  be  prepared  do  stop  if  needed

Test  in  production  but  before  the  product  is  announced/released Test  on  a  slice  of  the  production  infrastructure  

Only  by  testing  in  the  production  infrastructure  can  you  take  into  account  all  of  the   components  of  your  application  ecosystem,  including  those  outside  the  application   itself.  These  may  include:

Shopping  cart  applications

Content  delivery  network  performance  and  file  placement

You  can  only  discover  problems  with  these  third  party  application  integrations  by  

testing  in  the  production  environment.  And  you  only  uncover  problems  with  your  own  

larger  infrastructure  (load  balancers,  DNS  routing,  firewalls)  by  testing  from  across  the  

web  rather  than  a  tightly  controlled  test  environment.

2.  Test  for  2-­3  times  expected  capacity

What's  the  peak  capacity  that  you  expect  for  your  application?    Can  you  hazard  a   guess?  

No  one  is  really  good  at  predicting  peak  loads,  we're  all  just  guessing.    Even  compa-­

nies  that  have  been  in  the  business  for  years  and  have  history  to  look  back  on  make   the  mistake  of  assuming  that  past  performance  predicts  the  future.  That's  a  dangerous   assumption  in  today's  online  world  –  particularly  with  mobile  users  who  can  start   downloading  a  new  app  or  music  album  by  simply  pulling  out  a  phone  or  tablet.

For  example,  Internet  giant  Amazon  was  blindsided  when  it  offered  a  one-­day  promo-­

tion  for  a  new  Lady  Gaga  album  with  its  Cloud  Drive  and  Cloud  Player  services.    With   more  than  10  million  Twitter  followers  and  32  million  Facebook  fans,  Lady  Gaga   mobilizes  a  virtual  army  of  fans  on  fixed  and  mobile  networks.    The  response  

exceeded  expectations  and,  in  the  words  of  the  director  of  Music  for  Amazon,  "melted"  

some  servers.    Amazon  tried  a  'do-­over'  the  next  day.  Its  business  can  certainly  

sustain  the  negative  impact  of  this  kind  of  public  meltdown.  Your  business  might  not  be   so  lucky.

Many  factors  can  affect  your  traffic  in  unanticipated  ways:

Your  marketing  team  runs  a  brilliant  social  media  campaign  that  takes   off  beyond  their  expectations

A  prominent  blogger  raves  about  your  app

External  events  conspire  to  send  people  your  way

Because  of  this  uncertainty,  we  recommend  that  you  estimate  your  peak  traffic,  and   then  test  at  two  to  three  times  that  peak  capacity.  Testing  at  scale  like  this  gives  you   confidence  that  your  application  infrastructure  can  handle  an  unexpected  load  without   melting  down.

3.  Test  accurately  for  the  mobile  environment

Make  sure  that  your  performance  tests  can  model  the  actual  behavior  and  environ-­

ment  of  your  mobile  users.

Certain  test  platforms  inherently  restrict  testing  parameters  and  models.  For  example,   some  test  platforms  require  the  use  of  browser  plug-­ins  to  record  web  traffic  to  create   the  tests  –  essentially  limiting  testing  to  those  specific  browsers.    

In  the  diverse  mobile  application  environment,  you  need  to  be  able  to  accurately   reflect  the  user  environment,  including  web  services  calls  from  games  or  other  apps  as   well  as  mobile  browsers.    For  mobile  web  pages,  testing  should  include  multiple   mobile  platforms,  including  Blackberry  devices,  WebOS  and  Windows  Mobile.    This  is   only  achieved  by  emulating  the  traffic  just  as  the  phone  or  tablet  would  communicate   with  your  application  infrastructure.  

Once  you  have  created  accurate  scripts  from  different  mobile  devices  and  user   behavior  profiles,  you  can  then  combine,  parameterize  or  manipulate  them  to  create   the  complex  usage  patterns  and  high  traffic  volumes  needed  for  an  accurate  load  test.

Ideally,  you  should  be  able  to  correlate  end-­user  performance  on  real  mobile  devices  

with  performance  metrics  from  your  supporting  back-­end  infrastructure,  to  discover  

how  infrastructure  issues  impact  the  end  user  performance.

4.  Test  across  geographies

By  definition,  mobile  users  don't  stay  in  one  place.    And  the  Internet  defies  geographi-­

cal  borders,  so  it  is  important  to  test  across  geographies  for  global  reach.

Send  traffic  from  different  parts  of  the  country,  over  different  Internet  backbones,  and   from  different  countries  and  continents.  Make  sure  that  the  application  performs   consistently  and  reliably  across  the  entire  infrastructure.

5.  Test  for  real  time  results

With  fast-­moving  development  cycles  and  rapid  mobile  adoption  cycles,  you  cannot   take  weeks  or  months  to  test.  You  need  to  see  results  in  real  time,  so  you  can  make   changes  on  the  fly  and  test  those  changes  as  part  of  an  iterative  performance  test.  

When  testing  in  the  production  environment,  real-­time  insight  also  helps  you  shut   down  testing  that  might  be  affecting  actual  customers  or  application  performance.

Sound  too  difficult?    Use  the  cloud.

A  few  years  ago  the  guidelines  in  this  paper  would  have  sounded  impossible  and   impractical.  Who  can  afford  to  deploy  a  global  testing  infrastructure,  or  ramp  up   thousands  or  tens  of  thousand  of  virtual  users  in  a  few  hours?

Today,  however,  the  idea  of  deploying  testing  applications  across  geographies,  on  a   giant  scale,  is  not  unreasonable.  In  fact,  it's  surprisingly  easy  and  affordable,  with   multiple  cloud  ecosystems  like  Amazon  EC2  and  Windows  Azure  offering  the  possibil-­

ity  of  instant  scale-­up  with  a  global  reach.

When  testing  mobile  apps  at  scale,  you  do  not  actually  need  to  deploy  legions  of   mobile  users  around  the  globe  on  iPhones,  iPads,  Android  phones  or  other  devices.  

Instead,  use  a  proxy  to  capture  all  HTTP  or  HTTPS  from  each  tested  device  perform-­

ing  the  actions  you  want  to  test.  You  can  use  load  generators  to  simulate  the  behavior  

of  those  devices.  Creating  these  test  scripts  should  not  require  complex  scripting.  

New  technologies  also  offer  the  fast  test  deployment  and  real-­time  insight  necessary   to  get  the  testing  done  without  delaying  deployment  or  adversely  affecting  production   performance.    

If  you  are  deploying  mobile  applications,  you  really  have  two  options:  performance  test   yourself  to  find  and  correct  the  problems  created  by  running  at  scale  in  the  real  

environment,  or  leave  the  performance  testing  to  your  customers,  and  use  reactive   crisis  management  to  minimize  the  damage  to  your  reputation  and  market  share.    The   first  alternative  is  a  better  approach  from  a  business  perspective,  and  with  today's   technologies  it  is  easily  within  reach.    

About  SOASTA

SOASTA's  CloudTest  products  and  services  leverage  the  power  of  cloud  computing  to   quickly  and  affordably  test  consumer-­facing  Web  and  mobile  applications  at  scale,   providing  customers  with  the  confidence  that  their  business-­critical  applications  will  be   reliable  and  scalable,  even  under  extreme  loads.  SOASTA’s  customers  include  many   of  today’s  most  successful  brands,  including  American  Girl,  Chegg,  Gilt  Groupe,   Hallmark,  Intuit,  Microsoft  and  Netflix.  

For  more  information  about   SOASTA,  please  visit   http://www.soasta.com.  

©SOASTA,  Inc.  2010.  SOASTA,   SOASTA  CloudTest,  and  SOASTA   CloudTest  On-­Demand,  are  

trademarks  or  registered  trademarks  of  

SOASTA,  Inc.  All  other  trademarks  are  

the  property  of  their  respective  owners.