Phase 3: the improved questionnaire 45

During  the  third  phase,  the  full  questionnaire  is  improved  based  on  the  feedback   of  the  validation  inside  SIG  and  the  validation  with  the  consultants.  After  

completing  the  adjustments  of  the  questionnaire,  there  were  two  validations   done  with  external  organizations.  This  way  the  questionnaire  could  be  validated   in  practice  with  people  that  were  not  involved  in  the  development  process.   During  the  validation,  the  same  aspects  as  in  earlier  phases  of  the  development   are  checked.  The  validation  focus  on  these  three  aspects:  

• Is  the  questionnaire  useful?  

• Is  the  questionnaire  easy  to  answer?  

• Could  the  questionnaire  be  answered  in  a  short  time  period?   In  addition,  the  requirement  completeness  is  taken  into  account  as  extra   information  to  improve  the  questionnaire  on  missing  aspects.  

3.3.1  Processing  feedback  of  validations  

During  the  validations  sessions,  some  aspects  to  improve  were  provided  by  the   participants.  Most  of  this  feedback  only  required  some  minor  adjustments  to  the   text.    

Besides  the  adjustments  to  the  text,  there  was  some  confusion  about  how  to  fill   in  the  questionnaire  in  general  cases  (see  end  of  Section  3.3.2)  and  how  the   consultant  has  to  process  the  evaluation  framework.  An  introduction  for  the   questionnaire  is  created  to  provide  extra  information  for  the  client  organization.   After  finishing  the  questionnaire,  a  translation  from  the  answers  to  a  SIG  rating  

was  designed.  The  introduction  and  the  rating  system  solve  the  problems   encountered  by  the  consultants.    

3.3.2  Validation  with  external  organizations  

Validations  with  potential  clients  of  SIG  are  done  at  the  end  of  phase  three.  The   questions  are  divided  differently  than  inside  SIG.  In  the  validation  from  SIG   internally,  all  questions  are  spread  over  five  functions  (CSO,  IT,  Lab,  Office  and   HR).  However,  for  the  external  organizations  all  questions  are  spread  over  three   functions.  The  reason  to  do  the  division  differently  is  that  the  HR  and  Office  have   a  limited  amount  of  questions.  We  would  like  to  limit  the  number  of  contact   person  in  order  to  facilitate  the  further  sessions  planning.  In  many  cases,  the  CSO   has  the  knowledge  to  answer  the  HR  and  Office  questions.  This  means  that  the   CSO  could  replace  the  two  other  functions,  so  that  planning  a  conversation  would   be  easier.  In  case  the  CSO  has  not  enough  knowledge  on  the  HR  and  Office  

controls,  he/she  could  distribute  the  questionnaire  for  obtaining  the  correct   answers.  Further  it  is  preferred  to  divide  the  amount  of  questions  equally  over   the  employees.  In  the  validation  inside  SIG  this  was  not  the  case,  but  the  new   division  took  this  into  account.  Preferably  the  questions  are  divided  over  the   following  three  functions:  

• Chief  Security  Officer  (CSO)  or  a  similar  function   • Head  of  IT  or  a  similar  function  

• A  person  from  the  software  development  department  (if  applicable)   It  is  difficult  to  give  an  exact  reason  about  choice  for  above  functions.  The   division  is  based  on  the  current  division  of  the  ISO  controls  inside  SIG.    

For  the  validation  with  external  organizations,  there  were  found  two  

organizations  willing  to  participate  into  the  validation.  Both  organizations  are  IT   related,  but  do  not  develop  software  themselves.  One  organization  is  considered   small  (50-­‐200)  and  the  other  one  business  is  medium-­‐sized  (200-­‐500).  Both   organizations  were  positive  about  the  questionnaire,  but  there  were  some   (small)  remarks.  

During  the  validation  sessions,  it  was  stated  that  these  questionnaire  is  a   concrete  (check-­‐)  list  of  actions  for  what  has  to  be  done  by  the  organization  to   keep  their  information  protected.  Both  organizations  agreed  on  that  it  was  a   good  start  for  getting  ISO  27001  certified  in  the  future.  One  organization  said   that  when  they  started  to  get  an  ISO  certificate  themselves,  they  used  an  external   organization  to  help  them.  However,  this  questionnaire  would  have  helped  them   more  than  the  external  organization  did.  

A  critical  point  that  was  mentioned  was  that  some  questions  could  be   interpreted  in  different  ways.  This  makes  it  harder  to  compare  the  given  

answers,  because  possibly  you  compare  two  different  things.  This  point  has  to  be   solved  in  the  future  work.  In  the  end,  there  have  to  be  a  balance  between  the   clearness  of  the  questions,  the  generality  of  the  questions  and  the  time  to  answer   them.  

Furthermore  both  organizations  did  think  the  content  of  the  questionnaire  was   quite  complete.  One  organization  did  say  it  would  be  also  nice  to  include  a   question  about  who  is  ultimate  responsible.  For  example  this  could  be  a   manager,  but  also  other  employees  with  lower  functions.  The  reason  for  this   addition  is  that  it  is  important  that  a  high  function  (e.g.  a  manager)  is  the  

ultimate  responsible,  because  security  requires  high  management  commitment.   Another  addition  could  be  a  general  risk  analysis.    

Another  observation  was  that  the  third  part  of  the  questionnaire  about  software   development  has  to  be  handed  to  an  experienced  employee  such  as  the  head  of   development.  An  inexperienced  employee  has  possibly  not  enough  knowledge  to   fill  in  the  questionnaire  properly.    

The  participants  reminded  me  that  some  organizations  prefer  less  paperwork.   These  organizations  could  have  implemented  many  security  controls,  but  do  not   write  most  of  this  down.  In  the  questions,  it  is  asked  if  there  are  policies  and   documentation  about  implementation,  but  are  these  documents  really   necessary?  This  could  be  investigated  in  a  future  work.  

Furthermore,  the  time  spent  on  filling  in  the  questionnaire  was  in  both   organizations  under  the  defined  limit  of  one  day.