7.3 Validity of the Tests
Even though tests have been conducted in order to improve the design, determine the usability of the program and look for tendencies on privacy related topics, it is not given that the results are 100% reliable or correct. It is therefore important to consider the results with a critical eye.
7.3.1 Selection of Participants
The participants mainly consisted of friends and acquaintances that agreed to test the program. The distribution in age could have been better, since most of them were students and a bigger variety of age could have given a more realistic outcome that better represents the tendencies among most people.
The same applies for the background of the participants as most of them were students, whereupon most of them were studying within the fields of science or technology.
When it comes to the number of participants used in the tests, 19 should be more than enough when looking purely on usability testing. Most breakdowns were discovered after approximately 6-7 participants, while the same breakdowns were repeated for the rest of the participants. When looking at the statistics for the questionnaire on the other hand, even more users would be desirable in order to make the results more statistically correct as described in Oates [27].
The process of finding and recruiting participants for the tests could preferably been better planned in order to get a bigger variety and amount of people, with respect to both age and background, which in turn better represents society.
7.3.2 Validity of the Results
The actual results produced should be taken with a portion of reasonable judgment.
For example, the participant can be inhibited from acting as they would under normal circumstances because they are asked to perform predefined tasks under observation. The possibility of the participants’ acting as they think I want them to, rather than how they normally would are definitely present, as described in Chapter 3.4.2 as the Hawthorn-effect.
The same applies when answering the questionnaire, even though the received results were anonymous. The fear of doing something wrong or being perceived as stupid could also affect the participants during the tests.
This will however always be the case when conducting usability tests on other people and is therefore difficult to avoid. It is therefore not considered as a big prob-lem in this case, but the results are still being evaluated with reasonable judgment.
7.3.3 Tasks and Questions
When planning tasks and questions for use in research, it can be easy to be wiser after the event, concerning the tasks given and the outcomes or results from the research. This meaning that after the results are gained and interpreted, one often think that one could have reformulated the questions or the alternative answers to better fit the direction of the research to gain better statistical results. This could even happen, despite good planning.
This was also the case in this work. Some more time could have been used while planning the tasks and questions, in order for get better quality of the results. This was the case for some questions in these tests as well. For example when asking the age of the participants, the alternatives were quite broadly divided leading to difficulties in saying something useful on age differences. This could in this case also be due to the selection of participants, but could still have been improved.
The same applies to how often the participants used the Internet. If I knew that everyone would answer ”every day”, I should have provided more specific alternatives for answers. Even though some questions or answers could have been improved, I still find the results satisfying in terms of usability and developing the design which was the main focus of the tests.
When it comes to the practical tasks that were given on the system it might be an idea to include some less familiar webpages to the tests, to see if the users really use Privacy Advisor for help or if they only base their decisions on their own experiences. Even if familiar webpages were added to see what is really important for the users, service or privacy, some of the tasks could have been revised or some extra tasks could have been added. Still, I think most of the breakdowns were found and Privacy Advisor’s potential was shown.
Future Work
In this chapter I will present some ideas for future work based on the work done and presented in this thesis. These are ideas I had during the work on this thesis and should only be considered as inspiration rather than a definite future plan.
8.1 System Improvements
There are some ideas for improvements of the system; one includes functionality and one on the design.
8.1.1 Additional Features
As of today, there is no embedded functionality to withdraw specific policy data from the system’s advice engine, but it would be an idea to implement this because it would give the user the opportunity to get concrete statements in the advices on why a webpage not should be trusted. These statements could also be affected by the weighting of what elements the user thinks is important. For example, a user that thinks data collected for the purpose of telemarketing is unacceptable would weight purpose as an important factor to consider when comparing policies. The user would then probably appreciate to receive a warning if the webpage is looming to sell the users information to telemarketing firms. The potential statement could be something like this;
”This page does not take care of your privacy in a good way and should not be trusted because: - This webpage collect your information for telemarketing purposes and might sell your contact information to telemarketing firms.”