Chapter 3 Experimental Design
3.4 Data Collection
3.4.1 MySQL Database setup
A standard MySQL database [100] was chosen and setup to save users’ scores, comments, and relevant data for the web. The collectable data from the system were a users’ score, state of mind, user comments (optional). Other additional data is sourced from built-in-options such as login and a chat forum on the web pages.
Figure 3-11: MySQL database
The database is built using the MySQL database server which contains four tables described in more detail in Figure 3-11. The database contains four tables of
“result_table”, “comment”, “shouts”, and “member”. The table of “result_table” has
fields of ipaddress, time, userfeedback, and stateofmind in order to save the IP address,
time, user’s score and user’s state of mind respectively. The table identified as “Comment”
has fields of username, email, and comment in order to save user comments (optional).
With this comment option, users can enter any comments together with their numerical scores. The tables of “member” and “shouts” are created for additional built-in-options in
3.4.2 Human-Subjects
There were 12 users involved in our pilot study, 21 users took part in the second experiment with a three-level state of mind and two-level content factors integrated, and 12 users took part in the third experiment with a three-level state of mind and four-level content factors integrated.
Figure 3-12: Setup of users and website sources
All of them were post-graduate or undergraduate students studying at our university. They consisted of a mixture of individuals who have a computer background (e.g. Computer Science) and those coming from non-computer backgrounds (e.g. Food Science, Agronomy). Of the 45 users, there were 5 taking undergraduate studies and 40 involved in postgraduate study. Based on their sex, there were 14 female and 31 male. They accessed the same website sources simultaneously. In this experiment, the impairments that were applied were the same for all participants and experienced simultaneously.
While users accessed the website, the users’ actions were captured to see the behaviour that they exhibited through their selection of hyperlinks on the displayed page, together with the frequency and nature of these selection activities.
3.4.3 User score-MOS
The users’ scores were collected from them in the form of Mean Opinion Score (MOS) recommendations [12, 101] which are based on the traditional five level scale
{1, 2, 3, 4, 5} that links these numerical values to a user’s perception as defined by {Not recommended, Dissatisfied, Fair, Satisfied, and Very Satisfied}.
When users indicate a score of 5 or “Very Satisfied”, it means that performance quality from the web server was excellent as the user expected. Similarly, when users enter a score of “4”, “3”, “2”, or “1”, it means that performance quality from the web server was good, normal, poor, or unacceptable respectively.
Figure 3-13: MOS and its score meaning used in the project
3.4.4 User tasks
Users were asked to give an evaluation of their own experiences at two to five minute intervals through the use of a user scoring system for the particular web pages that they were surfing under different networking situations. Users gave their evaluation by sending their users’ scores via a feedback page accessed via the pre-prepared web pages.
Figure 3-14: Feedback page
The feedback page described by Figure 3-14 was designed as a radio button selection; Users selected their preferred scores: one for an overall performance of system, and one for their own state of mind, using two single clicks on the two options and a single click on the submit button. Therefore, it is expected that this task would not involve much time for the users in sending their user feedback experience.
These users were asked to access the prepared local websites; then to rate:
1) Score on performance: This score is to rate the web site performance as described in Figure 3-14 for each of the objects and to record a score based on their perception. The meaning of each score is described in detail in Figure 3-13. 2) Score on your state of mind: This scores show users’ feelings during the experiment whether or not they feel Normal/Bored/Stressed during the time they have to do with the experiment, rather, it should represent their general mood for the day and how it might be changing as the experiment proceeds. In more details, do they feel “Bored/Normal/Stressed” when they have to repeat various actions during the experiments. Please note that this score is not intended to be related to the networking performance when the users choose an option for their state of mind.
3.5 Summary
The experiment has been carried out to collect the users’ scores and assess objective and subjective factors which affect their QoE for the case of web based services. The output of this chapter is the subjective users’ score related to objective controlling network parameters which will serve as an input for our QoE assessment in the next chapter. The experiment has been designed using the Taguchi quality method approach which ensures that a quantitative number of observations were carried out. As noted earlier in this thesis, it consumes significant amounts of cost and time when using a substantial number of participants in any research involving human participants. However, by carrying out the experiments using the Taguchi method, we can mitigate both time and cost as well as collecting accurate quantitative observations.
The controlling sessions have been designed in order to limit boredom and memory effects that were mentioned in Section 3.2.2. The control factors and noise factors have been assigned based on Taguchi’s guidelines.
The format of the web design has been described and the elements of the web pages outlined. A database based on MySQL server has been built to save data for the purposes of both collecting users’ scores, and saving additional built-in data related to the various web functions.
User tasks were designed to be as simple as possible; therefore, participants would not be annoyed when asked to achieve their respective tasks.