Limitations

Although the longitudinal log study undertaken in Chapter 4 gained a valuable insight into users’ interactions on desktop operating systems, there are various limitations to

121

the research which, if addressed, could further increase the validity of the research and further benefit the research community as a whole.

7.5.1 Operating System

During the longitudinal log study only actions performed inside Microsoft Windows operating systems were logged. This limits the scope of the study as further valuable insights could be gained from studying other types of operating systems such as iOS or mobile platforms.

7.5.2 Participant Makeup

Within the log study only a small number of participants (17) were used limiting the better understanding which would have been achieved with a larger number of users. Valuable information such as whether users from different types of jobs or backgrounds interact differently would be an important addition to research in this area. Participants were recruited from family, friends and through connections within the School of Computing & Communications at Lancaster University. Various e-mails requesting assistance with the study were sent out to the department and once participants registered their interest in the study a participation information pack was sent which explained the study in detail and what they needed to do. 17 computer users from a variety of different backgrounds (11 frequent - more than 50 interaction days - and 5 female) took part in the user study for a period of 90 days. The age range of participants was between 18-74 years. Seven users had single screens, eight had two screens and two had three screens. Participants used a variety of Microsoft Windows operating systems with eight using Windows 7, three using Windows 8.1 and three using Windows 10. Seven users had laptops and eleven desktops.

As such, there is a certain bias which is a limitation of the study in that a lot of the participant cohort were work/research users and were found to generally use more programming IDEs/research tools/specialist programs, use more screens and usually be expert users. For each of the two main areas of analysis in Chapter 5 (window switching and data transfer), a brief analysis is provided in the following sentences on the participant cohort. Analysis conducted on the participant cohort for window switching,

122

clipboard and drag-and-drop showed actions were much more frequent among work users, possibly down to the fact they use their computers more frequently (of the total interaction hours recorded, 56.9% were among work users with 43.0% among leisure users). For window switching, over 80% of switches recorded were for work users, with less than 20% for leisure users). For clipboard usage these results were even higher for work users with over 90% of actions undertaken by work users and only around 9.5% by leisure users, indicating a higher usage rate of the clipboard among work users. The results for drag-and-drop followed a similar trend in that over 98% of drag-and-drop actions were performed by work users with just less than 2% of actions performed by leisure users. This data shows that the majority of window switches were for work users (80%) and work users utilised the clipboard and drag-and-drop operations much more frequently. These results were expected as it was observed that work users had a higher number of interaction hours and as such, it is to be expected work users will perform more window switches, clipboard and drag-and-drop actions than leisure users. As a result, this shows that it would be advantageous to utilise wider cohorts or various different cohorts in the future. It also shows that cohort-specific analysis is useful in understanding actions users undertake, due to the large variation in differences between work and leisure users.

7.5.3 Participant Makeup Reflection

The participant makeup could be seen to bias the results due to the fact the majority of participants came from the research community and from the School of Computing & Communications at Lancaster University. These users will generally be performing similar tasks (preparing lectures, writing research papers, performing research projects) and as such, these tasks could have had an impact on the results of the longitudinal log study. As such, the frequent use of some applications may be down to the fact these users are utilising the same applications to perform similar tasks (research/work). For example, high usage was seen in Mozilla Thunderbird, STEP 7 Manager and Rhino, all of which were from work users and higher than the usage of Google Chrome which was utilised by many more users. Of the top 10 used applications, 7 were directly related to a given work task while 3 (Windows Explorer , Google Chrome and Microsoft Outlook)

123

were less niche applications indicating high usage by individual users for an extended period of time.

A wider variety of participants would have been a useful addition to the study and would have given a better and more accurate overview of the ways in which users of computers perform interactions on them. Further deeper analysis into the different types of users would also have been beneficial and will be considered in any future work undertaken. Analysis by work vs home users and types of job performed for work users would have provided extra contextual information and may have led to being able to make more relevant statements surrounding these types of users and develop better tuned tools geared toward these users specifically.

Future studies undertaken will carefully take into account the participant makeup and extra work will be undertaken both in the participant recruitment stage and during the data analysis stage to fully understand the impact of the participant cohort selection. For example, participant recruitment will be expanded and participants from different backgrounds/workplaces will be recruited for future studies. Advertising regarding the longitudinal log studies will be more widely spread both in terms of across further departments across the university but also in external organisations/families. During the analysis stage of future studies, analysis by job or user type will be presented and compared to past results and with each other. This in turn will give a wider analysis taking into account the makeup of the participant cohort. It will also provide findings which are possibly relevant to different workers undertaking different jobs. Further analysing these interactions by type of participant may uncover different ways of working which require different means of assistance in order to increase efficiency. For example, work users may prefer and be more efficient with a different recommender tool (similar to QuickFileAccess) than home users. Extra analysis of the data would provide this valuable information. In summary, future studies will carefully take into account the participant cohort and analyse it fully.

7.5.4 Contextual Information

The study also had a limitation by the fact that no further contextual information was gathered. For example, no information relating to certain tasks the user was performing was gathered nor was additional information surrounding the rationale behind certain

124

user actions. Both these types of data would be valuable in gaining a deeper understanding of user interactions on desktop operating systems.

7.5.5 Data in Isolation

The analysis of the study described in Chapter 5 also only analysed data in isolation. For every participant, actions were only logged across one machine. Several of the participants of the study had multiple machines (for example an office computer (which interactions were logged on) and a laptop (which interactions were not logged on)). Logging all users’ machines would be the ideal solution to this limitation but this is not currently possible if they have machines running different operating systems with which the software is not compatible. It would also allow comparisons to be made as to whether users interact in a cross-application way differently across different machines.

7.5.6 Contextual Information

The QuickFileAccess system makes the assumption that the relationship between clipboard and file directory access is a good correlation measure for reporting back relevant directories to the user through the Windows Explorer Quick Access system. There are also many other algorithms available for use in selecting which file directories to report back to the user through the Windows Explorer Quick Access list.

In summary, various limitations exist with the research which include only logging events within Microsoft Windows operating systems, using a small number of participants, not gathering extra contextual information, analysing data in isolation and the assumption that the QuickFileAccess system should use the relationship between clipboard and file directory access as a good measure.