The data on the decision makers’ activities during the experiments were recorded as the Web-based event logs using the logging module of the system. The event logs are an indirect record of what a user has done (Zhang, 2007). They provide an efficient and non- intrusive method for collecting data from the participants for the purpose of analyzing human-computer interactions. The main incentives for using the logs in the data collection process are low implementation cost, high speed, and high accuracy. In addition, the logging method does not require the use of personally administered questionnaires or interviews (Atterer, Wnuk, & Schmidt, 2006).
There are a number of log storage techniques/formats, such as text-based log files, flat text files, and databases (see Chuvakin, Schmidt, & Phillips, 2012). In this study, a database logging approach was employed to record the log information. Each time a user performed an interaction with the system, the system continuously wrote records to the log database describing the nature of the action (see Appendix D). The main advantage of using the database logging approach is that it allows for structuring the log information in a format that can be quickly read, searched, reviewed, analyzed, and queried. In contrast
104
to the file-based approaches that take a lot of time and effort to read, filter, summarize, and analyze the log data, the database approaches allow for using standard SQL queries to combine all sorts of information from different entries and easily analyze the log records. The log data for information acquisition behavior include the information the subject seeks (information cells) in the decision table, how much information is examined, how long the information is examined for, as well as the sequence in which they are looked at in the decision table. In addition to recording the data on the use of the decision table, the system records decision makers’ activities during the use of the decision map. The records are date and time stamped, and when reviewed, provide a picture of the user interaction with the system. By querying the log data stored in the MySQL database, one can derive data for computing the information acquisition metrics defined in Chapter 4. Figure 25 shows an example of SQL query in the Navicat for MySQL7 environment, which aims at retrieving the number of information cells examined by each decision maker. This query returns the number of information cells acquired specifically for each user in a particular decision situation (task complexity level) and a particular decision mode. The query results for two example decision makers are shown in Figure 26. For instance, the query results show that the number of information cells examined in the decision table by the decision maker with “UserID=1” in decision situation “5×2” (task complexity = 1) and within the GIS-MCDA individual mode is 14.
7
105
Figure 25. Querying the log data using the Navicat for MySQL.
Figure 26. Results for the query “what is the number of information cells acquired by each participant in a particular decision situation within a particular decision mode?”.
106
Chapter 6
6
Results and analysis
The research hypotheses developed for this study call for an examination of the differences in the information acquisition metrics when task complexity or information load increases (low vs. high), the information aid varies (the map vs. table), and the decision mode changes (individual vs. group). Additionally, the relationships between the metrics, and the effect of the decision mode and task complexity on these relationships will be examined. The hypotheses were tested by conducting Repeated Measures ANOVA (within-subjects ANOVA), Linear Mixed Model (LMM) analysis, and Pearson correlation tests using the Statistical Package for the Social Sciences (SPSS) software (SPSS IBM., 2012). Sixteen sets of hypotheses were examined (see Chapter 1). The hypotheses from H1 through H9 examine the effect of task complexity on the information acquisition metrics. These hypotheses were tested using the Repeated Measures ANOVA test (with the Greenhouse-Geisser correction as needed), with task complexity as the independent factor and each of the information acquisition metrics as the dependent variable. This would enable a comparison of the means for the dependent metrics at different levels of task complexity. The set of H10 hypotheses look at the effect of the decision mode on the information acquisition metrics. To test the differential effects of the decision mode on the metrics, the LMM test was carried out. The LMM procedure extends the general linear model so that the data are permitted to be correlated (SPSS IBM., 2011). The term “mixed model” refers to the use of both fixed and random effects in the same statistical analysis8. The presence of the random effects often introduces correlations between the subjects. The LMM test allows for integrating and analyzing the correlated repeated measurements by explicitly modeling a variety of correlation patterns (or random effects) (SPSS Inc., 2005).
8
107
The set of H11 hypotheses examine the relationship between information acquisitions in the decision table and map. The hypotheses H12 look at the relationship between the information acquisition in the decision table and map. The H13 hypotheses investigate the inter-relationship among the information acquisition metrics in the decision table. The three sets of hypotheses, H11, H12, and H13, were tested by conducting the Pearson correlation test; however, some of the H11 hypotheses were also examined using the LMM test. The set of hypotheses H14 explore the effect of task complexity on the relationship between the information acquisition in the decision table and map. The H15 hypotheses look at the effect of task complexity on the relationship between the times spent on the decision table/map and the time spent viewing the group decision. The set of H16 hypotheses assess the influence of the decision mode on the relationship between the information acquisition in the decision table and on the map. The three hypotheses, H14, H15 and H16, were tested using the LMM test. All of the sixteen sets of hypotheses were examined at a significance level of α= 0.05. In addition to the significance level of α= 0.05, the Pearson correlation tests on the hypotheses were conducted at a level of α= 0.01.