Aims & Expectations

The conceptual cross over between social presence and team trust is interesting yet in need of clarification.

Is the crossover simply a coincidence due to a shared focus on cooperation, are the concepts the same in some regard, or are social presence and team trust separate concepts, occurring within the same situation, but conceptually distinct? As there are a number of core elements of team trust, to understand the interplay between team trust and social presence both concepts must be measured in relevant team-based gaming environments. A variety of games were chosen for this study to compare not only the core concepts of trust and social presence, but establish what effect contextual gameplay elements and the games themselves have on the experience. It may be that some games contain elements which promote trust while others suppress it, indeed there results of the study show a great variety in the level of team trust from game to game.

This study aimed to gather user experience data which would be used to examine the interplay between trust and social presence across a variety of game scenarios. This study will measure various elements of trust such as outlined in the literature such as familiarity and performance as variables.

Trust and Cooperative Social Presence:

To establish if conceptual crossover between team trust and cooperative social presence was more than simply coincidence or semantics, the levels of these concepts experienced by gamers in team-based games was be measured. This data was used to establish any correlation between the two scales. It was expected that due to the conceptual crossover the levels of trust and cooperative social presence would correlate.

Performance:

The team trust literature suggested that high trust leads to high performance, and the Wang [2013] study suggested that performance affects cooperative social presence. However more evidence was needed to confirm the findings of the Wang [2013] study, which had a small sample size. This study measured performance by asking players if their team won or lost, and compared the team trust and cooperative social presence scores of the winning and losing participants. The expectation based on the Wang [2013]

study was that high performance would lead to high cooperative social presence but would not affect competitive social presence. As performance is claimed to be evidence of trust, these two concepts were expected to correlate.

Familiarity:

The trust literature argued that familiarity affects early trust in teams, while data from the development of the CCPIG suggested that familiarity may lead to higher social presence. The relation between familiarity, trust, and social presence in games needed to be explored for this relationship to be clarified. It was expected that high degree of familiarity would lead to higher levels of trust and cooperative social presence.

Trust and Danger/Challenge:

The MMO literature suggested danger/challenge in games is a trust builder, but does this carry across to team-based games? To explore this question the interplay between between perceived overall challenge, competitive social presence (of which challenge is a factor) and team-mate trust was explored. It was expected from arguments made in literature that the level of danger/challenge would correlate with the team trust scores.

Monitoring Behaviour in Games:

While the trust literature states that monitoring team-members is evidence of a lack of trust, social presence literature would suggest this activity would increase social presence. By comparing the amount of monitoring behaviour to the level of trust it was be possible to establish whether it could be considered evidence of a lack of trust in the context of team-mates games. As this study was dealing with gaming, it was expected that monitoring behaviour would correlate with social presence.

5.3.2 Procedure

The data for this study was gathered using the online community survey methodology. The online ques-tionnaire was designed and lightly customised for each game community asked to participate in this study.

Once the online questionnaire had been constructed the calls for participants were posted on the com-munity forums, and before posting permission from the forum moderators was gained. The calls offered respondents a chance to win Steam games (worth around£20) by entering an optional random prize draw when submitting their data. Based on previous experience it was expected that a the call for participants would gain a response rate of one respondent per ten thread views, and so to encourage thread views links to the calls were posted on reddit. The online questionnaire was generally left active for seven days, after which the prize draw was conducted, winners contacted and results announced on the forums. The data was then compiled, coded, and factor analysis was conducted using and SPSS. The remaining statistical analysis was conducted using R.

Measures

In this study the following concepts were measured:

ˆ Social Presence

ˆ Trust

ˆ Familiarity

ˆ Performance

ˆ Monitoring Behaviour

ˆ Game Information

Familiarity, performance, monitoring behaviour and game information are all simple concepts which were measured relatively easily using self reported information, while social presence was of course be measured using the CCPIG. A full list of the questionnaire items can be found in Appendix 7.3. Game information, such as the game, game-mode, and team role a participant was playing, is information which was gathered throughout the development of the CCPIG and so forms part of the standard introduction section of any online CCPIG.

Familiarity was rated by respondents by checking relevant options in the following question:

How familiar were you with the other players?

Please show who you were sharing the server with by choosing any number of the following:

 Real-Life Friends

 Online Friends

 Clan-mates

 Acquaintance (server regulars)

 Strangers

These options were then converted to numerical values from 1-5, with ‘Real-Life Friends’ having a value of 5, ‘Strangers’ of 1, and so on. From these values it was possible to establish the minimum, maximum and mean levels of familiarity that a respondent had with the other players in their game. For example if a respondent stated they were playing with ‘Clan-mates’ and ‘Strangers’ their maximum familiarity score would be 3, their minimum would be 1, and their mean familiarity would be 2. An alternative familiarity measure could have been the four item familiarity scale found in the [Webber, 2008] trust questionnaire, however the items in this scale were work-place centric, with little relevance to online gaming scenarios.

For this reason and to keep the number of total items to a minimum the simple check-box measure was used. While this is a rather crude scale the item has high face validity and the resulting numbers give a good impression of who the player was sharing a virtual environment with.

Performance was measured with both numerical and binary measures. Performance was established using simple explicit questions, asking participants game specific performance questions (e.g. did your team win/lose?), and about how they would rate their team’s overall performance. Monitoring behaviour too was measured using straight-forward questions, where relevant, asking participants how much they checked the scores of their team-mates throughout the gaming session. While the Cummings and Bromiley [1996]

monitoring measure from the Langfred [2004] questionnaire could have been used, the four item scale seemed a unnecessarily repetitive, with items such as ‘We watch to make sure everyone in the team meets their deadlines’, rather irrelevant to gaming. In most team-based games there are a number of ways to monitor the activity of one’s team, checking a map of the game environment, using in-game communication, etc. However these methods are not available in all games, thus the core measure for monitoring behaviour this study will be checking the player score board, a monitoring action available in most games.

Information about the game the respondents were playing was also gathered. This information included their team size, their role, class or equipment in the game (if relevant), and whether they were playing in a public (pub) or organized setting. In this study ‘pub play’ is defined as any gaming session which took part on a server which is open to the public and which a player joined without specifically pre-planning.

The alternative to pub play in this study is organized play (org) which consists of various types gaming sessions, for example data categorized as organized play in this study includes play in passworded servers, clan matches, clan practise sessions, participation in regular or pre-scheduled community events, and so on. This information was gathered based on previous user feedback from the CCPIG development studies (see Appendix 7.2.13) in which users argued that organized and pub pay were very different experiences.

Measuring the complex concept of team trust needed a little more contemplation. While popular in its original and adapted forms the McAllister [1995] questionnaire was far too long to be used in combination with the CCPIG. The Webber [2008] is short, contains a section on monitoring, and contains a familiarity section which is similar to the measure of friends vs strangers vs clan-mates which was a feature of the CCPIG introduction items throughout the development. However the citizenship and reliability sections are too similar to CCPIG items regarding cooperative behaviour and team-mate value. While similarities between trust questionnaires and the CCPIG may be expected due to “cooperation [being] among the most proximal behavioral manifestations of trust”[Rousseau et al., 1998], the development of the CCPIG has shown that it is important to reduce repetition in questionnaires to avoid participant frustration. Similarly the Jarvenpaa et al. [1998] and Costa [2010] questionnaires cover all the core concepts of trust in teams but are again too long to accompany the CCPIG in full.

Much of the questionnaires documented in the literature review are constructed from combinations of novel items, adapted items, and established measures. While the Costa [2010] questionnaire is the closest in required content it was, as stated, too lengthy to include with the CCPIG. The Costa [2010] contains an adaptation of a well established Cummings and Bromiley [1996] measure of organizational trust, which was also too long to accompany the CCPIG, at 62 items [Vidotto et al., 2008]. It would have been possible to do as previous team trust studies had, to take sections of other questionnaires, adapt them to be more relevant to the research domain, and use them to measure the concept of trust. However as sections from validated measures are cherry-picked and adapted, it becomes increasingly difficult to establish if they are still valid and have not drifted conceptually from their original focus. For example, the Langfred [2004] questionnaire contained a four-item scale based on the Simons and Peterson [2000] trust scale, however the two seem to bare little resemblance. The Simons and Peterson [2000] scale contained five-items (Cronbach’s α of 0.89 in the original study) with a very narrow focus on interpersonal trust between executives, while the Langfred [2004] items simply refer to trust (Table 5.4).

Langfred [2004] Items Simons and Peterson [2000] Items We trust each other a lot in my team We absolutely respect each other’s

competence

I know I can count on the other Every executive present shows

team members absolute integrity

The other team members know they We expect the complete truth

can count on me from each other

I trust all of the other team members We are all certain that we can fully trust each other

We count on each other to fully live up to our word

Table 5.4: Comparison of Trust Items

However despite the unrelated appearance the Langfred [2004] version achieved a Cronbach’s α of 0.83 in their study, suggesting the scale remained internally consistent, perhaps due to the similarity of the items and the fact that all but one item contains the word trust. The early trust section within the Jarvenpaa et al. [2004] questionnaire is similar in that it too was taken from another source ([Schoorman et al., 1996]) yet achieved reasonably high Cronbach’s α scores (0.77 & 0.80) when user in a different context. These two scales were concise enough to be used with the CCPIG without over-inflating the overall number of items. Table 5.5 shows a comparison of the Langfred [2004] adapted Simons and Peterson [2000] scale and the Jarvenpaa et al. [2004] adapted [Schoorman et al., 1996] scale. Both scales have their strengths and weaknesses, the Langfred [2004] items are simple but rather general, where as the Jarvenpaa et al. [2004]

items are more task oriented but would need to be more heavily adapted for use in games. For example item three is largely the same as item one, yet with a focus on monitoring behaviour, as discussed, while monitoring behaviour is seen as evidence of a lack of trust in workplace teams, it is unclear if this concept carries across to gaming.

Langfred [2004] Items Jarvenpaa et al. [2004] Items We trust each other a lot in my team I feel comfortable depending on my team

members for the completion of the project

I know I can count on the I feel that I will not be able to

other team members count on my team members to help me

The other team members know I am comfortable letting other team they can count on me members take responsibility for tasks

which are critical to the project, even when I cannot monitor them

I trust all of the other team members I feel that I can trust my team members completely

Table 5.5: Comparison of the Trust Scales

Once the scales were adapted for use in a gaming context (Table 5.6) it was decided that the Langfred [2004] items would be used due to their simplicity, face validity, and concise nature. This decision was supported with the feedback from a number of gamers and colleagues.

Contextualized Langfred [2004] Items Contextualized Jarvenpaa et al. [2004] Items I felt the team trusted each other a lot I felt comfortable depending on my team

members for the completion of team goals

I knew I could count on the other team members I knew I could count on the other team members to help me (reversed)

I felt the other team members could count on me I was comfortable letting other team members take responsibility for critical tasks in game

I trusted the other team members I feel that I could trust my team members completely

Table 5.6: Scales Contextualized for Gaming

Statistical Criteria

This study aimed to investigate the interplay between various concepts, primarily social presence, trust and contextual variables. This investigation was achieved by establishing correlations between scales, and exploring the statistical significance and effect size of differences between variables. As the data set is so large and varied that the effect size of any differences was considered as a counterpoint to significance, as

significance was likely to appear from small differences in such a large data set [McCluskey and Lalkhen, 2007]. The statistical significance of any conditions, such as winning and losing, on concepts such as social presence were measured with a T-test, with a P < 0.05 being considered significant. While many T-tests are documented throughout this study effect size was used as the focus of analysis and so over-testing should not be a concern. To measure effect size Cohen’s D was used, with a score of 0.2-0.5 considered as a small effect size, 0.5-0.8 as medium, and 0.8 or more considered a large effect size[Cohen, 1992].

Establishing the Cronbach’s α of sub-scales was used to ascertain the internal reliability of each sub-scale, and establishing the measures of sampling adequacy (MSA) and Kaiser-Meyer-Olkin (KMO) scores was used to determine sampling adequacy. These statistics indicate the degree to which the items as a whole provide a consistent statistical structure. KMO and Cronbach α scores of over 0.6 are desirable indicators of statistically reliable items (Table 5.3.2)[Kline, 1999, Everitt, 1993, Nakazawa, 2007].

Cronbach’s α Internal Consistency

< 0.9 Excellent

0.9 - 0.8 Good

0.8 - 0.7 Acceptable 0.7 - 0.6 Questionable

0.6 - 0.5 Poor

0.5 > Unacceptable

Correlations between the concepts measured by each sub-scale were tested to reveal any potential interplay, a “correlation coefficient is an index of agreement between two sets of scores 1 is perfect, 0 is no agreement, and -1 complete disagreement”Kline [1998]. In this study correlation scores of 0.3 - 0.5 being considered weak correlations, while correlations scores over 0.5 were considered strong. While a correlation of 0.3 may not usually be considered weak these distinctions are meaningful within the context of this study as correlations range from non-existent to extremely high (over 0.8), and so these distinctions help to create a more fine-grained analysis of the data, especially when comparing individual game data sets. Correlations of course do not imply causation but the strong correlations seen throughout these results are perhaps indicative of underlying common mechanisms. Throughout the study such mechanisms are hypothesised about, though acknowledging that in all cases they would need to be investigated more rigorously through more controlled studies.

To make the results more readable scores of note in tables are colour coded, with weak-medium scores being coloured blue, and strong in green, for example correlations of over 0.3 will be blue, over 0.5 will be green, Cohen’s D’s over 0.3 will be blue, over 0.7 will be green, and so on. The R codes used for the statistical methods can be found in Appendix 7.3.

Game Communities

Respondents were be gathered from a variety of game communities which centred around team-based online games. Some game communities were chosen as they were involved in previous studies and provided valuable feedback, others were chosen due to their player base or because they offered a novel team-based experience. Calls for participants were posted on community forums, and links to that post were also

submitted to the relevant sub-Reddits.

Arma

The call for participants from the Arma community was posted on the general section of the Arma 3 forum¹. While most of the communities focus on one game, communities such as the Arma forums cater to all the Arma games in the series. The call for participants on the Arma community was posted in the Arma 3 forum, and it was requested that people play Arma 3 before responding to the questionnaire. However responses based on Arma 2 were not discounted as both games provide very similar experiences/core game mechanics, with Arma 3 providing upgrades to fidelity aspects such as graphics, physics, animations and sounds. Arma 3 was released in 2013, while Arma 2 was released in 2009. 75% of respondents based their answers on Arma 3, however for simplicity sake the data collected from the this community will simply be refereed to as Arma throughout this study. Arma is a modern military FPS game which shares its engine with military training simulators. It is described as a ‘military sandbox’ , providing a large virtual environment (290 km) in which players can design their own scenarios, or play pre-made missions. Arma can be played player versus player (PvP), player versus computer controlled enemies (cooperative only), or in ‘Zeus’ in which players cooperate in a scenario controlled by a human games master. Arma is a combined arms game, with players having access to ground, air and sea vehicles, and realistic weapons and equipment. Arma aims to be close to simulation, and as such a player’s character moves realistically and is very vulnerable to damage and bleeding, with only a few bullets being enough to kill a character.

The Arma series of games shares many elements with the Virtual Battle Space series of military simulation software and is based upon the same game engines.

Chivalry: Medieval Warfare

The call for participants for the Chivalry: Medieval Warfare (Chivalry ) was posted on the general sections of the Torn Banner forums². Released in 2012, Chivalry is a first person melee game, in a fictional medieval setting. There are a number of game modes but the respondents to this study predominantly played team

The call for participants for the Chivalry: Medieval Warfare (Chivalry ) was posted on the general sections of the Torn Banner forums². Released in 2012, Chivalry is a first person melee game, in a fictional medieval setting. There are a number of game modes but the respondents to this study predominantly played team