Applicability of the Results

Type of community: This study used a specific community but at the same time it represents a typical community of researchers (some people working on projects together, many of the projects not overlapping, members remotely located and people based in the same institution). This is a completely voluntary community and no incentives were used for participation. Since the community had a physical presence and a virtual presence discussions have not been examined (in fact, most of the discussions in this community happened in the physical context). Discussion forums can be helpful when people are remotely located and virtual presence is their

only way of communicating. However, using discussion data gives a biased view of the community since many people exhibit lurking behaviour.

Limited study period: The study was conducted within three months which was a relatively short period. Nevertheless, we observed some positive influence on the community related to TM, SMM and CCen. If the duration of the study was longer, we would expect stronger results, e.g. a better integration of newcomers (some newcomers remained active in the community even after the study ends). Longer period would provide larger corpus of tracking data, and thus the change patterns algorithms would have been employed and extract certain patterns that could not be detected in the limited period of this study. Consequently, notifications linked to these patterns would have been generated to make notification messages more appealing to individual members. This would allow us to evaluate what this study suggested that the more targeted the notifications the better for VC members. As a result, members would develop better awareness of what was happening in the community. There is a tendency based on the results obtained that people become more motivated after they have received the notifications. However, we cannot conclude with confidence that this would be manifested in a longer study (e.g. people may ignore notifications or find them distracting).

Use of ontology: Although the community under study was created under a common title, the expertise of people was heterogeneous. Different results might have been obtained if the topic of the VC was more closed. The effect of awareness in such VCs would have been different and the connections among people would have been stronger (every member would have been connected with every other member). The heterogeneous style of this VC allowed us to exploit a semantic structure (ontology) that showed the benefits of this approach (all members agree the notifications are relevant). It is important to note that without the ontology it would have been difficult to extract accurate connections between people.

BSCW system: This VC was using the BSCW system with all its advantages and disadvantages. BSCW is a robust system and the functionality is stable and well-tested. BSCW allowed us to keep the tracking data used as generic as possible. However, there were some negative aspects associated with BSCW. The system has its own style of interaction, many of the members were not used to this style. Most members had not used BSCW before and also, during the study, were not using BSCW for any other activities in their practice. For users this was yet another system to learn:

“I already use other bookmarking/reference tracking system and I didn't see the advantage of adding another place to keep track of papers”

A number of members commented that they had not used the system because they do not like BSCW style: “[…] also the community folder structure does not appeal to me”

We can argue here that some of the negative results obtained can be attributed to the BSCW platform following studies that people tend to perform best when the tools are similar to what they have used to and also what appeals to their working style (Uruchrutu et al., 2005).

Experimenter as a member of the community: Finally the experimenter was a member of this community, thus some remarks must be noted. The questionnaires were given before the CM was extracted in each period in order to mitigate the influence on the experimenter’s behaviour. When participants know the experimenter, they might reply in the questionnaire in a biased way (for example one member noted that the purpose of this community is: “for Stella's PhD”). Having this issue in mind the researcher tried to mitigate any noise in the data. In addition, the questionnaires have been structured in such a way that members’ replies could not be fixed in order to please the experimenter since they did not know what could be a correct answer. The experimenter’s replies to the questionnaires differ from what has been extracted in the CM. On the other hand being a member of the VC allowed the experimenter to be part of what was actually happening and being able to interpret the results in a more meaningful way.

8.6 Summary

This chapter presented and discussed the summative evaluation of the notification messages. Questionnaires and statistical approaches have been used - qualitative and quantitative data were collected and analysed. The results presented in Section 8.4 support the hypothesis that notification messages can have a positive effect on members (both newcomers and oldtimers).

The second message format (personalised information for each member with links to resources in the VC), was preferred by members. All oldtimers reported they had followed the links included in the notification messages. In all cases members rated the notification messages as relevant to them. This verifies that the information kept in the CM with respect to VC members was realistic and relevant. Furthermore, there was some evidences that notifications motivated members to visit the VC space and in the case of newcomers to upload and download resources. In general, notification messages can be used for motivating members to keep active in the VC.

The notifications had some positive effect on the confidence of VC members. Both oldtimers and newcomers felt more confident to contribute after they received the notification messages. Although statistical analysis did not show sufficient evidence for the development of TM or SMM for either oldtimers or newcomers, the comments of VC members show a slow development of TM and SMM since members are reporting they are becoming aware of the resources and people available in the VC. The pre study questionnaire with oldtimers and the first questionnaire with newcomers show evidences of SMM among members. Some

newcomers and oldtimers had increase in their activity after they received notification messages. Members have either uploaded or downloaded due to the information they received with notifications. Finally, there is evidence that monitoring the CCen of members can be used to support the knowledge sharing in a closely knit VC.

The questions addressed in the experimental study presented in this chapter have not been directly derived from any previous studies since there are no evaluation studies with VCs focusing on TM, SMM and CCen. Although this evaluation is driven by clear aims and questions, the potential impact of the notifications on members and the VC in general requires a long term study which is not feasible in the time limit of a PhD (considering that a community has to be established, start functioning, a series of interventions can be done, and the long term effect observed).

Chapter 9