Reflections on the Methods

Due to the limited availability of publicly accessible digital traces in most online platforms, this study aimed to experiment with exploring a sample of scholars’ scholarly acts across platforms based on their self-report. Specifically, this study targeted researchers who were Twitter users. There are several advantages of using self-report in surveys. First, it is an

inexpensive and relatively more efficient way of obtaining data. With the aid of electronic survey tools like Qualtrics, this study was able to reach a wide variety of research subjects than could be reached by other methods. More importantly, the self-reports can be made in private and can be anonymized to protect sensitive information and perhaps promote truthful responses. If not relying on self-report, it would be difficult, if not impossible, to obtain the footprints of their other scholarly acts (e.g., reading, saving, citing) on a relatively large scale.

However, self-reports have drawbacks due to factors including respondents’

178

accurately, instability of opinions and attitudes, as well as the lack of truthfulness (Dixon, Singleton, & Straits, 2016). Because the survey in this study heavily relied on self-report from the respondents, which might cause biases, caution needed to be used in the analysis and interpretation of the data. Previous research has recommended using self-report in conjunction with other methods (Fan et al., 2006). So, in this study, a test-retest approach was employed to assess the reliability of operational definitions. Specifically, for the 20 interview participants, their responses in the survey were cross-compared to their responses in the interviews as a data triangulation approach.

There were two levels of comparisons. The first one was between two types of self- reports: answering an online survey on the laptop or cellphone and talking to an interviewer. In this comparison, no significant unreliability was found. When asked about the rationale behind their survey responses in the interviews, their answers were specific and grounded in their previous descriptions of relevant questions. The second level of comparison was between reporting general behaviors and discussing specific incidents. In this comparison, only one participant (P19) commented not remembering the stories in multiple cases. Except for P19, the other participants all told detailed stories behind their critical incidents. Their answers also corresponded to their “typical activities” of workflow answered in the survey. This multi-method approach provided a more global and therefore likely more accurate picture of the stories.

6.2.2 Reliability Checking in the Survey Data

In addition to the multi-method checks conducted in the previous section to examine the reliability of self-report, another approach of reliability checking was conducted on the survey data. Specifically, in the survey, the researchers were asked: “When you posted tweets regarding scholarly articles, how often did you have a positive/neutral/negative opinion of the articles?”

179

This same question was asked four times regarding the four acts of tweeting, retweeting, replying and liking. In the survey data cleaning process, the answers from the participants were checked to see if there were contradicting or incongruent patterns of answers based on logic. For instance, one cannot always have a positive opinion on the articles they tweeted while at the same time always having a neutral or negative opinion on the articles they tweeted. In total, seven types of responses were identified and further examined. The process of data cleaning is described in more detail in Appendix IV.

This further examination revealed some unreliable and therefore erroneous data, which were deleted from the analysis. Meanwhile, it also revealed some problems in respondents’ interpretation of the survey questions. These might be due to the different introspective abilities of the participants, their degrees of devotion to the survey, or due to the wording and design of the survey.

Overall, two problems were found (and discussed in Appendix IV). First, in very few cases, participants might overuse “Always” and “Most of the time.” Particularly, “always” should mean and did mean “in all cases” to the majority of the participants. However, it seemed that in some cases, participants might “always” have a positive opinion on the articles they tweeted but at the same time additionally “sometimes” had a neutral opinion on them. Secondly, in a few cases, participants tended to overuse “Rarely” or “Never” for the acts that did not conduct very frequently in general. In the design of the survey, the anchors here (“Always,” “Most of the time,” “Sometimes,” “Rarely,” “Never”) should be in relevance to all cases within the four categories of acts. However, if an act was conducted very infrequently by a participant, it was possible they might answer “Never” or “Rarely” to all categories of “Positive,” “Neutral,” and “Negative.”

180

An additional minor problem was found based on the responses regarding the act of liking: a small number of participants “sometimes” had a positive opinion on the articles they liked while “sometimes” had a negative opinion on the articles they “liked.” While this might truly reflect their behavior, it is also possible that this was due to careless responses. Being the fourth question in a series of similar questions, there was a possibility that some participants provided this response due to the patterns of their answers in the previous three questions, which was similar to the straight-lining behavior when responding to surveys.