Summary of data obtained

3.1.1. GO annotations

In both the single-MOD and multi-MOD studies, the formal GO annotations were collected using a spreadsheet template distributed to all participants. The forms had a separate sheet for each article with predefined columns for each of the GO annotation elements. Each of the participants sent the completed forms to a non-participating curator, who then compiled them, ensured they were de-identified, and sent them to the study PI. In the single-MOD study, this person was a supervisory member of the MOD staff; in the multi-MOD study, this was a curator who was coordinating the meeting at which the study was based. The participants in both studies also used the same template to record their consensus annotations, which were collected in the same manner. The observations resulting from the studies are shown in Table 4. A total of 480 annotations composed of 3,915 instances were generated by the curators during the two studies. On average, there were about 2 instances per annotation in the SMS, and 8 per annotation in the MMS.

Section 3.3.4 below assesses incomplete instances during the evaluation of the Validity quality aspect. In cases where curators made no annotations, these observations were counted as having zero instances rather than as missing data, because it is a valid annotation outcome to conclude that an evidence source contains no annotatable

Table 4. Aggregate GO annotation data points by study

Single-MOD Multi-MOD Total

Individual annotations per paper *2/4 23 -

Total individual annotations **128 230 358

Total consensus annotations 32 90 122

Overall annotations (individual + consensus) 160 320 480

Total individual annotation instances 265 2,289 2,554

Total consensus annotation instances 103 1,258 1,361

Overall instances (individual + consensus) 368 3,547 3,915

*4 each from the 16-paper 4x coverage set, and 2 each from the 32-paper 2x coverage set **64 from the 4x coverage set, and 64 from the 2x set

consensus annotations, and 11 in the MMS. In the latter, most (n=8) of the cases were related to one curator, C23, while in the SMS the cases were distributed across all 8 curators; every curator had at least four papers where zero annotation instances were made, and 6 of the 12 consensus pairs had at least one case of zero instances. Thirty-four of the cases were from the 2-fold coverage set of 32 papers, which was not assessed in detail by the MOD leadership for the presence of putative annotations, while the 4-fold set was.

Within the spreadsheet templates, some curators made notes that describe why they made certain annotations, and the decisions underlying them. In some cases these notes provide additional contextual detail, but many of them relate to current database policy or other issues that do not directly affect the GO annotations, and so they were not

considered in the quality evaluation. In some cases, curators chose existing GO terms for their annotations, but also noted that in a real-life situation they might request a new term instead of using the existing terms.

Because curator IDs were assigned randomly to participants, and the contextual data were not linked to the annotation data, the study PI had no knowledge of the identities of the curators until after the quantitative analysis had been performed. Likewise, the data from the personal experience questionnaire pages of the MMS template was not compiled

until after the analysis of the GO annotations, since it could have been possible to identify curators based on their attributes.

3.1.2. Individual interviews (both studies)

Fifteen interviews with individual curators were conducted over the course of the two studies, ten with participants in the single-MOD study, and five with those in the multi- MOD study. Digital audio recordings were made of all interviews, which were then transcribed, corrected, and verified. Table 5 lists each interview’s acquisition date, duration in minutes, and transcribed length in pages and words.

Table 5. Audio recordings of individual curator interviews

ID* Date Duration Transcribed pages / words

SI01 2006-06-05 00:40:12 21 / 6,300 SI02 2006-06-05 00:37:34 21 / 7,200 SI03 2006-06-05 00:30:05 19 / 5,300 SI04 2006-06-06 00:30:35 15 / 4,700 SI05 2006-06-06 00:22:33 13 / 3,800 SI06 2006-06-07 00:32:21 18 / 4,300 SI07 2006-06-09 00:34:00 21 / 5,900 SI08 2006-06-09 00:44:48 29 / 8,900 SI09 2006-06-09 00:29:56 26 / 5,500 SI10 2006-06-09 00:23:20 16 / 4,000 MI01 2006-07-14 00:19:18 18 / 3,500 MI02 2006-07-14 00:25:00 18 / 4,200 MI03 2006-07-11 00:21:22 20 / 4,100 MI04 2006-07-12 00:27:26 14 / 4,500 MI05 2006-08-03 00:41:08 20 / 5,800 Total 07:39:38 289 / 78,000

*‘S’ prefix = single-MOD study, ‘M’ prefix = multi-MOD study

The interviews contained data on curators’ workflows and individual work

processes. These data were coded as described in section 3.2.2 and used for the workflow analysis in section 3.5

3.1.3. Curator background and experience data (both studies)

In the multi-MOD study, the template file in which curators recorded their

annotations also contained a four-item self-administered questionnaire which was used to gather information about each curator’s backgrounds and experience. All 23 curators completed the questionnaire, and the data are described in section 3.6 below. The questionnaire is attached as Appendix A2. Subsequent to the completion of the study, curators were asked a follow-up question by email about the number of years or months experience each had with GO annotation at the time of the study.

In the single-MOD study, similar questions were asked of the 10 participating curators during the individual interviews that were conducted. This information was extracted from the transcribed interviews, recorded, and analyzed as described in section 3.7 below.

3.1.4. Evidence sources and other artifacts (both studies)

Several types of additional artifacts were collected during the two studies. Six participants in the multi-MOD study provided complete sets of their paper copies of the 10 articles used in the study, for analysis of their intermediate manual notes and

annotations. These total approximately 630 pages, which included about 15 separate pages of associated notes in addition to the marked-up primary articles. Other documents and data included one curator’s manual notes and article annotations from the single- MOD study, and a chain of emails from one of the consensus pairs from the single-MOD study to document their process of consensus formation. Also available were the PI’s notes from both studies of observations and interviews, and screenshots of a MOD’s curator web interface, collected during a portion of the single-MOD study.