Chapter 4: Polycentric Resolution of Information Pollution in Open Collaboration Systems
4.6 Analysis Results
4.6.4. Robustness and Additional Analyses
We conducted several robustness tests and additional analysis related to the impact of five key factors: the variance in the number of pollution events across articles, the variance in the order of pollution events, provider rejection, talk page activity, and veteran vs, new providers.
Number of Pollution Events
To test for the consistency of our hypothesized model across articles with a varying number of pollution events, we conducted three analyses. First, we tested the model with articles that had only a single pollution event (N=4,020). As Table B1 in Appendix B shows, the model results are consistent with our main analysis of pollution events from articles with varying numbers of pollution events. Second, we tested the model with articles that had two pollution events (519 articles, N=1,038). As the model results (Table B2, Appendix B) show, incremental adaptation (revision) is no longer statistically significant in relation to pollution resolution time. Finally, we tested the model on articles containing three or more pollution events (119 articles, N=408). As
189
Table B3 in Appendix B shows, incremental adaption (revision) is positively associated with pollution event length, suggesting that it slows down pollution resolution. The results of these analyses suggest that incremental adaptation (revision) helps resolve pollution faster in articles with a single pollution event, but that this association reverses itself in articles with three or more pollution events.
Order of Pollution Events
To test for our hypothesized model’s consistency across pollution events based on their order, we conducted three analyses. First, we tested the model across all first events in our sample (N=4,679). As Table B4 in Appendix B shows, the direct relationship between boundary regulation and pollution resolution time is statistically insignificant. In all other respects, the model is consistent with the main model. Second, we tested the model across all second events (N=639). The model results (Table B5, Appendix B) are consistent with the main model, in which we do not distinguish between pollution event order. Finally, we tested the model across the third or more pollution event (N=169). As Table B6 in Appendix B shows, these results are inconsistent with our main model: (1) We detect no significant relationship between incremental adaptation and pollution resolution time. (2) We found no curvilinear relationship between provider rejection and pollution resolution time. This is slightly different from our research model findings. However, it is important to note that 97 percent of pollution events in the random sample has two or less pollution events; having a third or more pollution event is rather an atypical occurrence.
Provider Rejection
In the main specification, we incorporated U_PRJ and its squared term in the model specification (Equation 1), as PRJ is highly correlated with boundary regulation (policy citations) and
190
incremental adaptation (revision). By way of robustness, we evaluated a model with controls and only PRJ and SHA as per the following equation:
PEL=β0+ β1Controls+ β2PRJ+β2PRJ2+ β3SHA+ ε
We used a three-step hierarchical OLS regression analysis to estimate the equation, adding variables in the following order: 1) control variables, (2) provider rejection, and (3) squared value of provider rejection to test for the hypothesized curvilinearity.
The model explains 24.3% of the variance in pollution event length (where 11.9% is explained by the control variables). The addition of provider rejection and provider rejection squared uniquely contributes to variance explained (Table 4.8). The VIF values are less than five, suggesting that multicolinearity is not a major issue.
Table 4.8 Provider Rejection Results, DV: Event Length Time (N=4,325)
Variable Model 1 Model 2 Model 3 Model 4
Article age .314*** .329*** .337*** .335***
Length of article .065*** -.023** -.020* -.109***
Providers before pollution event -.521*** -.536*** -.609*** -.549***
Providers before: talk page .126*** .046** .114*** .057***
References before pollution -.025** .046*** .029*** .061***
Reverted revisions (PRJ) .274*** .528*** .263***
Reverted revissions2(PRJ2) -.274*** -.072***
Work distribution / providers (SHA) -.276***
Model F 773.6 1,044.7 954.2 1,156.1 F change 2,115.1 337.9 2,085.2 Adj. R2 .119 .179 .19.8 .243 ΔR2 .06 .019 .045 * p < .05; ** p < .001; *** p < .0001
Our results also support H4, which predicted that provider rejection (reverted revisions) will reduce the length of pollution events to a point and increase it thereafter. Provider rejection is associated with an increase in pollution event length (β = .263, p < .0001), and provider rejection
191
squared is associated with a decrease in pollution event length (β = -.072, p < .0001). Apart from references before a pollution event, the control variables are statistically associated with pollution event length.
Talk Pages
Previous research shows mixed results regarding the importance of explicit coordination; researchers have found evidence that explicit coordination helps to improve outcomes (Arazy and Nov 2010; Viegas et al. 2007), but also that a disproportionate amount of work is carried out in relative isolation, without communication between contributors working on the same article (Kittur and Kraut 2008). In our data, we find that almost all editing activity during pollution events is devoid of explicit coordination through the talk page. An average of only 1.9 providers (SD 3.5) participated on the talk page during an event, making an average of four revisions (SD 16.9) to it and citing on average only 0.05 policies (SD 0.6). We performed a hierarchical OLS hierarchical analysis adding variables in the following order: (1) control variables, (2) polycentric governance variables, and (3) talk page variables (see Table B7, Appendix B). We find that the addition of talk page variables adds only modestly to variance explained (adjusted R2 changes only 1.7 %) and that our hypotheses still hold when we include the talk page variables.
Veteran and New Providers
We further examined how work distribution between veteran and new providers impacts pollution resolution. Consistent with previous research (Panciera et al. 2009; Shah 2006), we find that greater involvement of veterans is associated with faster pollution resolution. Interestingly, however, we find that most of the work during pollution events is shouldered by providers new to the article rather than by previous contributors to it. Approximately 85% of providers active during the pollution event were not active on the article prior to the cleanup template posting; further,
192
new providers were responsible for approximately three-fifths of the total aggregate edit distance during the pollution event. This indicates that the cleanup template acts as a call for action and is more effective at injecting new blood into the article than in rousing existing providers to improve their work. We again performed a hierarchical OLS analysis adding variables in the following order: (1) control variables, (2) polycentric governance variables, and (3) variable on veteran providers. The veteran provider variables are: (a) the ratio of editors who remain active (veteran
editors of the article) in relation to all editors who were active before the pollution event; (b) the number of veterans; (c) the ratio of veterans to all editors active during the pollution event; and (d) the proportion, in characters, of the length change that veterans made As Table B8 in Appendix B shows, we find that the involvement of veteran providers reduces the length of pollution events, but including those variables adds only modestly to variance explained (adjusted R2 changes 2.6%). Again, our hypotheses hold when considering veteran’s activities.
4.7 Discussion
Our empirical analysis supports the general assertion that polycentric governance practices influence pollution resolution in Wikipedia. Specifically, we find that shared accountability and incremental adaptation are linearly associated with faster pollution resolution, while boundary regulation and provider rejection have more complicated relationships. Based on these findings, we first bring together theoretical contributions to the literature on addressing quality in OC systems (Arazy and Nov 2010; Chesney 2006; Giles 2005; Kane 2010
;
Kittur and Kraut 2008; Wöhner and Peters 2009). We then move to discuss the findings in light of PIC theory (Mindel et al. 2018) and the implications of our findings for resolving pollution in OC systems and other online platforms with low barriers to participation.193