Previous to the implementation of the NOSE dashboard, we conducted an OSS developers survey. This evaluation part brought new ideas for the design of our dashboard. This group did not evaluate the NOSE dashboard directly. The OSS developers were given a ten question long survey. Figure 8.7 displays the survey questions and the results. Many of the questions had an optional comment field. We posted a request with a link to our survey on the IRC of several bigger OSS projects: OpenStack, PostgreSQL, GIMP, Mozilla, Oracle VM VirtualBox, GNOME, TomCat. Some developers provided initial feedback directly in the chat. The survey was anonymous, therefore, it was not possible to trace from which project the participants originated. Nevertheless, based on the survey results and by observing each chat for four hours, no malicious users were detected. In total there were 32 responses with many constructive suggestions. Everyone who participated in the survey finished it, which is a further indication of the genuine interest shared by the participants of the survey.
!"#$%&'( )#$ *' +,%&'(-. !"#$"%#%&'#()*+,,- /01234456 .#(./* 012'#$"%#'2'3#4"56378%6'9#6"#15$#()*+,,#:3";'46#('<=#8%=#3':"36*- /312789756 >#(?<>/* +@A"@B#C760%8#159#&7D7A13#:A16E"3D&#E"4%&#"5#F"GH4"DD76#&6167&674&B# I@'3'1&#J#I156#6"#E"4%&#"5#4"DD%576$(K:'":A'*#'2"A%67"5<#!"#$"%#E759#6@'# 75E"3D167"5#6"#8'#756'3'&675=- 37128/9/56 >>#(?L<M/* J#I156#6"#:3"279'#&6167&674&#"5#6@'#I@"A'#4"DD%576$B#8%6#1A&"#'2'3$# 4"56378%6"3#E3"D#6@'#D17A75=#A7&6<#N"%A9#$"%#E759#76#%&'E%A- 0:12;<56 O#(PQ/* R3'#$"%#756'3'&6'9#75#1#5'6I"3S#27'I#(&"471A#=31:@*#"E#$"%3#()*+,,# 4"DD%576$- 0/12;39756 T#(PO<>/* R3'#$"%#756'3'&6'9#75#1#6'U6#D7575=#151A$&7&#"E#$"%3#()*+,,#:3";'46# 4"DD%574167"5#('<=<#9'6'3D75'#D175#6":74&#D17A75=#A7&6* 04128:9<56 >>#(?Q<Q/* R3'#$"%#756'3'&6'9#75#&'567D'56#151A$&7&B#'&67D1675=#6@'#VD""9V#"E#'14@# D'&&1='#159#:3"27975=#1==3'=16'9#&6167&674&#E"3#6@'#I@"A'#:3";'46- >.#(?><?/* 00128=9=56 !"#$"%#%&'#I'8W&'3274'&#A7S'#+@A"@B#C760%8#"3#"6@'3#&7D7A13#:A16E"3D&- 0:12;<56 Q#(>Q<L/* X"6#4%33'56A$B# 8%6#J#@12'# 9"5'#&"#75#6@'# :1&6Y#?#(T<Z/* N"%A9#1#:'3&"51A7['9#91&@8"139#('<=<#4@""&75=#9161B#27&%1A7[167"5#E"3D#'64<*# 8'#D"3'#%&'E%A#6@15#6@'#:3";'46W"37'56'9#27'I#"EE'3'9#8$#+@A"@#159# C760%8- O#(PQ/* P#(L<?/* X''9#6"#63$#76# E73&6Y#22 (68.8%) J@'3'#15$#&6167&674'=13975=#$"%3#()*+,,#:3";'46#6@16#$"%#I"%A9#A7S'#6"# @12'-#)"3#'U1D:A'B#7E#$"%#@12'#%&'9#C760%8#"3#+@A"@#159#76#I1&#D7&&75=# &"D'6@75=#7D:"36156< )3''#E"3D#15&I'3&
Figure 8.7: OSS Developer Survey Results.
Background The survey results showed that all of the evaluation participants use
OSS projects and 96.9% have contributed to at least one of them. In conclusion, the user group was very well captured.
Community Web-platforms like Ohloh or GitHub were used by 75% of the partic-
ipants. The participants had the possibility to name some additional applications which they had previously used. There were 16 comments in total, with GitHub being mentioned 13 times, SourceForge 5 times and Ohloh 4 times. Regrettably, the question was a bit ill-formed, since it was unclear whether the users of GitHub and SourceForge were also interested in the offered project analysis. This concern was confirmed in the next evaluation part, where some bioinformatics OSS developers stated that, although they used GitHub, they had never considered the provided statistical visualizations.
When asked, if the OSS developers were interested in the statistics related to community evolution, 63.3% of participants replied positively. However, in four of seven comments, participants mentioned that it was not completely clear what kind of information was meant or “How would/could I benefit from that information?” Considering the question of whether the interviewed OSS developers were in- terested in getting statistics from the mailing list regarding the whole community (including core and periphery), 75% answered “yes”. Based on the given comments, the participants recognized it as a useful metric for getting an approximate user base size. However, one participant also mentioned a negative aspect, that “[...] too
much statistical evaluation could put the community off as they feel “observed””.
According to our findings, 71.9% liked the idea of the network graph representa- tion of an OSS community. However, two answerers mentioned that they would find it more interesting than useful.
Dashboard A personalized dashboard was considered useful by 25% of the partic-
ipants. Whereas the majority of the participants (68.8%) replied that they would need to try it first. One participant mentioned that “The projects I contribute to
have their own dashboards, I don’t think an external dashboard would match my expectations”. Indeed many of the OSS hosting platforms offer their own built-in
analysis and visualization. Hence, an external dashboard had to be significantly different from the existing visualization platforms to gain the OSS developers attention. Alternatively, it should be integrated in the existing platforms or tools.
text mining (TM) Majority of the participants (64.5%) were interested in TM
analysis of the mailing list communication. The participants also suggested addi- tional uses for TM:
• “[...] determine the needs of the users in addition to voting and tagging in
bugtrackers”
• “[...] creating FAQs for new contributors”
• “[...] finding out in which direction the community wants to evolve”
When asked, if the OSS developers were interested in sentiment analysis (SA) of the community, the participants were mostly uninterested, with 68.8% replying with “no”. Unfortunately, this negative reaction could be a result of little awareness of the SA meaning. For example, one of the participants said that: “Only slightly
interested. I doubt that much useful information could be drawn from such an analysis, but I would need to know more about the methodology and findings to be sure, and it sounds interesting at least”. Another participant expressed the
concern that such analysis “Would be interesting/fun, but i’m not sure whether
its useful for me [...]”. However, there were also participants that were clearly
in favor of SA: “Definitely. I would have stopped before entering some projects
if I would have known about the mood swings of their contributors beforehand...”.
There were also responses that were clearly against it, for example “I don’t think
public statistics of the form “messages from developer X are mostly aggressive” would do anything good”. Some developers were rather interested in aggregated
statistics. For example, determining the mood of the whole community or specific groups without tracing it back to each individual member was proposed: “I think
this is only useful if combined with a certain segmentation of the user groups”.
Missing Functionalities Finally, the participants were asked if they missed any
statistics in the existing OSS navigators they used so far. The following responses were given:
• “[...] which wiki pages are consulted often, which problem appear in lists/fo-
rums frequently and so on.”
• “Some projects [...] allow Users to be credited in commits for their Testing
or Bug reports (e.g. Reported-By: Tested-By:) including those contributions in statistics would by nice [...]”
• “Last time on ohloh I wanted to see a simple list of contributions, but I only
found timelines and such, which I find hard to browse”
• “More information about project activity. Most FOSS project have stalled
development, are abandoned. This is for me the #1 FOSS problem”
Additionally, one replier provided the link to a service that provides a contributor- oriented statistics (OSRC in Section 8.1). Another survey participant questioned the accuracy of sites like Ohloh and GitHub. He argued that a user might not be socially present there, but he/she is still part of the project.
Summarized, the OSS developers showed strong interest in community and mailing list analysis. This supports the willingness of OSS communities to be aware of what is happening within their projects. However, there was some uncertainty about the meaning of some proposed metrics. To find out which statistical charts and their designs are truly useful, the users need an application that they can try out.