Chapter 2 Learning and Strategic Behavior in a Crowdsourcing Site
4. Users’ Continuing, Learning, and Adaptation
As we see that the properties of the tasks and the award offered by the tasks can influence people’s decision on whether to participate in a competition, we further examine what makes them continue. Merely winning appears to play an important role in contribution. The vast majority of the users on the Witkey websites actually get nothing from their contributions, since the probability of winning is so small. One might therefore expect that a lot of users would leave after a couple of failures. In fact, from 2006 June to May 2007, there were 66,182 users who had one, two, or three submissions during this period and never submitted anything else after May 2007. These users, one third of Taskcn's total participants, disappeared. The high number of registered users who have never attempted a task (89%) suggests that although there are many people interested in participating, they might be hindered by the very likely futility of their efforts.
For those who do elect to participate, the first attempt in the competition can be very important in influencing their subsequent participation in Taskcn. There are 2307 users who won on the first attempt and 169,456 others who failed on the first attempt. Figure 3 shows the portion of users in the winner and loser group who had 2, 3, 4 … j attempts.
Both groups have a heavy tailed distribution of attempts: the majority of users have a couple of attempts and a handful of users attempt many tasks. One can observe that, on average, the winners have more attempts than the losers group.
A
A Cox propor 9% lower pr hows, this tr roup. If the whether the nd subseque winning enco
Users Learn
Next we inve o users adap
he timing o hose to sub ormalize the
rtional hazar robability (si ranslates to
first win occ user continu ent, compet ourages user
n to Submit
stigate users ptively chang
f users’ sub bmit early, o e time of a u
Figur submiss
rds analysis s g.<10-4) of s approximat curs on the t ues participa
itions can be s’ contributi
t Later
s’ participati ge their beh
missions is a r wait to se user’s submi e 3: Distribu sions depend
or lost on
shows that u stopping aft ely 1 additio third attemp ating. This s e an importa on.
on pattern f avior over ti
an importan e how many ssion by the ution in the ding on whe n their first a
users who wi ter each sub onal attemp pt, there is a
suggests tha ant factor in
from a dyna ime?
nt participati y other subm e task period
total numbe ether the us attempt.
in on the firs bsequent att pt on averag
smaller diff at the result n later partic
mic perspec
ion dynamic missions a t d (the durati
er of ser won
st attempt h empt. As Ta ge for the w ference of 12 t of a user's cipation beh
ctive. That is,
c, since user task receives
on from the ave a
time to the end time of a task), so that a user who submits at the very beginning has a submission time of 0, and one who submits at the end has a submission time of 1.
Table 5: Comparison between the number of submissions for first time winners and losers
Winners in 1st attempt
Losers in 1st attempt
Mean 4.388817 3.20194 Variance 85.02092 25.54748
Observations 2307 169456
P(T<=t) two-tail 8.04E-10
t Critical two-tail 1.960985
We find that, for all users, task award correlates with users submitting solutions later. It may be an indication that people are more intent on winning higher awards ( = 0.067, sig. <10-4) and so either take longer to devise a solution or “sit” on it until they are certain it is their best effort. Interestingly, tasks of longer duration have a slightly later submission time (relative to the overall duration (= .026, sig. <10-4). One possible explanation is that users may notice the task after it has started and still have sufficient time to submit a solution.
Furthermore, we find that the number of submissions is negatively correlated with the time when people submit ( = -.128, sig. <10-4). A simple reason could be that most tasks with many submissions require little effort, and so users can complete and submit solutions sooner. An alternate explanation could be that when people see that many others have participated in the task, they may not want to follow up. This would result in a higher proportion of submissions having an earlier submission time.
Fi
igure 4 show winners alike
bserve that thers’ subm However, the
Users Learn
ince a user ompeting in elect tasks w
n general, th o choose ta
orrelation be 0-4). Note th ver time. In re choosing
Figure 4:
w
ws that user e, are likely
winners con missions, the later submi
n to Choose
r’s chance o n the task (Y with fewer co
here is a lear asks with f etween the hat this is occ
fact, overall the less po
Point of sub who have pa
rs, both tho to submit s nsistently su ere is little ssions may b
e Less Pop
of winning Yang et al., 2 ompetitors, i
rning pattern fewer comp
number of curring not l task popula
pular tasks a
bmission in rticipated 1
se who have lightly later bmit later in additional be an indica
pular Tasks
largely dep 2008 a), we
n order to e
n of users ov petitors (for
competitors because the arity is rising as they gain
the task per 5 or more ti
e won at le as they pa n the time p information tion of grea
pends on t hypothesize nlarge their
ver time: use all users a s and order ere is an ove
g very slight n more expe
riod for user imes.
ast once (w rticipate ove period. Since
they can ter effort ex
the number ed that user winning pro
ers are more and all par
of attempt:
rall decline i tly. Rather, o erience on th rs
inners) and er time. We e they canno gain by wa pended.
r of other rs would lea obability.
e and more rticipation le
= -0.23, s in task popu on average, he site. Note
th
hat the varia he variance i n first attem
n order to ta ame total nu t each attem
articipants w 28, and 3520 asks in which
Another direc me is to me ubmissions he task. Inde
ance explain is due to the mpts, mean=2
Figu
ake a closer l umber of tas mpt. Figure with exactly 0 users resp h they chose
ct way to se easure the a before and eed, we obse
ed by the le e difference
2362, standa
ure 5: Avera users in ea
look of this t sks and look 5 shows th 20, exactly ectively). We e to compete
ee that user average exp after the pa erve a negat
earning tren in popularit ard deviation
age size of th ch task that
trend, we se at the avera he trends of 12, and mo e plot the av e, for each at
s adopt a st erience of t articular task ive correlatio
nd is necessa ty of tasks ch n=1971 sub
he competit t they attem
elect subsets age characte f how users ore than 15 a
verage numb ttempt they
trategy favo the user (giv k) and comp on ( ~ -0.2,
arily low, giv hosen by dif missions).
tion for mpt.
s of users wh eristics of the s selected ta attempts (th ber of the su
make.
oring less po ven as their pare it with t , sig.<10-4) b
ven that mu fferent users
ho attempte e task they c asks, consid he sets have ubmissions o
number of views and number of submissions for a task and the average experience of the user.
We can further run a regression for the submission order of a user, as related to the recency of a task, represented by the order in which it appeared, and the total number of submissions for the task. We find that both variables are significant (sig. <10-4), with later submissions by users naturally corresponding to more recently posed tasks, but in addition also corresponding to less popular tasks.
Given that users tend to adopt the same strategy of choosing less popular tasks, it is of little surprise that experienced users find themselves attending to the same tasks. If we select two users at random for each task, we observe a positive correlation ( = 0.13, sig.
<10-4) for the number of submissions by each of the two users. This implies that inexperienced users are more likely to go up against other inexperienced users who are making their first attempts, while the old timers are likely to find themselves in the company of other old-timers.
Beyond simply attempting to increase their odds of winning, we find that the more experienced users have even more interesting selection criteria. Using the human-rated sample of 157 tasks, we find that, on average, experienced users are more likely to participate in tasks with a higher skill requirement ( = 0.253, sig. = 0.002). In addition, the higher workload of the task actually hinders experienced users from attempting the task ( = -0.242, sig. = 0.003). The result suggests that the serious users of the site have a combination of multiple strategies when choosing the next task to participate in. In addition to selecting tasks of higher winning probability and expected award, they also tend to challenge themselves by participating in tasks requiring greater skill; but they are thrifty with their effort by selecting tasks of lower workload.
Users Learn to Choose Tasks with Higher Winning
O
ddsS ome tasks n
eople to pa he number robability of might be exp WinChance to
he user (=
imilarly, we participants
f WinChanc which means
articular tas epresenting
tasks have head of tim need multipl rticipate. Th of submissio f a task in ge pected to sel
o be increas 0.19, sig. <1
can also c with exactly e. Figure 6 s that each sks. In addi
users having
more than o e) and this e people to hus we defin ons for the eneral, witho lect the task sing very sli 10-4).
ompare the 20, exactly shows that group has ition, the g g more attem
one winner can also affe o complete t ne WinChanc task. Intuiti out regard t ks of a highe ghtly on ave
e task select 12, and mo all three gro
successfully group that t
mpts in total
(in most ca ect the chan the task or ce as the nu ively, this ra o a particula er WinChanc
erage with e
ting pattern re than 15 a oups presen y improved tends to st l) selects tas
ses, multiple nce of winni
simply want umber of wi atio can den
ar participan ce. Indeed, w each subseq
ns of the th attempts) se nt increasing
their chan tay longer ks with high
e winners w ing. For exam t to attract
nners divide note the win nt. Strategic we found tha quent attem
hree user gr parately in t g average tr
ces by sele (the blue p
T asks that use
Users Also R
We just saw robability. N kelihood of or a winner o
igure 7, the ward expect
n summary, ppear to be hey are likely
d WinChanc
tation over ti
these result mainder is ted in increa
r Award Ex
users gain k whether th th a higher a e divide the ificant but v ime (= 0.04
ts indicate t d by the awa ess popular t gure 7: Aver
rd)/(# of wi
part to the u explained b ases over tim
xpectation
experience, hey may als award expec total award very weak t 4, sig. < 10-5
that those u ard. They adj tasks, which rage expecte nners for ta
participat
users selecti by the actua me.
they tend so be attem ctation. To o by the num trend of use
5).
users who re just their pa
yield higher ed award (am
sk) for each te in
ing less pop al number o
to enhanc mpting to co
obtain the ex mber of winn ers increasin
pular tasks, a of winners o
ce their win ombine a h xpected ear ners. As show ng their win
e on the we
th ser groups w mproving w
ivided by th
Figure 8: Agg submission or the set of
15 times
What is wors me in term ategories th ategories, us how the per Design categ eclined sligh
hus, the que
e the winning ees that it is
ox: Users Fa
y, for most who had 8, in rates (de e total attem
gregate num s by the ord f users parti s in the desig
se, although s of wining here is actua
sers perform rformance o gory. We ca
htly over tim
estion arises
g award exp distributed b
ail to Impro
users this e 12, 15, 10, a efined as th mpted subm
mber of win der of attem icipating at gn category
h there is no rate or win ally a very s m even wors
over time of n see that and 25 attem
e number o issions) or in
ning
were able t
y choosing t
significantly mpts and th of winning ncreasing th pend more t
who had mo verall winni
to enhance
asks with hi
y rewarded:
here is no em submissions e actual mo
Total award e who have mo in the desig
all users’ pe esign and S , which mea time on the ore than 15 ng rates an
their winnin
gher awards
we investig merging tre s by those
ney won.
earned by th ore than 15 gn category
erformance a Strategy Plan ans in these site. Figure 5 attempts in nd earned a
ng probabili
p his group te won at least
ubmissions p or their first 0). Each add mean of 5.3 worsen their
ven though full 19.9% o hat this core
has grown w winning with ditional win submissions chances of w
there were o of the total w
set of users
worse instea h those who
ure 10: Inter wit
gnificant lea more efficien
and observ heir first win
we observe comes 0.68 s between w winning, the
only 231 use wins on the s learned effe
ad. We can a are not. Tha
rval until the th 5 or more
rning effect nt at winning ve the inter n, the interva
a quickenin submission wins (median
winners lear
ers winning 5 site. It is th ective strate
answer this b at is the goal
e next win fo e wins.
for a group g over time.
rval between al between t ng in the su egies for win
by contrastin l of the next
or users
p of winners . We take al n wins, i.e.
their 1st and uccession of
= -0.12, sig while most mprove them
asks, their wi ecially intere ning.
ng users wh section.
s, or at least l users who the numbe d second win
wins (see F g. <10-4) out users manag m.
ins accounte esting to ob
o are
S ot visible u thers' solut ndicates thes
he winners, lapsed). The rom the winn winners tend
igure 4 show he sequence hey are alwa
F
imilarly, whe
that winne e defined as he users, in take longer t
sers before ions). All su se users ten
are likely t e difference ner group (w
to submit ws that like e of attempt ys submittin
Figure 11: Av 5 attempts p and w en consideri
rs do differe users who w that they pa to submit th
the task fi ubmitters e nd to submit o submit la is statistical who won at l later than o all users, wi ts, but they ng later than
ve-submissi participated winners are ng popularit
ently? In ma won at least articipate in he task (note nishes, so t except for w
t solutions in ater at 0.617 ly significan least once) a other users nners also t also have a others.
ions of the t d: losers are those who w ty, winners a
any ways, th once) tend less popula e that the co
that one do winners hav n the middl 76 (after 61 nt at p =.001
and the rest (mean diffe tend to dela
consistent d
tasks users w those who h won at least
are selecting
hey are just to have the ar tasks ove ntent of oth oes not ben ve a mean
e of the tas 1.8% of the
1. Similarly, of the users erence = 0.0
y their subm delay relativ
who had at l have never w t 5 times.
g less and le
like other u same strate er time. How her submissio nefit from se of .5039, w sk duration;
task period
ess popular users.
ev
ven lower p sig. < 10-4).
ower average ad never wo hese users a ractice the robability.
n addition, t verage. The
However, the xpectation o rend.
he comparis thers at star
opularity, st As we can s e number of on in all thei are more exp
strategy of
the winner g difference is
Figure 1 at least have n
ere is no d over time, alt
son between rting and stic
arting with t see in the F
f competitor ir at least 5 pert, but it choosing u
group is oft s also statist
12: Ave-Win 5 attempts never won a
difference b though aver
n the winne cking with a
their very fir igure 11, th rs than the l attempts). T does show t npopular ta
ten able to f ically signific
Chance of t participated nd winners
least 5 tim
between wi ages for the
r group and strategy tha
rst attempt.
e winner gr loser group This result do
that they on asks, in orde
find tasks o cant (sig. < 1
he tasks use d: losers are are those w mes.
inners and e two groups
d others imp at will impro
The differe roup has alw
(defined by oes not dire n average m er to enhan
of higher win 10-5). (See Fi
ers who had e those who who won at
others in s both hint a
plies winners ve their win
ence is signif ways a signif those users ectly suggest more aggres
nce their win
nning chanc
result is consistent with our previous finding that the best predictor of whether an individual will win is the size of the competition, only then followed by the expertise of the user. Winners are simply better at executing this strategy.