Micro-task Crowdsourcing: A Time
Allocation Perspective
Research-in-Progress
Ling Jiang
City University of Hong Kong
Tat Chee Avenue, Kowloon, Hong
Kong SAR
[email protected]
Christian Wagner
City University of Hong Kong
Tat Chee Avenue, Kowloon, Hong
Kong SAR
[email protected]
Abstract
Small payments in micro-task crowdsourcing markets appear unreasonable compared with remunerations for regular work in the workplace, yet hundreds of thousands of micro-tasks are completed each day, and frequently by highly educated individuals. To explain this perplexing anomaly, we investigate individuals’ continuous participation in micro-task crowdsourcing from a time allocation perspective. Drawing upon the theory of the allocation of time, relative advantage over alternative activities and reservation wage of micro-task crowdsourcing affect intent to continue and expected wage respectively, which in turn have effects on intent to increase participation level. Based on previous research on time structure, we propose time structure as another indicator of continuous participation in micro-task crowdsourcing. More importantly, the negative moderating effect of time is conjectured as a salient driver of continuous participation in micro-task crowdsourcing. IT-enabled time structuring thus helps individuals fill dead time with micro-tasking online in spite of low payments.
Keywords: Time structure, micro-task crowdsourcing, relative advantage, reservation
wage, intent to continue, intent to increase
Introduction
Crowdsourcing, often described as an “
act of a company or institution taking a function once performed
by employees and outsourcing it to an undefined (and generally large) network of people in the form of
an open call”,
has grown exponentially with advances in information and communication technology (Howe, 2006a, 2006b). Significant numbers of individuals with discretionary time are present online, thus providing an opportunity for recruiting workers through the cloud (Hoßfeld, Hirth, & Tran-Gia, 2011). A variety of crowdsourcing applications have emerged to harness distributed intellect for problem solving. Crowdsourcing platforms such as Threadless, iStockphoto, or InnoCentive (Brabham, 2008a, 2008b, 2010), are characterized by innovative co-creation in specific domains (Zwass, 2010). Micro-task crowdsourcing markets, such as Amazon’s Mechanical Turk (Ipeirotis, 2010a) or Microworkers (Hirth, Hoßfeld, & Tran-Gia, 2011), in contrast, foster a paradigm where small and simple tasks are transacted at low cost within small chunks of time (Kittur, Chi, & Suh, 2008; Kittur, Smus, Khamkar, & Kraut, 2011). At present, Amazon Mechanical Turk for instance offers more than 400,000 such tasks at any point in time. The micro-task paradigm, though, appears to go contrary to economic logic for task workers. Although the task completion has been recognized as similar to regular work (Kaufmann, Schulze, & Veit, 2011), the small payments (frequently about $0.01 to $0.10 per task) are considerably less than regular minimum wages in the traditional workplace. Moreover, typical micro-task crowdsourcing platforms such asAmazon’s Mechanical Turk are dominated by highly educated workers who possess bachelors or advanced degrees (Ipeirotis, 2010b; Ross, Irani, Silberman, Zaldivar, & Tomlinson, 2010). Are these highly educated individuals acting irrationally by completing low-pay micro-tasks? Are they possibly rationalizing the underpaid online work by re-framing it (e.g., as leisure); or are other benefits that accrue to make the effort rational? This research seeks to understand the economic paradox of worker participation in crowdsourcing platforms.
Research questions
Previous studies have extensively investigated participants’ motivations of performing tasks in crowdsourcing platforms (Brabham, 2012; Hossain, 2012; Kaufmann et al., 2011; Quinn & Bederson, 2011; Rouse, 2010; Zheng, Li, & Hou, 2011; Zwass, 2010). Although the literature identifies extrinsic and intrinsic motivations as typical antecedents of participation in crowdsourcing, the unique characteristics of micro-task crowdsourcing markets appear to transform the connotations of these motivations, while may introduce temporal cost-and-effect rationality. Unlike problems at the project scale in co-creation or open innovation crowdsourcing platforms, the crowdsourcing micro-tasks are fine-grained in terms of difficulty and duration. They can thus be fit into small chunks of time, and bring temporal structure to fragmented discretionary time, which might otherwise be idle time, “wasted” time, or under-valued time. On Amazon’s Mechanical Turk, for instance, participants perceive micro-task completion as a fruitful way to spend free time instead of watching TV, playing online games or even wasting that time (Brabham, 2012; Ipeirotis, 2008).
The literature though has attributed little attention to the temporal structuring emerging from micro-task crowdsourcing platforms or the benefits associated with time structuring. Focusing on the potential driving force of time structure beyond extrinsic and intrinsic motivations, our study aims to explore two research questions: 1) what factors affect individuals’ continuous participation in micro-task crowdsourcing from a time allocation perspective? 2) How does time structure shape the effects of identified factors on individuals’ continuous participation in micro-task crowdsourcing?
Intention of the study
The small granularity of micro-tasks, which was neglected in prior research, motivates this study to justify the low payment from a time allocation perspective. This study proposes a research model to reveal the mechanisms by which workers allocate to micro-task crowdsourcing platforms. In the light of the theory of the allocation of time (Becker, 1965; Gronau, 1977), two primary factors, namely relative advantage and reservation wage, are identified as the enabler and the inhibitor of continuous participation in micro-task crowdsourcing. We further introduce time structure as another main factor, and also examine its moderating effects on the relationships deduced form the theory of the allocation of time.
This study attempts to extend the theory of the allocation of time by generalizing its applicability to the domain of online labor markets. Framing micro-task completion as a device to structure fragmented spare time in a purposeful way sheds light on how to rationalize worker’s acceptance of small payments in micro-task crowdsourcing platforms. The focus on time structure plays an important role in exploring the temporal benefits enabled through participating in micro-task crowdsourcing. This study also has significance for task providers who may wish to leverage the time structure through decomposing online tasks so as to sustain worker participation.
Theoretical Background
Theory of the allocation of time
The study of time allocation stems from Becker’s (1965) seminal work on household production model. Becker proposed the theory of the allocation of time to economically analyze the allocation and efficiency of time between different activities. Each activity is conceptualized as a basis commodity whose price consists of the price of the market goods and of the time. For instance, seeing of a play is considered as a commodity depending on the input of actors, script, theatre and the playgoer’s time. The core assumption of the theory is that households are both producers and consumers who combine market goods and time to produce commodities. All commodities produced by households directly enter the household
production function. As allocating time to different activities, households aim to maximize the utility function of the commodity set, which is subject to a time constraint and a budget constraint.
Following Becker’s household production model, many time allocation studies proposed models for the demand for different activities, such as sleep (Biddle & Hamermesh, 1990), watching television (Corneo, 2005), in-home and out-of-home discretionary activities (Kitamura, 1984; Yamamoto & Kitamura, 1999). To distinguish housework from leisure in nonmarket time domain, prior studies of time allocation formulized the trichotomy of work in the market, work at home (home production), and leisure (Gronau, 1977; Solberg & Wong, 1992; Wales & Woodland, 1977).
Online labor markets blur the traditional dichotomy work-for-pay and nonmarket activity such that Becker’s household production model did not take into account the online market production during non-working time. Furthermore, due to the differences in psychological cognitions (e.g., preference, perception, and expectation) between traditional workplaces and online labor markets, individuals’ behaviors might deviate from the way predicted by economic theories (Mullainathan & H., 2001; Rabin, 1998). The limitations of the theory of the allocation of time inspire us to examine time allocation to micro-task crowdsourcing from both economics lens and psychology lens through psychologizing the propositions of the theory of the allocation of time (Camerer, 1999; Mullainathan, 2005; Rabin, 2002).
Research on time structure
Since people need to have a sense of what lies ahead (Hagestad, 1986), the need for structure is inherent in the nature of humans (Kelvin, 1981). A person is prone to experiencing boredom when he or she attends to the passage of time itself (Grubb, 1975; James, 1890). Temporal structuring allows people to plan, organize, segregate, prioritize and to balance activities so that it influences all human behavior and attitudes by setting up background expectancies of a normal life (Zerubavel, 1981, 1985). Previous studies suggested that filling one’s time with interesting activities may reduce the perception of boredom (Csikszentmihalyi, 1975; Vodanovich & Watt, 1999; Watt & Blanchard, 1994). For an unemployed worker, the ability to structure his or her time is even the best single predictor of mental health (Hepworth, 1980). However, individuals always have difficulty with occupying their time in a structured way. Prior qualitative research found that people’s spare time is characterized by a lack of structure (Grossin & Hantrais, 1986; Jahoda, 1988; Kay, 1989). Findings of time-use surveys conducted in multiple countries also showed that people had time on their hands that they did not know what to do with (Baxter, 2011; Frederick, 1995; Robinson & Godbey, 1999). Due to the inherent desire to structure each block of time during each day, people’s difficulty with filling in discretionary time brings about disorientation, feelings of helplessness, and problems with mental health (Kielhofner, 1977). As a result, watching television becomes a universal coping mechanism for the use of unstructured time (Csikszentmihalyi & Kubey, 1981; R. Kubey & Csikszentmihalyi, 1990; R. W. Kubey, 1986; Robinson, 1981; Shaw, Kleiber, & Caldwell, 1995). To examine the level in which individuals use their time in a structured way, Feather and Bond (1983) proposed the concept of time structure, which is defined as the degree to which individuals perceive their use of time to be structured and purposive. To operationalize the construct of time structure, Bond and Feather (1988) developed an instrument, Time Structure Questionnaire (TSQ), to measure an individual’s sense of time structure. Prior research has investigated the positive effect of time structure on individual’s psychological and mental status, such as self-esteem, and mental health (Bond & Feather, 1988; Hepworth, 1980), work and life satisfaction (George, 1991), as well as individual behaviors, such as Type A behavior (Bond & Feather, 1988; Mudrack, 1999).
Research Model and Hypotheses Development
Key constructs are briefly described in Table 1 and the research model is shown in Figure 1. We include both behavioral intentions (intent to continue and intent to increase) and economic expectation (expected wage) as dependent variables. Intent to continue, which is adopted from IS continuance studies (Bhattacherjee, 2001; Bhattacherjee & Premkumar, 2004), focuses on the extent to which participants intend to continue completing tasks on micro-task crowdsourcing platforms. Intent to increase further examines the extent to which participations would intend to increase their levels of participation in the
future. Considering the sensibility of inquiring participants’ actual crowdsourcing earnings, we use expected wage as an approximate indicator of economic outcome.
Table 1 Construct Interpretations
Construct Iterpretation
Time structure Extent to which people intend to make fragmented discretionary time structured and purposive
Relative advantage Relative advantage of participation in micro-task crowdsourcing over alternative activities
Reservation wage Wage below which participation in micro-task outsourcing is refused
Expected wage Wage expected to be earned from participation in micro-task crowdsourcing Intent to continue Intent to continue to participate in micro-task crowdsourcing
Intent to increase Intent to increase participation in micro-task crowdsourcing
Table 1. Construct Interpretations
Drawing upon the theory of the allocation of time, we identify relative advantage as the antecedent of intent to continue, and reservation wage as the antecedent of expected wage. We further propose the direct impact of time structure on all three dependent variables. More importantly, we emphasize the moderating effects of time structure on the relationships deduced from the theory of the allocation of time. The following sections discuss the model and corresponding hypotheses in detail.
Figure 1. Research Model
Effect of time allocation on crowdsourcing participation
Although the theory of the allocation of time takes the form of an economic modeling, its claim of utility maximization informs the rationale of time allocation in the non-work domain. On the one hand, workers’ spending time on micro-task crowdsourcing implies its relative advantage to maximize utility. Ideally, any selected activity has an advantage over alternative activities such that replacing the selected activity with an alternative will reduce the total utility. Therefore, the relative advantage of completing task on micro-task crowdsourcing over alternative activities promotes individuals’ intentions of continuous participation.
Hypothesis 1 (H1): Relative advantage positively influences intent to continue.
On the other hand, the price of participating in any activity indicates the economic cost of working in micro-task crowdsourcing. Following the commoditization of activity in the theory of the allocation of time, workers complete micro-tasks on crowdsourcing platforms at the expense of the market goods and of the time as inputs. Furthermore, forgone incomes becomes an opportunity cost, as any time spend on
micro-task crowdsourcing entails losing potential earnings possibly derived from other activities. Taking into account the costs associated with completing tasks on micro-task crowdsourcing platforms, participants probably set up a reservation wage in their minds as an acceptable threshold to compensate the costs (Lippman & McCall, 1976). As economists’ job search models suggest, the optimal policy for the job searcher is to reject job offers below a single critical number, i.e., reservation wage, and to accept those above this critical number (Lancaster & Chesher, 1983; Lippman & McCall, 1976). Once the payment of a task goes below reservation wage, self-selected workers would reject the task in micro-task crowdsourcing as the way they do in the workplace. Considering the reservation wage as a self-defined borderline of earnings on micro-task crowdsourcing platforms, workers’ expected wage in micro-task crowdsourcing is bound to vary with the reservation wage accordingly.
Hypothesis 2 (H2): Reservation wage positively influences expected wage.
Effect of time structure on crowdsourcing participation
The concept of time structure uncovers an understudied reason of participation in micro-task crowdsourcing from a time allocation perspective. Time structure was defined as the extent to which individuals perceive their use of time to be structured and purposive (Bond & Feather, 1988; Feather & Bond, 1983). We conjecture that the rationale of structuring time is applicable to the context of micro-task crowdsourcing by our extension of time structure. On the one hand, compared with the general time use (e.g., lifetime from day to day) examined in the time structure literature, the time spent on micro-task crowdsourcing is more specific and more fine-grained (e.g., several minutes during a coffee break). On the other hand, individuals’ inherent need for temporal structure (Hagestad, 1986; Kelvin, 1981; Zerubavel, 1981, 1985) stimulates us to consider time structure as an antecedent of, rather than a consequence of, workers’ participations in micro-task crowdsourcing. Instead of focusing on post hoc perception of time use, we re-conceptualize time structure as a priori intention to organize small chunks of unoccupied time. Taken together, the construct of time structure proposed in our study anchors the time domain to discretionary time slot at a micro level and focuses on a priori intention of structuring time.
In this study, we propose that structuring small pieces of discretionary time through completing micro-tasks drives workers’ participations in crowdsourcing platforms. A micro-task in a crowdsourcing platform is characterized by small granularity (Hoßfeld et al., 2011). It takes little time and pays a small amount of money. Workers freely select micro-tasks and complete them within several minutes. Such an activity becomes a means for people to fill the fragmented unoccupied time with something meaningful or enjoyable (Ipeirotis, 2008). Small time slots are thus organized in a fruitful way. The concept of time structure captures this unique form of time allocation promoted by micro-task crowdsourcing. Workers thus benefit from allocating small chunks of discretionary time to micro-tasks crowdsourcing by avoiding boredom, creating diversion and even broaden knowledge.
As a driver of participation in micro-task crowdsourcing, time structure can enhance workers’ intentions of continuous participation, and can even increase their levels of participation in the future. Furthermore, the non-monetary benefit accrued from making full use of fragmented time, to some extent, compensates for financial incentives such that expected wage would go down as a compromise. Therefore, we hypothesize that time structure has a positive effect on both intent to continue and intent to increase, but has a negative impact on expected wage.
Hypothesis 3a (H3a): Time structure positively influences intent to continue. Hypothesis 3b (H3b): Time structure positively influences intent to increase. Hypothesis 3c (H3c): Time structure negatively influences expected wage.
Proximal antecedents of intent to increase
Among the three outcome variables in the research model, we differentiate according to the purpose of further explaining individuals’ intent to increase. Apart from investigating whether individuals would continue to participate in micro-task crowdsourcing, this study is also interested in the trend of their participation. Even though individuals intend to continue participating in micro-task crowdsourcing, they might change their level of participation either positively (increase) or negatively (decrease). When individuals have greater intentions of continuous participations, they would probably increase their
participation in the future. Therefore, the extent to which individuals intend to continue participating in micro-task crowdsourcing positively influences their intentions of increasing participation levels.
Hypothesis 4 (H4): Intent to continue positively influences intent to increase.
From an economic perspective, participants’ expected wage prevents them from increasing their participation levels given the negligible payments offered by micro-task providers. For participants who establish a higher level of expected wage, the relatively low payment makes the gap between expected and actual wage much more significant. In that case, participating in micro-task crowdsourcing is not as worthwhile as individuals initially expect, thus making an increase in their level of future participation less likely.
Hypothesis 5 (H5): Expected wage positively influences intent to increase.
Moderating effects of time structure
Apart from the main effects of time structure on individuals’ participations in micro-task crowdsourcing, time structure is further considered as a shaper to moderate the strengths of relationships proposed in previous sections. Time structure distinguishes individuals who intentionally plan out discretionary time slots from those who have no awareness of making use of fragmented unoccupied time. For participants with stronger intentions of using idle time in a structured way, intentions of structuring time make relative advantage a less notable driver for their intentions of continuous participation. Thus, the effect of regarding relative on intent to continue would be weaker when time structure is at a higher level.
Hypothesis 6a (H6a): Time structure weakens the positive relationship between relative advantage and intent to continue.
A high level of time structure indicates that participants are more likely to recognize the non-monetary benefit gained from structuring time through completing micro-task on crowdsourcing platforms. Since micro-task crowdsourcing caters for participants who seriously intend to structure their small chunks of discretionary time, economic costs of micro-task crowdsourcing participation (reflected by reservation wage) would no longer be the sole consideration for expected financial rewards (reflected by expected wage). The convenience and efficiency of structuring micro-task via crowdsourcing will compensate for monetary rewards, thus lowering the predictive power of reservation wage for expected wage. Therefore, due to the additional reward induced from structuring time, the extent to which participants’ expected wage is benchmarked by their reservation wage would be attenuated.
Hypothesis 6b (H6b): Time structure weakens the positive relationship between reservation wage and expected wage.
In previous sections, we proposed the relationships among the three outcome variables (i.e., intent to continue, expected wage and intent to increase). High level of intent to continue indicates individuals’ increasing the level of participation, whereas high level of expected wage implies their reduction of participation level in the future. Intent to continue and expected wage act as a positive and a negative predictor of intent to increase respectively. Bearing on the direct effect of time structure on intent to increase, we further investigate the interaction effects of time structure, intent to continue and expected wage on intent to increase.
With the inclusion of time structure as an indicator of intent to increase, we envision that time structure would overweigh the impacts of intent to continue and expected wage. Workers of micro-tasks crowdsourcing always experience a struggle between the desire for making money and the discouraging pay rate. This struggle might become a barrier to enhancing future participation level, regardless of high level of intent to continue or low level of expected wage. However, when workers are more oriented toward time structure, they are probably keen on utilizing their fragmented spare time, which otherwise might be wasted instead of getting paid through completing simple tasks. Workers are able to reconcile themselves to small payments, such that the struggle between earning money and low pay rate can be relieved.
Thereby, the effects of intent to continue and expected wage on future change in participation level vary with the extent to which workers intend to structure time. Intent to increase would more rely on their intentions of structuring small amounts of time, in contrast to their intentions of continuous participation
or expected wage. As workers are more willing to structure small amounts of time, the predictive power of intent to continue and expected wage for intent to increase would decline.
Hypothesis 6c (H6c): Time structure weakens the positive relationship between intent to continue and intent to increase.
Hypothesis 6d (H6d): Time structure weakens the negative relationship between expected wage and intent to increase.
Research Design
Data collection
To test the proposed research model, we plan to conduct an online survey in a micro-task crowdsourcing platform, Amazon’s Mechanical Turk (AMT, https://www.mturk.com/mturk/), at the individual level. AMT is a highly popular crowdsourcing marketplace, introduced by Amazon in 2005, and today offering approximately 300,000 tasks at any point in time. In AMT, employers are known as requesters and post tasks, which are called HITs (human intelligence tasks). These HITs are generally simple tasks requiring human intelligence, such as tagging objects found in an image, document relevance assessment, or podcast transcription. Online participants, referred to as workers, freely choose from available HITs and complete them in exchange for a small payment (Ipeirotis, 2010a). This crowdsourcing marketplace has been analyzed in several studies (Ipeirotis, 2008, 2010a; Kaufmann et al., 2011; Ross et al., 2010). Given the growing popularity of surveys and experiments conducted on crowdsourcing platforms, a substantial number of studies have sought to evaluate the quality of data collected through crowdsourcing platforms. Their findings have provided empirical support for the validity and reliability of online labor portals for scientific research (Behrend, Sharek, Meade, & Wiebe, 2011; Buhrmester, Kwang, & Gosling, 2011; Mason & Suri, 2011; Paolacci, Chandler, & Ipeirotis, 2010; Rand, 2012). For this study, we chose AMT’s workers as the target sample and distributed the survey as a HIT on AMT.
Measurements
The research model consists of six variables: time structure, relative advantage, reservation wage, expected wage, intent to continue, and intent to increase (see Table 2). Specifically, relative advantage employed the instrument established in prior studies (Moore & Benbasat, 1991), adjusting the context from IT innovation to micro-task crowdsourcing participation. Intent to continue was measured by items widely used in IS continuance studies (Bhattacherjee, 2001; Bhattacherjee & Premkumar, 2004). Measurements of reservation wage and expected wage derived from the definitions of these two types of wage in a form self-report numbers. Items for time structure was self-developed based on its definition proposed in this study as well as the existing time structure instrument TSQ (Bond & Feather, 1988; Feather & Bond, 1983). Intent to increase would measured by an agreement scale statement, as well as a continuous slider labeled from “nonexistent”, “decreased”, “kept the same”, “increased”, to “vastly increased”.
Table 2 Construct Measurements
Construct Item
Time structure (Bond & Feather, 1988)
TMS1 I prefer to structure my small pieces of spare time intentionally.
TMS2 It is against my intention to leave small free time amounts unorganized. TMS3 I seek to organize my small chunks of unobligated time.
TMS4 I intentionally make the fragmented discretionary time available to me more structured. Relative advantage (Moore & Benbasat, 1991)
RAD1 Compared with other things I could do at the moment, completing HIT(s) on Mechanical Turk benefits me more.
RAD2 I can find other things to do at the moment, but completing HIT(s) on Mechanical Turk is more worthwhile.
RAD4 There are other things I could do at the moment, but completing HIT(s) on Mechanical Turk has advantages over them. Reservation wage (Lippman & McCall, 1976) RWG1
Considering the low-pay HITs you recently completed and were paid for on Mechanical Turk, what was the payment rate at which you regretted completing a HIT ($ per hour)?
RWG2 For all the HITs you have seen on Mechanical Turk, what is the payment rate below which you would never consider accepting a HIT ($ per hour)?
Expected wage EWG1 How much do you expect to earn (in US$) on Mechanical Turk ($ per HIT / per hour)?
Intent to continue (Bhattacherjee,
2001)
ITC1 I intend to continue participating in Mechanical Turk rather than discontinue the participation. ITC2 My intention is to continue participating in Mechanical Turk. ITC3 I will continue to participate in Mechanical Turk in the future. ITC4 I plan to continue to participate in Mechanical Turk in the future. Intent to
increase
ITI1 Looking forward into the future, I will be completing HITs on Mechanical Turk at an even higher participation level.
ITI2
Envisioning myself in the future, my participation in Mechanical Turk will be (move the slider accordingly)
nonexistent / decreased / kept the same / increased / vastly increased
Table 2. Construct Measurements
Finally, to rule out the alternative explanations, we also included several demographical variables as control variables, including age, education, employment status, gender, tenure, and current level of participation.
Expected Contributions
This study expects to make several contributions to theory. First, the time allocation perspective introduced by our study contributes to the existing theoretical lenses for explaining the recent surge of individuals’ participations in poorly paid online labor markets. Breaking through the dominance of motivational perspective in the literature (Brabham, 2008b, 2010; Hossain, 2012; Ipeirotis, 2008; Kaufmann et al., 2011; Schulze, Seedorf, Geiger, & Kaufmann, 2011; Zheng et al., 2011; Zwass, 2010), we employed the time allocation perspective (Becker, 1965; Gronau, 1977; Solberg & Wong, 1992), as an economic lens, to tease out the temporal rationale for continuous participation in micro-task crowdsourcing. Second, our study contributes to the theory of the allocation of time by extending its applicability to the domain of online labor markets. The inclusion of time structure into the economic theory (time allocation theory) further clarifies the boundary of the theory by shaping the relationships deduced from the theory. Third, our study advances the generalizability of time structure by re-conceptualizing it from a macro level to a micro level and from post hoc perception to a priori intention. This study is also expected to inform task providers on the importance of task granularity in enabling task workers’ time allocation and structuring. Micro-task crowdsourcing dramatically changes the granularity and organization of work in the Internet by splitting jobs into short, low-pay micro-tasks which can be completed rapidly by the human cloud (Hoßfeld et al., 2011). Instead of simply posting a complex task at a macro level, decomposing and redesigning the task into a fine-grained micro level is a promising approach to facilitating task completion.
Acknowledgement
This research has been supported in part by GRF project No. CityU 194613 awarded by the HK SAR Research Grants Council to Christian Wagner.
References
Baxter, J. (2011). An examination of the characteristics and time use of those who have unfilled spare time. Electronic International Journal of Time Use Research,, 8(1), 30-61.
Becker, G. S. (1965). A theory of the allocation of time. The Economic Journal, 75(299), 493-517.
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behav Res Methods, 43(3), 800-813. doi: 10.3758/s13428-011-0081-0
Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation model. MIS Quarterly, 25(3), 351–370.
Bhattacherjee, A., & Premkumar, G. (2004). Understanding changes in belief and attitude toward information technology usage: A theoretical model and longitudinal test. MIS Quarterly, 28(2), 229–254.
Biddle, J. E., & Hamermesh, D. S. (1990). Sleep and the allocation of time. Journal of Political Economy, 98(5), 922-943.
Bond, M. J., & Feather, N. T. (1988). Some correlates of structure and purpose in the use of time. Journal of Personality and Social Psychology, 55(2), 321–329.
Brabham, D. C. (2008a). Crowdsourcing as a model for problem solving: An introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75-90. Brabham, D. C. (2008b). Moving the crowd at iStockphoto: The composition of the crowd and motivations for participation in crowdsourcing applications. First Monday, 13(6),
http://firstmonday.org/ojs/index.php/fm/article/view/2159/1969.
Brabham, D. C. (2010). Moving the crowd at Threadless: Motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8), 1122-1145.
Brabham, D. C. (2012). Crowdsourcing: A model for leveraging online communities. In A. Delwiche & J. Henderson (Eds.), The Participatory Cultures Handbook. New York: Routledge.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3-5. doi: 10.1177/1745691610393980
Camerer, C. F. (1999). Behavioral economics: Reunifying psychology and economics. Proceedings of the National Academy of Sciences, 96(19), 10575-10577.
Corneo, G. (2005). Work and Television. European Journal of Political Economy, 21(1), 99-113.
Csikszentmihalyi, M. (1975). Beyond Boredom and Anxiety: Experiencing Flow in Work and Play. San Francisco, CA: Jossey-Bass.
Csikszentmihalyi, M., & Kubey, R. (1981). Television and the rest of life: A systematic comparison of subjective experience. Public Opinion Quarterly, 45(3), 317-328.
Feather, N. T., & Bond, M. J. (1983). Time structure and purposeful activity among employed and unemployed university graduates. Journal of Occupational Psychology, 56(3), 241–254.
Frederick, J. A. (1995). As Time Goes by - Time Use of Canadians. Ottawa: Statistics Canada.
George, J. M. (1991). Time structure and purpose as a mediator of work-life linkages. Journal of Applied Social Psychology, 21(4), 296-314.
Gronau, R. (1977). Leisure, home production, and work-the theory of the allocation of time revisited. Journal of Political Economy, 85(6), 1099-1123.
Grossin, W., & Hantrais, L. (1986). The relationship between work time and free time and the meaning of retirement. Leisure Studies, 5(1), 91-101.
Grubb, E. A. (1975). Assembly line boredom and individual differences in recreational participation. Journal of Leisure Research, 7(4), 256-269.
Hagestad, G. O. (1986). Dimensions of time and the family. American Behavioral Scientist, 29(6), 679-694.
Hepworth, S. J. (1980). Moderating factors of the psychological impact of unemployment. Journal of Occupational Psychology, 53(2), 139-146.
Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2011). Anatomy of a crowdsourcing platform - Using the example of Microworkers.com. Paper presented at the International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), Seoul, Korea.
Hossain, M. (2012, May 21-22). Crowdsourcing: Activities, incentives and users’ motivations to participate. Paper presented at the International Conference on Innovation, Management and Technology Research, Malacca, Malaysia.
Hoßfeld, T., Hirth, M., & Tran-Gia, P. (2011). Modeling of crowdsourcing platforms and granularity of work organization in future internet. Paper presented at the 23rd International Teletraffic Congress, San Francisco, CA.
Howe, J. (2006a). Crowdsourcing: A definition. Retrieved June 11, 2013, from
http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html
Howe, J. (2006b). The rise of crowdsourcing. Wired Magazine. Retrieved June 11, 2013, from
http://www.wired.com/wired/archive/14.06/crowds.html
Ipeirotis, P. G. (2008). Why people participate on Mechanical Turk. Retrieved June 11, 2013, from
http://www.behind-the-enemy-lines.com/2008/09/why-people-participate-on-mechanical.html
Ipeirotis, P. G. (2010a). Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2), 16-21.
Ipeirotis, P. G. (2010b). Demographics of Mechanical Turk. Retrieved June 13, 2013, from
http://archive.nyu.edu/bitstream/2451/29585/2/CeDER-10-01.pdf
Jahoda, M. (1988). Time: A Social Psychological Perspective. In M. Young & T. Schuller (Eds.), The Rhythms of Society (pp. 154-172). London: Routledge.
James, W. (1890). Principles of psychology. New York, NY: Holt.
Kaufmann, N., Schulze, T., & Veit, D. (2011, August 4-8). More than fun and money: Worker motivation in crowdsourcing – A study on Mechanical Turk. Paper presented at the Americas Conference on Information Systems, Detroit, USA.
Kay, T. (1989). Active unemployment - A leisure pattern for the future? Loisir et Société / Society and Leisure, 12(2), 413-430.
Kelvin, P. (1981). Work as a source of identity: The implications of unemployment. British Journal of Guidance & Counselling, 9(1), 2-11.
Kielhofner, G. (1977). Temporal adaptation: A conceptual framework for occupational therapy. American Journal of Occupational Therapy, 31(4), 235-242.
Kitamura, R. (1984). A model of daily time allocation to discretionary out-of-home activities and trips,. Transportation Research Part B: Methodological, 18(3), 255-266.
Kittur, A., Chi, E. H., & Suh, B. (2008, April 5-10). Crowdsourcing user studies with Mechanical Turk. Paper presented at the ACM CHI Conference on Human Factors in Computing System, Florence, Italy.
Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. (2011, October 16-19). Crowdforge: crowdsourcing complex work. Paper presented at the ACM symposium on User interface software and technology, Santa Barbara, USA.
Kubey, R., & Csikszentmihalyi, M. (1990). Television and the Quality of Life: How Viewing Shapes Everyday Experience. Hillsdale, NJ: Lawrence Erlbaum Associates.
Kubey, R. W. (1986). Television use in everyday life: Coping with unstructured time. Journal of Communication, 36(3), 108-123.
Lancaster, T., & Chesher, A. (1983). An econometric analysis of reservation wages. Econometrica, 51(6), 1661-1676.
Lippman, S. A., & McCall, J. J. (1976). The economics of job search: A survey. Economic Inquiry, 14(2), 155-189.
Mason, W., & Suri, S. (2011). Conducting behavioral research on Amazon’s Mechanical Turk. Behav Res Methods, 43(1), 1–23.
Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of adopting aninformation technology innovation. Information Systems Research, 2(3), 192-222.
Mudrack, P. E. (1999). Time structure and purpose, type A behaviour, and the Protestant work ethic. Journal of Organisational Behaviour, 20(2), 145-158.
Mullainathan, S. (2005). Development economics through the lens of psychology. In F. Bourguignon & P. B. (Eds.), Lessons of experience: Annual World Bank Conference on Development Economics (pp. 45-70). New York, NY: Oxford University Press.
Mullainathan, S., & H., T. R. (2001). Behavioral economics. Paper presented at the NBER Working Paper 7948.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411-419.
Quinn, A. J., & Bederson, B. B. (2011, May 7–12). Human computation: A survey and taxonomy of a growing field. Paper presented at the ACM CHI Conference on Human Factors in Computing Systems, Vancouver, Canada.
Rabin, M. (1998). Psychology and economics. Journal of Economic Literature, 36(1), 11-46.
Rabin, M. (2002). A Perspective on Psychology and Economics. Paper presented at the U.C. Berkeley Working Paper E02-313.
Rand, D. G. (2012). The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. J Theor Biol, 299, 172-179. doi: 10.1016/j.jtbi.2011.03.004
Robinson, J. P. (1981). Television and leisure time: A new scenario. Journal of Communication, 31(1), 120-130.
Robinson, J. P., & Godbey, G. (1999). Time for Life: The Surprising Ways Americans Use Their Time (2nd Ed.). Pennsylvania: The Pennsylvania State University Press.
Ross, J., Irani, I., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010, April 10-15). Who are the Crowdworkers: Shifting Demographics in Mechanical Turk. Paper presented at the ACM CHI Conference on Human Factors in Computing Systems, Atlanta, USA.
Rouse, A. C. (2010). A preliminary taxonomy of crowdsourcing. Paper presented at the Australasian Conference on Information Systems, Brisbane, Australian.
Schulze, T., Seedorf, S., Geiger, D., & Kaufmann, N. (2011, June 9-11). Exploring task properties in crowdsourcing–An empirical study on Mechanical Turk. Paper presented at the European Conference of Information Systems, Helsinki, Finland.
Shaw, S. M., Kleiber, D. A., & Caldwell, L. L. (1995). Leisure and identity formation in male and female adolescents: A preliminary examination. Joumal of Leisure Research, 27(3), 245-263.
Solberg, E. J., & Wong, D. C. (1992). Family time use: Leisure, home production, market work, and work related travel. Journal of Human Resources, 27(3), 485-510.
Vodanovich, S. J., & Watt, J. D. (1999). The relationship between time structure and boredom proneness: An investigation within two cultures. The Journal of Social Psychology, 139(2), 143-152.
Wales, T. J., & Woodland, A. D. (1977). Estimation of the Allocation of Time for Work, Leisure, and Housework. Econometrica, 45(1), 115-132.
Watt, J. D., & Blanchard, M. J. (1994). Boredom proneness and the need for cognition. Journal of Research in Personality, 28(1), 44-51.
Yamamoto, T., & Kitamura, R. (1999). An analysis of time allocation to in-home and out-of-home discretionary activities across working days and non-working days. Transportation, 26(2), 231-250.
Zerubavel, E. (1981). Hidden Rhythms: Schedules and Calendars in Social Life. Berkeley, CA,: University of California Press.
Zerubavel, E. (1985). The Seven Day Circle: The History and Meaning of the Week. Chicago, IL: University Of Chicago Press.
Zheng, H., Li, D., & Hou, W. (2011). Task design, motivation, and participation in crowdsourcing contests. International Journal of Electronic Commerce, 15(4), 57-88.
Zwass, V. (2010). Co-Creation: Toward a taxonomy and an integrated research perspective. International Journal of Electronic Commerce, 15(1), 11-48.
