CrowdFlower
Other Microtask
US-based microtasking platform. Largest English-speaking microtask platform (according to 2014 estimates). Offers "full service" and "self service." Founded 2007 as Dolores Labs.
Terms of Service
Rejection:negative rating Change:negative rating Warranty:negative rating Client Contact:positive rating Worker Contact:positive rating
Rejection:negative rating
Change:negative rating
Warranty:negative rating
Client Contact:positive rating
Rejection:positive rating
Platform Overview
Crowdflower was launched in 2007 as a service for clients who wanted help managing large tasks on crowdwork platforms. They now describe themselves as an AI “human-in-the-loop” platform: a platform with machine learning and processing features in which human labor completes tasks that can not yet be automated.
-
Platform History
The firm now called CrowdFlower was founded in 2007 by Lukas Biewald and Chris Van Pelt as “Dolores Labs” in San Francisco. In 2010 it was renamed “CrowdFlower”; as of January 2017 it has around 65 employees. Biewald is CEO and Van Pelt is Chief Technical Officer. After originally serving as an intermediary for clients looking to manage large task batches on Amazon’s Mechanical Turk platform, CrowdFlower expanded into other “channels” and in December 2013 stopped recruiting workers through Mechanical Turk. Companies including Ebay, Twitter, and Microsoft (also a CrowdFlower investor) post tasks that are offered through CrowdFlower’s own platform and through platforms operated by over 40 channel partners (including Clickworker, ClixSense, Crowd Guru, DDD, GetPaidTo, instaGC, and NeoBux) to workers all over the world. These workers are considered self employed according to the CrowdFlower Terms of Service. The tasks offered are “microtasks” including data research tasks, transcription, categorization, text production for product descriptions, etc. [3][4][5]
-
Business Model
Crowdflower charges a monthly fee of $1,500 USD for basic access to their platform (more for large clients), where clients can then pay workers for individual tasks. A “Data for Everyone” plan is available for researchers and students. Clients on this plan pay worker fees plus a percentage to the platform, but no monthly fee.[6]
-
Worker Classification
Workers are self-employed.
-
Jobs and Clients
The tasks offered are “microtasks” including data research tasks, transcription, categorization, text production for product descriptions, etc.
Example Clients
CrowdFlower advertises the following clients [7]:
- ebay
- intuit
- Autodesk
- Mozilla
-
Work Process
Customers register on the CrowdFlower site; workers may register directly on the CrowdFlower site or through channel partners. Tasks are distributed to workers as appropriate based on CrowdFlower’s collected information about workers’ abilities and qualifications. Quality of the completed work is measured and recorded, and used in allocation of future tasks. CrowdFlower takes a percentage of the reward paid to the worker for each task. According to CEO Biewald, hourly wages average between USD 2 and USD 3 per hour. Workers are self employed, not employees of CrowdFlower. [8][9][10][11]
-
This information was collected from 25-100 verified workers on the platform in 2016 and 2017. More information
-
Pay
Hourly Wages MINIMUM: €0.51 MAXIMUM: €15.00 AVERAGE: €2.93 MEDIAN: €1.57 Payment Overview
With a median of €1.57 and an average of €2.93, effective hourly wages on CrowdFlower were some of the lowest in our survey.
Nonpayment, however, was much less common than other platforms, with only ten percent of respondents reporting any experience with nonpayment, and 100% of these workers saying it almost never happened.
Hourly Wages
Although a small handfull of workers reported receiving higher wages — as much as €15 per hour — well over half of respondents (58%) were making less than €2.50 per hour.
However, like Prolific workers, CrowdFlower workers were geographically diverse. Many survey respondents lived in countries where these wages — very low by German standards — would be more reasonable:
Crowdflower is truthworthy platform and I am able earn more money then in regular job for same hours.
I work at CF because in my country (venezuela) is really hard to find jobs right now, we are struggling, and CF allows me an somewhat easy way to get money so can help my family with food
Hourly Wage Distribution
Percentage of survey respondents whose hourly wages fell into each category
CrowdFlower specializes in repetitive micro-tasks, with a focus on data processing. Its advertised use cases include tasks such as image categorization. The system is set up so that a company might upload thousands (or more) images; and each individual worker might work through a large batch of data points within a single project, paid per individual task.
This means that the payment for each individual task — e.g., categorizing a single image — is very very small, and a worker can only accumulate significant payment if they can find a large batch of data that they can work through very quickly. As one worker notes, it can be very challenging to actually build up to a livable hourly wage in 1 cent increments:
Since many jobs only pay 1 cent, it is quite difficult to reach an average hourly wage.
As with other platforms, workers spent a significant amount of time looking for tasks — on average workers reported spending about 45 minutes looking for work for every 1 hour spent actively working. If this time were taken into account, hourly wages on the platform would be even lower.
Nonpayment experiences
10.64% of survey respondents have experienced non-payment at least once
Only 10% of survey respondents reported experiencing non-payment, and all of those reported experiencing it only once or twice, suggesting that nonpayment is not a significant issue on CrowdFlower.
Frequency of Nonpayment Experiences
However, when conducting the survey, we were surprised to find that while we received 47 answers on our external survey platform, only 40 workers had completed the task according to the CrowdFlower site. We do not know what went wrong – if it was worker error in not completing the task on the CrowdFlower side, or an error in the CrowdFlower site. Running a survey like this is not what the site was designed for and requires some manual customization of the normal task setup. Nonetheless, we found it very concerning that such a high rate of task completions were not shown as complete (and thus weren’t paid) in the CrowdFlower interface. We did not have this issue on any other platform.
-
Communication
Workers’ responses to our survey raise questions about the quality of communication with CrowdFlower representatives, and draw attention to the lack of features for client communication.
Communication with other workers was rated highly, though it all ocurred off-site — either on social media groups or via official platforms on CrowdFlower partner websites.
Communicating with management
Although there is a built in ticketing system for support requests, many workers complained that platform support was slow to respond to these tickets. Only about 46% of respondents said that platform operators were prompt with responses to their inquiries.
Sometimes the support takes a while to respond.
Workers’ of course had different opinions as to what “prompt” meant, with some workers describing their experiences with a 1-2 day turnaround time on tickets as “great” and many others reporting that the platform could improve in this area.
More troubling, only 62% of workers reported that platform responeses to their tickets were helpful more than half the time.
Percentage of respondents who said that communication with management is...
Communicating with other workers
There are three official social media communities for CrowdFlower:
- Twitter: www.twitter.com/cfcomm
- Facebook: www.facebook.com/crowdflowercommunity
- Tumblr: crowdflowercommunity.tumblr.com
Percentage of respondents who said that communication with other workers through the official platform is...
CrowdFlower also integrates with other online work sites such as NeoBux — a “pay to click” site where workers are paid to click on advertisements and browse advertiser websites. Many workers reported using the official worker forums on NeoBux as well as ClixSense (clixsense.com/).
Workers also used other unofficial media for communicating with eachother, including Facebook groups, WhatsApp groups, twitter, Google+, and Skype Conferences, as well as a few other forum sites such as the “Money Maker Group.”
-
Evaluation
Workers are evaluated on CrowdFlower in several different ways — by clients, by other workers, and by the platform itself.
Percentage of respondents who said that evaluations of workers are...
Individual customers/clients have the power to accept or reject any submission by a CrowdFlower worker, as well as to “flag” the account of workers in such a way as to prevent workers from receiving future work.
In addition, most tasks are set up such that multiple workers each repeat a task — or “judgement” — for any given data point. Agreement among each worker about the correct answer is then used to automatically judge accuracy/correctness.
Finally, CrowdFlower offers workers a series of “badges” that can be earned for completing a certain number of tasks while also maintaining a certain accuracy/approval rating. Badges are then used to qualify for future jobs. The platform also has certain skill assessments that can be used to qualify for future jobs.
Only 62% of respondents felt that these kinds of evaluations were fair more than half the time. Many respondents had a story to tell about a time when they had had at least one unfair evaluation.
When workers had a task — or “judgement” — rejected, they did not always know why it was wrong, nor what the ‘right’ answer would have been.
In some tasks you got expelled but no corrections marked so you do not even know where you made a mistake
Also, several workers were frustrated with the ability of clients to “flag” their accounts without specifying a reason.
Task authors tend to flag us for no reason and it takes a lot of time to get that flag deleted.
Sometimes workers were successful at getting unfair and incorrect evaluations overturned through the CrowdFlower support ticket system. However, even when they were successful it could take several days or more to go through the process, during which time their work on the platform was either constrained or not allowed at all.
-
Tasks
Workers on CrowdFlower had mixed opinions about the quality of tasks on the platform. Most workers reported that tasks were meaningful, interesting, fun, and satisfying most of the time. However, a few workers workers also reported that tasks were dangerous or psychologically harmful most of the time. And nearly a quarter of participants reported that the work was ethically questionable more than half the time.
Positive features of tasks on CrowdFlower
Most workers were fairly happy with the tasks on CrowdFlower, and several workers mentioned in comments that they found the work interesting and fun.
A few workers especially liked the content of some of the tasks they did on CrowdFlower, such as viewing photos on Pinterest.
How often is the work ...
... meaningful?
... interesting?
... fun?
... satisfying?
Negative features of tasks on CrowdFlower
Thirty percent of respondents reported that tasks on CrowdFlower are ethically questionable about half the time or more. However, comments in the survey do not point to specific examples of ethically questionable tasks.
Almost 18 percent of respondents said that tasks on the platform were psychologically harmful about half the time or more. While comments in the survey do not give details about this aspect of the work, content moderation is the first use case that CrowdFlower advertises on its website[12]. As has been documented elsewhere, content moderation work can be psychologically challenging and harmful for workers.[13]
How often is the work ...
... physically dangerous or harmful?
... demeaning or psychologically harmful?
... ethically questionable?
-
Technology
Workers were generally satisfied with the CrowdFlower technology, although there were some concerns with reliability, in particular.
Percentage of respondents who said the technology is...
Reliability
Workers in our survey reported issues with site reliability which ranged from periods of slowness, to times when the site was entirely frozen, and not updating worker stats (such as a worker’s approval rating, which is an important qualifier for some jobs).
Sometimes the dashboard freezes and won’t update our stats. But other than that yes.
There are some tasks which depend on pictures and there are days when it is impposible to open them.
their website is usually fast, sometimes the servers overload but it doesnt happen often, maybe from time to time, but is great in general
In addition, some workers reported difficulties with the browser plugins required for some tasks.
The site is reliable, but the browser extension used for some specific tasks often has problems.
-
Things Workers Like
A Place to Learn New Skills
great opportunities to earn extra money ..and learning opportunities as well..
Many survey respondents liked CrowdFlower because it was a place to learn new things, and especially to improve their English language skills while also making some money.
What I like most is to make money while improving my English knowledge.
I like very very very much to work in CF because I earn money and help to improve my level in English.
Variability of Tasks & Task Content
Some workers commented that they liked the variability of tasks on the platform. Unlike survey sites, there are a variety of different things to work on on CrowdFlower.
The tasks are very varied, and some pay very well.
Others liked that they could work on tasks with certain kinds of content, for example, working on image categorization tasks with images that they thought were interesting or fun to look at.
Flexibility
Like other platforms, CrowdFlower offers workers flexibility. Respondents liked being able to work from home, as well as having control over their schedule.
The versatility of it. I can work whenever I like to, or when I have time.
I can work from home whenever I choose to.
It allows me to generate an extra income with my free time, without rigid schedules and with compensations.
Because of this flexibility, part-time CrowdFlower workers were able to convert what would have been leisure time into “something useful” for the client, and income-producing for themselves:
It’s a great opportunity to use my leisure time in something useful for someone and that would bring me a profit.
High Payment
Although wages on the platform are low by German standards, for many respondents wages on CrowdFlower were far higher than their own local standards:
I love that I can work from my house without a fixed schedule and that the economic income is very good.
The gloabal nature of the platform makes it possible for workers in low-wage countries to work across traditional geographic-economic boundaries without even leaving their own house. This makes the internet-based micro-task site an effective platform for a kind of micro-arbitrage to the benefit of workers living in low-wage areas and, of course, also to the benefit customers based in areas with higher local labor costs.
Prompt Payment
In general, respondents were happy with how quickly they were paid for their work on CrowdFlower.
Always paid fast, faster than other sites.
Jobs are paid very fast after it is completed.
Workers who had experienced problems getting paid on other (unspecified) sites, found CrowdFlower to be prompt, by contrast.
Contributing to Large Companies’ Data Quality
A couple of workers mentioned that they liked feeling like their work was contributing to something bigger. Much like Prolific workers liked contributing to science by taking surveys, some CrowdFlower workers liked knowing that their work was having an impact improving the data and algorithms that large companies use — and that, in turn, impact many people’s lives:
Most of my motivation is due to the income. But knowing that my work is helping to improve data from large companies is really cool.
CrowdFlower Community
Finally, some workers mentioned liking the community of CrowdFlower workers.
I can work from home whenever I choose to. Extra income. Friendly people that do CrowdFlower tasks with me.
They liked participating in the “very nice Facebook community” as well as chats on WhatsApp, Skype, and other media.
-
Worker Concerns
Low Pay
The rate of pay on CrowdFlower came up as both a positive and negative in our survey, relative to individual workers’ place of residence and experience on the platform.
Workers in countries with higher standard wages and higher costs of living believed that pay on the platform was “really low” and that for some of the tasks that they did, workers “deserve more.”
With a median of only €1.57 / hour, wages on the platform were the lowest of any platform that we have surveyed.
Unfair Flags & Rejections
As mentioned in the Evaluation section above, many workers had experienced at least a few unfair evaluations of their work on the platform. In particular, several respondents commented about their account being unfairly “flagged” by a particular customer, which then limited their ability to get future work.
The major problem all testers on CrowdFlower have is the power that Customers have. If one customer doesn’t like your work, he has the power to give you a Flag (punishment) that remove all your badges (you can reach up to level 3 badge) and you won’t work in any jobs that requires badges (level 1, 2 or 3 badges) anymore.
Sometimes very slow staff. Task authors are very abusive with flags when you didn’t even do any mistakes and it takes forever to get flags removed.
There are less tasks to work on than when I first started. Some of the client tasks/test questions are graded unfairly and decline further work.
Waiting for Jobs, Not Enough Work
As noted above, workers in our survey spent about 45 minutes looking for work for every 1 hour spent actively working. Like workers on many other platforms, several CrowdFlower workers reported frustrations with this situation, and would like it if there was more work available on a consistent basis.
-
Ability to Refuse Payment Rejection:negative rating
Nonpayment / rejection is allowed; no recourse for workers. -
Change to Terms of Service: Change:negative rating
Can be changed unilaterally at any time. -
Warranty Warranty:negative rating
No possibility to revise rejected work. -
Contact with Employers Client Contact:positive rating
No prohibition. -
Contact with Workers Rejection:positive rating
No prohibition.
-
Notes
- CrunchBase. 2016. CrowdFlower, Inc. Funding Rounds. Accessed 17 Jan 2017. https://www.crunchbase.com/organization/crowdflower/funding-rounds
- Harris, Derrick. 2014. CrowdFlower raises $12.5M to deliver better data for better models. GigaOm, 17 Sep 2014. https://gigaom.com/2014/09/17/crowdflower-raises-12-5m-to-deliver-better-data-for-better-models
- CrowdFlower Master Terms of Service. https://www.crowdflower.com/legal/. Accessed 17 Jan 2017, last modified 13 Oct 2015.
- Rothenberger, Jan. 2014. Mikrojobs im Selbstversuch. Tagesanzeiger, 14 May 2014. http://www.tagesanzeiger.ch/digital/internet/Mikrojobs-im-Selbstversuch/story/28585542.
- Schischka, Benjmain. 2010. 6 seltsame Wege für Geld verdienen im Internet. Freenet, 22 Jun 2010. http://www.freenet.de/digitalewelt/onlinesicherheit/6-seltsame-wege-fuer-geld-verdienen-im-internet_1746724_4734620.html
- CrowdFlower Pricing. https://www.crowdflower.com/pricing/. Accessed 11 Apr 2017, last modified 13 Oct 2015.
- Crowdflower Success Stories. Accessed 17 Apr 2016.
- Harris, Mark. 2016. Say goodbye to your highly skilled job. It’s now a “human intelligence task.” Backchannel, 6 Jun 2016. https://backchannel.com/its-not-just-robots-skilled-jobs-are-going-to-meatware-954adf770710.
- Karpie, Andrew. 2016. CrowdFlower Receives $10M to Advance New AI/Machine Learning Direction. Spend Matters, 10 Jun 2016. http://spendmatters.com/2016/06/10/crowdflower-receives-10m-to-advance-new-aimachine-learning-direction.
- Rothenberger, Jan. 2014. Mikrojobs im Selbstversuch. Tagesanzeiger, 14 May 2014. http://www.tagesanzeiger.ch/digital/internet/Mikrojobs-im-Selbstversuch/story/28585542.
- Gutierrez, Daniel. 2016. Human-in-the-loop is the future of machine learning. Inside Big Data, 11 Jan 2016. http://insidebigdata.com/2016/01/11/human-in-the-loop-is-the-future-of-machine-learning.
- Use Cases | CrowdFlower, https://www.crowdflower.com/use-cases/, accessed 2017-05-24.
- Sarah T. Roberts. 2014. Behind the Screen: the Hidden Digital Labor of Commercial Content Moderation. Ph.D. dissertation, University of Illinois at Urbana-Champaign. Abstract accessed 3 May 2017.
