Amazon Mechanical Turk Overview

Amazon Mechanical Turk (https://requester.mturk.com), a crowdsourcing marketplace, adds value by processing work that cannot be computerized.

Typically, these include large-scale [data] projects - think Amazon.com's 500MM product database - which requires some form of human judgment to help digitize, validate, moderate, tag, cleanse, categorize, dedupe, etc...This is accomplished by breaking down the project into micro-tasks, distributing to qualified cloud-based workers, who then deliver the results.

1. Introducing Amazon Mechanical Turk A Brief Overview of What, Why, and How John Hoskins

2. Agenda What is Mechanical Turk? When would you use it ? How are others using it - Case Studies How – Implementation options

3. What is Mechanical Turk?

4. Mechanical Turk is a M arketplace .

5. Mechanical Turk is a marketplace for work . Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce. Flexibility : Scale your workforce up and down quickly Accuracy : Get high quality, efficient and cost effective results. Price : Pay only when you are satisfied with the results. Speed : Start receiving results in minutes

6. How it works: . “Design & Publish” “Validate, Pay & Go”

7. Workforce 500,000 Workers 190+ Countries Who are the Workers? Workers are global Workers represent numerous skill sets 24X7 Follow the sun availability Managing Your Workforce You can choose who is allowed to work on your assignments Can narrow to US based (i.e. when you need western culture competence) Leverage qualifications to find your best Workers

8. When would you use Mechanical Turk?

9. Business Challenges Our Community is generating 10,000 comments per day – how can we eliminate SPAM? We pull information from hundreds of sources – how do we normalize and cleanse? We have thousands of legacy assets – how do we make them searchable? Our heuristics need jump started – how can I generate thousands of reference data points fast? We need to add a new attribute to our data?

10. Business Challenges Requires Human Judgment: Recognition, Reasoning , Context, Prior Knowledge Cannot be completely solved algorithmically Has unpredictable fluctuations in volume

11. How are businesses using it? Data Management Data Verification Data Entry & Collection Data De-duplication Algorithm Training Content & Media Moderate Photos & Content Content Creation & Editing Transcription Categorization Classification Tagging Keywords Sentiment Analysis Business Services Search Relevancy Data Gathering Enhancement

12. How are others using Mechanical Turk?

13. Search Enhancement Make your assets more discoverable Categorize them into your search tree Add descriptions, attributes and keywords Enhance listing data How it works:

14. Data Handling Resolve Data conflicts directly in your business process Remove duplicates Normalize data entries Resolve conflicts and collisions Enhance entries How it works :

15. How to apply your task to Mechanical Turk?

16. Your Project  Tasks “Working Backwards” What are the business results that you want? How will you use those results? What format do the results need to be in? Which tasks benefits from human intelligence? Break your overall project into smaller “tasks” Consolidate like items and processes Define instructions for each task

17. Example Workflow Content Moderation with API integration

18. The Key components of your HIT Instruction Guidelines for how to do a HIT Data “batch” concept Question(s) What kind of dress is this? Cocktail Sun dress Bridal dress

19. Work Catalog Workers discover available HITs through the listings catalog Name of Requester, Name of HIT, Price and Qualifications are displayed

20. HIT Example Specific instructions on what to do – and how to do it Providing as much information as available

21. Adjudication Strategies Leverage different methods of determining the correct answer by designing for accuracy in your HIT: Multiple Choice Plurality Known Answers (“Gold Standards”) Free-form Multiple HIT Workflow’s where a new Worker “edits” or “grades” previous Worker submissions Creative Tasks Bonus Payments

22. Your Workers Identify and manage your best Workers

23. Balances & Levers Leverage market dynamics to optimize speed, price and accuracy to meet your business needs.

24. Requester access:

25. Self Service Applications Mechanical Turk supports a robust ecosystem of providers that have built or can build solutions to meet your needs.

26. Custom Solution Providers

27. Advantages of Mechanical Turk Converts a fixed cost into a variable cost Significantly reduces ongoing headcount expenses Offers a pay-as-you-go staffing model Lowers Costs Eliminates lead time of interviewing and staffing Work gets done faster, by working in parallel Work continues around the globe, around the clock Reduces work turnaround time Gives you access to an on-demand workforce 100% Efficiency Maintain the flexibility your business demands Use only the capacity you need, when you need it Eliminates Staffing Issues

28. Thank you. John Hoskins [email protected] 206.266.8105 @ amazonmturk Facebook.com/ amazonmturk mechanicalturk.typepad.com