Post A Survey On Mechanical Turk And Watch The Results Roll In | WAMU 88.5 - American University Radio

Post A Survey On Mechanical Turk And Watch The Results Roll In

Play associated audio

You can buy just about anything on — clothes, books, electronics. You can buy answers, too. College students and professors are doing all sorts of research on an Amazon site called Mechanical Turk.

Need 200 smokers for your survey on lung cancer? Have a moral dilemma to pose for your paper on Kierkegaard? Now researchers can log in, offer a few pennies in payment and watch the data roll in.

When you type in the Web address, you'll see a page full of gray rectangles, each one a different task. When I visited the site recently, I see one out of UCLA. They're paying 25 cents for a 5-minute survey on consumer decision-making. The next one is 10 cents and it's about dating preferences. When I scroll down, there's one from Ohio State paying 50 cents for workers to fill out a survey about their political beliefs.

'It's Very Fast. It's Very Cheap'

"It's very efficient. It's very fast. It's very cheap," says Adam Berinsky, a professor at MIT who studies public opinion. Some of those gray rectangles are his studies and surveys.

"There's been a huge explosion. So about four years ago, not a lot of folks were doing it. But now everyone I talk to at conferences is doing that," he says.

It all started about a decade ago. Amazon had a problem. It has millions of Web pages, each with a product for sale, but some were identical. The company wanted to weed out those duplicates, which is hard for computers but easy for people to do. So Amazon offered to pay people a few cents to do it for the company. Soon, other companies were posting tasks too and Mechanical Turk was born.

Why the name? Well, it references a famous hoax from the 1700s. There was a chess playing machine that could beat everyone. But actually, it was a chess master hiding inside. So, too, with Mechanical Turk. It seems like a computer is doing the work but it's actually a person.

"I do between six to eight hours and earn about $20 a day," says Ana Vasquez, who is unemployed. Between job interviews she Turks from her Brooklyn apartment. She does the math. She makes roughly $2.50 an hour.

"Which is basically, I guess, waitressing without the tips," she says.

'I Need The Money'

Mechanical Turk, or MTurk as it's called, has been criticized as a digital sweatshop. Vasquez knows the pay is peanuts but she really wants a job in social services.

"I'm enjoying the flexibility to keep pursuing that and not have to, say, choose something right now because I need the money," Vasquez says.

She's strategic. There are lots of different tasks on MTurk, such as Googling websites to boost their ratings, transcribing and copy editing. Research studies pay the most, Vasquez says.

"Recently, I did one on memory and it just involved clicking different numbers on the screen. It's probably for someone's Ph.D. or something like that," Vasquez says.

Quite likely. Jonathan Phillips is a Ph.D. student at Yale and he uses MTurk all the time. Compared to what he did before, it's a dream, he says. He used to print out questionnaires and hit the street.

"You'd spend two to three hours before you got too exhausted to continue and you'd get 15, 20 responses," Phillips says. "It's just so painstaking."

Not anymore.

How Reliable Are The Data?

"You come up with a study and then, an hour later you have, like, 200 responses, which is pretty fundamentally different," he says.

Berinsky, over at MIT, says researchers save not just weeks of effort, but great amounts of money as well. He's pays a couple cents per participant, compared to the $10 or $15 he used to pay.

OK, so MTurk is fast and cheap. Is it good? How reliable are the data?

The research shows that the population of Turkers is pretty representative, more so than signing up college students. There are other benefits too.

"I think it's actually improved the extent to which we can really be sure about our results," Berinsky says.

From his tucked away office, Phillips gives me an example. Say you put up a survey but the wording isn't quite right. That could skew the findings.

"People are much more willing to say, 'Well, I didn't really like that you used that one word there. So can you just change it and rerun it on Mechanical Turk,' knowing that it's not going to take you that long?"

Taking More Chances

Berinsky has another upside. "It's enabled people to take more chances. To do things that might seem crazy or sound crazy but are really clever and interesting and turn out to be right," Berinsky says.

What about the downsides? Maybe it's made research too easy and too cheap. Berinsky says some scholars might be putting up bad surveys.

"Maybe it would be better if people had to invest a little more of their own money before they ran with an idea," he says.

Not all research is suited to Mechanical Turk. If your study involves children, that's a no go. Need to observe your subject's facial expressions? Can't do that on MTurk. Those researchers can, of course, just do their research the old-fashioned way. But Phillips worries some researchers might be shying away from tough topics.

"I don't think we have any good sense of exactly which questions are now being neglected because they are harder to study," he says. "But I think it would be crazy to deny that it's having an influence on which questions we ask and how we ask them."

There's another change underway. We think about academics toiling away in a cubicle or sterile laboratories. One of Berinsky's colleagues surveys American voters on MTurk — but he does it from Indonesia. He might even be on the beach.

Copyright 2014 NPR. To see more, visit


Church Of Scientology Calls New HBO Documentary 'Bigoted'

The filmmaker says Going Clear, harshly critical of the Church of Scientology, is about the dangers of "blind faith." The church has hit back with an aggressive public relations effort of its own.

Think Nobody Wants To Buy Ugly Fruits And Veggies? Think Again

Remember that old movie trope, in which the mousy girl takes off her glasses to reveal she was a beauty all along? A similar scenario is playing out among food waste fighters in the world of produce.

Amazingly, Congress Actually Got Something Done

The leaders and members must, in a word, compromise. And on this occasion, Speaker John Boehner and Minority Leader Nancy Pelosi did just that, with skill and savvy.

Can Republicans Get Ahead In The 2016 Digital Race?

When Sen. Ted Cruz threw his hat into the ring, it happened first on Twitter. Political news is breaking more and more on social media, and both sides face different challenges in reaching out.

Leave a Comment

Help keep the conversation civil. Please refer to our Terms of Use and Code of Conduct before posting your comments.