Quick, Cheap Usability Testing with Amazon’s Mechanical Turk

Recently my father shared an interesting story from Slashdot with me about a man that pays other people to argue with him on Amazon‘s Mechanical Turk. The Mechanical Turk is a place where you can offer up money in exchange for the performance of a task. Typically it’s used to accomplish things that a computer isn’t good at, like image recognition or in this case, forming a coherent counter argument.

Here’s a brief excerpt from the story:

But there were a few reasons I found this preferable to the conventional ways of gathering interesting rebuttals to your own reasoning. If you send out a sample argument to all of your e-mail buddies, you will probably get some useful replies, but they may start to think you’re a little weird for asking them to evaluate your thought processes, especially if you do it over and over.

This hits home because I find myself constantly bothering friends and family for feedback on my new site, Feed Each Other. Everyone is happy to pitch in, but there’s a fine line between being charmingly inquisitive and being just plain annoying. After the 100th revision of the site’s registration flow, friends aren’t going to be interested in testing it anymore. It would be great if there were other people I could bother. People that could approach my site with a fresh, detached point of view.

Yahoo and Google pay people quite a bit of money to come in and do such usability testing. They have fancy labs with video cameras and one-way mirrors. It’s immensely useful to watch people play with your products in this manner, but it’s also expensive. Amazon‘s Mechanical Turk offers a poor man’s alternative.

After reading the story about the guy using the Turk to argue, I realized I could use it to get feedback on my website. I spent 5 minutes and whipped up a quick script asking people to visit my site, sign up, play around a little bit and write down their impressions, thoughts and questions along the way. I asked the Turk to let 10 people give it a go for $1 a piece. In the first hour I had my 10 responses.

Here’s why this is really cool:

  1. It only took an hour. That’s pretty fast feedback.
  2. The responders ranged from people that hadn’t ever heard of RSS all the way up to a big time blogger and JS geek that I read regularly.
  3. 7 out of the 10 responses were reasonably interesting. They contained the sort of stream of thought information you’d expect from a proper usability test in a lab (although a lot less of it)
  4. The other 3 were also interesting, but a bit too brief to glean anything other than a general impression. This is also still useful though.
  5. This only cost $10 and not $1000
  6. No one on my staff had to spend the testing time with the 10 users. All you have to do is write up one script.
  7. These people are totally detached from my site and can offer honest, fresh perspectives.
  8. I just got 10 new users! :)

Is this as effective as having a trained facilitator sit down with a user? Absolutely not. But, it’s a fantastic way to quickly get candid feedback from a variety of different people.

I plan on experimenting with this idea a bit more. It seems that $1 was enough to entice people to participate rather eagerly. I’d like to see how low I can go and still get reasonable responses. Also in question is the total size of the Mechanical Turk user pool. When will I run out of fresh users? I’d also like to try posing more specific questions. My initial script was only asking for general impressions of the site and of the registration process and I limited them to half an hour which was probably too short. A few of them mentioned that they were running out of time.

Here are some excerpts I enjoyed from the responses:

This is a fascinating site! I’ve known about Feeds but never got around to actually subscribing to any. I thought it was too much work. But your site makes it all so easy. Your site will save me so much time browsing to many different sites one at a time.

That put a smile on my face.

Generally I like the mission of the site and it does seem rich with content and perhaps a user-friendly experience.

Perhaps? :)

It seems like this site is a giant forum type site where people can post any random things that they’d like.

Hmmm. Kinda, I guess.

I will definitely join in because you fill a “need to know” I have re RSS. I feel like you are going to make it possible for me to organize the information I am interested in seeing in the best and most efficient way possible. More importantly, the sharing aspect is so very appealing to me. It’s like looking at what people have on their bookshelves. How better to learn something new, or something new about the person who owns the bookshelves than to see what they are reading? This is a fantastic concept and I wish you all the very best of luck with this project as I am sure it will be a great and positive success.

That’s what I’m talking about!

The links at the bottom need larger font size IMO. We baby boomers get tired of squinting!

That one’s for you Ofer. Done.

Wow, this registration thing was really good. Probably the simplest you can get, right?

I’m really proud of my simple little registration page. More than meets the eye.

Overall I really like the idea of the site and how it works. My only compliant would be that when a different theme is selected ( I chose black) the logos and images in the header get all blurry (I’m a graphic designer, sorry… ) But really, it’s a cool site. Like a combination of stumble upon and a feed reader, pretty awesome idea!

Not a bad way to put it.

I think it to be an Aladdin’s cave, so much to explore and to know about.This site will be very useful to explore the web n find out new n interesting facts and news etc. I think this site would prove very useful.

Aladdin’s cave!

This should give you a small taste of what type of feedback you can get with this method. I didn’t post the more critical items here because I’m not that daring, but there were indeed some and they were by far the most useful part of this exercise.

5 Responses to “Quick, Cheap Usability Testing with Amazon’s Mechanical Turk”

  1. Philip wilkinson Says:

    great post. Can you share the ‘script’ you used as an example of what we need to submit?

  2. Doug Says:

    For anyone who doesn’t want to take the time to set everything up on Mechanical Turk, we’ve set up a small company to provide this service for you – http://www.EasyUsability.com. We’re still in beta, so sign up if you want some free tests.

  3. Ralph Says:

    This was an interesting post. I found another website (http://www.UserTesting.com) was a lot less work. I was able to focus on writing my desired usability tasks (no small feat), then tap into their pre-recruited and vetted set of user testers. It was a bit more per test, but I saved a huge amount of time.

  4. usability testing with MTurk - usability fail. Says:

    […] with this problem, I remembered something that a teammate of mine did once, and this post from Udi’s Spot, and tried something a little different. I mishmashed user testing with A/B testing, deployed to […]

  5. Hung Groden Says:

    FYI, your blog doesn’t look suitable in Opera.