Quick, Cheap Usability Testing with Amazon’s Mechanical Turk

December 12th, 2007

Recently my father shared an interesting story from Slashdot with me about a man that pays other people to argue with him on Amazon‘s Mechanical Turk. The Mechanical Turk is a place where you can offer up money in exchange for the performance of a task. Typically it’s used to accomplish things that a computer isn’t good at, like image recognition or in this case, forming a coherent counter argument.

Here’s a brief excerpt from the story:

But there were a few reasons I found this preferable to the conventional ways of gathering interesting rebuttals to your own reasoning. If you send out a sample argument to all of your e-mail buddies, you will probably get some useful replies, but they may start to think you’re a little weird for asking them to evaluate your thought processes, especially if you do it over and over.

This hits home because I find myself constantly bothering friends and family for feedback on my new site, Feed Each Other. Everyone is happy to pitch in, but there’s a fine line between being charmingly inquisitive and being just plain annoying. After the 100th revision of the site’s registration flow, friends aren’t going to be interested in testing it anymore. It would be great if there were other people I could bother. People that could approach my site with a fresh, detached point of view.

Yahoo and Google pay people quite a bit of money to come in and do such usability testing. They have fancy labs with video cameras and one-way mirrors. It’s immensely useful to watch people play with your products in this manner, but it’s also expensive. Amazon‘s Mechanical Turk offers a poor man’s alternative.

After reading the story about the guy using the Turk to argue, I realized I could use it to get feedback on my website. I spent 5 minutes and whipped up a quick script asking people to visit my site, sign up, play around a little bit and write down their impressions, thoughts and questions along the way. I asked the Turk to let 10 people give it a go for $1 a piece. In the first hour I had my 10 responses.

Here’s why this is really cool:

  1. It only took an hour. That’s pretty fast feedback.
  2. The responders ranged from people that hadn’t ever heard of RSS all the way up to a big time blogger and JS geek that I read regularly.
  3. 7 out of the 10 responses were reasonably interesting. They contained the sort of stream of thought information you’d expect from a proper usability test in a lab (although a lot less of it)
  4. The other 3 were also interesting, but a bit too brief to glean anything other than a general impression. This is also still useful though.
  5. This only cost $10 and not $1000
  6. No one on my staff had to spend the testing time with the 10 users. All you have to do is write up one script.
  7. These people are totally detached from my site and can offer honest, fresh perspectives.
  8. I just got 10 new users! :)

Is this as effective as having a trained facilitator sit down with a user? Absolutely not. But, it’s a fantastic way to quickly get candid feedback from a variety of different people.

I plan on experimenting with this idea a bit more. It seems that $1 was enough to entice people to participate rather eagerly. I’d like to see how low I can go and still get reasonable responses. Also in question is the total size of the Mechanical Turk user pool. When will I run out of fresh users? I’d also like to try posing more specific questions. My initial script was only asking for general impressions of the site and of the registration process and I limited them to half an hour which was probably too short. A few of them mentioned that they were running out of timevelcro wall for sale.

Here are some excerpts I enjoyed from the responses:

This is a fascinating site! I’ve known about Feeds but never got around to actually subscribing to any. I thought it was too much work. But your site makes it all so easy. Your site will save me so much time browsing to many different sites one at a time.

That put a smile on my face.

Generally I like the mission of the site and it does seem rich with content and perhaps a user-friendly experience.

Perhaps? :)

It seems like this site is a giant forum type site where people can post any random things that they’d like.

Hmmm. Kinda, I guess.

I will definitely join in because you fill a “need to know” I have re RSS. I feel like you are going to make it possible for me to organize the information I am interested in seeing in the best and most efficient way possible. More importantly, the sharing aspect is so very appealing to me. It’s like looking at what people have on their bookshelves. How better to learn something new, or something new about the person who owns the bookshelves than to see what they are reading? This is a fantastic concept and I wish you all the very best of luck with this project as I am sure it will be a great and positive success.

That’s what I’m talking about!

The links at the bottom need larger font size IMO. We baby boomers get tired of squinting!

That one’s for you Ofer. Done.

Wow, this registration thing was really good. Probably the simplest you can get, right?

I’m really proud of my simple little registration page. More than meets the eye.

Overall I really like the idea of the site and how it works. My only compliant would be that when a different theme is selected ( I chose black) the logos and images in the header get all blurry (I’m a graphic designer, sorry… ) But really, it’s a cool site. Like a combination of stumble upon and a feed reader, pretty awesome idea!

Not a bad way to put it.

I think it to be an Aladdin’s cave, so much to explore and to know about.This site will be very useful to explore the web n find out new n interesting facts and news etc. I think this site would prove very useful.

Aladdin’s cave!

This should give you a small taste of what type of feedback you can get with this method. I didn’t post the more critical items here because I’m not that daring, but there were indeed some and they were by far the most useful part of this exercise.

Are you stuck in the short tail?

December 4th, 2007

Joshua Porter asks the question, “Did the Long Tail beget social design?” I think the answer is absolutely yes.

Here’s what Joshua had to say

Netflix rents most of its movies from the catalog of past movies, not from the current list of blockbusters. Same with Amazon and books, iTunes and music. Christopher Anderson goes into a lot more details in the book he wrote on the subject: The Long Tail.

When content is digital, a public good, it is freely distributable by electronic means. It is infinitely copyable at 100% fidelity. Moreover, as the Long Tail shows, libraries of content can be built cheaply which provide value for the long term. Once Google digitizes all the books in the world they won’t ever have to again.

In other words, all content is available at all times.

What does this lead to? The Paradox of Choice! There are simply too many things to choose from. Which of the thousands of movies on Netflix do I rent? Which of the books on Amazon do I read? Which of the songs on iTunes do I listen to?

He’s right on. Too many choices leads to confusion. The only way to help people find personally relevant things in such large collections is by giving them new ways to slice through the choices. Amazon, and later Netflix, did this brilliantly, and necessarily. Without such tools, their sites would be pretty useless.

The thing I’d like to point out about all this is that there are way more pieces information out there on the internet than there are movies and books. Yahoo! and Google index billions of web pages. Billions. And, that number is growing every day. How the heck does someone dive into that sea of information and come up with a small set of things that are worth actively monitoring? It’s not easy. In fact, it’s really, really hard. It’s similar to the problem that Amazon and Netflix have been tackling, but it’s a new space. Google did a pretty good job with search (and they did so with social design). But what about browsing? What about keeping up with the new stuff? What’s the canonical example of a tool to tackle the long tail of the browseable web? What comes to mind? Anything?

It feels to me like there is a void to be filled, and this is exactly why I felt the burning need to build Feed Each Other. A feed reader is a great way to keep track of information sources you care about, but how do you find out which sources to care about?

All of the leading feed readers were dropping the ball on this. They were giving people access to the largest set of information sources that has ever existed and providing them with an efficient way to monitor them, but they were putting almost no effort at all into helping people explore and discover new things. I just couldn’t believe that the market leaders were missing this opportunity. It seemed so obvious. I felt like I was taking crazy pills. Something had to be done.

The reason that RSS reading has not caught on with a mainstream audience is that until now it has been designed as a solitary experience. You pick the feeds you want to read. You read them. That’s it. Well, if you ask me, that’s really boring.

The only people that can think up a list of feeds worth reading at this point are geeks. We’re ahead of the curve. The sites we read every day were the first ones to add rss feeds. We’re willing to go through the trouble of cutting and pasting ugly urls, and we already have a huge list of bookmarks to work with.

But John Q. Citizen? He opens up a feed reader and goes “uhh, now what?”.

John Q. Citizen gets his online news from the NY Times. He reads his sports at ESPN. That’s about all he needs, and he’s pretty happy with that. He doesn’t even know that there is all of this other amazing stuff out there that he’s missing. He doesn’t know that there are 20 interesting blogs out there that could teach him more about his favorite hobby. He isn’t aware that there is some guy out there publishing an unofficial feed of set-lists from recent shows by his favorite band. He has no idea where to start looking or even that he should be looking. He has no need for a tool that lets him efficiently monitor many information sources because he doesn’t have many information sources that he wants to monitor. He doesn’t know that they’re out there.

John Q. Citizen needs a place where this long tail of the fresh web is exposed. Where he can see what his friends are reading every day. Where he can browse the fresh web by topic. Where he can find other people who share his interests.

Feed Each Other is an attempt to do this. It’s a full fledged feed reader. It does everything you’d expect from a good reader. But, it does more. It shows you feeds related to the feed you’re looking at. It points you to like-minded people who have similar subscriptions. It lets you go to the profiles of those people and see what else they’ve chosen to read. It lets you share the things you find in your feeds with your friends, family or co-workers (with very granular group & privacy controls of course). It shows you that your friend Steve just subscribed to a feed about stamp collecting (and you love stamp collecting!). Well, chances are that Steve found a keeper. You’ll probably want to jump on that and check it out.

Feed Each Other gives you the tools you need to explore that mess of stuff out there and find what is most personally relevant to you. It makes this herculean task into a more accessible group endeavor. You get introduced to narrowly focused news sources that you’ll love. This lets you unsubscribe from other noisy, broadly focused news sources that only rarely hit on the topics that you’re really interested in. The result is that you read more interesting things in less time. It’s just way more fun and interesting this way. Every day you can discover something new and potentially get rid of something old.

I tell people that I’ve built a new, social website and their eyes glaze over. “Social” has been done to death. I know this.

But you know what? This makes sense. It was done this way for a reason. It’s actually useful.

Facebook is slow, but there’s an easy fix

November 22nd, 2007

It seems like the Facebook engineers add another javascript and css file to their pages with every new feature. As of today, viewing the Facebook homepage loads 36 javascript files from Facebook and 4 more from advertisers. In addition to these 40 files, there are 12 css files.

That’s a lot of files! If you’ve ever noticed the excessive flickering happening in your browser’s status bar when you go to Facebook, this is the cause.

How can they improve the situation? The total file size isn’t too terrible at about 100k of JS and 40k of CSS. The best thing to do would be to use a simple build script that concatenates all these files together into one larger file. If they can’t do that for some reason, they could at least serve the files from more than one domain to take advantage of parallel downloads in the browser. They could also try a service like Akamai, but that’s a bit more complex and expensive.

This seems like a really silly oversight from an organization that in general produces beautiful work. My guess is that when the company was getting started they set up a relatively simple build process that has now become ingrained and difficult to alter as their setup has quickly become larger and more distributed. However, they’ve reached the point where it’s time to revisit things and do it properly.

Jake Shimabukuro and the exploding niche

November 9th, 2007

Last night I went with a few friends to Yoshi’s in Oakland to see Ukulele player Jake Shimabukuro put on a sold out show.

This guy is a Ukulele player. He’s outrageously talented, but he’s still a Ukulele player. I’ll bet most people in the Bay Area have never even seen a Ukulele before, but somehow this guy managed to sell out multiple shows.

How did that happen? Well, Jake told us.

A while back he was asked to do a performance for a tv show called Ukulele disco. Clearly Jake was already a star in the Ukulele world. They went out to Central Park in NY and taped him playing his beautiful arrangement of “While my guitar gently weeps”. Some time later, all of a sudden, more and more people started showing up at his performances. They told him that they saw his video on Youtube. He said, “You-what”?

It turns out that someone posted his clip from Ukulele disco to Youtube and the clip quickly went viral. Since then millions of people have viewed it. “That video changed my life”, Jake said before he performed the song for us. “It allowed me to play all over the world in all kinds of places that I never could have before”. He was genuinely appreciative.

That clip is how I was introduced to his music and I bet most of the people in the room at Yoshi’s were in the same boat.

Youtube allowed Jake to escape a rather obscure niche. I’m really glad I got to see him play, and without Youtube it would never have happened. It’s the modern day equivalent of being on the Late Show. The difference is that today many, many more people get the opportunity to be seen.

Check out the clip:

Random tips & tools (nerd alert)

October 30th, 2007

Nothing ground-breaking here, but I wish someone had told me about these things sooner. Maybe these will help some of you out.

  • Firefox
    • ctrl+1 . . . ctrl+n will switch to that specific tab.
    • ctrl+t (new tab), ctrl+k (search), ctrl+l (address bar) and ctrl+w (close tab) are also musts.
    • You can add new items to the list of feed handlers (Like this)
    • After Firebug, my favorite extension is Resizable Textarea
  • Linux
    • The locate and updatedb commands are very useful. Every mention of “find . -name foo -print” online should also say, “trying locate foo first is usually MUCH faster and easier”. But, no one ever seems to talk about it.
    • Here’s my .vimrc file. Can’t live without it.
    • When running top, shift+> and shift+< are your friends. Also, hitting c is usually the first thing I do.
    • nohup is cool.
    • nmap -sV or netstat -tulpen will show you what’s running on what port.
    • Learn about the load average numbers that you see on the top of top.
    • Java processes can run away with all of your memory. Be careful. Use java -Xms100m -Xmx500m to limit things. The Xms is the start amount, the Xmx is the max amount.
  • Tools
  • Any other tips I should know about? Please share.