I tried out usertesting.com for the first time this week. I've been hankering for an excuse since I learned about the service. Essentially, they provide remote usability testing services via a network of hundreds of testers across North America and the UK. I have to say, I was impressed.
Task Set-up
The sign-up process is very straight-forward, divided into 3 parts. The first screen is where you set up the tasks of the test. The form asks for the URL to be tested, the scenario and the specific tasks to be carried out. The default is 4 tasks, and I left it at that. I only had a short window to review and analyze the results so I could brief our web designer.
Tester Demographics
Screen 2 recruited the testers. I chose to have 3 testers carry out the tasks. The demographic options include age, income, geographic location, and level of web savviness. In my experience, the demographics are less important and shouldn't be fussed over too much. Regardless, it's nice to have the options if you have a very specific user set. The minimal downside of restricting demographics, is that you may have to wait longer for the tests to be completed, as there won't be many testers who meet narrow criteria.
The third screen was payment. This was my only hiccup, which I already blogged about, but it didn't dampen the experience.
Great Video and Commentary
In less than an hour I had email notification that one test had been completed. When all 3 were done, I logged into my account to watch the results. The video playback was smooth and clean and the audio clear, with little background noise. I imagine usertesting.com sets minimum standards for software and equipment for all testers. The video is not of the tester, but of the screen and their mouse movements and clicks. The audio is the play-by-play commentary given by the tester as they perform the tasks.
It was evident that all were experienced testers. They spoke slowly and loudly, reading the tasks aloud and communicating their expectations about what should happen as well as their thoughts as they performed the tasks. When finished, all testers also gave commentary on the overall positives and negatives of their experience.
There's No Substitute for Real Users
I am a usability professional, but every time I do user testing I'm reminded that I am not a typical user (and this is a good thing). All three testers identified issues (the same ones) that hadn't occurred to me in my initial audit.
In addition to the videos I could read the responses to 4 post-test questions that gave me more insight. I also got to see the demographic profiles of each tester.
Overall, it was very successful and I will definitely use the service again. When I do I'll do 2 things differently:
- I'll make my tasks a bit harder and involved - I went a little easy on a few as I was unsure what to expect from the testers.
- I will provide more tasks to be performed. It's a flat fee per test so whether you ask 3 tasks or 10, the main factor is the time you'll have for review and processing the information. If you're showing the tests to exec's, go big. Even if you don't use all the results immediately, you'll have the feedback available to you in future.
And the cost...$29US per tester (I think this was a first-time user special) + approximately 4 hours of my time to create the tasks, review the completed tests and document the results and recommendations = great value!
Comments