Collecting User Feedback From a Beta-Test Site

by Jakob Nielsen on January 10, 1999

Sidebar to Jakob Nielsen 's column on how to collect user feedback from users of an archive.

One of the replies to my reader challenge about the design of an archival website was a case study of a totally different project: adding a search engine for a commercial travel site. This case study is so good that it deserves its own sidebar: not only does it present good advice on how to collect user feedback, but it also shows the value of doing so. The results showed that the original design didn't work and so the company killed it rather than inflict a useless site on the users. A rare and wonderful respect for customers . A completely new design with much better usability resulted.

Jay Virdee , Operations Director for the British Hotel Reservation Centre , writes:

The challenge was to test how users would react to our site after a major upgrade; i.e. would they continue to use it and make hotel bookings as they were doing with the existing site. The reason for the upgrade was to make the site user-friendly by adding a search engine for finding hotels. Conventional wisdom would be to go for this because that is what users want! I wasn't convinced that we should barge ahead with a major change without first finding out how users would react. The existing site consisted of hyperlinked pages which made it easy to use - could the search engine on the new site have an adverse effect?

The Solution

We decided to create a beta test site. We would recruit users who had used our site in the past, had volunteered feedback and had requested updates to our website. Each page within the site would carry a feedback section which asked users what they thought of the content e.g. was it easy to read, was there too much or too little useful information, was the download time acceptable etc. On the homepage we explained the purpose of the site and asked general questions like, the users speed of connection, monitor size and resolution etc. In return for the favour of testing our site, we gave every user that made a test transaction, a free discount book worth nearly US $ 20.00 which they could use at over 200 establishments in London and get a minimum 20% discount e.g. at restaurants, major tourist attractions and of course hotels.

When all was set we did an e-mail shot to over a thousand users and they came in droves.

The Problems

Logically one would assume that if you told users that a site was a beta and explained no real transaction would result from using it - they would do so. Wrong - we found that even though every page carried a feedback section/footer that had a message to that effect, users were testing the search engine but not making bookings which was the whole point. We had each page numbered i.e. Page 1 of x, and users would only qualify for the free gift if they reached the end of a transaction. Eventually some did get through to the end after we made further changes to the messages and encouraged users to make bookings. There were also some links that were not functional and we had alot of feedback asking why not. This was a beta after all and we hadn't made all the links functional e.g. Special Offers for Hotels.

The Result

We decided not to launch the website since we couldn't give users what they wanted. This was mainly due to the way the database was structured - getting to the information users wanted would lead to long delays that we considered unacceptable. We went back to the drawing board and redesigned the whole database and booking engine.

The beta was conducted during Q2 1998 and the new site will now not go live until Q1 1999. In the interim we revamped our existing site and put in alot of the features of the new site but still without the search engine. This revamp has been a major success and we have now been able to test more of the features which will eventually end up in the new site. The new site will now offer one of the fastest and easiest hotel search and booking engines on the Internet - we have been able to cut the number of steps for making a booking from over ten to just five for new users and four for those already registered. None of our competitors site can match that today!.

 

  • Was the feedback method successful - Yes
  • Would we do it again - Yes
  • Would we do it differently - Yes, by designing the beta site for beta testing only and not bolting on feedback sections later.

Share this article: Twitter | LinkedIn | Google+ | Email