Avoiding Bias from the Survivor Effect

by Jakob Nielsen on June 24, 2002

Sidebar to Jakob Nielsen 's column on Improving Usability Guideline Compliance .

In selecting websites for our mid-2002 e-commerce usability survey, we avoided any site studied for our 2000 report . Why? To avoid the survivor effect.

Only a few of the 2000 survey sites are still around. We can safely assume that the surviving sites are not a random sample of the original group, but rather that significant differences exist between the sites that made it and those that died. Survival might be due partly to luck, but it is mainly a result of good management and an understanding of Internet fundamentals . Thus, the surviving sites are likely to be disproportionately clued-in about what it takes to run an online business.

Usability is one of the main factors in Web success . Not the only one, of course. We all know e-commerce sites that failed because of bad pricing or overly aggressive expansion. Still, there is a strong correlation between good management and simple design .

The survivor effect therefore implies that the remaining websites from our 2000 survey sampling will tend to have significantly above-average usability. Thus, if we included these sites in our 2002 survey, we'd likely find dramatic increases in the average numbers, purely caused by a selection bias.

The stock market offers another example of survivor effect bias: Say that you want to calculate the average return on investment from buying stocks and holding them for twenty years. You might do this by picking a group of companies from today's stock market that had been public for more than twenty years. Doing so would let you see what you would have paid for their shares twenty years ago and compare this figure to current prices. Unfortunately, this approach will lead you to overestimate investment gains. Because of the survivor effect, you would have eliminated failed companies from your sample. A real investor who bought stock twenty years ago would have invested a percentage of money in companies that went out of business in the subsequent years.

Share this article: Twitter | LinkedIn | Google+ | Email