Sample bias in business
Sample bias is a simple concept. If you do a poll about the economy standing outside the unemployment office people will probably say its bad. If you do it inside the Gucci store you might get a different answer. Reality might be somewhere in between.
In many businesses, we don’t get to talk to all our customers to understand how satisfied they are. We might talk to a couple each day, and over the course of a few months feel like we have a large sample and a good handle on customer satisfaction, having talked to 150 customers.
Imagine two executives, the COO and CTO, each meeting 150 customers. Lets look at their samples.
The COO flies all around the globe meeting customers. When he is going to Japan, he calls the head of Japan two weeks ahead and asks to have some customer meetings. The head of Japan thinks a little bit. I could take him to customer A who is really mad because we screwed up their project. Or I could take him to customer B, who loves how their project went and wants to buy more. In most companies, he sees customer B. That pattern repeats itself the world over. The head of sales sees a lot of happy customers, and thinks that customer satisfaction is great and the team is doing a great job.
The CTO stays at home. Besides working on the next version of the product, he is the final escalation point for complex technical issues. The company has 20,000 customers. Each of them has some form of issue roughly every 3 months, so each day there are 200 issues logged. 90% of those issues are resolved quickly by the support analyst assigned to it. 10% of them – 20 a day – need some extra attention from an engineer. Again, 90% of them are resolved fairly quickly. The other 10% are somehow problematic: the engineers can’t figure it out either. No problem, enter the CTO. He solves the 1% of the support cases that the support team and engineers can’t solve, which ‘only’ amounts to two issues a day. Not so many for a base of 20,000 customers, but a lot for the CTO to deal with. Now ask the CTO about customer satisfaction and how the support team is doing. Not so well.
Same product, different sample. One selected for satisfaction, the other for dissatisfaction. Completely different conclusions.
Max Schireson's blog 
Perfect.
I was helping a local businessman work though his product segments and how the buyers/influencers view the available products/services. He has 30+ years in his field and has an encyclopedic knowledge of teh segments in two important groups:
-> People who looked and bough;
-> People who looked and did not buy
But he’d never made an attempt to reach the vast majority of the market: people who’d never heard of his product. So he has no idea if there addressable segments inside the 98% of his TAM that he’s never reached.
It was a lightbulb moment in the workshop. I love that kind of stuff.
-XC