5 common mistakes when analyzing survey results

Designing a good survey is only half the work. The other half is analyzing the results without drawing conclusions the data doesn't actually support. These are the most common mistakes we see over and over.

1. Drawing conclusions from very small samples

If 3 out of 5 people said they prefer option A, that does NOT mean "60% prefer A" in any statistically useful sense. With small samples, a couple of different answers change the percentage drastically. Before generalizing a result, ask yourself whether the sample is large enough for the number to be stable.

2. Ignoring who didn't respond

A 40% completion rate isn't just an operational detail, it can be biasing your results. If the most dissatisfied people are the ones who drop off earliest (which is common), your results will look more positive than they really are, because you're only seeing the people who stuck around.

3. Confusing correlation with causation

If users who use feature X have a higher NPS than those who don't, it's tempting to conclude "feature X improves NPS." But it could be the other way around: the most satisfied users are the ones who explore more features. The survey tells you there's a relationship, not what's causing it.

4. Not filtering by segment

An overall average can hide important differences between groups. If overall NPS is 40, but new customers have an NPS of 60 and long-time customers have an NPS of 10, that average of 40 doesn't describe anyone well. There's a specific retention problem with long-time customers that the overall number doesn't show.

• Always cross-tab results by relevant segments (new vs. existing, plan, region) before drawing conclusions from the overall average.
• Look at completion rate alongside the results, not separately.
• Be skeptical of any finding based on fewer than ~30 responses.

5. Not acting on open-ended questions

Free-text questions often go unread because it takes more work than glancing at a bar chart. But that's usually where the "why" behind the numbers lives. It's where people tell you, in their own words, what's not working or what they'd like to see done differently.