How to compare survey results across months or locations without fooling yourself

Comparing is the most natural way to read a survey. Are we doing better than last month? Which location gives better service? Which product do people like more? The catch is that two different numbers don't always mean a real difference. Sometimes it's just the luck of who happened to answer that week.

If you make decisions based on differences that are really noise, you end up praising one team for good luck and pressuring another for bad luck. This guide shows you how to compare results fairly, without complicated formulas.

Why an NPS going from 32 to 38 might mean nothing

Every survey is a sample. You don't ask all your customers, only the ones who answered. If you ran the same survey with a different group of customers, the result would shift a bit even if nothing in your business had changed. That natural variation is the margin of error.

NPS is especially sensitive to this, because it's calculated by subtracting two percentages (promoters minus detractors). With few responses, two or three people switching groups moves the score by several points.

Imagine that in March you got 40 responses and an NPS of 32, and in April 45 responses and an NPS of 38. It looks like a 6-point improvement. But with around 40 responses a month, the margin of error on each NPS can be more than 20 points up or down. A 6-point difference sits well inside that margin. With that data you can't say you improved. You can't say you got worse either. Only that there isn't enough evidence of a change.

That doesn't mean the survey is useless. It means you need more responses, or a trend over several months, before drawing strong conclusions.

The size of each group matters more than the total

A common mistake is thinking about total responses. If you have 300 responses overall but split them across 6 locations, each one has about 50. And if one of them has 8, its result is very unreliable, even though the total looks large.

Before you compare, write down how many responses each group has. A few practical rules:

• Always show the number of responses next to each result. A 4.8 from 5 responses isn't worth the same as a 4.5 from 200.
• If a group has very few responses, combine it with another period (for example, look at the quarter instead of the month) or mark it as “not enough data”.
• The smaller the difference you want to detect, the more responses you need per group.

Same question, same scale, same moment

For two results to be comparable, the only thing that should change between them is what you want to measure: the month or the location. Everything else has to stay the same. Check these points:

• The same question, with the same wording. Changing “How satisfied were you?” to “How happy were you?” looks minor, but it can move the answers.
• The same scale. An average on a 1 to 5 scale can't be compared with one on 1 to 10, and a 5-point scale doesn't give the same results as a 7-point one.
• The same moment. A survey sent right after a purchase doesn't get the same answers as one sent two weeks later.
• The same channel. People who answer a QR code at the table aren't the same as people who answer an email. If one location uses QR and another uses email, the comparison is already skewed.
• The same kind of customer. If December brings lots of first-time customers and March mostly regulars, results will change even if the service stays the same.

If you changed anything on this list, write it down. When you see a jump in the chart, you'll know whether it could come from that change rather than from your service.

A full example: three locations and one percentage

Imagine you have three locations and you ask customers “Were you satisfied with the service?” with a yes or no answer. This month you get:

• Downtown: 18 of 20 said yes (90%).
• North: 160 of 200 said yes (80%).
• South: 3 of 3 said yes (100%).

At first glance, South is the best and North the worst. But South has only 3 responses: with that number, the result could be 100% or 60% by pure chance. Downtown has 20, still few: its 90% has a wide margin. North, with 200 responses, is the only fairly precise result.

If you compare Downtown and North with a test for the difference between two percentages, the 10-point gap isn't clear enough with only 20 responses at Downtown. The honest conclusion would be: “Downtown might be doing better than North, but we need more responses to know.” And South, for now, stays out of the comparison.

The same goes for averages. If you ask on a 1 to 5 scale and one location gets 4.3 and another 4.1, the difference may or may not be real, depending on how many responses each has and how spread out they are.

Weighted ranking: so a location with 3 responses doesn't top the list

When you rank locations, products or salespeople by their average, groups with few responses tend to land at the extremes. Three very happy customers put a location first. One angry customer sends it to the bottom. Neither position tells you much.

A weighted ranking fixes this. The idea is simple: each group gets a small “nudge” toward the overall average. If it has few responses, the nudge weighs a lot and its result stays close to the average. If it has many, the nudge barely shows and its result holds.

Imagine the overall average across all your locations is 4.3 on a 1 to 5 scale. Location A has 3 responses, all 5s. Location B has 120 responses and an average of 4.6. If you rank by simple average, A (5.0) comes first. With a weighted ranking that adds the equivalent of 20 responses at the overall average, A ends up at 4.39 and B at 4.56. Now B comes first, which makes more sense: it has a high score and backs it up with many responses.

You don't have to do this math by hand. There are free calculators on the site for it: “NPS comparison calculator”, “Compare two percentages”, “Compare two averages” and “Weighted ranking calculator”. You enter your numbers and they tell you whether the difference is clear or still noise.

How to do it with SurvIQ

In SurvIQ you can use the same survey month after month so the question and scale don't change. Share it by link, email, WhatsApp or QR code, and customers answer from their phone without creating an account. Results update in real time with charts, and AI can summarize them and answer questions about them. To compare two months or two locations properly, take the numbers to the free calculators on the site. You can start on the free plan.