Pilot test and Cronbach's alpha: which items to fix before collecting data

After expert judgment comes the pilot test: you give the questionnaire to a small group similar to your sample to see whether it is understood and whether the scales measure consistently. That consistency is summed up with Cronbach's alpha, which is what your advisor will look for in the reliability section.

How many people to include

Thesis pilots usually have 20 to 30 people, or about 10% of the sample. What matters is that they resemble the people who will answer later (same occupation, age or education) and that they are not part of the final sample. At the end, ask them what was hard to understand and how long it took.

How to read Cronbach's alpha

Alpha goes from 0 to 1 and measures how much the items of a scale point in the same direction. A widely used guide (George and Mallery, 2003) is:

• 0.90 or more: excellent.
• 0.80 to 0.89: good.
• 0.70 to 0.79: acceptable.
• 0.60 to 0.69: questionable.
• 0.50 to 0.59: poor.
• Below 0.50: unacceptable.

Calculate alpha for each variable and each dimension, not only the total. A dimension with few items (2 or 3) tends to have a lower alpha even when it is well built, so look at the correlation between its items as well.

Which items to review

Two numbers tell you which item lowers reliability:

• Corrected item-total correlation: how much the item relates to the rest of the scale. Below 0.30 it is worth reviewing.
• Alpha if item deleted: if alpha goes up when you remove it, the item is not in line with the others.

In the example below, the full burnout scale gives an alpha of 0.69. Items 9 and 10 have a negative item-total correlation: they go the opposite way from the rest because they measure personal accomplishment, the opposite of exhaustion. Item 8 in the same dimension was already marked as reverse-worded; 9 and 10 were not.

A negative correlation rarely means the item is bad. Most often it is worded the other way and needs to be recoded, or the dimension should be reported separately. For established instruments, the author's manual says how each dimension is scored; follow it before changing anything.

When not to drop an item

Dropping items just to raise alpha can leave a dimension uncovered. If the judges said the item is relevant and the pilot shows a problem, try rewriting it first. If you remove it, note it and explain it in your thesis.

The pilot in Thesis Mode

In SurvIQ you mark when the pilot starts and when data collection starts. Responses from each phase stay separate, so the pilot doesn't mix with your sample and your tests before starting don't count. You see alpha per variable and dimension, the per-item table and, on the Pro plan, McDonald's omega, which some universities ask for alongside alpha.

As always, SurvIQ runs the calculation and flags what is worth a look. The decision about each item and how you report it in your thesis are yours. If you only need the number, SurvIQ's Cronbach's alpha calculator is free.