Cognitive Interviewing: How to Test Survey Questions
Test how participants understand, recall and answer survey questions before fieldwork, with a worked example and revision log.
A survey question can be grammatically clear and still produce answers to different questions. One respondent interprets “recently” as this week; another thinks of the past year. Both answer confidently. Your dataset quietly combines two time frames.
Cognitive interviewing helps investigate how people understand a question and arrive at an answer. It is not a statistical test of whether a questionnaire is valid. It is a way to identify response problems before they become columns in your dataset.
In this guide
Examine the response process
Cognitive interviewing investigates how a respondent reaches an answer.
The CDC CCQDER. Cognitive Interviewing. describes investigating processes such as comprehension, recall, judgement and response selection. These provide useful starting points for probing.
For a fictional question—“Have you recently had difficulty accessing services?”—ask what “recently”, “difficulty” and “services” mean to the participant. Then explore which experiences they considered and how they chose their answer.
Do not begin by explaining what you intended. That would test whether a respondent can follow your explanation rather than whether the questionnaire works on its own.
Use probes that reveal reasoning
After the participant answers, useful probes might include: “What time period were you thinking about?”, “What kinds of services did you include?” and “How did you decide between sometimes and often?”
Think-aloud approaches can also be useful, but may feel unfamiliar. Choose and practise a method appropriate to the participants and instrument. Avoid turning the session into a lesson in how to give the desired answer.
Document the problem before revising
Suppose one participant includes transport to the clinic, another thinks only about entering the consultation room, and a third includes obtaining medicines after the visit. The question mixes several possible constructs.
| Problem observed | Possible revision | What to test next |
|---|---|---|
| Different recall periods | Specify “during the past 30 days” | Can relevant events be recalled within that period? |
| Different meanings of access | Ask about a specific step in the care pathway | Does the narrower wording match the research objective? |
| “Often” interpreted inconsistently | Consider a count or defined frequency categories | Can respondents provide that information reliably? |
These revisions are examples, not automatically better questions. A precise 30-day period is unhelpful if the event is rare or that period does not match the study objective.
Include the people likely to encounter problems
Consider variation in language, education, experience of the service and other characteristics relevant to interpretation. The aim is to explore response processes, not estimate population percentages from the cognitive-interview sample.
Use iterative rounds where feasible: test, revise and retest. Keep a record of the questionnaire version, observed problem, proposed change and evidence supporting the decision.
Respect existing instruments
For an established scale, changing wording, response options or recall periods may affect comparability and measurement properties. Check permissions and developer guidance before adapting it. Document all changes.
Boateng et al. Best Practices for Developing and Validating Scales: A Primer. 2018. situates item development and pretesting within a broader measurement-development process. Clear wording is necessary, but it does not by itself establish reliability, construct validity or suitability for a new population.
End with a decision, not a satisfaction score
“Participants liked the questionnaire” is not the main result. Identify which items worked as intended, which require revision and which remain uncertain. If a question consistently measures something different from your objective, consider replacing it rather than polishing its grammar.
Before launching the survey, ask one practical question: Can you explain how intended respondents interpret each important item—not just how the research team reads it? Continue with the research design guidance to check whether the revised instrument still answers the study question.
Continue your research
Three useful next steps for this topic.
Research notes, once a week
Practical research guidance, useful tools and funding opportunities in your inbox.