Writing better questions for surveys and unmoderated tests
As one of the lead product designers at Delphia, I developed a series of guides to help our team with user research and discovery.
Surveys don’t give you the why; they give you the what.
– Nikki Anderson, UX Researcher & Founder of User Research Academy
It is essential to always remember that surveys do not provide us with insights about the “why” behind a user behaviour. We may ask users “why” questions, however the data you collect will likely be surface-level and should be treated as such. To deeply understand user behaviour, we recommend moderated user interviews.
Define your research goal and understand what type of data you’re looking for.
Before you send out a survey or unmoderated test, it’s important to think about your research goal, the type of data you’re looking for, and how the results will impact your decision-making. This is important because the data you are seeking and the type of survey you are running, either qualitative or quantitative, will impact what you ask.
Before creating your survey, answer the following questions:
What are we trying to learn about our users?
How will this survey or test help us?
What do we expect from the results (assumptions)?
What do we expect to do with the results?
What type of data am I looking for (what type of survey is best suited for my goals)?
Example: We want to produce a roadmap of prioritized features that we are going to build next quarter.
✅ Do this
You’ve done qualitative interviews, you understand the problem(s) users are hoping to solve, and you’ve validated that these features will solve their problem(s).
Now, you’re looking for quantitative data to prioritize the features that would satisfy your customer base.
Multiple Choice: Which feature would you love to see next?
Monthly budgeting
Multi-account or family budgeting
Notifications
Monthly spending summaries
Other, please specify
💫 In an ideal world, the multiple choice options have been validated by previous research.
❌ Not this
You have not done any qualitative interviews, but you want to use your user’s feature requests to inform your roadmap.
Open-Text: Tell us what feature you would like to see next!
⚠️ Open-text survey responses are not “wrong” and can be effective when used properly and at the right time (i.e. qual survey early in discovery).
Why it isn’t recommended in this example is because surveys do not tell us why users are requesting this feature, nor does it validate that this feature will actually solve the problems they’re facing.
Keep it short and simple
You want to design a clear and brief survey that is easy to complete. Respondents will drop off or get fatigued during long tests – this means your last few questions may not get the same level of attention or accuracy.
Keep your survey as short as possible
Ask one question at a time
Front-load your survey with your most important questions
💡 Tip: Critically think about the questions you ask and only ask questions necessary for what you’re trying to learn. If you don’t think you will act on the data, then don’t ask the question.
Use simple, easy-to-understand language
Use language a child would understand and replace jargon, acronyms, or technical/business language with simpler or familiar words.
✅ Do this
Describe the most difficult part of tracking your monthly spending.
❌ Not this
Describe the primary pain point you have when it comes to tracking your monthly outgoing cashflow.
Use plain, neutral language. Avoid bias, leading, and loaded questions.
Always use plain, neutral language when writing questions to ensure you don’t imply particular answers or give away your expectations. The way a question is written can impact your results.
Leading Question: Assumption Based
Questions written with preconceived assumptions from the researcher. Often, these questions will skew towards either a positive or negative and does not allow the respondent to state what their experience was like.
✅ Do this
Rate the experience you had with our budgeting tool using to the following scale:
1-Poor to 5-Excellent
Open-text: Describe why.
❌ Not this
Which of our budgeting features did you find most useful?
⚠️ This assumes that the respondent found the feature useful and does not allow them to state if they did not find it useful.
Leading Question: Direct Implication
Questions that ask respondents to respond with a prediction of a possible reaction to something if something else happens. It’s extremely difficult for a person to give an accurate prediction about their future behaviours. We prioritize questions about past or present behaviour, as past/present behaviour is a better indicator of future behaviour.
✅ Do this
What is difficult about budgeting?
Time-consuming
I don’t have enough money to budget
My expenses are shared and it’s difficult to manage multiple people’s spending
Other
How do you prefer budgeting?
On-the-go
Desktop
Other
Follow-up: Why do you prefer budgeting this way?
What would motivate you to download a mobile budgeting app?
Automatic tracking for spending
Suggestions on where I can optimize my spending
Household or shared budgeting
Other
❌ Not this
If you could save time with a mobile budgeting app that automatically tracks spending, how likely would you download it?
⚠️ Avoid assuming the user’s problem, the solution for that problem, and asking the user to predict their future behaviour.
You’ll get richer and more useful data if you objectively ask about each of these questions (ideally using multiple choice options from previous research insights).
Leading Question: Unbalanced Scales
Questions that sway answers by limiting users' choices.
✅ Do this
Rate the experience you had with our product using to the following scale:
1-Poor to 5-Excellent
❌ Not this
Rate the experience you had with our product using the following scale:
1-Enjoyed it a little to 5-Enjoyed it a lot
Double-Barrelled Questions
These are sentences that ask two questions at once.
✅ Do this
Rate the your experience product using to the following scale:
1-Poor to 5-Excellent
Rate your experience with customer support:
1-Poor to 5-Excellent
❌ Not this
How would you rate the quality of our budgeting app and customer support?
Vary your question types
Mixing up the types of question on your survey can help you collect insightful data.
Open-ended survey questions
Single open-ended question
(e.g. “How could we improve budgeting for you?”)Pairing closed and open-ended survey questions
(e.g. Question 1: How do you budget? [Multi-select answers from previous research] Question 2: Why do you budget this way?)
Closed survey questions
Multiple Choice
Single select (e.g. radio buttons)
Multi-select (e.g. select all that apply)
Likert or rating scale (e.g. strongly disagree to strongly agree)
Matrix (e.g. evaluate one or more row items using the same ratings)
Ranking order
Dichotomous (e.g. Yes/No)
💡 Helpful tips for multiple choice questions
Your multiple choice options should come from previous research
Always offer the “Other” option
Ensure your response options are mutually exclusive (e.g. you shouldn’t make “budgeting” and “spreadsheet budgeting” two separate options)
Make use of logic functionality