How to Ask Unbiased Questions in User Research

Andi Galpern

When conducting user research, you want to ask unbiased questions.  These are questions designed to uncover how a person feels and what they want.  Your goal is to make that person feel comfortable, authentic, and important.  You listen because what they have say matters.  

In this article, I'll provide a few tips on how to improve your questioning skills. Use these techniques to write surveys, interview candidates, and engage in everyday conversation.

Are We All Biased?  

Bias occurs when you allow your beliefs to shape the research or conversation.  According to Merriam-Webster, bias is an inclination of temperament or outlook; especially a personal and sometimes unreasoned judgment or prejudice.  Most people don’t intentionally make judgements or act biased.  They do this because it’s ingrained within them. 

Many researchers, including Rina Jensen, a User Researcher at Mozilla, believe that all data contains bias. 

“There’s bias in all data.  Even big data with large sample sizes has bias.  Humans create surveys, machine learning, and algorithms from their point of view.”

“Your analysis of the data is your observation on how you observe that data. What you report is a reflection of what you see happening.  When create a report, it's from your analysis and interpretation. "

Rina Jensen, User Researcher & Service Designer at Mozilla

Reduce Bias with Neutral Questions

What is a Neutral Question?

Neutral questions do not carry a hidden-agenda.  They are open-ended and allow people to think deeply about the answer.  As a result, you and the respondent will form a deeper connection.

Rina also recommended a book called “Surveys in Social Research” by David de Vous.  

 

Rina Jensen, User Researcher at Mozilla, recommends "Surveys in Social Research" by David de Vaus. This is a thick book that she references throughout her workday.

Question Your Questions (HOW META!)

Here is an excerpt on how to neutralize questions from David de Vaus's book "Surveys in Social Research": 

Is the language simple? Can the question be shortened?

Is the question double-barreled?

Is the question leading?

Is the question negative?

Is the respondent likely to have the necessary knowledge?

Will the words have the same meaning for everyone?

Is there a prestige bias?

Is the question ambiguous?

Is the question too precise?

Is the frame of reference for the question sufficiently clear?

Does the question artificially create opinions?

Is personal or impersonal wording preferable?

Is the question wording unnecessarily detailed or objectionable?

Does the question have dangling alternatives?

Does the question contain gratuitous qualifiers?

Is the question a ‘dead giveaway’?


Common Mistakes

Here are a few examples of common bias questioning mistakes, and how to fix them.  I've added a little Andi Pandi branding for your enjoyment.

Leading Questions

Leading questions subtly prompt the respondent to answer in a particular way.  They include emotional phrases to encourage a desired answer from the participant.


What’s wrong with this design?


Share your thoughts on this design. 

The word "wrong" implies that the design does not work. Don't make assumptions or integrate your judgements into the question. Stand back and observe.

Double-Barreled Questions

A double-barreled question groups two questions into one. Defense lawyers often object to these types of "compound" questions – another name for double-barreled questions.


How often and how much time do you spend on Reddit?

How much time do you spend on Reddit?
How often do you visit Reddit?

To avoid confusion, separate the question into two parts.  Ask one question at a time. 

Loaded Questions

Loaded questions include unverified and emotionally charged assumptions that a respondent might disagree with.   


How terrible is gluten for your health?  


Do you think gluten is healthy, unhealthy, or neither?  

More Tips: 

  • Try open-ended phrases like "Tell me about", "Describe a time when you..." "Explain when you..."
  • Don't inject your opinion into the question
  • Do not assign judgement to your question
  • Eliminate emotional words like love, hate, wrong, or right
  • Don’t always settle for the first answer you receive.  Dig deeper until you gain clarity. 


Become an Explorer!

You are not an expert, you’re an explorer. Remove any knowledge that you have about the subject, and observe.  Focus on the participant.   Ask what they want and learn about them. When you’re in a conversation, put aside judgement, stay quiet, and listen.  Ask better questions to improve the quality of your relationships, and everyday conversations.

Thank you Rina Jensen for the interview!

Read more from the CloudApp blog — Inside The Workflow

Join the Discussion