GOOD DATA FROM BAD QUESTIONS?
IMPOSSIBLE!
University of Wisconsin Cooperative Extension
Wisconsin Tobacco Control Board Monitoring and Evaluation Program
This document was produced through a grant from the Wisconsin
Tobacco Control Board, dated April 2002
The purpose of the following checklist is to help you construct
or choose clear, meaningful questions that will produce
useful evaluation results. Use this checklist to review
the quality of questions you use in evaluation instruments (surveys,
interview questions, focus group protocols, etc.). The checklist
addresses some common mistakes people make when constructing questions
for instruments (surveys, interviews, focus group protocols) to
collect evaluation information.
Before you write or select questions:
- Clarify what you want to learn from your
questions. Do your questions address what you really want
to know? Are you asking the question because you have always
asked it? Has it provided useful information in the past? If
you answered NO to any of these questions, revise the question
or cast it aside! Even if you have used the question over time
to collect trend data, consider supplementing your approach
with a more useful question.
- Know how you will USE the information.
How will you use information from each question? Have your analysis
plan in place before your start.
- Don't reinvent the wheel. Has someone
already developed good questions? Ask colleagues, look at published
literature, contact authors, or conduct an online search for
tools. If you're looking for tobacco-specific questions that
have already been pilot tested, check out our Build
Your Own Survey tool. Your regional Evaluation Specialist
may also have suggestions about where to find existing questions.
- Know your population. Clarify for yourself
how your sample may interpret the questions you're asking. Will
smokers see the questions differently than non-smokers? Will
older folks read the questions differently than younger ones?
This will help you avoid ambiguous questions. You don't want
respondents to think to themselves, "That depends on what you mean by smoke-free,"
or, "Do you want me to answer as a parent or as
a day care operator?"
- Be sure you pilot test your questions. Because
we cannot always see the hidden meaning in our own words, test
any evaluation tool with a small group of respondents, whether
it's a questionnaire, interview questions, or a focus group
protocol, to ensure the questions and response options will
make sense to your respondents.
As you select or write questions:
- Be specific. Is the
question specific enough for your population to answer?
- Ask whether your respondent
will have the knowledge to answer the question.
- Make sure the question is not "loaded"
or "leading."
- Avoid questions that
ask more than one question ("double-barreled"
questions).
- Avoid jargon and acronyms.
- Avoid double negatives.
- Avoid embedding response
options within your question.
- Make sure people can choose only one appropriate
response option. (Of course, this does not apply for
questions to which respondents can "check all responses
that apply.") For example, a question that asks one's religion
should not provide response options that include both "Christian"
and "Catholic." Ensure response options do not exclude
certain respondents from answering the question. To decrease
missing data and improve validity, it may be important to add
options like "other" or "none of the above"
to your list of response options.
- Provide reasonable
ranges of variation in your response options.
- Avoid questions likely to elicit socially desirable answers. This problem can arise
when respondents are not anonymous, when the question is posed
among peers, or the respondent does not trust the source or
possible uses of the survey.
- Test the question for age,
cultural and literacy appropriateness. Cultural appropriateness
extends beyond race and ethnicity to social class, rural vs.
urban residence and gender identity.
- Consider demographic
questions that will help you analyze the data in a more
meaningful way.
Back to checklist
Tips on writing good questions
Be specific: What are you asking people
to rate?
|
Not so good
|
Better
|
|
Please rate the following speakers as excellent, fair,
or poor:
- John Anders
- Ron Trudale
- LouAnn Marks
- Don Jacobs
|
How useful were the following sessions to your work
in tobacco control?
|
|
John Anderson: Winning strategies for policy work
|
Not useful
|
Some-what useful
|
Very useful
|
|
Ron Trudale: Working effectively
with youth
|
Not useful
|
Some-what useful
|
Very useful
|
|
LouAnn Marks: Building a strong coalition
|
Not useful
|
Some-what useful
|
Very useful
|
|
Don Jacobs: Evaluating your cessation efforts
|
Not useful
|
Some-what useful
|
Very useful
|
Back to checklist
Respondents should be able to answer
the question
|
Not so good
|
Better
|
|
Was the presenter a expert in her field? or Was the presenter knowledgeable
about the subject?
|
Did the presenter effectively answer questions from the
class?
|
Back to checklist
Avoid questions with more than one
answer
|
Not so good
|
Better
|
|
Please tell us if today's session helped you set clear,
reasonable goals for action with the coalition that are
within your control.
|
Please list three goals for your own action with the coalition
that you set today. Next to each goal, write one thing you
plan to do in the next month to progress toward that goal.
Three personal goals for action with the coalition
1.
2.
3.
My plans to act
1.
2.
3.
|
Back to checklist
Avoid "loaded" or "leading"
questions
|
Not so good
|
Better
|
|
Don't you think teenagers caught with cigarettes should
be fined in order to prevent them from smoking?
|
Teenagers caught with cigarettes should be fined.
- Strongly disagree
- Disagree
- Agree
- Strongly agree
|
Back to checklist
Avoid jargon and acronyms
|
Not so good
|
Better
|
|
Do you think the TFCWC has done an excellent, fair or poor job in informing
community members about the Quitline?
|
For community members: In the past year, did you
receive information about the Wisconsin Tobacco Quitline,
a toll-free telephone counseling service to help people
quit smoking? (Please check only one response.)
- Yes
- No
- I did not receive information about the Quitline
in the past year
|
Back to checklist
Avoid double negatives
|
Not so good
|
Better
|
|
Do you agree or disagree that teens should not be fined
for not obeying the local outdoor smoking ordinance?
|
Teens should be fined for violating the local outdoor smoking
ordinance.
- Strongly disagree
- Disagree
- Agree
- Strongly agree
|
Back to checklist
Avoid embedding response options within
your question
|
Not so good
|
Better
|
|
Do you agree or disagree that smoking should be banned
in public buildings?
|
Smoking should be banned in public buildings.
- Strongly disagree
- Disagree
- Agree
- Strongly agree
|
Back to checklist
"Anchor" your questions in specific behaviors
|
Not so good
|
Better
|
|
Do you emphasize leadership in your local FACT chapter?
|
How often do you review progress in leadership skills with
members of your local Fight Against Corporate Tobacco (FACT)
Youth Group?
- Annually
- Quarterly
- Monthly
- Weekly
- Never
|
|
Do you provide your employees with information on how to
quit smoking?
|
How often do you provide your employees with information
on how to quit smoking?
- Annually
- Quarterly
- Monthly
- Weekly
- Never
What information do you provide?
(list of possible pieces of information)
|
Back to checklist
Do response options make sense?
Unless you ask respondents to "check all that apply,"
response options should be mutually exclusive. Make sure choosing
one response doesn't preclude choosing another.
|
Not so good
|
Better
|
|
What is your religion?
- Christian
- Catholic
- Lutheran
- Jewish
- Episcopalian
|
What is your religion?
- Catholic
- Lutheran
- Jewish
- Episcopalian
- Islamic
- Other
- None of the above
|
Back to checklist
Response options should provide
reasonable ranges
|
Not so good
|
Better
|
|
What do you think is the effect on one's health of second-hand
smoke?
- Harmless
- Somewhat negative
- Very bad
|
Breathing secondhand smoke is...
- Very harmful to one's health
- Somewhat harmful to one's health
- Not very harmful to one's health
- Not at all harmful to one's health
|
|
Not so good
|
Better
|
|
What is your annual income?
- < $5,000
- $5,000-70,000
- $70,000 or over
|
What category best represents your annual household income?
- Less than $20,000
- Between $20,000 and $39,999
- Between $40,000 and $59,999
- Between $60,000 and $79,999
- Between $80,000 and $99,999
- $100,000 or more
|
|
|
|
Back to checklist
Avoid questions that seek socially
desirable answers
|
Not so good
|
Better
|
|
Interviewing a pregnant woman:
Do you smoke?
|
Interviewing a pregnant woman:
How often are you exposed to cigarette smoke?
- All day, every day
- A few times a day
- A few times a week
- Once a month
- Less than once a month
- Never
|
Back to checklist
Are questions appropriate to culture,
age and literacy level?
|
Who buys the groceries in your family?
|
Please circle the person who most often purchases
the groceries for your household.
- Head of household
- Child/teenager
- Grandparents
- Other relatives
- A neighbor
- Other
|
|
Questionnaire Design: Asking questions with a purpose