Memo 1
Rarely do Extension bulletins get evaluated for impact. For example,
our bulletins for farmers have the best information in the world, but
do farmers actually change their practices after reading them? We hope
so, but we don’t really know.
The Parenting the First Year series of
age-paced, instructional newsletters for new parents are different.
They are not just an Extension bulletin, but a fully-fledged educational
program of your Extension office. In a quasi-experimental study, the
first year series was evaluated for impact, and found to be effective.
Parents who receive Parenting the First Year
have attitudes that are significantly less like child abusing parents,
compared to parents who do not receive the series, and they report actually
spanking or slapping their babies fewer times per week.
This memo describes how to do an evaluation of the Parenting
the First Year newsletter series. If you haven’t done this
kind of impact evaluation before, don’t worry. In practice, it
is easier than this letter full of procedures might make it appear.
I will have supportive materials to help you through any difficult steps.
Included in this memo you will find
a:
Purpose of this evaluation
- It shows people that we are hard-nosed and self-critical, making
sure our efforts are really effective.
- The results of an evaluation become the content for educational
presentations you can make in your county.
- And a local evaluation can be used to generate continued financial
support for the newsletter project.
Evaluation questions
This evaluation is designed to answer the following questions:
- Do parents use the newsletters?
- How useful do parents find the newsletters, compared to their other
sources of child
rearing information and advice?
- Do parents report changing their child rearing behaviors in key
areas that (a) are
important for child development and (b) have been emphasized in the
newsletters?
- Can parents name specific changes they have made in their child
rearing behaviors,
as a result of reading the newsletters?
- Are parents in “risk” categories, as compared to parents
not at risk, more likely to
report benefits from reading the newsletters?
Methods
Data collection will be by questionnaire survey. The research design
will be a single group, cross-sectional design (data is collected only
once). The sample will be parents who have received the Parenting
the First Year series for at least six months (i.e. child age
6 to 12 months).
The survey questionnaire...
...is ready for your use, and a copy
is included with this memo. It has 32 questions fitting on four
pages (so it can be copied on two sheets of paper).
The questionnaire includes two improvements over the forms we have
used for the earlier newsletter evaluations. First, we include more
opportunities for parents to tell us how they have changed their childrearing
practices due to the newsletters. This should help us establish the
validity of the self-reports of behavior change (if people can name
a specific, concrete way they have changed, then their report of having
changed becomes believable). These quotes will also make for very
readable reports.
The second innovation is the inclusion of two "lie scale"
items. These are items 18 and 21 on the questionnaire, in which parents
are asked if reading the newsletters caused them to change how they
diaper their baby, or change how they wake their baby in the morning.
Our newsletters give no advice on either of these topics. By including
these items, we will be able to see how many respondents have adopted
a "response set" (answering all questions the same way, instead
of considering each carefully) or have adopted a "desirability
bias" in their responses (in which they try to give positive answers,
rather than strictly true answers). We will be able to report the percentage
of respondents who answer "yes" to both these items, and then
delete them from our analyses. This is a level of rigor we have not
attempted in our questionnaire surveys in the past.
A note on terminology
Many people use the terms “survey” and “questionnaire”
as if they have the same meaning. But this is not true, and I’d
like to alert you to my using these words with their correct meanings.
There are many kinds of surveys, for example a questionnaire survey,
interview survey, and a survey of public records. Only the first kind
uses self-report questionnaires, as we will in this evaluation project.
These questionnaires are sometimes called “survey forms.”
That is OK. But one of these “forms” or questionnaires cannot
be called a “survey.” The survey is the overall process
of sending out forms to a large number of respondents (i.e. to a sample).
For example: if you conduct a questionnaire survey and get 100 questionnaires
returned, you have 100 questionnaires but only one survey. You would
be incorrect to say you got 100 surveys back.
If you evaluate the Parenting the First Year
series, you will do so with a single survey.
Procedures
If we do this evaluation, you and I will be co-investigators. We will
write the report as co-authors. I will be responsible for some parts,
and you for others.
Local project organization
While we are co-investigators on the evaluation, the distribution
project for Parenting the First Year
is yours. Deciding if you want to have partners, and who they
are, is your decision (most Family Living Agents form a partnership
for this project with a maternity hospital and/or health department,
and one or more service clubs, most commonly Kiwanis). I will
normally not attend local meetings of your group, but will communicate
only with you. People in your county will come to see this project
as yours, certainly not mine; and they will see me (if at all)
as a resource you bring to the county.
Distributing the questionnaire
You will be responsible for copying and distributing the questionnaire
(and for other materials like return envelopes, postage, etc.).
I recommend you include the questionnaire with the final issue
of Parenting the First Year, along
with a stamped and addressed return envelope. When I say “you
will be responsible,” I don’t necessarily mean you
personally. I mean your Extension Office budget and / or your
organizational partners (hospital? Kiwanis Club? Health Department?)
will need to cover these costs.
Why do I suggest waiting for the final issue (month 12)? The reason
is what researchers call the dose-response
relationship. The greater the dose of an intervention,
the greater the response we expect. Think, for example, of research
on new drugs. If we give some people twice the dose (twice as
many pills), then we expect a larger impact. The same is true
of the newsletter series. The more issues parents receive, the
greater the likelihood that we will be able to document an impact.
On the other hand, some of you will have deadlines or impatience
that will make you want to avoid waiting that long. So while I
advise you to wait until parents have received all 12 issues of
Parenting the First Year, I am willing
to help you evaluate the series after parents have received as
few as 6 issues (that means the newsletter issue for months 6-7).
That is my guess on the minimum number of issues parents must
receive before we have a decent chance of documenting an impact.
Additionally, you may evaluate all parents who have just received
issues 6 through 12.
Sample size
I will normally recommend at least 100 participants. This is enough
so we can begin to do some sub-group comparisons. For example,
we can begin to contrast the answers of first-time parents versus
experienced parents, and can produce a third group of parents
who live in difficult (“at risk”) circumstances, such
as low income or single parents. To get a good estimate of effect
in any of these sub-groups, we need about 25 respondents in the
group. (With that many, even one “odd ball” or person
who lies on their questionnaire cannot change the average response
a great deal.) Usually we can form these sub-groups with at least
25 in each group if we have 100 or more questionnaires returned.
You may need to collect data for several months to get 100 returns.
How long will it take you to collect enough questionnaires? Let's
use 50 percent as a rough estimate of your eventual response rate.
Let’s also suppose that your county averages 600 births
per year (that would be 50 births for each month). So, at an expected
50% response rate, you should expect to get about 25 questionnaires
back each month. Thus, it would take four months to get 100 questionnaires
returned.
If you have data on more than 100 families this is even better,
up to about 200 or 300, after which additional data doesn’t
add much to our accuracy. I will consider supporting the evaluation
with fewer than 100 returned questionnaires, but you and I need
to agree on a specific number in that case (before we begin).
I have approved evaluations with as few as 50 completed questionnaires
for counties with few births.
Response rate
When it comes to your sample, two different numbers are important.
I have just told you the overall size we need in order to perform
some minimal statistical comparisons. The other number is response
rate, meaning the percentage of possible respondents who actually
returned completed questionnaires. Even if your sample size is
huge, say 10,000 parents, if the response rate was small (say
25%) then we would question whether we could believe your results.
We would question whether your findings really represent
the population of parents. We would wonder if those who
didn’t answer might have answered differently. We might
think that your findings apply to the 25% who answered, but that
you couldn’t fairly generalize your
findings to all the parents receiving our newsletter series.
This would make your findings fairly useless.
Therefore, do what you can to increase your response rate. Included
with this memo is a copy of a cover letter you can use with the
questionnaire. It is very brief (long cover letters discourage
people from reading any further). And it explains enough about
our intentions so people might understand the importance of their
responses. It is in Microsoft format so you can download and edit it.
You must keep very careful records on how many questionnaires
are mailed out. Equally important, keep records on how many questionnaires
are not delivered. For example, some people will move, and the
newsletter series might not be forwarded. Since they did not receive
the evaluation questionnaire, we do not have to include them in
the calculation of response rate (this will make our response
rate both higher and more accurate). One thing you can do that
has helped a lot in earlier projects is to send a 3-by-5 inch
postcard to everyone in the sample about 3 or 4 days after they
receive the questionnaires. It can say something simple like this:
|
Dear Parent,
We sent you a questionnaire a few days ago about the Parenting
the First Year newsletter series.
If you have already returned the questionnaire, THANKS! If not,
could you still do so? It takes only about 10 minutes. Your answers
help us learn how to be more helpful to future parents.
Thanks again,
Your Name
Your County Extension Office
|
This postcard does two things. First, it reminds people to answer
the questionnaire. My best estimate is that this will increase
your response rate about 14%. The second thing you accomplish
is to find out how many people did not receive the questionnaires
you mailed, which allows you to calculate your response rate more
accurately (and upward!). You accomplish this by sending each
postcard with first class postage and stamping the words “Return
Service Requested” below your return address. For every
returned postcard, you know that our actual number of delivered
questionnaires was one less than anticipated.
Data entry
You will be responsible for entering the data onto a computer,
but I will help. Memo 2 includes the materials and instructions
to do this. You can enter the data into an Excel spreadsheet. When data
entry time arrives, look at Memo 2 for more specific instructions.
It does not amount to a large amount of time. With 100 questionnaires,
I estimate data entry will take about 2 to 3 hours. You can send
us the data as an attachment to an email message!
I will do the statistical computing…
… and will send you a paper version of the printouts.
I will also keep a copy, so we can easily discuss them if the
need arises. Memo 3 provides some brief info on how to read the
printouts. Some Family Living Agents find this part easy, and
others hard. I will provide whatever level of assistance you need.
We can co-author the report
A copy of a model report is included
with this memo for our use. I will ask that you take the first
turn at modifying the standard report to fit your results (Memo
3 provides instructions on this part of the process). Mostly,
you will just insert your own numbers and change a few sentences
to fit your findings, and then I will check your changes and add
my own. I consider it essential that you learn to find your results
in your printouts, and see exactly how they become sentences of
conclusions, because you (and not I) are most likely to have to
answer questions like “how do you know that?” I will
help you learn to do this, you will enjoy it, and you will really
know your data! I promise!!
My office can handle the final report preparation (including
a power point version for you to use in giving presentations). Memo
4 also includes press releases for your use. This evaluation report
can produce a variety of local teaching opportunities in which
you present not just “research-based information,”
but the results of your own local research.
|
Before You Begin This Project
If you wish to conduct this evaluation, please let us know. Send an
email to let us know when you expect to begin distributing questionnaires.
Here is how to reach us:
Email: dariley@wisc.edu or costergr@wisc.edu
We will be happy to answer any questions you may have. We think this
is a terrific project, and we will be excited to work on it with you!
|
