Tips on Data Analysis
Overview
- Review
definitions
- Planning
your analysis: Purpose and questions to address
- Limitations
of the data, making generalizations
- What
are some ways to look at your data?
- Other
Resources
1. Review of
definitions:
You should not have to perform complex analyses of your survey data. You already
know most of what you need to produce meaningful information.
REMINDER: BEFORE YOU START ARCHIVE A COPY OF YOUR DATA. Saving a
copy of your data in another place (i.e. both on a floppy disk and on your
hard drive) before you begin your analysis is very important. If you make a
mistake in your calculations or accidentally erase a part of your data, you
will have the original untouched data to go back to.
Statistics: Making estimates about a whole population based on a
sample of the population. American Heritage Dictionary:
"...collection, organization, and interpretation of numerical data,
especially the analysis of population characteristics by inference from
sampling."
Population: In survey research, the "population" refers
to any group of people, organizations or groups you are studying. You can
have a "population" of worksites, restaurants, associations,
churches or schools. But you can also have a "population" of
older adults, teens, or pregnant women.
Sampling: Selecting members from a larger population. There are a
variety of sampling techniques. Please see the tips on implementing a worksite survey
for a more in-depth discussion of sampling.
Response rate: Number of completed surveys / Total number
attempted
Descriptive statistics: i.e. those used to describe our data (See
UWEX Publication, Analyzing
Quantitative Data)
Frequency: Count of a particular response.
Percentage: Proportion with a particular response or
characteristic within the total group of interest.
Frequencies and percentages are generally used for questions with
categorical responses, such as number of men and women, "not
important, somewhat important, very important," or number of worksites
that have formal written smoking policies.
Measures of central tendency are generally used with responses
that are given in numbers rather than categories, like the mean number of
seats in smoking sections, median number of employees in worksites, or the
mode, or most common, professional title of respondents to the worksite
survey. These describe where the most common characteristics of your data
lie.
Mean: Average, or Sum of all relevant response values / Number of
responses for that question
Median: Midpoint of responses
Mode: Most common answer among all responses
In the following sections, we will use some examples to illustrate how
to use these descriptive statistics in looking at your data.
2.
Planning your analysis: What is the purpose of the survey?
What questions do you want to address?
Before moving on to look at some examples of how to use descriptive
statistics to analyze your data, it is important that you clarify the
purpose of the survey. What questions would you or other stakeholders
like the survey information to address?
Be explicit. For example, if you define the purpose of your survey
as collecting baseline data, you can be clear about the questions you need
to ask to save time on your data analysis and have it give you the
information you require.
Step 1: Begin by recording some standard information about the
survey you are conducting. This will help you focus your analysis and will
serve you well once you begin to communicate findings. The following is an
example of the kinds of documentation you should record for any survey you
conduct. Its basic elements include:
|
Purpose of survey
Situation in your County that compelled us to conduct a survey
Geographic region where the survey was conducted (this may not
look important at first, but can be very helpful if anyone leaves or
enters the survey process or for those who come after you in the
coalition)
Methods
Method to collect information
Source of information
Instrument
Dates interviews conducted (be clear as to when you started the
survey - especially important with baseline data collection)
Sampling frame (the source of your worksite or restaurant names -
the "master list" that represents the population of worksites
or restaurants)
Sampling procedure (how did you draw your sample? maybe you
surveyed everyone - i.e. a census sample, or oversampled
if you were interested in the top 10 employers in your county like
Kenosha is so you oversampled large employers
since this group can affect a large number of people). Please see the notes on implementing a worksite
survey for a more in-depth discussion of sampling.
Sample size
|
Below is a completed example for the fictitious
"Wick County."
This is also available to download as a word document. A blank version is
also available for download for you to fill in and use for your own survey
work. This form will help you be explicit about the details of your survey
and help you to focus your analysis.
|
Baseline data collection:
Worksite smoking policies in Wick County,
WI
|
|
Purpose of survey:
|
Assess current status of worksite smoking policies in Wick
County. Plan follow-up survey
in June 2003.
|
|
Situation in Wick
County:
|
High suspected level of exposure to environmental tobacco smoke at
work. No systematic information about status of worksite smoking policies
in Wick County.
Need information to target programming of Wick County Tobacco-Free
Coalition.
|
|
Geographic region:
|
Wick County, Wisconsin
(population 143,000)
|
|
Methods
|
|
|
Method to collect information:
|
Telephone survey
|
|
Source of information:
|
HR managers. If HR manager n/a, another knowledgeable of worksite
smoking policies
|
|
Instrument:
|
WTCB Monitoring and Evaluation Program
Worksite Questionnaire, 32 questions (developed May 2001)
|
|
Dates interviews conducted:
|
June 20 - August 31, 2001
|
|
Sampling frame:
|
Wisconsin Department of Development list of
employers (updated 2000), Wick County Business Association list of
businesses and non-profit organizations with more than one employee
(updated 1999). After duplicates removed, 4,753
worksites.
|
|
Sampling procedure:
|
Simple random sample. Oversampled businesses
larger than 200 employees.
|
|
Sample size:
|
375 worksites. Sampled all 26 businesses over 200 employees.
|
Step 2: Depending on the survey you are conducting, begin by
recording in detail the questions you would like to answer. If you are
using the MEP questionnaires, we have provided a condensed list of the
survey questions for both the worksite
and restaurant surveys as Word documents. This may help you identify
the questions more easily than looking through the survey instrument. If
you are working on another survey, it may help to "condense" your
questions into one or two pages so that you can more easily see your
potential analysis strategies.
Step 3: Prioritize the questions in order of importance to achieving
your objectives. In planning your survey, you may have identified a group
of people - your coalition members, a partner hospital or clinic, the
chamber of commerce - who might be interested in the results. You talked
with them about the kinds of things they would like to learn from the
information you collect. This will give you ideas for focusing your data
analysis.
Step 4: For each question (you might begin with your top five
questions), choose the descriptive statistic or statistics you might use to
answer the question. So far, you might have a table of these items:
|
Questions we would like to answer
|
Priority ranking (1 = most important)
|
Descriptive statistic
|
Coming up with a clear idea of what you want to get out of the analysis
will help you organize your data.
Step 5: Before you go any farther, archive a copy of your
electronic data file in two places - on a disc and on your hard drive.
When analyzing your data, make a copy of your original data file!
3. What are
the limitations of your data? Can you generalize from your data to the
broader population of worksites in the county?
For now, consider the following limitations as you begin to look at your
data:
- Sampling frame:
What were the possible shortcomings of your original list? Were
worksites or restaurants systematically excluded? Talking to the
person who "keeps the list" can help clarify some of these
issues.
- Sampling bias:
If you are conducting a household survey by telephone, did you miss
households who don’t have a phone?
- Selection bias:
How did you conduct your sample? Was it truly random?
- Response bias:
How was the survey administered? Did you provide training to ensure
questionnaires were administered consistently? If more than one person
conducted the survey, did they use similar approaches and language?
- Did your
administration exclude businesses that may only be open on weekends or
evenings?
- Was there some
extraneous factor that influenced whether your survey excluded
particular businesses? For example, was there a large storm recently
that had all of the insurance companies busy handling claims, so no
one was available to take your call or were recent layoffs keeping the
HR person too busy to have time to speak with you?
You should note the possible limitations of your survey and consciously
report them when you are ready to interpret your results. Noting and
understanding possible sources of bias may help you look at your survey
information more realistically. It can help save you from coming up with a
"liberal" interpretation of the data that doesn't represent your
population.
4. What are some ways to look at your data?
We will work through the following example to give you a better idea of how
to use descriptive statistics to answer questions you may be asked about
you data.
Worksheet: Choosing descriptive statistics
The County Board has asked you to present the results of your worksite
survey.
They have three questions:
1. Who was surveyed and when? How many worksites responded? Look at
the sample results below. How would you present this information?
|
Selected results:
|
|
Response rate: 251 completed surveys (67%)
|
|
Characteristics of worksites:
198 worksites (79%) had fewer than 50 employees.
|
|
Indoor worksite smoking policies:
52% of all worksite smoking policies did not allow smoking anywhere
inside worksite buildings, 38% said smoking was allowed only in
designated areas, and 10% said smoking was allowed anywhere inside
worksite buildings.
35% of worksite smoking policies did not allow customers or visitors
to smoke anywhere inside worksite buildings; 21% said the smoking policy
allowed customers or visitors to smoke only in designated areas; and 33%
said their policy allowed customers or visitors to smoke anywhere inside
worksite buildings. 11% of respondents didn’t know how their smoking policy applied to
customers and visitors.
|
|
Cessation benefits:
200 (80%) of worksites offered health insurance to employees. Of
those, 150 (75%) said they had health insurance plans that offered some
for of cessation benefit, and 50 (25%) did not know whether their plans
offered cessation benefits.
|
Frequencies (counts)
Counts may be enough to use on their own. Or they may be used as the base
(denominator) for other calculations. Look at the sample
data above. The "response rate" and "characteristics of
worksites" results are expressed both in terms of counts and
percentages.
Response rate: 251 completed surveys (67%)
Characteristics of worksites:
- 198 worksites
(79%) had fewer than 50 employees.
Why do we sometimes report both counts and percentages?
We report both to give the reader a sense of the "real number" of
surveys or worksites, and we report percentages to give a sense of what
this number means in proportion to the total number we surveyed.
2. What is the relationship between size (number of worksite employees)
and whether a worksite has a smoking policy?
Q: What items would you need to examine to give the County
Board an answer?
Characteristics of worksites:
- 198 worksites
(79%) had fewer than 50 employees. (This is an interesting result in
and of itself. One of the first questions you want to answer is the
makeup or nature of the sites. The majority of worksites in Wisconsin
have less than 50 employees. It is interesting to see if your county
fits this and it can also help focus your program and analysis).
- Of 26 businesses
over 200 employees, 6 did not respond. 20 (8%) had more than 200
employees. (We oversampled larger
businesses because exposure to ETS affects
a large group there).
- 13% had between 51
and 199 employees. (This puts our sites into 3 tiers of sizes and
answers one portion of the question).
You might construct a table that lists these three categories of worksite
size along the top and their smoking policy status along the side:
|
|
Number (percent) of
worksites
(N = 251)
|
|
Smoking policy
|
Less than 50 employees
|
50 - 199 employees
|
200 or more employees
|
|
No smoking allowed indoors
|
|
|
|
|
Smoking allowed only in designated places
|
|
|
|
|
Smoking allowed anywhere inside
|
|
|
|
This table (size of worksite vs. kind of smoking policy) represents the
answer to question 15 on the worksite survey. This is called a cross tab
and can be a very useful tool. There are some models of this presented in
the Wisconsin Medical Journal articles listed in the Other
resources section.
Q: (Adding to the example above) What if employees are allowed to
smoke outside on worksite grounds?
A: Many worksites do not allow smoking inside the building, but do
allow it in designated places outside. Simply add this as another option
under smoking policy in the table above.
Q: Why would you hesitate to give the board a mean number of
employees across all worksites?
A: Look at the sampling technique. Large businesses are oversampled: "Sampled all 26 businesses over 200
employees." You can't come up with mean if you've oversampled
large businesses because it will be skewed towards the larger groups.
Q: What are some other options to present this data?
A: You could present the median or midpoint of the number of
employees.
Remember to be aware of the limitations of your data in using it to target
your program. Think of your strategy in light of these limitations.
3. How many worksites offer cessation benefits through a health plan?
This directly relates to question 23 on the worksite questionnaire and may
be of great interest to local stakeholders. If you can't smoke in the
workplace, but no cessation assistance is offered, it brings about an
interesting catch 22.
Caution: Select the appropriate denominator to calculate your percentage.
Percentages:
When using percentages, the following steps should help you avoid making
errors in your analysis:
Tip 1: Use the correct base, or denominator. Some survey questions
will not apply to everyone, and your total number of responses will be less
than the total number of "completed" surveys. Let’s
look at an example:
Response rate: 251 completed surveys (67%)
Cessation benefits (Question 21 on worksite survey): (80%) of
worksites offered health insurance to employees. Of those, 150 (75%)
said they had health insurance plans that offered some form of cessation
benefit, and 50 (25%) did not know whether their plans offered cessation
benefits.
So, you need to calculate percentages based on those who said they offer
health insurance (201 or 80% of 251), not on the entire 251 worksites that
completed the surveys. Using the entire response rate won't give an
accurate number.
|
Number (percentage ) of
worksites
who offer health insurance
N=251
|
|
|
Yes
|
No
|
|
Does worksite offer health
insurance?
|
201
(80%)
|
50
(20%)
|
|
Number (percentage) of
worksites with health
insurance plans featuring cessation benefit
N=201
|
|
|
Yes
|
Don’t
know
|
|
Do health plans offer
cessation benefit?
|
150
(75%)
|
50
(25%)
|
Q: If you don't include all of the responses (i.e
251 instead of the 201) isn't this also deceiving?
A: All you need to do is clarify that 50 of the worksites don't
offer health insurance.
Tip 2: Avoid averaging percentages. Say you have calculated the
percentage of worksites in 4 different zip codes who do not allow smoking
anywhere inside worksite buildings. You might be tempted to add all those
percentages and calculate an average percentage for worksites in those 4
zip codes. However, this can introduce error, and it is sometimes
significant.
To calculate averages, use worksite sizes for all worksites in those 4 zip
codes, and calculate the percentage.
For more information on working with percentages, see the UWEX publication,
Analyzing
Quantitative Data
Q: Are there any standards we should be comparing our data to? How
can we find out how our data compares to the rest of the state? rest of the country?
A: We will let you know as results come in from other coalitions
with finished data sets and as we work on building the minimum data set for
the state. Unfortunately, we have not come across county level data from
other states to compare with yours. We will be happy to share anything we
do come across.
5. Other
resources:
(will open in a new browser window)
(will open in a new browser window)
Two articles from the most recent issue of the Wisconsin Medical Journal
should help you see examples of how these descriptive statistics are used:
(will open in a new browser
window)
|
