Why survey worksites at the local level AND at the state level?

Mary Michaud
State Coordinator, Local Program Evaluation, UW Extension
WTCB Monitoring and Evaluation Program
March 2002

Over the past few months, tobacco-free coalition staff have posed the following questions:

• Why did the Wisconsin Tobacco Control Board Monitoring and Evaluation Program complete a statewide survey of worksites, when many local coalitions are doing their own surveys? Isn't that redundant?

• Why can't MEP simply distribute the data to counties like they did with the results of the statewide municipal/county building survey?.

Good questions!

There are three principal reasons for conducting surveys at both state and local levels:

1. There are far too many worksites in Wisconsin to get a meaningful sample size for every county (it's cost-prohibitive)

2. Establishing statewide baseline data on worksite smoking policies means that the sampling and survey administration process must be uniform across the state, rather than done in distinct ways, county by county

3. The data from the statewide survey and the local surveys will be used in different ways

First, there are almost 50,000 Wisconsin worksites. The cost of survey administration prohibited us from drawing a statewide sample large enough in each county to generate meaningful data for that county. (Can you imagine what it might cost to survey 10,000 worksites and enter all the data into a database? This is the sample size for some of today's largest national health surveys.) The statewide worksite survey used a random sample of 2,200 worksites across the state. This seems like a lot of worksites, but the total list of worksites in Wisconsin (from which MEP drew the sample) was more than 40,000. In some counties, because of the random sampling, very few worksites (or perhaps no worksites) received a survey, and in counties where several worksites were sampled, the randomization may have resulted in an unrepresentative group of worksites for that county. Coalitions would not find that kind of data useful in establishing solid baseline data for the coalitions.

Someone may say, "How can you generalize about worksites across the state if you didn't even survey a worksite in my county?" MEP conducted a stratified random sample and had a good response rate. Using this method, we can be about 95% sure that the results we achieved fairly represent worksite smoking policies across the state, but not necessarily in your county. In other words, we are 95% confident that the patterns that we see across the state are not due to chance.

Both state and local surveys establish baseline data so that we can measure change over time. However, many coalitions used local surveys for a different reason: To actively identify worksites interested in assistance to develop their smoking policies. This is a key difference between the purposes served by state and local surveys.

Another difference was that coalitions used distinct survey processes to collect their data.

• Some coalitions telephoned worksites and others adapted the questions to a mail survey. Unfortunately, the myth that telephone surveys are more costly and more work than mail surveys remains prevalent, and some coalitions were willing to accept low response rates for their local purposes. (This is only a myth if you are concerned about getting a decent response rate--more than 60%--as mail surveys generate a much lower inital response rate and require significant follow-up mailings and sometimes even telephone follow-up to generate the 60% response rate.)

• Some coalitions used random samples of worksites while others surveyed only worksites with more than 100 employees. This was their target group, as they felt they could impact the most number of employees at one time by changing policies in these worksites.

• Some coalitions pared down the MEP questionnaire or added questions to it, making the instrument inconsistent across counties.

• Other coalitions defined worksites in a way meaningful to their own county context (e.g., housing contractors who worked out of their vehicles made up a sizeable number of one northern county's "worksite" population, and they were interested in that group of employees).

• Not every county conducted a worksite survey.

Because of these distinct approaches to the survey, the individual data sets cannot be pooled at the state level in a way that allows us to generalize about worksite policies across the state. If we compared data across these counties, we might as well compare apples and oranges, which might be tasty but wouldn't mean too much.To establish a baseline of worksite smoking policies across the state, MEP drew a stratified random sample (will open in a new browser window) and administered the questionnaire in a uniform way across the state. They needed to ensure a high response rate and reliable data so that they could generalize findings to worksites statewide. One strategy to ensure a high response rate is ensuring confidentiality of respondents. However, this means that MEP cannot name worksites when they report results. MEP is preparing to publish data from the statewide worksite survey by DHFS region, which may be a helpful reference for the local survey work you have completed.

That said, MEP would like to compare how local coalitions administered their own surveys, and your data can prove extremely valuable to helping understand worksite policies at the local level. Our experience assisting coalitions with worksite surveys has told us that the process has helped many coalitions solidify their understanding of issues among worksites and employees in their counties. To submit your data, go to our data submission instructions page, complete the brief Excel description of your survey, and follow the instructions to send it to MEP staff.

We hope this helps. The work you are doing at the local level is extremely important for your purposes, and you have not duplicated efforts at the state level.

Feel free to contact your regional evaluation specialist if you have any questions about MEP surveys or your own survey implementation, analysis or reporting.