 |
|
|
|
||||||||||
 |
|||||||||||||
|
Outcome Measurement Past questions and answers from Harry Hatry from live broadcasts aired by the Learning Institute for Nonprofit Organizations collaboration* Other Q&A topics:
Q1) Are there low cost, user-friendly computer programs available that will help organizations get started in setting up Outcome performance measurement? A. United Way's Outcome Measurement Resource Network recently compiled descriptions of products from nine vendors. A 21 page description (Adobe pdf document) of these Outcome Measurement Data Systems can be found on the Outcome Measurement Resource Network at: http://national.unitedway.org/outcomes/publctns.htm#R Five of the products are software products (Client Management System, Client Tracker, Outcome Results System, Tailored Outcome-Based Evaluation, TARGETTrac) and four our web-based systems (CAREeval, Online Data Manager, Outcome Toolkit, Tapestry) Outcome Measurement Past questions and answers from Meg Plantz from live broadcasts aired by the Learning Institute for Nonprofit Organizations collaboration* Q1. What kind of outcomes might make sense for a reading program for kindergartners from immigrant families whose native language is not English that are in ESL (English as a Second Language) classes - where senior adults and children get together once a week for 35 minutes to increase literacy and instill a love for reading? A: You already have identified the intended outcomes: the kindergartners' literacy skills increase and they have a love for reading. The next question is whether it seems reasonable that in 35 minutes a week, the program's activities can lead to these outcomes. If it does, then you're on to the challenge of measuring literacy and love for reading. Q2. If our caller records indicate that 28% of our caller's needs were unmet when looking for temporary shelter, can this be turned around and 72% of our callers were successful in finding shelter. This then becomes an outcome? A. If you are an agency that assists individuals who lack housing and what you want to be able to say about them as a result of your services is that they secure temporary shelter, then that is indeed your intended outcome. Further, if 72% of these individuals report success in finding shelter, then you have achieved that intended outcome with 72% of your clients. (Note that if you only have data for 28% who said their needs were not met, it may be problematic to assume that the other 72% were successful. You need to make that decision based on what you know about how the data are collected and how accurately they measure the intended outcome.) Q3: The HTFM example seemed to still just be counting numbers (pounds of food distributed). How are they measuring "full food security" for individuals? A: The video case study featuring the Hunger Task Force of Milwaukee helps point out why it is so important to specify a program's clients or target group as part of the process of identifying outcomes. HTFM does not serve individuals directly. Its "clients" are food pantries in the area, and one of the things HFTM wants to be able to say about them is that they have enough food to distribute to meet the needs of their own clients. Thus, that food pantries with large client loads have more pounds of food to distribute as a result of HTFM efforts is a measure of one of their intended outcomes, just as an increase in the number of calories consumed by hungry families could be a measure of an intended outcome of one of those pantries. Another outcome HTFM wanted to achieve was that food pantries would pack more nutritionally balanced food bags. After they trained food pantry staff on this topic, the pantries began putting more balanced assortments of food in bags they distributed, indicating this outcome also was being achieved. Q4. My agency is Recreation/Social Service based. The only outcomes we've identified deal with (1) awareness of social service programs and (2) actual participation in social service activities. How can we improve or expand on these? A: Without knowing more about your program, it's hard to be sure of an answer that is really on target. If you run social service programs and are doing things to make the public aware of your programs (advertisements, flyers, posters, etc.), these "things" are not really services for participants. They are outreach activities to draw participants into the programs. In this context, awareness of social service programs isn't an outcome (which doesn't mean it isn't important!). On the other hand, if you're running a teen recreation program and some of your activities are intended to familiarize participants with programs offering other services they need, then awareness of these programs may well be an outcome, somewhat analogous to the outcome of many information and referral services. Participation in an activity usually is not an outcome, because just showing up doesn't mean that any benefit or change takes place, although there are exceptions. To identify outcomes for each of your programs, define clearly who is participating in the program and what services you provide for or with them. Then describe what you want to have happen for them as a result of their participation: they think differently, know more, can do more, behave differently, change their situation, etc. The tips on page 24 of your participant packet may help you with this task. Q5. What about situations where clients outcomes and agency outcomes are different? E.g., a client may want to eat a meal. The agency may want to increase the skills and tools to make the client more self-sufficient. A. The first question is whether the outcomes are mutually exclusive (the battered women's shelter wants women not to return home; the women want to go home but not be battered again), or instead represent different time horizons or different emphases. In the example you cite, if the mission of the organization is to increase self-sufficiency of individuals that are experiencing hunger, an appropriate initial outcome meeting participants' immediate need would be that participants have adequate food for a defined period of time. Once that basic need is met, longer-term outcomes might be that participants gain skills and then obtain employment as part of becoming more self-sufficient. If part of the intake procedure involves telling potential participants what the program does and what outcomes it hopes will result, and they want something different, then they can decide that this is not the appropriate program for them. Q6. How do you measure the outcome? If the participants are done participating in whatever program, how do you know if their knowledge, attitudes, behaviors have changed? A. After the challenge of figuring out the outcome comes the challenge of measurement. Questions here include: A: You already have identified the intended outcomes: the kindergartners' literacy skills increase and they have a love for reading. The next question is whether it seems reasonable that in 35 minutes a week, the program's activities can lead to these outcomes. If it does, then you're on to the challenge of measuring literacy and love for reading. Q4. My agency is Recreation/Social Service based. The only outcomes we've identified deal with (1) awareness of social service programs and (2) actual participation in social service activities. How can we improve or expand on these? A: Without knowing more about your program, it's hard to be sure of an answer that is really on target. If you run social service programs and are doing things to make the public aware of your programs (advertisements, flyers, posters, etc.), these "things" are not really services for participants. They are outreach activities to draw participants into the programs. In this context, awareness of social service programs isn't an outcome (which doesn't mean it isn't important!). On the other hand, if you're running a teen recreation program and some of your activities are intended to familiarize participants with programs offering other services they need, then awareness of these programs may well be an outcome, somewhat analogous to the outcome of many information and referral services. Participation in an activity usually is not an outcome, because just showing up doesn't mean that any benefit or change takes place, although there are exceptions. To identify outcomes for each of your programs, define clearly who is participating in the program and what services you provide for or with them. Then describe what you want to have happen for them as a result of their participation: they think differently, know more, can do more, behave differently, change their situation, etc. The tips on page 24 of your participant packet may help you with this task. Q5. What about situations where clients outcomes and agency outcomes are different? E.g., a client may want to eat a meal. The agency may want to increase the skills and tools to make the client more self-sufficient. A. The first question is whether the outcomes are mutually exclusive (the battered women's shelter wants women not to return home; the women want to go home but not be battered again), or instead represent different time horizons or different emphases. In the example you cite, if the mission of the organization is to increase self-sufficiency of individuals that are experiencing hunger, an appropriate initial outcome meeting participants' immediate need would be that participants have adequate food for a defined period of time. Once that basic need is met, longer-term outcomes might be that participants gain skills and then obtain employment as part of becoming more self-sufficient. If part of the intake procedure involves telling potential participants what the program does and what outcomes it hopes will result, and they want something different, then they can decide that this is not the appropriate program for them. Q6. How do you measure the outcome? If the participants are done participating in whatever program, how do you know if their knowledge, attitudes, behaviors have changed? A. After the challenge of figuring out the outcome comes the challenge of measurement. Questions here include: Exactly what are we going to measure (e.g., if an intended outcome is "parents are knowledgeable about children's development," just what will we observe, count, weigh, or otherwise measure to tell us whether we've achieved this outcome)? As programs make these decisions, they can identify procedures they need to put in place to make the data collection possible. For example, if it is important to follow up with participants a year after they complete the program, then the program must be sure to get contact information while participants are still in the program. It also may want to verify the currency of the information after 6 months to minimize contact problems when the time comes for data collection. Q7. What is the most common outcome tool used? Please list some tools other then surveys. A. Surveys, used to obtain data through in-person or telephone interviews, are the first measurement tools most people think of. Other common tools are record extraction forms, used for pulling data from existing records; observation guides, used by trained observers to record data about behavior or environments; questionnaires that individuals fill out at the program or return through the mail; and tests, measuring physical changes or attainment of knowledge or skills. Q8: Can you give suggestions on how to control for the other factors that contribute to behavior change? In other words, knowledge alone may not make a difference? A: The methodology of outcome measurement does not control statistically for the effect of factors other than the program that may influence change (i.e., no control groups). However, programs are encouraged to think about what those factors might be. For factors that the program can influence, it may seek to expand its activities to address that factor (e.g., conduct activities that address attitudes and/or skills in addition to knowledge). For factors that the program can't influence itself, there may be other programs that can help. For example, if parents know they should read to their children and want to do so but don't have age-appropriate books, another program that distributes children's books can help address the access factor. A measurement approach is to collect information concerning those other factors for each program participant and then sort outcome findings based on that factor. For example, if access to public transportation may influence whether someone acts on their new knowledge, the program could determine how close each participant lives to a bus stop. If data analysis shows that those who live within 6 blocks of a bus stop follow through more often that those who live more than a mile away, this indicates that access to public transportation does in fact influence the outcome. To learn more about factors that may influence outcomes, programs can ask participants what factors that they perceive to contribute to or interfere with achieving an intended outcome. Q9. How does one measure outcomes when trying to influence embedded, systemic problems? E.g., the goal is: "Eliminating negative media images of girls." The problem is the media and marketing system makes billions in profits from doing business this way. A. The first step is to identify the intended outcomes you're going to try to measure, which are not the same as the goal. Your example goal is huge, and as you suggest, there is a lot working against it. An outcome, however, is a change in target individuals or groups that a program can reasonably expect to influence. Out of this huge goal, what are some key targets groups that a program really has a shot at influencing and what change could a program plausibly bring about? If this is a local program, maybe it decides to start with one newspaper in the community as the focus of attention and wants it to be true that in the coming year, the paper gives girls' sports as much coverage as it gives boys' sports. If the program thinks that local advertising influences the newspaper, maybe an intermediate outcome (one that helps bring about the longer-term outcome) is that major advertisers express their concern about coverage of girls' sports in writing to the managing editor. If the program thinks public recognition offers leverage, perhaps another intermediate outcome is that local girls' sports teams designate "Daily Bugle Nights" where they invite sympathetic reporters from the paper, feed them well, introduce them to the cheering crowd, and present them with plaques touting them as "Champions of Tomorrow's Women." Once the program has identified a series of target groups and outcomes it believes can lead to the desired change at the local paper, it then can develop specific strategies for making these outcomes happen. It also now has a manageable outcome measurement task. Q10. How can we manage and account for the difficulty of constructing measurable outcome indicators? That is, how reliable can measuring meaningful behavioral and community outcomes actually be? A. Identifying outcome indicators that are both meaningful and measurable can be challenging, and many programs modify or replace their initial indicators as they gain experience with outcome measurement. Locating or developing valid and reliable measurement methods also is a challenge. However, researchers in the social sciences have many tools and approaches that can be applied to the task to help produce quality data on individual change or performance. The challenges are even greater at the community level, and this is less-charted territory. In any measurement task, a basic question for programs and funders is: Given what we are trying to measure and how we plan to measure it, how much confidence will we have in the data we are getting? How much credence will we give the data in making resource allocation and program change decisions? Q11. If an outcome is not being achieved to the level that is satisfactory, how do you begin to analyze inputs and activities to see what to change and what to keep? A: After the challenge of measurement comes the challenge of using outcome findings to benefit the program. A few steps to take are: Display the number and percentage of participants achieving each outcome directly on your program logic model diagram. This helps you see if there are large drops in the percentage of participants achieving outcomes as they move from initial outcomes (e.g., increased knowledge) to intermediate outcomes (e.g., changed behavior) to longer-term outcomes (e.g., altered condition). Q12: As programs initiate an outcome management process, measurable outcomes and/or data results may be somewhat slim. Agencies cannot make valid program evaluations/modifications as a result, yet are asked to. A: An initial concern that the program's outcomes are not measurable is common, but as programs work through the process, nearly all find meaningful ways to measure the extent to which they've been successful in achieving their outcomes. In the early stages of outcome measurement, programs focus on modifying their measurement systems to assure they are collecting reliable and useful data. Once they are confident of the quality of the outcome data, they find that the data, even when slim, provide insights for staff engaged in program improvement. Q13: The comment was made on how the strategic planning process compliments the program outcomes process. Could you elaborate on this? Which process takes on more importance? A: Much of what happens in a nonprofit organization revolves around the strategic planning process. This is one reason that the Learning Institute for Nonprofit Organizations selected Strategic Planning as the first program in its certificate series. Outcome measurement is a very useful tool in the larger strategic planning process. Information on how well a program is achieving its intended results for participants is valuable input for the plan, helping identify programs for expansion, improvement, or elimination, for example, and suggesting where resources might be targeted for greatest impact. Outcome-focused strategic planning is especially powerful. Here, an organization identifies the individuals or groups that it seeks to influence and what it wants to be true of their knowledge, attitudes, behavior, condition, etc., as a result of its efforts—that is, its intended outcomes. These outcomes are clearly tied to the mission and vision of the organization, which are at the heart of any effective strategic plan. Then, the organization crafts goal statements, strategies, and objectives describing how it will bring about those changes. This helps the organization view its activities and outputs, not as ends in themselves, but as tools for achieving a larger purpose. Q14: Briefly describe the differences between: Goal, objective, strategy, task, vision, mission, and outcome. A: These terms are defined and used in many different ways. In the field of program outcome measurement, the most common characterization of an outcome is that it describes a change or benefit for program participants or targets groups. That is, it's not about the program itself (e.g., program staff being more knowledgeable because of a training workshop is not a program outcome) and it's about more than just being the recipient of a service (e.g., clients receiving 10 hours of counseling is not a program outcome). An additional common characterization of a program outcome is that it is a change or benefit the program reasonably can expect to influence. This is a key distinction between a program outcome on one hand and a mission or vision on the other. Some of the terms you list are strategic planning terms that are defined as follows by Frank Martinelli, who teaches the Learning Institute's Strategic Planning satellite program: Mission is a broad description of what the organization does, with/for whom the organization does what it does, its distinctive competence, and WHY the organization exists (the ultimate end). Mission conveys a sense of "fundamental purpose." Strategic Vision is a statement that describes what we want the organization to look like in ideal terms in the future - the results we will be achieving and characteristics the organization will need to possess in order to achieve those results. The strategic vision statement provides direction and inspiration for organizational goal setting and conveys a sense of "future direction." Goals are broad statements of what the organization hopes to achieve over the next 3-5 years. Goals focus on major outcomes, improvements or results and are qualitative in nature. Strategies are statements of major approach or method for attaining goals and resolving specific issues. Strategies are a bridge between the goals contained in the strategic plan and the objectives contained in the annual or operational plans set on a yearly basis. Objectives are specific, concrete, measurable statements of what will be done to achieve a goal, generally within a one-year time frame. An objective answers the following questions: Who? Will do what? By when? At what cost? How will accomplishment of the objective be measured? Q15. Our biggest challenge is showing how we compare to similar organizations. Are there any resources you can suggest for standardized outcome measurement? A. This is a mine field. It assumes (a) that you can identify organizations that truly are "similar" in all key aspects, (b) that seemingly similar organizations ought to have the same intended outcomes, and (c) that the outcome performance of those other organizations is somehow a marker for what your performance should be. In reality, even different locations for the same program are likely to be different in such important ways—participant characteristics, attributes of the environment, staff resources, etc.—that comparing their outcome results is not meaningful. Far more appropriate and productive are internal comparisons. Are you having different levels of success with different participant groups? Is one staff unit showing better outcome results than another? How does your program's outcome performance this year compare to its performance last year? That is, did you learn from last year's findings, make changes as a result of what you learned, and become even more effective at achieving your outcomes? Q16. Our funders are increasingly asking for standardized measures where available. How do we reconcile multiple stakeholder requests? A. There are very few standardized outcome measures in human services. The newness of the effort and the wide differences among programs are two reasons for this. Why are your funders asking for standardized measures? Are they wanting assurance that grantees are using high-quality measures? Are they wanting to make it easier for them to compare programs? Is there another motivation? If you can determine their purpose, you may be able to identify another way to satisfy their concern. Multiple stakeholder requests can present a real challenge, especially if they are contradictory. Unfortunately, programs often have little leverage in this situation. One strategy is to include these multiple stakeholders in your outcome measurement planning process and try to get them to reconcile their own differences. If that doesn't work, decide what you need and want from outcome measurement and be sure you pursue those ends, adding whatever adjustments are essential to satisfy the key stakeholders. Q17. Outcome based pressure from external forces like funders and board members is causing the "most difficult to serve" to be served less. How can we have conversations with these external folks about this issue? A. This is a serious potential downside to focusing on outcomes. In order to assure good outcome performance, programs may feel pressured to shift the balance of their client load, serving fewer "hard-to-serve" individuals who are less likely to achieve the intended outcome. Minimum performance thresholds, comparing one program's outcome performance to another's, and funding based only on outcome performance are among funder practices that increase this pressure. One useful step is to identify clearly the factors or characteristics that distinguish more-difficult and less-difficult cases for your specific program. This is different for every program, of course, and must be considered carefully. Using that profile, you can assess the level of difficulty of each case and then separate your outcome results for more-difficult and less-difficult cases. Also consider developing two vignettes, one depicting the situation and progress of a hard-to-serve participant and another describing a typical easier-to-serve case. Now, in talking with your funders, you can define the characteristics of those who most need your services; present vignettes illustrating the challenges they face and the difficulties you encounter in serving them; and provide data demonstrating the difference in success rates for the two groups of participants. Assuming funders agree that you should be serving those who are hardest to serve, you will be helping the funders understand what it takes to do this and what the implications are for your likely outcome performance. (Incidentally, the suggestion that board members are among an organization's "external forces" undoubtedly raises flags for many readers about the relationship between board and staff and how both see the board 's role in the organization.) Q18. We have spent most of the time focusing on the benefits of using outcomes to agencies and staff. What are the benefits of outcome measurement for clients/consumers? A. If programs use outcome measurement to increase the effectiveness of their services, their clients benefit. If they use outcome measurement to communicate the value of what they do, they are likely to attract more clients and additional resources, which means that more clients benefit and their effectiveness can increase even further. Q19: The evaluation consultant discussed a web based results program. What is the web page address that we may further research this? A: You might be referring to the web-based Outcome Measurement Resource Network described on pages 34-35 of your print packet. The address is: http://www.unitedway.org/outcomes. Also mentioned was the second chapter of the Kellogg Foundation's evaluation handbook, which was one of the pre-readings for the program. This can be found in the Learner Resource Center (Click on "pre-program readings" in the upper left hand corner of this page) or visit: http://www.wkkf.org/publications/evalhdbk/1569.htm Q20. Why doesn't United Way of America provide an inventory of outcomes, indicators, or logic models for the types of programs United Ways typically fund? A. For outcome measurement to be successful, it has to be useful to the program that implements it. To be useful to a program, it has to reflect the goals, client needs, activities, environment, and other characteristics of that program. The experience of United Ways and agencies who have been working in outcome measurement indicates that "one size fits most" lists of outcomes and indicators really don't fit most and can cause programs to loose time and other resources in trying to measure things that don't relate well to their circumstances. United Way of America's outcome measurement resources encourage each program to apply the logic model thought process and seek measurement methods that make sense for them. See page 32 (Question #4) of the participant packet for further information about this "Frequently Asked Question." Many issue-focused web sites are devoted to exploring possible outcomes and indicators for related programs, and a few of the more helpful sites are listed as annotated links in the Outcome Measurement Resource Network web site (see page 34 for URL). Links to general accreditation and performance resources can be found on this site at: http://www.unitedway.org/outcomes/accredit.htm Q21. In spite of what you said in outline item I-D [page 2 of the participant packet], statistical validity and experimental design validity seem essential for developing valid conclusions and assertions. For example, in learning activity #2, the difference in apparent outcomes of the different curriculums COULD BE explained by the difference in outcomes between single and double adult households OR vice-a-versa. The results as charted are not able to support any conclusion. Your comments? A: The results as charted represent a starting point for the staff of the program in looking at their outcome data. If the staff suspect there may be a relationship between which curriculum was used and how many adults are in the household—i.e., participants who are the only adult in their household took part in programs that used one curriculum while participants from multi-adult households were exposed to the other curriculum—then one of the next steps they will want to take is to see if that is in fact true. Another step they certainly will want to take is to talk with participants to gain their perspectives and insights. They also undoubtedly will talk with the three program leaders. They may want to contact other programs that have used one or the other curricula to learn of their experiences. They may want additional analyses of the data, or to examine the curricula to see how much time each devotes to various topics and with what materials or learning activities. Their quest here is exploring and learning, with the outcome data serving to generate questions and suggest avenues to explore. Assuming that the staff decide to make some changes—perhaps in one or both curricula, in program leader training, in supplemental services, in how participants are assigned to programs, or anything else—the data also will provide a point of comparison next year to help staff determine if the program's effectiveness has increased. No, the data won't prove that the changes in the program caused the changes in the outcome performance, but the more diligent staff have been about considering issues and options, the more satisfied they will be, even in the absence of experimental proof, that the changes they made have resulted in better service for parents. Q22. In the parenting education program model, how do you account for the difference in effectiveness between dual-parent households and single-parent households? A: This certainly is one of the questions that the staff of this program will want to explore. The data indicate that parents from one-adult households were as successful at learning positive discipline methods as parents from multi-adult households, but were not as successful at implementing what they learned. Perhaps parents from one-adult households are more overwhelmed by the responsibilities of parenting than are parents who share those responsibilities with another adult. Perhaps they missed more sessions of the 8-session program. Perhaps they have more children at home, making their challenge greater to begin with. Fortunately, the staff know that the parents themselves are the best source of insights into this question. Through questionnaires, telephone interviews, focus groups, or even informal conversations when parents are in the neighborhood center for other purposes, they will ask parents for their thoughts about why these differences occurred and what the program might do to be more effective. Q23. In the parenting education program activity, how do you decide which curriculum is better? They have mixed results. A: You're right about the mixed results. The old curriculum is more effective in helping parents learn about positive discipline methods. If this program had said that parents' knowing the information is its only intended outcome, and only measured whether participants learned the information or not, the first curriculum would be the clear choice. However, the program recognized that learning the information is not very meaningful unless it is put into practice, so it identified and measured an additional intended outcome related to behavior change. The findings show that while the new curriculum is not as effective in helping parents gain knowledge, it is more effective in helping them apply the knowledge. Therefore, if the program has to use one curriculum or the other, the new curriculum seems to be preferable. Even better would be to identify the effective components of each curriculum and see if they can be combined. Q24. Dr. Plantz stated at the beginning of the session that outcome measurement does not help establish a link between outputs and outcomes. I'm currently struggling with this issue in my developing a monitoring and outcome evaluation plan for our statewide cancer control initiative that's led by many cancer control organizations who will be implementing a variety of activities/assessments and interventions around the state. Funders are asking me to speak to this issue (this coming Thursday 5/11/00) to whether they'll be able to justify their existence and take credit for the outcomes of the Initiative. The ultimate goal is to reduce cancer mortality rates in the state but that takes at least two decades to see. In the mean time we plan to monitor outputs of the implemented activities and we can observe changes in screening behaviors (intermediate outcome). Can you suggest ways we can investigate some sort of relationship or association to demonstrate that program activities have at least contributed to outcomes. How can we take credit for desired outcomes? A. If you want to demonstrate that your activities have a causal relationship with changes in screening behaviors, which is what someone usually means when they say they want to "take credit," you need a more methodologically rigorous tool than outcome measurement. Outcome measurement does not assess causality; demonstrations of causality require experimental (or perhaps quasi-experimental) research designs--random assignment, control groups, tests of statistical significance, etc. In outcome measurement, an association between activities and outcomes is based on logic or plausibility: Does it make sense that these activities implemented in this way with these individuals will have a meaningful influence on the changes we want to see? To assert defensibly that program activities have at least contributed to outcomes, in the absence of experimental research, one needs Prior to 1995, a number of United Ways across the country had recognized the value of outcome measurement and its long-range importance for United Ways and the agencies they fund. These organizations had begun outcome measurement programs and encouraged United Way of America to provide resources and national leadership on the issue. In April 1995, UWA established an internal team charged to help United Ways document and improve their impact on community problems by developing and supporting approaches to measuring the outcomes of United Ways’ investment in health and human services. UWA’s outcome measurement program is advised by a 30-member national task force whose membership includes representatives of United Ways, human service organizations, foundations, corporations, public agencies, and research institutions. In August 1996, with The Urban Institute as its technical consultant, UWA published Measuring Program Outcomes: A Practical Approach. This step-by-step manual is designed to help health, human service, and youth- and family-serving agencies identify and measure their outcomes and use the results. The manual is in its sixth printing, with 80,000 copies in distribution in the U.S. and several other countries. A related publication, the Measuring Program Outcomes Training Kit, provides presentation notes, activities, handouts, and other resources for delivering hands-on training based on the manual. Q26) Where can I get a copy of United Way of America’s outcome measurement manual, Measuring Program Outcomes: A Practical Approach? The manual is distributed by Sales Service/America: 703/212-6300. (In the U.S., call toll free at 800/772-0008.) The item number for the manual is 0989. Excerpts from the manual, as well as other outcome measurement resources, are posted on UWA’s Outcome Measurement Resource Network: http://www/unitedway.org/outcomes/). A description of the network is provided on page 34 of this Participant Packet. Q27) How many United Ways are asking agencies they fund to identify and measure their outcomes? In November 1999, nearly 400 United Ways—at least one in every state in the U.S.— reported being involved in or planning to implement program outcome measurement. It is important to note that United Ways are independent organizations, incorporated and governed locally. Not all United Ways that are implementing outcome measurement are using the resources or approach developed by United Way of America. Q28) Why doesn’t United Way of America provide suggested outcomes, logic models, or indicators for the types of programs United Ways typically fund? Programs that appear from the outside to be similar—similar clients, similar missions, similar service delivery strategies—may in fact have very different outcomes. United Ways and others who have worked in outcome measurement have learned that, for outcome measurement to be successful, programs must identify outcomes and indicators for themselves. This assures that they are relevant to the program’s design and useful to program managers. In addition, programs report that the process of identifying their outcomes is in itself a valuable benefit of outcome measurement, providing greater clarity and focus for staff, volunteers, and clients. And, of course, we all are more committed to an effort if we have helped shape it and thus know its results will be useful to us. Q29) Does outcome measurement really provide benefits for programs? This question merits serious research that explores not only whether there are benefits, but also under what circumstances, for what types of programs, and other issues of applicability and whether or not results can be generalized. A 1999 confidential, exploratory survey of the programs involved in outcome measurement that are funded by six United Ways, conducted by an independent contractor for United Way of America, examined both positive and negative aspects of measuring program outcomes and using the results. In preliminary data (N=258 programs) 75 percent of program executives agree or strongly agree that "On balance, implementing program outcome measurement has had a positive impact on this program’s ability to serve clients effectively." More specifically, they report that implementing program outcome measurement has helped their program: - Communicate program results to stakeholders (87%) *Responses were transcribed by Prof. Andrew Lewis from the live broadcasts produced by the Learning Institute for Nonprofit Organizations collaboration*.
*Responses were transcribed by Prof. Andrew Lewis from the live broadcasts produced by the Learning Institute for Nonprofit Organizations collaboration. |
||||||||||||
