What is the difference between outcomes and impact?

Is your program REALLY making a difference?

This question is deceptively simple. “Yes, of course…” and we share stories or output measurements like, “X number attended our events,” or “Y number successfully completed our program.” Logic models help define the relationship between our planned work and intended results.

Inputs: Resources needed for program activities.

Activities: Services or components of a program.

Outputs: Direct result of activities.

Outcomes: Intended benefits of activities.

It’s important to distinguish outputs and outcomes because outputs can easily inflate our perceived impact. During my freshman year of college, I started a weekly meeting in a local prison with men approaching reentry. It was a form of kind-hearted social malpractice that I thought was making a real impact. Why? The room was packed every week! And the feedback of inmates was always positive. The difference between outputs (attendance) and outcomes (benefits) could be revealed with the follow-up, “so that…” to the outputs. “Well, they come to the meetings, so that they can develop a reentry plan, identify housing and job prospects, and strengthen positive relationships.” These were some of the (unmeasured) program outcomes. How many inmates obtain stable housing and employment? Such an outcome measurement could be distinguished by short and long-term estimates.

But do outcomes equate to impact?

No, outcomes alone cannot fully answer the question, “Is your program making a difference?” We could show some convincing pre and post-test data that is statistically significant. However, what outcomes alone cannot tell us is what would have happened if the person did not participate in the program. This is known as the counterfactual. The impact of a program is simply “what happened” minus “what would’ve happened without it.”

Impact = factual — counterfactual

Unfortunately, the simplicity of the concept belies its complexity in practice. Since counterfactuals aren’t observable, causal inference methods are necessary to try to create the context where we can come as close as possible to observe the unobservable. In the next few posts, I’ll share the primary concepts, terms, and methods for measuring social impact. Before we jump in, let’s make sure to distinguish between simple difference and treatment effect (impact).

Simple difference is NOT the treatment effect

Treatment effects (impact) are changes in outcomes due to changes in treatment (activities) holding all other variables constant. This last phrase is really important. In order to answer the question, “Is the program REALLY making a difference?” we need to isolate the effect of the program, which means that a simple difference between pre and post-test scores cannot be the treatment effect. Post-test data could show significant positive change during the time of the program, but how do we know the program is responsible for the change? We need to create a control group in order to ‘hold all other variables constant.’ The gold standard for such an evaluation is a randomized controlled trial (RCT); however, our focus will be social programs where resources are limited and RCT’s aren’t feasible. How can we identify the effects of our programs with limited resources? That’s our focus in this series of posts.