A panorama of a research room taken at the New York Public Library

Prove Your Impact

In an earlier post, I mentioned that Logic Models describe a set of “If…Then…” statements, such as “If we teach 200 driving students, then there will be fewer auto accidents.” But how do I know this is true?

Grant reviewers are a hard group. Well, the good ones are. You need to be able to prove your statements. Reviewers won’t, and in fact, can’t, assume that outputs have impacts just because “It’s intuitive” or “Everyone knows that.”

I once heard a reviewer say, “If you tell me that today is Tuesday and that Tuesday is always followed by Wednesday, you need to show documentation of both statements.”

Here are the levels of evidence from a recent grant application review guidelines, along with the point value of each:

No Evidence – Zero points

The applicant just asserts that the outputs will generate the desired outcomes. They haven’t collected any data and don’t know of anyone who has. Sometimes they present some anecdotes.

Pre-preliminary Evidence – 1 point

The applicant has collected quantitative or qualitative data from their program and that it has been used for performance improvement.

Example: Feedback forms at the end of a presentation or a class.

Preliminary Evidence – 2 points

Applicant presents at least one non-experimental study from their program that supports the contribution to observed outcomes. [A non-experimental study is a study that does have a control group or where the researcher cannot manipulate a key variable.]

They may also use a non-experimental study from a similar program that uses a comparable intervention. This is useful for new programs that do not have a track record.

Examples: Tracking beneficiaries through each step of a process and measuring their response at the end of the program, or using pre-testing and post-testing to determine changes.

Moderate Evidence – 4 points

Notice the larger jump in points. Moderate evidence requires one or more quasi-experimental studies on the program or a similar program. [A quasi-experimental study has a control group, but the researcher does not assign subjects to the control group in a random manner. This could introduce subject bias. A quasi-experimental researcher may be able to manipulate variables.]

Two or more non-experimental studies on the proposed program are also rated as Moderate evidence.

A well-designed experimental study on a program using a similar intervention is also considered Moderate evidence.

Example: A study that compares outcomes between a group that receives the intervention and a similar group that does not.

Strong Evidence – 8 points

An even bigger jump in points, but very difficult to achieve. Strong evidence requires a well-designed and well-implement experimental study conducted on the proposed program. [An experimental study has a randomly selected control group and the key independent variables are under the control of the researcher.]

Strong evidence is very hard to obtain on social programs. Key variables cannot be manipulated and it may be unethical to exclude individuals from an intervention that is available.

Summary

Overall, the reviewer tries to judge whether or not the proposed outputs will produce the desired outcomes. This is a measure of effectiveness, as opposed to efficiency. It is up to the applicant to provide the proof.