How to Measure
Policy – whether at the program, department, organization or jurisdiction level – is measured through the accomplishment of the policy objectives defined in the logic model. Measures look at changes that have occurred since the implementation of a policy and consider the extent to which changes can be attributed to the policy.
Key questions to consider in measuring the impact of the policy lever:
What external considerations, such as other policies, the economic environment and business choice, may impact the results of the policy implementation?
What is the nature of the relationship between the policy and the desired impact?
Did the policy contribute to a change in the outcomes and impacts of interest? Why or why not? If not, what could be done differently?
Were there any unintended consequences of the policy? If so, how could these have been avoided or reduced? What actions need to happen now to address those consequences?
Did contextual/environmental factors influence the level of impact? Was the impact the same across the jurisdiction, agency, department or program? Were there variations by population?
What was the economic impact of the policy to the jurisdiction, agency, program or department?
What approach was most effective? Are there trends? Does it vary by type of policy?
What is the expected magnitude of the change? Is the timeframe over which I am measuring impact reasonable based on this magnitude?
Remember other external forces, programs, and agendas may be occurring simultaneously. While this is unavoidable, it is important to be aware of the current state prior to designing, implementing or drawing conclusions from the data collected. Methods for measuring policy should be fit-for-purpose, meaning they are relevant and specific to content and target groups, taking into consideration the interpretations of the policy (e.g., undocumented individuals likely experience the policy differently, even within the same community).
|
Table
|
Sample Measure
|
|---|---|
Incentives |
• # of times/amount of resources provided and resulting change |
Requirements |
• Reductions in incidents (# or %) of undesirable behaviors or increase in incidents (# or %) of desirable behavior |
Policy/Practices |
• Positive or negative perceptions of involved individuals |
Tips for Measuring
Look for ways to engage a diverse set of workers (e.g., employees, participants, partners or even local residents) in the development of the policy, and the implementation and evaluation approach.
Identify a control group for comparison. This might be achieved by piloting the change with one team, office or geography before rolling it out more broadly. Randomized control trials are also useful tools but the cost, level of rigor and need to maintain strict control groups can be prohibitive.
Consider collecting data at different stages of the process, including during communication of the policy, training on the policy, and technical assistance provided to support the policy implementation, for more holistic insights into the change.
Set reasonable expectations for the timeline necessary to see change from the policy implementation. The greater the change the more time is generally required to see the results.
Be aware of other changes occurring in the environment at the same time as they can make it difficult to determine which results are attributable to the policy versus other shifts in the environment.