How to Start Measuring Job Quality
- Use the outputs and outcomes in your logic model as a starting point
- Examine what data is available to you now
- Select your metrics based on available data
- Think about the story you want to tell with data
- Plan for continuous improvement
1. Use the outputs and outcomes in your logic model as a starting point.
If you haven’t developed a logic model, use this as a guide. To advance development of your logic model with multiple stakeholders, check out this facilitation guide.
Different projects may require different types of data collection and metrics. If addressing eligibility, enrollment, or employer engagement, you may collect more information about whether a particular policy or practice is in place (e.g., does an anti-harassment policy exist or what type of organizations have you funded to date) and how it is used. As you measure procurement processes, programs, or the quality of placements, consider disaggregated performance output and outcome measures (e.g., did the % of Black women who completed a program improve when childcare was provided, was funding and time directed to build the field of BIPOC-focused service providers).
Example
Area of Focus
- Incorporate job quality language in RFPs
- Diversify respondent pool and dollars allocated to such organizations
- Implement evidenced-based scoring preferences for organizations with a history of placing workers in good jobs
Resources + Inputs
- Procurement process owner
- Staff providing outreach/training
- Funding for provision of technical assistance or training
- System for data storage/analysis
Activities
Assess prior awards (types of orgs, amount, results)
Develop agreed upon job quality definition for inclusion in RFP
Define specific job quality measures (e.g., wages, access to benefits) that respondents are required to track (and what calculator will be used for living/self sufficiency wage)
Review current RFP response history to determine baseline (e.g., on average how many responses, type of respondents)
Develop a scoring rubric which allocates priority points to job quality and diverse respondents
Identify a location to house data provided by respondents throughout program (e.g., existing database)
Develop draft version of a scorecard template and determine whether to set targets, or if the initial version will simply report data
Develop guidelines for establishing selection committee
Update procurement documentation
Update procurement policies, if necessary, on allowable expenses (e.g., compensating participants for lending their voice)
Secure approval for revised procurement documentation
Develop communication plan to socialize definition, rubric and related documentation with potential respondents (e.g., bidder calls)
Develop bidder focused technical assistance and/or training to increase knowledge of job quality and related data collection
Determine cadence for regular meetings to track/assess progress
Outputs
# of diverse organizations that participated in the procurement process
# of hours/session of pre-submission technical assistance provided to BIPOC organizations
Qualitative survey/interview/focus group results from potential RFP respondents
Use of feedback loops throughout process (e.g., submission of questions in response to RFP, participation in internal groups)
Breakdown gender/race by level of respondent’s organization (e.g., executive, supervisors, staff)
$ paid to participants for lending their voice to the work (e.g., survey, focus groups)
Outcomes
Short Term
Income increase or % change in wage as a result of living wage requirement
# or % of awardee staff with increased job quality (e.g., access to benefits, access to learning pathway) as a result of procurement
# or % increase in engagement of BIPOC-focused organizations
Perception changes pre and post training
Shifts made in procurement process (e.g., changes to administrative requirements, inclusion of training) based on quantitative and qualitative feedback
Long Term
Change in diversity of awardees over time
Increase in % of dollars directed to BIPOC organizations that create or place individuals in good jobs
% or # of awardee staff with increased job quality (e.g., access to benefits or career pathways) as result of procurement
% improvement in service delivery to BIPOC participants based on diversification of service providers
Reduction in participation gaps of particular groups (e.g., BIPOC led organization responses to RFPs)
Employer practice change (e.g., hiring approach, retention)
Impact
The work serves as an example for other workforce development organizations who are seeking to advance equity and job quality
The respondent pool is more representative of the diverse population groups in the local area
Respondents become more knowledgeable about the importance of job quality and are better positioned to incorporate their learnings into their work
The agency’s programmatic results significantly improve as a result of using evidence-based practices
- Incorporate job quality language in RFPs
- Diversify respondent pool and dollars allocated to such organizations
- Implement evidenced-based scoring preferences for organizations with a history of placing workers in good jobs
- Procurement process owner
- Staff providing outreach/training
- Funding for provision of technical assistance or training
- System for data storage/analysis
Assess prior awards (types of orgs, amount, results)
Develop agreed upon job quality definition for inclusion in RFP
Define specific job quality measures (e.g., wages, access to benefits) that respondents are required to track (and what calculator will be used for living/self sufficiency wage)
Review current RFP response history to determine baseline (e.g., on average how many responses, type of respondents)
Develop a scoring rubric which allocates priority points to job quality and diverse respondents
Identify a location to house data provided by respondents throughout program (e.g., existing database)
Develop draft version of a scorecard template and determine whether to set targets, or if the initial version will simply report data
Develop guidelines for establishing selection committee
Update procurement documentation
Update procurement policies, if necessary, on allowable expenses (e.g., compensating participants for lending their voice)
Secure approval for revised procurement documentation
Develop communication plan to socialize definition, rubric and related documentation with potential respondents (e.g., bidder calls)
Develop bidder focused technical assistance and/or training to increase knowledge of job quality and related data collection
Determine cadence for regular meetings to track/assess progress
# of diverse organizations that participated in the procurement process
# of hours/session of pre-submission technical assistance provided to BIPOC organizations
Qualitative survey/interview/focus group results from potential RFP respondents
Use of feedback loops throughout process (e.g., submission of questions in response to RFP, participation in internal groups)
Breakdown gender/race by level of respondent’s organization (e.g., executive, supervisors, staff)
$ paid to participants for lending their voice to the work (e.g., survey, focus groups)
Short Term
Income increase or % change in wage as a result of living wage requirement
# or % of awardee staff with increased job quality (e.g., access to benefits, access to learning pathway) as a result of procurement
# or % increase in engagement of BIPOC-focused organizations
Perception changes pre and post training
Shifts made in procurement process (e.g., changes to administrative requirements, inclusion of training) based on quantitative and qualitative feedback
Long Term
Change in diversity of awardees over time
Increase in % of dollars directed to BIPOC organizations that create or place individuals in good jobs
% or # of awardee staff with increased job quality (e.g., access to benefits or career pathways) as result of procurement
% improvement in service delivery to BIPOC participants based on diversification of service providers
Reduction in participation gaps of particular groups (e.g., BIPOC led organization responses to RFPs)
Employer practice change (e.g., hiring approach, retention)
The work serves as an example for other workforce development organizations who are seeking to advance equity and job quality
The respondent pool is more representative of the diverse population groups in the local area
Respondents become more knowledgeable about the importance of job quality and are better positioned to incorporate their learnings into their work
The agency’s programmatic results significantly improve as a result of using evidence-based practices
2. Examine what data is available to you now
Examine what data is available. This includes data you own and data you access through data sharing agreements, data lakes or other integrated systems. This worksheet is useful to assess your data infrastructure.
If you don’t have the necessary data, there are three main ways to consider obtaining it.
If no one has it - New data collection is needed. May need to add fields to a system, implement a survey or conduct interviews.
If someone has it and could share it - New partnerships and collaborations are needed. May need to expand or deepen relationships or implement sharing processes.
If someone has it but can’t/won’t share - New policies, regulations or agreements are needed. May need to negotiate how to protect, store or limit use of data.
Example
After assessing our current data collection and data partnerships, we have the most data related to our goal of diversifying our respondents. This includes:
Current procurement statistics around the # of awards over time, the types of organizations, and the amount awarded
Amount of technical assistance provided
Customer satisfaction survey that can be updated to include job quality questions
Median wage data
Example
Given our goals and our available data, we are starting with the following:
# of BIPOC organizations that participated in the procurement process / % change from prior procurements
$ directed to BIPOC organizations as % of total budget, changes over time
# of hours of pre-submission technical assistance provided to BIPOC organizations
Pre/post TA qualitative survey/interview/focus group to measure perception changes and knowledge shifts
Median wage of placements vs living wage for the area
4. Think about the story you want to tell with your data
Assess how you will tell a story with your metrics.
Who is the audience for the data?
How can the data best be shared (e.g., charts, graphs)?
What is the right frequency for publishing the information?
For more on data storytelling, check out this recording and slides.
Example
In reviewing our procurement processes, we have determined:
We want to tell a story about the change in our procurement processes
Data will be published within 30 days of each completed procurement
Data will be made available to workforce board leadership, the procurement team and the relevant program team
Data will be displayed in tables and include qualitative write ups
5. Plan for continuous improvement
Establish a continuous process improvement process. This could include a monthly discussion, a feedback email box, or a regular survey to incorporate changes and make adjustments to reporting as the work moves forward.
Go Deeper With Uses Cases
See how government agencies might implement job quality measurement