Job Quality Playbook
 

Where to Start

How to Start Measuring Job Quality

  1. Use the outputs and outcomes in your logic model as a starting point
  2. Examine what data is available
  3. Select your metrics
  4. Tell a story
  5. Continuous improvement

1. Use the outputs and outcomes in your logic model as a starting point.

If you haven’t developed a logic model, use this as a guide. To advance development of your logic model with multiple stakeholders, check out this facilitation guide.


Different projects may require different types of data collection and metrics. If addressing eligibility, enrollment, or employer engagement, you may collect more information about whether a particular policy or practice is in place (e.g., does an anti-harassment policy exist or what type of organizations have you funded to date) and how it is used. As you measure procurement processes, programs, or the quality of placements, consider disaggregated performance output and outcome measures (e.g., did the % of Black women who completed a program improve when childcare was provided, was funding and time directed to build the field of BIPOC-focused service providers).

Example

  • Incorporate job quality language in RFPs
  • Diversify respondent pool and dollars allocated to such organizations
  • Implement evidenced-based scoring preferences for organizations with a history of placing workers in good jobs

  • Procurement process owner
  • Staff providing outreach/training
  • Funding for provision of technical assistance or training
  • System for data storage/analysis

  • Assess prior awards (types of orgs, amount, results)

  • Develop agreed upon job quality definition for inclusion in RFP

  • Define specific job quality measures (e.g., wages, access to benefits) that respondents are required to track (and what calculator will be used for living/self sufficiency wage)

  • Review current RFP response history to determine baseline (e.g., on average how many responses, type of respondents)

  • Develop a scoring rubric which allocates priority points to job quality and diverse respondents

  • Identify a location to house data provided by respondents throughout program (e.g., existing database)

  • Develop draft version of a scorecard template and determine whether to set targets, or if the initial version will simply report data

  • Develop guidelines for establishing selection committee

  • Update procurement documentation

  • Update procurement policies, if necessary, on allowable expenses (e.g., compensating participants for lending their voice)

  • Secure approval for revised procurement documentation

  • Develop communication plan to socialize definition, rubric and related documentation with potential respondents (e.g., bidder calls)

  • Develop bidder focused technical assistance and/or training to increase knowledge of job quality and related data collection

  • Determine cadence for regular meetings to track/assess progress

  • # of diverse organizations that participated in the procurement process

  • # of hours/session of pre-submission technical assistance provided to BIPOC organizations

  • Qualitative survey/interview/focus group results from potential RFP respondents

  • Use of feedback loops throughout process (e.g., submission of questions in response to RFP, participation in internal groups)

  • Breakdown gender/race by level of respondent’s organization (e.g., executive, supervisors, staff)

  • $ paid to participants for lending their voice to the work (e.g., survey, focus groups)


Short Term

  • Income increase or % change in wage as a result of living wage requirement

  • # or % of awardee staff with increased job quality (e.g., access to benefits, access to learning pathway) as a result of procurement

  • # or % increase in engagement of BIPOC-focused organizations

  • Perception changes pre and post training

  • Shifts made in procurement process (e.g., changes to administrative requirements, inclusion of training) based on quantitative and qualitative feedback

Long Term

  • Change in diversity of awardees over time

  • Increase in % of dollars directed to BIPOC organizations that create or place individuals in good jobs

  • % or # of awardee staff with increased job quality (e.g., access to benefits or career pathways) as result of procurement

  • % improvement in service delivery to BIPOC participants based on diversification of service providers

  • Reduction in participation gaps of particular groups (e.g., BIPOC led organization responses to RFPs)

  • Employer practice change (e.g., hiring approach, retention)


  • The work serves as an example for other workforce development organizations who are seeking to advance equity and job quality

  • The respondent pool is more representative of the diverse population groups in the local area

  • Respondents become more knowledgeable about the importance of job quality and are better positioned to incorporate their learnings into their work

  • The agency’s programmatic results significantly improve as a result of using evidence-based practices

2. Examine what data is available

Examine what data is available. This includes data you own and data you access through data sharing agreements, data lakes or other integrated systems. This worksheet is useful to assess your data infrastructure.

If you don’t have the necessary data, there are three main ways to consider obtaining it.

  • If no one has it - New data collection is needed. May need to add fields to a system, implement a survey or conduct interviews.

  • If someone has it and could share it - New partnerships and collaborations are needed. May need to expand or deepen relationships or implement sharing processes.

  • If someone has it but can’t/won’t share - New policies, regulations or agreements are needed. May need to negotiate how to protect, store or limit use of data.

Example

After assessing our current data collection and data partnerships, we have the most data related to our goal of diversifying our respondents. This includes:

  • Current procurement statistics around the # of awards over time, the types of organizations, and the amount awarded

  • Amount of technical assistance provided

  • Customer satisfaction survey that can be updated to include job quality questions

  • Median wage data

3. Select your metrics

Select 3-4 metrics, based on your available data, for your primary lever(s) or job quality principle. Consider what is most important and add over time as more data is available. See examples by Lever and Principle.

Example

Given our goals and our available data, we are starting with the following:

  • # of BIPOC organizations that participated in the procurement process / % change from prior procurements

  • $ directed to BIPOC organizations as % of total budget, changes over time

  • # of hours of pre-submission technical assistance provided to BIPOC organizations

  • Pre/post TA qualitative survey/interview/focus group to measure perception changes and knowledge shifts

  • Median wage of placements vs living wage for the area

4. Tell a story

Assess how you will tell a story with your metrics.

  1. Who needs access to the information?

  2. How can the data best be shared (e.g., charts, graphs)?

  3. What is the right frequency for publishing the information?

For more on data storytelling, check out this recording and slides.

Example

In reviewing our procurement processes, we have determined:

  • We want to tell a story about the change in our procurement processes

  • Data will be published within 30 days of each completed procurement

  • Data will be made available to workforce board leadership, the procurement team and the relevant program team

  • Data will be displayed in tables and include qualitative write ups

5. Continuous improvement

Establish a continuous process improvement mechanism. This could include a monthly discussion, a feedback email box, or a regular survey to incorporate changes and make adjustments to reporting.

Go Deeper With Uses Cases

See how government agencies might implement job quality measurement