This community is in archive. Visit community.xprize.org for the current XPRIZE Community.
Measuring success
jordangiali
Posts: 43 XPRIZE
Is job retention the right success metric for us to measure? What might be a better measurement of success?
Tell us how YOU would measure success in this competition. We'd love for you to get creative and think outside the box!
Tell us how YOU would measure success in this competition. We'd love for you to get creative and think outside the box!
0
Comments
The latest version of our prize design would challenge teams to develop and deploy a scalable training, job placement, and retention solution to:
@jordangiali's question relates to the third point, but we also want to know if you would suggest changing or adding any criteria.
Thank you!
I would suggest counting in weeks - 4/6/12 for train/place/retain respectively. I would also try to quantify what "living wage" and "growing faster" mean.
1. Backward looking indicator based on some assumptions of a tight labour market (previous of Jan 2020)
2. Research has shown that automation and other job disruption occurs in recessions, making some occupations / skills obsolete vs the labour market picture in 2019 (Nir Jaimovich & Henry E. Siu, 2020)
I would suggest looking at the BLS projection system as this is a 10 year outlook on jobs based on a pretty sound methodology (2018-2028), in conjunction with broader automation risk or AI potential indicators to give you alternative views of how a job might evolve in the future (5-10 years).
On defining success:
1. Job retention measured against a baseline measure from talent supply data sources would probably be more informative (i.e. what is the average attrition or time spent in this job for a comparable population sample?) - with this, you will ultimately battle a question of causal effect of the program in assessing what would have happened if you did nothing with the participants.
2. Other measures to consider are pre and post survey measures to get a sense on a) career / skill confidence and b) career / skill satisfaction for a more holistic measure on longer term impact. These are prone to self-reporting biases, but give another perspective.
In the meantime, we'd love more input on this question. @eperkins, @mhenrynickie, @AlmaSalazar, if you have any thoughts on this, please let us know!
I appreciate that these are specific metrics.
I also think the retention metric time Frane is not a very long period although workers choosing to leave is different that workers not succeeding.
- 90 days is a very short time frame. I understand the constraints of the contest, but many firms have grace periods that are that length. It would be easily imagined that workers keep jobs for 90 days but not 6 months (or a year). Long term outcomes matter.
- What about earnings? Given the current economic situation, I might be fine if the worker-firm match lasts 2 months before the worker moves but trainees continue to earn good incomes.
- The process by which workers are matched to firms is going to be very important. That does not seem to be incorporated in to the evaluation. I would argue that matching is just as important as the training.
- More abstractly, the definition of "living wage" occupation and "growing faster than average" is going to be based on pre-virus data. How do you define these categories given the new normal?
We want to partner with Workforce Development Boards for this. They would do the matching, so teams competing in our prize competition can focus on the training.
Good point! But I'm not sure we can do better than use pre-crisis data. It's probably too soon to assess how these metrics will be affected by the pandemic?
We've interviewed representatives from a number of workforce boards to understand how they can help teams place workers into jobs. In general, the boards have close relationships with local employers. They collect data on who is hiring and for which particular roles, along with the skill sets that are needed. The boards will provide these data to the competing teams and help them make contact with local employers.
Teams will also be able to form and leverage partnerships with additional organizations, such as staffing agencies. XPRIZE also plans to administer an online collaboration platform where teams can connect with mentors and other stakeholders that could lead to placement opportunities. Teams will have up to 60 days to complete the placement process.
Designing evaluation criteria with these intermediaries playing such an important role may be difficult. A solution might have better performance simply because it is paired with / has access to a more effective set of placement partners, which could depend on factors beyond the solution team's control.
If you haven't yet, you may want to consider which one of these evaluation models you want to pursue, because the design would be different:
Option 1: Placement partners (e.g. Workforce Dev Boards) are controlled for in evaluation. For a given workforce board partnership, solution X was more effective than solution Y. This could also involve random assignment of solutions to placement partners.
Option 2: Effectively leveraging placement partners is part of the challenge of the solution design. A solution that has better results because it builds better strategic partnership should get higher evaluation scores, even if all other aspects of the solution are not superior to others.
For Option 2, how would you suggest measuring a team's performance in building partnerships? Total number of partnerships formed?