Modern airliners can fly most parts of the journey on autopilot, but unlike me wishing for more than ‘traffic jam assist’ in my new car, most pilots don’t rely on autopilot, and that’s because regulators kept asking one question: can the pilot still fly when autopilot is switched off?
How many other organisations and professions ask the equivalent question about their people? And are they it driven purely by risk?
Then, there’s the autopilot your learning budget runs on, with the same programs renewed year after year, the same metrics reported and/or the same vendors signed… In this blog, I’m proposing you take a leaf out of the aviation book and switch off autopilot from time to time.
What is your dashboard measuring?
Donald Kirkpatrick’s four levels have shaped training evaluation for decades: reaction, learning, behaviour and results. In practice, the easiest numbers to collect sit at the first two. The harder and more useful questions sit at levels three and four.
That gap matters, as a study by Saks and Burke found that, of Kirkpatrick’s four levels, only evaluating behaviour and results was associated with higher reported training transfer of knowledge/skills. Organisations measure what’s easy, and the easy measures aren’t the ones linked to training that sticks.
AI brings a new set of easy numbers: adoption rates, prompts sent, hours saved. Essentially, reaction-level data in a new font that tells you the autopilot is engaged… Not whether anyone can still fly!
Four signals that show whether capability is growing
Performance: Borrow the method from the maths study discussed earlier in my blog series. Every so often, have people complete a representative task without a supportive tool. If performance drops noticeably, treat that as a warning sign of dependence, not proof that the tool caused it.
Decision-making: Look at the cases AI handles badly: exceptions, escalations, problems nobody has seen before. Are your people catching AI errors, or passing them on?
Behaviour someone else has seen. Can a manager or stakeholder name a specific change in real work? This is Kirkpatrick’s third level, and it’s the one most often skipped.
The stretch pipeline. Are your emerging leaders still getting the hard assignments that build judgement, or has AI absorbed them?
“We can’t isolate AI’s effect”
… and you don’t need a controlled trial to act. Three things are enough to decide whether to renew:
a before-and-after comparison on the same task,
a small group doing it with the tool switched off, and
an honest read from a manager or key stakeholder.
Example questions for your 6-month Scorecard
| Signal | Question to ask | Warning sign |
|---|---|---|
| Performance | Can people do the core task without the tool? | Noticeably worse than before the rollout |
| Decisions | Who's catching the AI's mistakes? | Errors reaching customers or the board |
| Behaviour | What has someone else seen change? | Only self-reports and satisfaction scores |
| Stretch / Pipeline | Are future leaders still getting hard assignments? | AI now does the work that used to stretch them |
If most of your answers are hitting the warning sign column, your budget is flying on autopilot.
Where Plexity fits
Plexity works where business acumen and people performance meet. I sit independently on the buyer’s side, with no product to sell. I help executives switch off the budget autopilot: deciding what to measure before signing, holding providers to it, and backing the development that passes the switch-off test.
Coming in 2027: a free capability health check. It’s a short diagnostic that shows whether your investments in organisational development and workplace training are building capability or creating dependence.
Coming next: Tips on writing a learning procurement brief.
References
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122
Federal Aviation Administration. (2013). Manual flight operations (Safety Alert for Operators 13002). U.S. Department of Transportation. https://www.faa.gov/sites/faa.gov/files/other_visit/aviation_industry/airline_operators/airline_safety/SAFO13002.pdf
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler.
Saks, A. M., & Burke, L. A. (2012). An investigation into the relationship between training evaluation and the transfer of training. International Journal of Training and Development, 16(2), 118–127. https://doi.org/10.1111/j.1468-2419.2011.00397.x