facebook-script

We've Been Measuring the Wrong Thing

We've Been Measuring the Wrong Thing

cover image

Subscribe to the Ausmed Toolbox

Toolbox Newsletter

I have spent most of my career in and around education. First, as an intensive care nurse, trying to stay current. Then, working with educators trying to help others do the same. It was the focus of my Master's thesis: how do we evaluate the effectiveness of continuing professional development (CPD), and, for the past several years, leading clinical governance, learning, and capability functions at Ausmed.

In that time, one question has followed me everywhere:

How do we know any of this is actually working?

It's a deceptively simple question. And the honest answer, for most mandatory training, is: we don't. We know whether people completed it. We don't know whether it made a difference.

That is the problem Knowledge Verification (KV) is designed to address. Not just as a feature, but as an educational proposition. And I want to make that case properly here, because the why matters just as much as the what.

Completion is Not the Same as Learning

Mandatory training is predominantly completion-based. Someone is assigned a module, clicks through to the end, answers a few questions, and receives a certificate. That completion forms the basis of a learning record. The compliance box is ticked.

But completion tells us almost nothing. It tells us the module was opened. It does not tell us whether the person understood it, retained it, or will apply it. It certainly does not tell us whether they needed it in the first place.

This matters because training volume is increasing. The increase is appropriate - it reflects genuine regulatory change and rising expectations for the quality and safety of care. In response to increased regulatory requirements for training and associated costs, auditors and Boards, are rightly so, asking:

  • Did your staff actually learn?
  • Can they demonstrate what they know?
  • Does your training system generate evidence of comprehension, not just participation?
  • Did the investment in training provide the return we're seeking?

Most organisations are not yet equipped to answer those questions. Not because they aren't committed to quality - they clearly are - but because the tools we have relied on for years were never designed to answer them.

Three Models That Changed How I Think About This

KV is underpinned by solid educational thinking; models that I have been working with and incorporating into my practice for a long time. These three frameworks in particular have shaped how I understand the problem and why I believe KV is a legitimate response to it.

1. Benner's Novice to Expert Model

Patricia Benner's 1984 model describes how people develop expertise over time, moving from novice through advanced beginner, competent, proficient, and finally to expert. It was originally applied to nursing, but it has since shaped workforce development thinking across healthcare.

The key insight is this: training alone does not build expertise. Expertise develops through accumulated experience, through doing, reflecting, making decisions in context, and building judgment over time. Repeating a module does not accelerate that process. For experienced staff who already have the knowledge, it may actively work against engagement and professional confidence.

Benner's model asks us to meet people where they actually are. If someone is already competent, training them as if they are a novice is not just inefficient - it is disrespectful of what they have already built. KV operationalises that principle. It allows experienced staff to demonstrate what they already know, rather than repeat what they already learned.

Benner's Novice to Expert Model

2. Miller's Pyramid of Clinical Competence

George Miller's 1990 model describes the stages through which a learner moves on the way to competence: Knows, Knows How, Shows how, and then Does.

Each stage builds on the one before it. You cannot meaningfully assess skill until knowledge is in place.

KV sits at the base of that pyramid: the knows stage. This is important because it means KV is not trying to be something it is not. It does not assess skill. It does not replace direct observation or workplace-based competency assessment. It verifies whether the foundational knowledge is there.

It also means that for experienced staff who have the knowledge, KV can genuinely, from a clear educational standpoint, free up time and training resources to focus on the stages of Miller's pyramid that actually require more intensive investment: the shows how and does stages, where competency-based education genuinely lives.

Miller's Pyramid of Clinical Competence

3. The New World Kirkpatrick Model

This final model goes to the heart of the 'so what' question.

Donald Kirkpatrick's model of training evaluation describes four levels: Reaction, Learning, Behaviour, and Results.

Most organisations measure Level 1: did people complete the training? Did they find it relevant?

Very few measure Level 2: did they actually learn anything? And fewer still have the tools to assess Level 3 or 4 in any systematic way.

KV moves organisations from Level 1 to Level 2. That is not a small step. It is the difference between a completion record and a comprehension datapoint. It is the difference between knowing that someone sat through a module and knowing whether they understood it.

For providers investing significant costs in training and preparing for audits, this distinction is increasingly important when demonstrating the impact of training. It moves us closer to evidence of learning, not just evidence of participation. KV generates that evidence in a way that completion-only models simply cannot.

The New World Kirkpatrick Model

What This Means in Practice

These three models converge on the same conclusion:

The way we have been measuring mandatory training is not adequate.

Not because providers haven't been trying, but because the measurement tools we have available have been focused on the wrong metrics.

Completion rates tell us something. They are not worthless. But they are not sufficient evidence of a trained and capable workforce.

The question is no longer just 'did they complete it?' It is 'do they actually know it, and can we show that they do?'

KV addresses that question directly. It introduces a structured, summative assessment that maps to defined learning outcomes, draws from randomised question banks, requires active demonstration of knowledge, and generates a defensible record of comprehension. It does not replace mandatory training. It does not assess competence. It verifies knowledge.

Over time, the data KV generates also becomes evaluation intelligence. If pass rates on a particular module are consistently low, that tells you something. If a cohort of experienced staff verifies their knowledge on a topic with ease, that tells you something, too. Neither of those insights is available from completion records alone.

A Word on What KV Is Not

I want to be precise here, because precision matters when the stakes are high.

KV is not a shortcut. It is not a way to skip mandatory training. It is not a competency assessment. It is not a solution for competency-based education. This requires an entirely different tool, for an entirely different purpose.

Knowing something and being competent in it are not the same thing. A staff member can know the correct steps for managing a deteriorating patient and still need direct supervision and assessment before they can be considered competent.

KV verifies the 'knows' stage of Miller's pyramid. It does not reach the 'shows how' or 'does' stages. That is not a limitation - it is an honest description of what the tool is designed to do.

Used well, KV is one part of a broader training system. It generates better evaluation data. It targets training effort more efficiently. It recognises what experienced staff already know. And it gives organisations stronger evidence to take into an audit. It is not a replacement for a training system, but a meaningful contribution to one.

Why Now

I've spent the past three years reviewing regulatory changes, making sense of reform and assessing genuine need for training. The evidence base for training in response to regulatory change is solid. The cost of continuing that, beyond what's essential, is becoming too high.

That is where we are.

Training volumes are increasing. Staff are spending hours repeating content they already know. Organisations are investing heavily in compliance at the expense of capability. And the regulatory environment is now explicitly asking for evidence of comprehension, not just completion.

The educational frameworks we have had for decades have always pointed toward this. The question is whether we act on them or continue measuring completion because it is easier.

KV is one answer to that question. Not the only answer. But a rigorous, educationally grounded, and timely one.

Learn more

  • Read the companion guide: Practical Guide to Knowledge Verification.
  • View eligible modules via the Ausmed KV Savings Calculator.
  • Check out the KV launch webinar recording for a video walkthrough of the staff and manager experience.

You can also reach me directly at zoe@ausmed.com.au - I am always interested in hearing how providers are thinking about this.


Author

Zoe Youl - Head of Community at Ausmed

Zoe Youl 

Zoe Youl is a critical care nurse with over 15 years’ experience in healthcare education, workforce development and regulatory strategy. At Ausmed, she leads sector expertise, translating regulatory change into practical guidance for providers, and ensuring Ausmed's products, content, and commercial conversations are grounded in current regulatory knowledge, governance and educational best practices.

She has previously led Ausmed's learning and capability and clinical governance functions, and brings clinical expertise, deep familiarity with accreditation standards, and a grounding in educational theory to her work.