You measure whether AI training worked by recording a baseline for a few specific workflows before the session, then checking at 30, 60 and 90 days whether people are still using AI on that work, whether it takes less time or effort, and whether the quality held up.
Satisfaction scores and attendance tell you whether people liked the session. They don’t tell you whether anything about the work changed, and that is the question leadership is really asking.
Who this is for
COOs, CFOs, heads of people and team leads who have run AI training, or are about to, and need a clear answer on whether it paid off and what to do next.
The short version
- Pick two or three named workflows per team and measure those, not “AI usage” in general.
- Capture a simple baseline before training, even if it is rough.
- Look at four layers: whether people use it, how often, what changed in the work, and whether quality held.
Why most training measurement falls short
Training is often judged on the day it happens: attendance, ratings and confidence. Those say little about behavior a month later.
The other common approach is to count logins or messages in the AI tool’s admin dashboard. That is better, but activity is not the same as value. Someone can send many messages while doing low-value work, and someone else can send a few that save a morning.
Useful measurement ties training to specific work the firm cares about.
Step 1: Choose what you will measure before training
For each team, pick two or three workflows the training will focus on. Good candidates are recurring, take real time, and produce something a reviewer can judge, such as a first draft of a memo, a meeting summary, a weekly report or a research brief.
For each workflow, write down:
- who does it and how often
- roughly how long it takes today
- who reviews it, and what they usually send back
- what “better” would look like: faster, fewer revisions, more consistent, or freeing senior time
If you can’t name the workflows, the training itself is probably too generic.
Step 2: Capture a light baseline
You don’t need a time-and-motion study. A short survey or a five-minute conversation with each participant is usually enough:
- How long does this task take you today?
- How often do you do it?
- How many review rounds does it usually go through?
- How often do you use AI for it now, if at all?
Estimates are fine as long as you ask the same questions the same way afterwards. Consistency matters more than precision.
Step 3: Measure four layers at 30, 60 and 90 days
| Layer | What to look at | How to get it |
|---|---|---|
| Use | Are trained people still using the approved tool at all? | Admin dashboard or a quick survey |
| Habit | Are they using it on the chosen workflows, regularly? | Short check-in or self-report |
| Work outcome | Has time, effort or turnaround changed on those workflows? | Same questions as the baseline |
| Quality | Is the output as good or better, with no new errors reaching clients? | Reviewer feedback, revision counts |
Checking at 30 days shows whether the habit started. At 90 days you can see whether it stuck or quietly faded. If usage peaks and then drops, that is useful to know, not something to hide.
Step 4: Ask reviewers, not only participants
People tend to overestimate their own time savings and underestimate quality problems. The person who reviews the work is often the best judge of whether output improved. Ask reviewers three simple questions:
- Is the work arriving faster, slower or about the same?
- Is it better, worse or about the same?
- Have you seen any new kinds of errors?
If reviewers report new errors, that is a training gap to fix, usually around checking output against sources.
Step 5: Turn the results into a decision
Group what you find into three outcomes:
- Working. Usage held and the workflow improved. Write down how the team does it, share it with similar teams, and consider whether the workflow now deserves a more permanent setup.
- Partly working. Some people use it, others stopped. Find out why. Common reasons are that the examples were not close enough to their work, they lacked time to practice, or they were unsure what was allowed.
- Not working. Usage faded and nothing changed. Before blaming the training, check whether the workflow was a good fit for AI in the first place, and whether managers made room for it.
Report these in plain language to leadership, with a few concrete examples alongside the numbers.
A simple measurement template
For each team, one page is enough:
- Workflows covered and why they were chosen.
- Baseline time, frequency and review rounds.
- Results at 30, 60 and 90 days on the same measures.
- Reviewer comments on quality.
- What we will do next: scale, adjust, retrain or stop.
Common mistakes
- Measuring only satisfaction. A well-liked session can still change nothing.
- Skipping the baseline. Without a before, any after is a guess.
- Counting messages as value. Activity is a starting signal, not the result.
- Measuring once. A single check right after training will usually look better than reality.
- Treating weak results as failure. They usually point to a specific, fixable gap.
When you do not need us
If a team lead or someone in operations can own the baseline and run three short check-ins, you can measure training well with the template above. Outside help is most useful when you want training designed around measurable workflows from the start, or an independent read on results before you scale a program across the firm.
How Ruby helps
- AI training: sessions built around named workflows, so there is something concrete to measure.
- AI strategy: setting up the baseline and check-ins, and turning results into a clear next step.
- AI engineering: building a more permanent system for workflows that prove their value.
A practical first step is one hour of hands-on AI training for one team, with a baseline captured beforehand.