Most AI training ends with a feedback form and a vague sense that it went well. Then, three months later, someone asks whether the money was worth it and nobody has an answer. The fix is cheap: decide what you will measure before the session, capture a baseline, and look again at fixed intervals. This guide sets out how we do that with Chicago clients, what each measure can and cannot tell you, and the claims we refuse to make.
One caveat up front. The honest return on AI training is partly unmeasurable in a small company. You will not get a laboratory-grade number for a 20-person office. What you can get is evidence good enough to decide whether to expand, repeat or stop, and that is the point of measuring.
What does AI training ROI mean?
AI training ROI is the value of the changes in how people work after training, compared with the full cost of that training. The full cost includes the fee, the hours people spent in the room instead of working, and any licenses bought because of it.
The most widely used structure for evaluating training is the Kirkpatrick Model, which has four levels: Reaction (did people find it relevant), Learning (did they gain knowledge, skill and confidence), Behavior (do they apply it on the job), and Results (what changed for the organization). Mapping AI training onto those levels stops you from treating a happy feedback form as proof of business value.
| Level | What to measure for AI training | How | When |
|---|---|---|---|
| 1. Reaction | Relevance to daily work, pace, clarity | Short end-of-session form, including "how useful was this for your daily work" on a 0–10 scale | Same day |
| 2. Learning | Self-rated confidence, ability to write a usable prompt and spot an error | 0–10 confidence before and after; a short practical check on a practice task | Before and at the end |
| 3. Behavior | Weekly use, saved instructions in use, tasks moved to AI with checking | Admin usage reports, prompt-library activity, a two-minute follow-up survey | 2, 6 and 12 weeks |
| 4. Results | Time on specific tasks, turnaround, rework, error rates | Time studies on named tasks; operational data you already keep | 6 to 12 weeks |
What baseline should you capture before AI training?
Capture a baseline for the specific tasks and people you expect training to change, in the two weeks before the session. Without it, every later number is a guess.
- Pick three to five tasks per role that recur weekly and that AI could plausibly help with: a client follow-up email, a meeting summary, a weekly report, a proposal section, a spreadsheet clean-up.
- Time each task a few times as people do it today. Ask them to log start and stop times in a shared sheet; a rough median is enough.
- Record quality the way you already judge it: how many revision rounds a proposal needs, how often a report has errors, how long a customer waits for a reply.
- Inventory current AI use. Who uses what, on which accounts, for what. Expect to find personal accounts. Record them without blame, or people will stop telling you.
- Ask for a 0–10 confidence rating on using AI at work, with the exact wording you will reuse afterward.
- Note the cost side: training fee, hours in the room, and licenses.
If you have not yet decided what the training should cover, the baseline work overlaps with an AI training needs assessment; do both at once.
Which adoption metrics matter after AI training?
The adoption metric that matters most is the share of trained people who still use the tool for work each week, six to twelve weeks later. Logins in the first week say little; habits form later.
Business plans give you some of this data. The Microsoft 365 admin center's Copilot usage report shows enabled users, active users and the active-user rate, and it counts someone as active only when they submit a prompt, not when they merely open the Copilot pane. ChatGPT Business gives owners and admins a workspace analytics view. Personal accounts give you nothing, which is one more reason to move work onto business plans.
Useful adoption measures for a small company:
- Weekly active users as a share of trained users, by team
- Number of saved instructions or shared prompts in active use, and who maintains them
- Recurring tasks that are now routinely done with AI and then checked by a person
- Questions coming into the internal AI channel (a rise early on is a good sign)
- Share of AI use on company accounts versus personal ones
Keep these at team level. If you ever use an AI system to evaluate individual employees for promotion, discipline or similar decisions, Illinois law requires notice to employees; our Illinois AI employment notice kit covers what that involves.
How do you run a time study on AI-assisted work?
A time study compares how long the same kind of task takes before and after training, with checking time included. It is the most persuasive measure for an owner and the easiest to get wrong.
Self-reported time savings are unreliable. In METR's early-2025 randomized study, 16 experienced open-source developers took 19% longer to finish tasks when allowed to use AI, yet afterward still believed AI had sped them up by 20%. METR has since published newer data on later tools, and coding is not office work, but the lesson about perception holds: ask people to time tasks, not to estimate savings.
To keep a time study honest:
- Use the same task types as the baseline, with comparable size and difficulty.
- Include the time spent checking and fixing AI output. A draft in two minutes that takes twenty to correct is not a saving.
- Collect several instances per person and use the median, not the best case.
- Record tasks where AI made things slower or worse; they tell you where not to use it.
- Note anything else that changed in the period (a new hire, a slow season, a new system).
Research shows why the last two points matter. In a Harvard Business School and Boston Consulting Group field experiment with 758 consultants, those using GPT-4 on tasks inside the AI's capabilities completed 12.2% more tasks, 25.1% faster, with more than 40% higher quality. On a task chosen to fall outside those capabilities, consultants using AI were 19 percentage points less likely to reach a correct answer. The same tool helped on some tasks and hurt on others.
How should you use a confidence survey like the 0–10 self-rating?
A confidence survey measures learning: whether people feel able to use AI on their work after the session. Ask the same question, with the same scale, before and after, and keep the individual pairs so you can see who moved.
We ask: "How confident are you using AI tools in your daily work? 0 means not at all, 10 means completely." In the results from the two-day program our founder led, attendees who started under 6 finished at an average of 8.2, up from 4.7. The average gain across all attendees was 35%, 83% rated themselves higher afterward, and all 16 of 16 sessions raised the group's average. Those numbers, and the program they came from, are on our AI training results page.
What this shows, and what it does not:
- It shows that people left the room more willing and able to try. That matters, because people who do not feel confident do not use the tool.
- It shows where the gain is concentrated. In our data the people with the lowest starting scores gained the most, which is useful when deciding who to train first.
- It does not show productivity, quality or hours saved. Confidence is a Level 2 measure. It needs Level 3 and 4 measures beside it.
- It can drift. Run the same question again at six or twelve weeks to see whether confidence held once people were back at their desks.
What should you not over-claim about AI training?
Do not claim more than your data supports. Over-claiming is the fastest way to lose a skeptical owner's trust, and AI training invites it.
- Extrapolated hours. "Each person saves five hours a week, times 50 weeks, times 20 people" multiplies an estimate by an assumption. Report measured time on named tasks instead.
- Borrowed research numbers. An NBER study of 5,179 customer-support agents found a generative AI assistant raised issues resolved per hour by 14% on average and by 34% for novice and low-skilled workers. That is evidence that the effect is real in some settings. It is not your team's number.
- Attribution. If you trained the team and bought licenses and changed a process in the same month, the training did not cause all of the change.
- Survivorship. Reporting only the people who kept using AI hides the ones who stopped, and why they stopped is often the most useful finding.
- Unchecked output as productivity. More documents produced is not a gain if someone has to fix them. Count finished, checked work.
- Testimonials as data. A quote saying "this saved me hours" is a good story and a weak measure. Keep quotes beside the numbers.
Our own rule: we report confidence gains and what clients tell us they kept using, and we do not promise hours saved.
How long after AI training should you measure?
Measure at the end of the session, again at two weeks, and a final time at about twelve weeks. Most drop-off happens in the first month, and most durable habits are visible by the third.
- Day of training: reaction form and the after-session confidence rating.
- Two weeks: a short survey on what people have used since, plus first usage numbers from admin reports.
- Six weeks: repeat the time study on the baseline tasks; review the shared prompt library.
- Twelve weeks: confidence again, weekly active users, and the time-study comparison. Decide whether to expand, run a follow-up session, or stop.
What does a simple AI training scorecard look like?
A one-page scorecard with six to eight measures is enough for a small company. Here is the shape we suggest; fill in your own numbers.
| Measure | Source | Baseline | 2 weeks | 12 weeks |
|---|---|---|---|---|
| Average confidence, 0–10 | Same survey question | Before session | After session | Re-survey |
| Weekly active users, % of trained | Admin usage report | Current use | Report | Report |
| Median time on task A, B, C | Time log | Timed | Optional | Timed |
| Revision rounds or error rate | Your existing records | Recorded | Not yet | Recorded |
| Saved instructions in use | Prompt library | Zero or current | Count | Count |
| Work on personal AI accounts | Survey | Inventory | Survey | Survey |
The training cost goes at the top of the page, next to any Illinois ETIP reimbursement you expect. ETIP can reimburse up to 50% of eligible training costs for Illinois employers, but it needs pre-approval before training starts and, for workshops, a certificate of completion; funding AI training in Illinois explains the steps, and the AI training cost guide covers the price side.
Can 312 help measure AI training in Chicago?
Yes. Every 312 IT Consulting session includes the before and after confidence rating, and in the first call we agree on one workflow to watch afterward. We train on-site across Chicago and the suburbs and live online, on your team's own ChatGPT, Copilot, Claude or Gemini accounts. A 2–3 hour AI workshop in Chicago for up to about 15 people typically starts from about $2,500; company-wide programs, like our corporate AI training, are quoted in writing after a free first call.