D6 · Measurement and ROI

This domain carries 16% of the mock — roughly 8 of 50 items. It tests whether a leader can prove value honestly: choosing leading and lagging indicators, baselining before deployment (the most-skipped and most-costly discipline), measuring quality and cycle time, understanding the unit economics of tokens and seats, confronting the attribution problem, and reporting to a board without the overclaiming that eventually destroys credibility. It closes the loop opened by opportunity sizing: the number you promised there is the number you must now measure.

What you need to know

Measurement fails at the start, not the end: if you did not baseline the current state before deploying, you can never prove improvement — so baselining is the first act, not an afterthought. Distinguish leading indicators (active use, cycle time, adoption — visible in weeks) from lagging indicators (cost, revenue, retention — visible in quarters). Measure quality alongside speed, or you will optimise throughput while degrading outcomes. Understand unit economics: cost is tokens (usage-priced per million) plus seats (per user), and value must clear that cost with margin. Confront attribution honestly — AI is rarely the only variable — and report to the board with baselines, ranges and counterfactuals rather than hero numbers, because one exposed overclaim discounts everything that follows.

Learning objectives

By the end of this page you should be able to:

1. Baseline the current state before deployment so improvement is provable.
2. Choose leading and lagging indicators appropriate to each initiative and time horizon.
3. Measure quality and cycle time together, not throughput alone.
4. Model unit economics from token and seat costs against realised value.
5. Address the attribution problem with controlled comparison where possible.
6. Report to a board credibly, with ranges, counterfactuals and no overclaiming.

6.1 Baseline before deployment

The single most common measurement failure is deploying first and asking “did it help?” afterwards — by which point there is no clean before-picture to compare against. Baseline first.

  RIGHT ORDER                     WRONG ORDER
  ───────────                     ───────────
  1. define the metric            1. deploy the tool
  2. measure the baseline         2. announce success
  3. deploy                       3. try to prove it later
  4. measure the change           4. discover there is no baseline
  5. attribute honestly           5. claim vibes as ROI

6.2 Leading vs lagging indicators

You need both, on different clocks. Leading indicators tell you early whether the initiative is on track; lagging indicators confirm the financial outcome later.

Reporting only lagging indicators leaves you blind for a quarter; reporting only leading indicators lets you mistake activity for value. Boards want the lagging outcome; the programme is steered by the leading ones. A healthy report shows leading indicators trending now and states when the lagging outcome will be measurable.

6.3 Quality and cycle time together

Speed without quality is a trap: AI can make people faster at producing worse work, and a cycle-time win that raises error or rework rates is a net loss. Always pair a throughput metric with a quality metric.

The leadership discipline: define the quality guardrail before celebrating the speed gain, and treat a speed improvement that breaches the guardrail as a failure, not a success.

6.4 Unit economics: tokens and seats

A leader must understand the two cost drivers, because “the pilot was free” on a trial becomes a real bill at scale.

  Value per period  >  (seats × seat price)
                     +  (tokens used × token price)
                     +  human review / oversight cost
                     +  run + support overhead

  If not, redesign: right-size the model tier,
  reclaim unused seats, tighten context length,
  or narrow scope — before scaling.

Two leadership levers follow directly. First, match the model tier to the task: routing high-volume, simple work to a lighter, cheaper model and reserving the top reasoning tier for genuinely hard work is often the largest single cost lever. Second, reclaim unused seats: paying for provisioned-but-idle seats is pure waste that a usage report exposes.

6.5 The attribution problem

AI is rarely the only thing that changed. Sales also hired, the market shifted, a process was tidied at the same time. Overclaiming attribution is the fastest route to losing board trust when a sceptic pulls the thread.

Where the stakes justify it, design an A/B or holdout from the start. Where you cannot, state the confounds openly and claim a range rather than the whole delta. Honesty about attribution is itself credibility: a leader who says “AI contributed an estimated 30–50% of this improvement, the rest is the process change” is believed; one who claims 100% is not.

6.6 Reporting to a board without overclaiming

The board report is where every earlier discipline is cashed in or squandered. One exposed overclaim discounts every future number you present.

  CREDIBLE BOARD LINE
  "Support handle time fell 34% (baseline 6.2 → 4.1 min) on the
   60% of tickets in scope. ~30% of the saved time is redeployed
   (cashable), the rest is capacity. Leading indicators are strong;
   the cost-to-serve outcome will be measurable next quarter.
   An A/B against two control regions attributes ~80% of the drop
   to the assist. Net run cost is below benefit with margin."

That is what “reporting without overclaiming” looks like: specific, bounded, attributed, and therefore trusted.

Decision framework

The P-R-O-V-E measurement plan

Attach this to every funded initiative before it deploys, so measurement is designed in, not retrofitted.

Common mistakes

Scenario challenge

Scenario. Your CEO wants to tell the board that “AI delivered £4m of value this year”. The claim rests on: a support team whose handle time “feels much faster” since ChatGPT assist launched (no before-measurement was taken); a sales region that grew 12% in the same period it adopted AI proposal drafting (and also hired two senior reps and entered a rising market); and a company-wide “£4m” figure produced by multiplying every user’s estimated minutes saved by their loaded rate — including 40% of seats that show no active use. The CFO is sceptical and the audit committee will ask questions.

Expert reasoning trace.

1. Refuse the composite hero number. “£4m” is exactly the claim that collapses under one sharp question, and collapsing it would discredit the entire AI programme. My job is to make the number smaller and defensible rather than large and fragile.

2. Fix the support claim — or downgrade it. “Feels faster” with no baseline is unprovable. I check whether handle-time logs predate the launch; if a baseline can be reconstructed, I report the measured delta on the in-scope tickets; if not, I report it as a leading-indicator trend to be measured properly next quarter, not as banked value. No baseline, no ROI claim.

3. Confront the sales attribution. A 12% regional growth coinciding with AI adoption is not evidence AI caused it — two senior hires and a rising market are large confounds. The honest move is a cohort or holdout comparison against similar regions; absent that, I claim only a range of the uplift and state the confounds. Claiming the full 12% would be the overclaim the audit committee is built to catch.

4. Strip the seat inflation. Multiplying estimated minutes by loaded rate across all seats — including 40% idle — is double nonsense: it counts capacity as cash and counts non-users as beneficiaries. I remove idle seats entirely (and flag the wasted seat spend as a cost to reclaim, per D4), and split the remaining benefit into cashable vs capacity.

5. Rebuild the report credibly. Present the measured support delta (or a labelled trend), a ranged and attributed sales contribution, and a defensible productivity figure net of idle seats and split cashable/capacity — with the run cost (tokens + seats + oversight) subtracted. The total will be well below £4m and utterly defensible.

6. Give the board the method, not just the number. Baselines, ranges, counterfactuals and the reclaimed-seat action signal a leader in control — which is worth more than a large fragile figure.

Board-ready outcome: the £4m composite is declined; the support claim is baselined or downgraded to a trend; the sales uplift is attributed with a holdout/cohort and reported as a range; idle seats are stripped and flagged for reclaim; the final number is smaller, attributed, cost-net and trusted — and the programme’s credibility survives the audit committee.

Assessment traps

Practice questions

Each item states how many responses to select. Commit before revealing.

Key takeaways

• Baseline before deployment; without a before-picture, improvement is unprovable and the board discounts the claim.
• Report leading indicators (use, cycle time, quality — weeks) and lagging indicators (cost, revenue, margin — quarters) together.
• Pair speed with quality; a cycle-time win that breaches the quality guardrail is a net loss.
• Model unit economics — tokens plus seats plus oversight — and right-size the model tier; reclaim idle seats.
• Confront attribution honestly with A/B or holdout comparisons, or state the confounds and claim a range.
• Report without overclaiming: baselines, ranges, counterfactuals, cashable-vs-capacity — one exposed overclaim discredits everything.
• Attach P-R-O-V-E to every initiative before it deploys: pre-baseline, right indicators, own the economics, verify attribution, evidence to the board.