D6 · Repeatability and Improvement

This domain is worth 14% of the mock — roughly 7 of 50 items. It is where “a thing I do” becomes “a thing our team does”: a workflow that is documented so someone else can run it, versioned so changes are traceable, measured so you know whether it is any good, and improved on evidence rather than on how it feels. It is the payoff of the whole track — decomposition, contracts, capability choice and oversight only compound if the workflow is repeatable and gets better over time.

What you need to know

A repeatable workflow is one a colleague can run end to end without asking you a question, because it is documented (a runbook: trigger, steps, inputs, outputs, review points, what to do when it fails). Prompts and configurations are versioned — dated, with a change note — so you can tell what changed and roll back if a change makes things worse. Quality and cycle time are measured against the acceptance criteria and the clock, so improvement is judged on numbers, not vibes. Iteration is evidence-based: change one thing, measure, keep it only if the metric improved. The failure mode is the workflow that lives only in the author’s head, changes silently, and is “improved” by intuition with no way to tell if it actually got better.

Learning objectives

By the end of this page you should be able to:

1. Document a workflow as a runbook a colleague can execute without you.
2. Version prompts and configurations so changes are traceable and reversible.
3. Measure a workflow on quality (against acceptance criteria) and cycle time.
4. Iterate on evidence: change one variable, measure, keep or revert.
5. Recognise vibes-based “improvement” and the single-author bottleneck as anti-patterns.

6.1 The runbook — documenting for handover

A workflow that only you can run is a liability, not an asset. The test is blunt: could a colleague run this end to end without asking you a single question? If not, it is not repeatable. A runbook makes it so.

RUNBOOK: Weekly deal-risk digest
  TRIGGER      Monday 09:00, or on demand
  INPUTS       this week's rep notes; CRM export; (Project holds style guide + template)
  STEPS        1. Extract slipped deals (data analysis on the export)
               2. Pull latest news per top-5 account (search)
               3. Draft digest in the house template
               4. REVIEW gate: check figures reconcile, tone on-brand
  OUTPUT       one-page digest posted to #sales
  DONE         digest posted; no unreconciled figures
  ON FAILURE   if CRM export missing → request it, do not estimate
  OWNER        Sales Ops; backup: Ops lead

6.2 Versioning prompts and configuration

Prompts and Project/GPT configurations drift as you tweak them. Without versioning you cannot answer two essential questions: what changed? and can we go back?

Worked example. A support-reply prompt is edited to “be more concise.” Replies get shorter — and the customer-satisfaction sample dips because they now omit a needed next step. Because the change was a single dated version with a note, you roll back to the prior version and instead change only the length guidance while keeping the next-step instruction. Without versioning you would be guessing which of several edits caused the dip.

6.3 Measuring quality

You cannot improve what you do not measure, and “it seems better” is not a measurement. Quality is measured against the step’s acceptance criteria from Domain 3.

The review gate and sampling from Domain 5 are your data source: every sampled item is a quality measurement. A workflow with a sampling regime already has a quality metric — the sampled error rate — and just needs it tracked over time.

6.4 Measuring cycle time

Quality is half the picture; the other half is cycle time — how long the workflow takes end to end, and how much of that is human vs automated. Measuring it tells you where the value (and the bottleneck) actually is.

Manual baseline:   ██████████████████████  120 min/run (all human)
After workflow:    ████  20 min/run  ── of which ──
                   ██ 12 min automated   ██ 8 min human review

Two uses:

• Justify the workflow. 120 → 20 minutes is the payback you screened for in Domain 1.
• Find the next improvement. If 8 of the 20 minutes are human review, the biggest remaining lever is the review design (Domain 5), not a better prompt. Measure before you optimise, or you will optimise the wrong step.

6.5 Iterating on evidence, not vibes

Improvement is a loop: hypothesis → change one thing → measure against the metric → keep if better, revert if not. “Vibes” improvement — changing several things because the output “feels” better — cannot tell you what helped, cannot be defended, and often makes things quietly worse.

EVIDENCE-BASED                          VIBES-BASED (anti-pattern)
 1 form a hypothesis                     1 tweak several things at once
 2 change ONE variable                   2 read a couple of outputs
 3 measure against the metric            3 "feels better", ship it
 4 keep or revert on the number          4 no metric, no attribution, no rollback

6.6 Closing the loop with the other domains

Repeatability is where the track’s pieces connect: acceptance criteria (D3) are the quality metric; the review sampling (D5) is the data source; the runbook documents the decomposition (D2) and capability choices (D4); and the measured payback validates the opportunity you scoped (D1). A workflow that is documented, versioned, measured and iterated is the finished product the whole Applied AI course is teaching you to build — and it is what a colleague can run, a manager can trust, and you can keep improving without guessing.

Decision framework

The DRIVE loop — the maintenance cycle for a live workflow.

Run DRIVE whenever the workflow is live and being improved. The discipline is in V and E: one change, measured, kept or reverted on the number — never a bundle of tweaks judged by feel.

Common mistakes

Scenario challenge

Scenario. Sofia built a contract-review workflow that flags risky clauses and drafts a summary memo. It saves her about 90 minutes per contract and she is proud of it. Three things are now going wrong. First, she is going on leave and her backup cannot run it — it exists as a set of prompts Sofia has “in her head” and pastes from a scratch doc. Second, over the last month she has edited the flagging prompt “a bunch of times” to catch more clause types, and lately it flags far more false positives, but she can’t tell which edit caused it. Third, her manager asks “is it actually better than before?” and Sofia can only say it “feels more thorough.” She wants to keep tuning the prompt until it feels right.

Expert reasoning trace.

1. Diagnose all three as repeatability failures, not prompt-quality failures. The backup can’t run it (documentation gap), the false-positive regression is untraceable (versioning gap), and “feels more thorough” (measurement gap). None is fixed by more prompt tuning; tuning-by-feel is precisely what created the regression.
2. Fix handover with a runbook. Write trigger, inputs, the ordered steps with their capabilities, the review gate, the definition of done, and what to do when a contract is missing a section — so the backup runs it unaided. “Record a video” or “they’ll figure it out” fails the no-questions test.
3. Introduce versioning to isolate the regression. The false positives appeared after “a bunch of” undated edits, so there is no way to attribute or roll back. Adopt dated versions with one-line change notes, restore the last version known to have a low false-positive rate, then re-apply the individual clause-type additions one at a time, measuring after each, to find the edit that introduced the noise.
4. Instrument quality and cycle time. Define the acceptance criteria (which clause types must be flagged; acceptable false-positive rate) and measure the sampled error/false-positive rate and the end-to-end time. Now “is it better?” has an answer: a number, before and after, against a baseline.
5. Switch to evidence-based iteration. Replace “tune until it feels right” with the DRIVE loop: hypothesise, change one clause-type rule, measure the false-positive rate on a fixed sample, keep or revert. This both fixes the current regression and prevents the next one.
6. Answer the manager properly. Not “it feels more thorough” but “against a 40-contract sample it flags 96% of the target clause types (baseline was manual) at an 8% false-positive rate, down from 19% last week after we reverted the bad edit, and cuts review time from ~110 to ~20 minutes.”

The decision: treat the problems as documentation, versioning and measurement gaps — write the runbook, adopt dated one-change-at-a-time versioning to isolate and revert the regression, and instrument quality and cycle time so iteration is evidence-based — not “keep tuning until it feels right,” which is the vibes anti-pattern that caused the regression in the first place.

Assessment traps

Practice questions

Each item states how many responses to select. Commit before revealing.

Key takeaways

• A repeatable workflow is documented as a runbook — trigger, steps with capabilities, review points, definition of done, failure handling, owner — so a colleague runs it without asking you a question.
• Version prompts and configurations: date them, note the change, keep the prior version, change one thing at a time — so effects are attributable and regressions are reversible.
• Measure quality against acceptance criteria (using the sampling regime as your data source) and measure cycle time to find the real bottleneck.
• Iterate on evidence, not vibes: hypothesis → one change → measure → keep or revert. “It feels better” is not a measurement.
• Measure before you optimise, or you will tune the prompt when the review step was the bottleneck.
• Repeatability closes the loop on the track: D3 criteria are the metric, D5 sampling is the data, the runbook records D2 and D4, and cycle-time payback validates the D1 opportunity.
• The DRIVE loop — Document, Record versions, Instrument, Vary one thing, Evaluate — is the maintenance cycle for a live workflow.