D1 · Scoping AI Solutions

This domain is about 12% of the OAI-API mock – roughly 7 of 60 items – and it mirrors the Academy Scope AI Solutions course (30 min). It tests the work you do before a single API call: turning a vague business ask into a defined job, a measurable success criterion and a cost envelope, and deciding whether AI is even the right tool. Most items describe a stakeholder request and ask what you should establish first, or whether the problem is a good fit for a language model at all.

What you need to know

Scoping is requirement elicitation for probabilistic software. You clarify the job to be done, the inputs and outputs, the quality bar and how it will be measured, the volume and latency the system must sustain, and the cost and risk envelope the business will accept. You decide build-vs-buy (a Responses API integration, a hosted product, or no AI at all), and you judge feasibility honestly – some tasks a model does well, some it does badly, and some it cannot verify. The output of scoping is a short solution plan that names the metric you will move and the smallest thing you can ship to test it.

Learning objectives

By the end of this page you should be able to:

1. Reframe a business request as a well-formed AI task with defined inputs, outputs and constraints.
2. Elicit requirements – volume, latency, quality bar, data sensitivity, integration points.
3. Define a success metric that is measurable before you build.
4. Estimate a cost envelope from token volume and model price, and check it against the business value.
5. Judge feasibility and choose build-vs-buy, including “do not use AI here”.
6. Write a solution plan that scopes the smallest useful first version.

1.1 Reframing a request as a task

Stakeholders describe outcomes (“make support faster”); you need a task (“draft a first-response reply from the ticket text and the last three tickets from the same customer”). A well-formed task names the inputs the model will see, the output shape it must produce, and the boundary of what it must not do.

1.2 Eliciting requirements

Six questions decide almost every design choice downstream. Ask them before you pick a model.

1.3 Defining a success metric

If you cannot measure it, you cannot improve it and you cannot prove it works. A scoping metric must be defined before build and be evaluable on a held-out dataset (this is where D1 hands off to D3, evals).

Task type            Good success metric                Bad metric
─────────────────    ───────────────────────────────    ─────────────────
Classification       accuracy / F1 on a labelled set    "feels accurate"
Extraction           field-level exact match            "looks complete"
Grounded Q&A         % answers supported by a citation  "sounds right"
Summarisation        rubric score by an LLM grader      "reads well"
Support drafting     % accepted by agent without edit   "faster" (unmeasured)

1.4 The cost envelope

Scope the cost before you build; a solution that works but costs more than the value it creates is a failed scope. Estimate from expected tokens and the model price. The September 2026 prices per million tokens:

Worked example: a triage classifier on gpt-5.6-luna, 800 input tokens and 40 output tokens per ticket, 50,000 tickets/day.

Input:  50,000 × 800  = 40,000,000 tok/day = 40 MTok × $0.20 = $8.00/day
Output: 50,000 × 40   =  2,000,000 tok/day =  2 MTok × $1.20 = $2.40/day
Daily total ≈ $10.40   →  ≈ $312/month

The same task on gpt-5.6-sol would be roughly 40 × $4 + 2 × $20 = $200/day (~$6,000/month) for no accuracy gain on a task Luna handles – the cost envelope is what rejects that choice.

1.5 Build versus buy versus don’t

1.6 Judging feasibility

             Does a ground-truth or a cheap check exist?
                        │
         ┌──────────────┴───────────────┐
         │ Yes                          │ No
         ▼                              ▼
   Measurable → good AI fit.      Can a human review each output
   Build with an eval loop.       at the required volume?
                                   │
                        ┌──────────┴──────────┐
                        │ Yes                 │ No
                        ▼                     ▼
                  Human-in-the-loop      Reconsider scope: narrow
                  assist is feasible.    the task, add a check, or
                                         don't ship AI here.

1.7 The solution plan template

The deliverable of scoping is a one-page plan. Every item on the mock that asks “what should the plan include” expects these fields.

Problem:        the business pain, in one sentence
Job to be done: the specific task, with inputs and outputs
Users & volume: who, how many requests/day, peak/min
Success metric: measurable, on a held-out dataset, target value
Constraints:    latency, data sensitivity, residency, budget
Cost envelope:  $/request estimate × volume vs. value created
Approach:       build/buy/no-AI, model, retrieval? tools?
Risks & gates:  failure modes, human review points
First version:  smallest shippable slice + how you'll evaluate it

Decision framework

Use the SCOPE framework to turn any request into a plan you could act on tomorrow.

The most valuable habit the course teaches is finishing C before starting P: teams that pick a model before defining the metric almost always over-buy.

Common mistakes

Scenario challenge

Scenario. You are a developer at a logistics company. The VP of Operations says: “Our dispatchers waste hours reading driver incident reports. Build an AI that handles them.” There is no dataset, no defined output, and the VP expects a demo in a week. Incident reports are free text, sometimes contain injury details (regulated), arrive at roughly 3,000/day with spikes after storms, and currently a dispatcher reads each one and either files it, escalates to safety, or requests more detail.

Expert reasoning trace.

1. Reframe the request into a job. “Handle them” is an outcome, not a task. The real job is: given a report, classify it into file / escalate / need-more-detail and extract structured fields (date, location, severity, injury flag). That is a classification-plus-extraction task, not open-ended chat.
2. Define the metric before building. Ask the safety team for 200 historical reports with the decision a dispatcher actually made. That labelled set gives an accuracy target and, critically, a way to check the escalation decision – the high-stakes one where a false negative is dangerous.
3. Surface data sensitivity early. Injury details are regulated. That drives retention (store: false or short retention), residency questions, and a mandatory human review gate on any escalate decision – you never auto-close a safety-relevant report.
4. Estimate the cost envelope. 3,000/day at ~1,200 input + 60 output tokens on gpt-5.6-luna is ~3.6 MTok × $0.20 + 0.18 MTok × $1.20 ≈ $0.72 + $0.22 = $0.94/day. Trivial versus dispatcher hours saved – the envelope is not the constraint here; correctness on escalation is.
5. Scope the first version, not the ideal one. Ship the classifier with human confirmation on every escalate, measured against the 200-report set. Defer auto-filing until the eval proves the escalation recall is high enough.
6. Reject the demo trap. A week-one demo that classifies with no eval and no review gate would look impressive and be dangerous. The plan says: build the eval set first, then the classifier, then measure.

Exam-correct decision: establish the labelled dataset and the escalation metric first, scope a human-in-the-loop first version, and size cost from real volume. Not “start prompting gpt-6-astra and demo it”, not “auto-close low-severity reports before measuring recall”.

Assessment traps

Practice questions

Each item states how many responses to select. Commit before revealing.

Key takeaways

• Scoping turns an outcome into a defined task with inputs, outputs and a measurable metric.
• Define the success metric on a held-out dataset before choosing a model.
• Elicit volume, latency, quality bar, data sensitivity, inputs and integration – they drive the architecture.
• Estimate the cost envelope from tokens × price/MTok × volume and compare it to business value.
• Build the differentiator, buy the commodity, and don’t force AI onto provable-exact deterministic work.
• Use the SCOPE framework; finish Criteria before choosing the Path.
• Ship the smallest first version that tests the metric, with review gates on high-stakes or irreversible steps.