D2 · Model Selection and Optimization

This domain is roughly 9 of 53 items. It tests whether you can pick the cheapest model that meets the quality bar and then drive cost and latency down with caching, batching, routing and output control – while pinning versions and handling breaking changes across releases. Expect worked-cost scenarios and “which model / which lever” trade-off questions.

Learning objectives

By the end of this page you should be able to:

1. Explain LLM fundamentals: tokens, tokenisation cost, context window, sampling and non-determinism.
2. Configure model options: fast mode, extended/adaptive thinking, effort levels, and where budget_tokens still applies.
3. Compare the model tiers (Haiku 4.5, Sonnet 5, Opus 5, Fable 5.1) on capability, price and latency, and do worked cost calculations.
4. Identify breaking changes across releases and plan pinning and migration.
5. Apply cost levers: token budgeting, prompt caching, batching, and routing/cascading.
6. Apply latency levers: streaming, smaller models, shorter output, caching.

2.1 LLM fundamentals

• Tokens are sub-word units; billing and limits are in tokens, not characters. Roughly 1 token ≈ 4 characters ≈ 0.75 words in English.
• Tokenisation cost: both input and output are metered in tokens; longer prompts and outputs cost more and add latency.
• Context window is the total tokens (input + output) the model can attend to in one request – 1M on Fable 5.1 / Opus 5 / Sonnet 5, 200k on Haiku 4.5. max_tokens caps output within that window.
• Sampling: the model outputs a probability distribution; temperature and top_p control how it is sampled. Lower = more deterministic.
• Non-determinism: even at temperature: 0, outputs are not guaranteed byte-identical across runs or model versions. Design for it – validate output, do not assume exact repeats.

2.2 Model options: thinking, effort and fast mode

• Use xhigh effort only for the hardest coding/agentic work (Opus 5 / Fable 5.1) – it costs more thinking tokens and adds latency.
• budget_tokens returns 400 on Fable 5.x / Opus 5 / Sonnet 5. Only Haiku 4.5 still accepts it, and Haiku has no effort parameter.
• Fable 5.1 always thinks; you cannot disable it.

2.3 Model tiers, pricing and trade-offs

Rule of thumb: start at Sonnet 5; drop to Haiku 4.5 for high-volume simple work; move to Opus 5 for hard agentic/coding tasks; reserve Fable 5.1 for the most demanding reasoning (accepting its constraints and 30-day retention).

Worked cost calculation

A task sends 3,000 input tokens and generates 500 output tokens, 20,000 times/day.

If Haiku 4.5 meets quality, it is 5× cheaper than Opus 5 here. Never default to the most capable model for simple, high-volume work.

2.4 Breaking changes across releases

2.5 Pinning and migration

• Pin a snapshot in production so behaviour does not shift under you. Model IDs from 4.6 onward are dateless but are still pinned snapshots.
• Store the model ID in one env-driven place so migration is a single change.
• Migration checklist: re-run the golden eval set on the new model; check for removed parameters (budget_tokens), changed tool_choice behaviour, latency and cost deltas; roll out behind a flag.
• Use client.models.list() / .retrieve(id) for live limits and availability.

import os
MODEL = os.environ.get("CLAUDE_MODEL", "claude-sonnet-5")   # pinned, one place

2.6 Token budgeting

Control tokens on both sides:

• Input: trim context, cache stable prefixes, prune old tool results, summarise long histories.
• Output: set a realistic max_tokens, ask for concise/structured output, stop early with stop_sequences.
• Thinking: adaptive by default; use effort levels rather than paying for xhigh everywhere.

Every token in the context window is billed each turn, so long conversations get expensive – budget with caching, editing and compaction (Domain 4).

2.7 Cost levers: caching, batching, routing

Request ─► Haiku 4.5 (cheap, fast)
             │
             ├─ confident / simple ─► return
             └─ hard / low-confidence ─► Sonnet 5 ─► (rare) Opus 5

2.8 Latency levers

2.9 Worked cost comparison: a 10k-document job

Consider a batch extraction job over 10,000 documents. Each request sends a 2,000-token shared prefix (system + schema + instructions) plus 1,000 tokens of unique document text (3,000 input tokens total) and generates 500 output tokens. Below is the arithmetic for each tier, first synchronous and uncached, then with caching, then via the Batches API.

Baseline (synchronous, no caching)

Cost per request = (input tokens × in-price + output tokens × out-price) ÷ 1,000,000, times 10,000 documents.

With prompt caching on the 2,000-token shared prefix

The 2,000-token prefix is written to cache once (≈1.25× the base input price for the 5-minute TTL) and read at ≈0.1× thereafter. The 1,000 unique tokens per document are never cacheable. For Sonnet 5:

• Cache write (first request): 2,000 × $2 × 1.25 / 1e6 = $0.005 (once, negligible).
• Prefix cache reads (remaining 9,999 requests): ≈2,000 × 10k × $2 × 0.1 / 1e6 = $4 (versus $40 uncached for the prefix portion).
• Unique input (never cached): 1,000 × 10k × $2 / 1e6 = $20.
• Output unchanged: $50.
• Sonnet 5 with caching ≈ $74 versus $110 — the shared prefix drops from $40 to about $4.

With the Message Batches API (50% off input and output)

Batching halves every line above and returns results within 24 hours — ideal because this job is latency-tolerant.

2.10 Latency levers in depth

Total latency has two visible parts: time-to-first-token (TTFT) and total generation time. Different levers move different parts. Confusing them is a classic exam trap.

Request ──► [cache read prefix] ──► [thinking effort] ──► [generate output]
   │              (caching)            (effort/fast)         (max_tokens)
   └── stream deltas to user as soon as generation starts (perceived latency)

2.11 Pinning and migration checklists for breaking changes

Pin a snapshot in production and drive the model ID from one env-backed constant. When you migrate — or when a release ships a breaking change — walk a checklist rather than swapping IDs blind.

General migration checklist

1. Change the single model constant behind a feature flag (do not edit IDs across many files).

2. Re-run the golden eval set on the candidate model; compare quality, latency p50/p95, and cost per task.

3. Scan the request payload for removed parameters (see below) and changed behaviours (tool_choice, thinking binding).

4. Roll out to a small traffic slice, watch error rates (400 spikes signal a rejected parameter), then ramp.

Breaking-change specifics

2.12 A routing / cascade implementation

Routing keeps most traffic on a cheap model and escalates only the hard tail. The decision to escalate should come from a structured signal (a classifier label, a validation failure, or a refusal/low-confidence tool result) — never from parsing prose.

import os
from anthropic import Anthropic

client = Anthropic()

CHEAP = "claude-haiku-4-5"
MID = "claude-sonnet-5"
HARD = "claude-opus-5"

def answer(messages, tools=None):
    """Cascade: try cheap, escalate on refusal/low confidence or validation failure."""
    resp = client.messages.create(model=CHEAP, max_tokens=1024, messages=messages, tools=tools or [])
    if resp.stop_reason == "refusal" or not passes_quality_gate(resp):
        resp = client.messages.create(model=MID, max_tokens=2048, messages=messages, tools=tools or [])
        if resp.stop_reason == "refusal" or not passes_quality_gate(resp):
            resp = client.messages.create(model=HARD, max_tokens=4096, messages=messages, tools=tools or [])
    return resp

def passes_quality_gate(resp) -> bool:
    """Programmatic gate: schema validity, required fields present, no empty answer.
    Do NOT use the model's self-reported confidence (anti-pattern #4)."""
    text = "".join(b.text for b in resp.content if getattr(b, "type", None) == "text")
    return len(text.strip()) > 0 and "I cannot" not in text

Request ─► Haiku 4.5 ──(gate passes)──► return
              │
              └─(refusal / gate fails)─► Sonnet 5 ──(gate passes)──► return
                                            │
                                            └─(rare)─► Opus 5 ──► return

2.13 A cost model you can compute in your head

The exam expects fast, defensible arithmetic. Memorise the per-MTok prices and the four multipliers, then estimate.

Prices per MTok: Haiku 4.5 $1/$5, Sonnet 5 $2/$10, Opus 5 $5/$25, Fable 5.1 $10/$50.

Worked drill. 4,000 input + 800 output tokens, 30,000×/day, on Haiku vs Sonnet:

Haiku is exactly half here — but only choose it if it clears the quality bar. Then, if the workload is latency-tolerant, halve again with Batches, and cut the shared-prefix portion to ~10% with caching.

2.14 Choosing thinking effort deliberately

Effort and thinking mode trade quality for latency and token cost. The exam tests picking the minimum that clears the bar.

2.15 Common misconceptions

2.16 Scenario walkthrough: taming a runaway model bill

Scenario. A team runs everything on Opus 5 ‘to be safe’. The workload is 95% short FAQ-style classifications and 5% genuinely hard multi-step reasoning. The bill is 5× budget. Latency for the classifications must stay interactive; the hard cases can take longer. They also run a nightly 20,000-item eval on the same Opus 5. A proposal on the table: ‘escalate to Fable 5.1 whenever Opus 5 reports confidence below 0.9.’ You must cut cost without hurting the hard cases or the eval’s fidelity.

Expert reasoning trace.

1. Split the traffic by difficulty. 95% is trivial → move it to Haiku 4.5 (cheapest, fast enough to stay interactive). Keep a path to a bigger model for the 5%. Running everything on Opus 5 is the constraint-blind default that caused the overspend.
2. Design the escalation gate correctly. Escalate on a programmatic signal — a classifier label, a schema-validation failure, or a refusal — not on the model’s self-reported confidence (anti-pattern #4). The proposed 0.9-confidence gate is the wrong design.
3. Pick the escalation target by need, not prestige. Most hard cases clear on Sonnet 5; reserve Opus 5 for the rare hardest tail. Jumping straight to Fable 5.1 is over-engineered and adds its 30-day-retention and append-only constraints for no benefit here.
4. Handle the nightly eval separately. It is latency-tolerant and bulk → run it via the Batches API (50% off) on the cheapest model that clears the eval’s quality bar. Do not confuse the eval’s model with the production classifier’s.
5. Verify with arithmetic. If the 95% moves from Opus ($5/$25) to Haiku ($1/$5), the bulk cost drops ~5×; only the 5% tail pays Sonnet/Opus rates. That alone likely brings the bill within budget.
6. Reject the tempting alternatives. ‘Keep Opus 5 but lower max_tokens’ — trims output only, not the model-tier overspend. ‘Use Fable 5.1 for the hard cases because it is most capable’ — over-engineered and adds retention/append-only constraints. ‘Escalate on confidence’ — self-report reliance.

Correct decision. Cascade Haiku 4.5 → Sonnet 5 → (rare) Opus 5 with a programmatic escalation gate; run the nightly eval on the cheapest adequate model via Batches. Pin snapshots and gate the change behind the golden eval set before rollout.

Exam traps in this domain

Practice questions

Key takeaways

• Tokens drive cost, latency and limits; the context window holds input + output, max_tokens caps output only.
• temperature: 0 reduces variance but is not byte-deterministic; pin a snapshot for reproducibility.
• Start at Sonnet 5; Haiku 4.5 for cheap high-volume, Opus 5 for hard agentic work, Fable 5.1 for the hardest.
• budget_tokens is Haiku-4.5-only; effort levels apply to Opus 5 / Sonnet 5 / Fable 5.1.
• Know the breaking changes: budget_tokens removal, Fable 5.1 tool_choice/thinking-binding/append-only, Opus 4.1 retirement.
• Cost levers: caching (~10% on hits), batching (50% off), routing/cascading cheap→expensive.
• Latency levers: streaming (perceived), smaller model / shorter output (actual), caching, fast mode.
• Compute cost in your head: input×price + output×price, then ×0.5 for Batches and ×0.1 for cached-prefix hits; the cheapest adequate model wins.
• Pick the minimum thinking effort that clears the bar; xhigh is for the hardest work only, budget_tokens is Haiku-4.5-only, and ‘Haiku + effort’ is invalid.
• Rate limits are about token/request volume — fix them with caching, trimming, batching or a higher tier, not by changing model tier.
• Treat a production model choice and an offline eval’s model choice as independent decisions; evals are latency-tolerant, so use the cheapest adequate model via Batches.