D2 · How Language Models Behave

This domain is worth 16% of the mock – roughly 10 of 60 items, the second-heaviest on the track. Domain 1 said the model predicts likely text; this domain is about the behaviours that fall out of that fact and how they shape your day-to-day use. It tests whether you can predict how the model will act in a given situation – when it will drift, when it will agree with you too readily, when it will invent, and when a different effort setting or surface would fix it.

What you need to know

A model has a finite context window – the total tokens it can attend to at once, and in ChatGPT that limit depends on your plan and whether you are using a fast or a reasoning model. Reasoning effort trades thinking time and cost for quality on hard problems. Hallucination is fluent, plausible content that is not grounded in truth; sycophancy is the tendency to agree with the user’s stated view. Output is non-deterministic – the same prompt can produce different wording. Models are unreliable at exact arithmetic and counting. Above all, fluency is not accuracy: reading well is a property of the generator, never evidence that the content is correct.

Learning objectives

By the end of this page you should be able to:

1. Explain what a context window is and read the ChatGPT plan-dependent limits.
2. Choose an appropriate reasoning effort for a task and justify it by stakes.
3. Recognise hallucination mechanics and where they cluster.
4. Detect and counter sycophancy in your own prompting.
5. Explain non-determinism and its consequences for reproducibility.
6. Predict where arithmetic and counting will fail and route around it.

2.1 The context window

The context window is the maximum amount of text – measured in tokens – the model can consider in one interaction. It has to hold everything: the system instructions, any memories, your uploaded files, the whole conversation so far, and the reply being generated. When you exceed it, the earliest content is squeezed out and effectively forgotten.

In ChatGPT the size is plan-dependent and model-dependent:

Two things new users get wrong:

• The total window is larger than the space available for your input: system instructions, memories and the model’s own internal processing consume part of it, so the usable input share is smaller than the headline number.
• A bigger window is not a filing cabinet. Everything in the window competes for the model’s attention; stuffing it with marginally relevant text can reduce answer quality even before you hit the limit (see D5 on context hygiene).

┌──────────────── context window (total tokens) ─────────────────┐
│ system instructions │ memories │ files │ conversation │ reply   │
└────────────────────────────────────────────────────────────────┘
                       ▲ everything shares one budget;
                         the oldest turns fall out first when full

2.2 Reasoning effort

Some models can spend extra internal “thinking” before answering. In ChatGPT this appears as a choice between fast/instant models and reasoning models (and, in the API, as explicit reasoning-effort levels from none up to max). More effort helps on multi-step, ambiguous or high-stakes problems; it costs more latency and, on the API, more money.

The guidance from OpenAI’s own docs is to use the lowest reasoning effort that still gets the result. Reaching for maximum effort on a simple task wastes time and money; using a fast model on a genuinely hard problem produces a confident, shallow answer.

2.3 Hallucination mechanics

A hallucination is output that is fluent and plausible but not grounded in the input or in reality. It is not a bug you can switch off; it is a direct consequence of a system that generates the most probable continuation rather than a verified one. Hallucinations cluster predictably:

• Specific numbers – percentages, prices, dates, counts, especially on niche topics.
• Citations – paper titles, authors, URLs, case names, often a real-sounding blend of real elements.
• Proper nouns – products, people, internal names the model never saw.
• Gap-filling – when the source does not contain the answer, the model may produce one anyway.

question ─► [model wants a probable continuation]
   │
   ├─ grounded in supplied/retrieved text ─► usually reliable, still check
   └─ no grounding, sparse training data  ─► plausible invention (hallucination)

Mitigations that actually work: ask for sources and open them, ground the model in supplied documents or retrieval, ask “what in the provided material supports this?”, and verify anything checkable. Things that do not fix hallucination: lowering temperature (reduces variance, not ungroundedness), asking “are you sure?” in the same chat, or switching to a more expensive model.

2.4 Sycophancy

Sycophancy is the model’s tendency to agree with the position you state, praise your idea, or soften a critique because you signalled a preference. It happens because agreeable, affirming continuations are common and well-received in training data.

The reliable counters: frame questions neutrally, ask for the strongest counter-argument, and – for a genuinely independent critique – ask in a fresh chat without stating your preferred answer.

2.5 Determinism

LLM output is non-deterministic by default: the same prompt can produce different wording on different runs because tokens are sampled from a probability distribution. Lower randomness (temperature, on the API) narrows the variation but does not guarantee identical text, and it does not make the content more true – only more consistent.

Practical consequence: if you need a specific good output, save it rather than assuming you can regenerate it; and never conclude an answer is verified just because two runs agree in wording.

2.6 Arithmetic and counting weaknesses

Because the model predicts likely tokens rather than computing, it is unreliable at exact arithmetic, long multi-step calculation, and precise counting (items in a list, words in a passage, letters in a word – the last compounded by tokenisation from D1).

"add these 40 numbers" ─► model guesses a plausible total  ✗
"add these 40 numbers" ─► data-analysis tool computes it    ✓ (real arithmetic)

2.7 Fluency is not accuracy

This is the sentence the whole track turns on. Because the model is optimised to produce natural, confident, well-structured text, its fluency tells you nothing about whether the content is correct. Polished formatting, a confident tone, and speed of delivery are all properties of the generator, not evidence of truth.

The behavioural takeaway that Domain 6 will operationalise: treat anything you will act on as a draft to verify, proportional to the stakes.

Decision framework

Use the behaviour-to-fix map to turn an observed misbehaviour into the correct response instead of a random knob-twiddle.

Common mistakes

Scenario challenge

Scenario. Devin is preparing a pricing recommendation. Over a long chat he has pasted three reports, a spreadsheet, and dozens of follow-ups. Late in the conversation he asks ChatGPT to “confirm the total addressable market from the first report”, and it returns a figure that does not match what he remembers. He also asked earlier, “our margin is healthy, agree?” and it agreed enthusiastically. Finally he asked it to sum a 30-row cost table and it produced a total that a colleague says is off by a few thousand. Devin is on a Business plan using a fast model, and the deadline is in an hour.

Expert reasoning trace.

1. The forgotten figure is a context-window symptom. After a long chat with three reports pasted in, the first report’s detail may have been evicted from the window. The right move is not to argue with the model but to re-supply the specific passage (or move the reports into a project/file) so the number is back in context – then verify it against the source.
2. The “margin is healthy, agree?” answer is sycophancy. He signalled the conclusion, so agreement is worthless as evidence. He should re-ask neutrally – “what are the strongest arguments that our margin is not healthy?” – ideally in a fresh chat.
3. The wrong sum is the arithmetic weakness. A 30-row total is exactly where next-token prediction fails; the fix is the data-analysis tool (real computation) or a spreadsheet, not trusting the confident number.
4. Consider effort and plan. For a genuine multi-step pricing judgment, a reasoning model would serve him better than the fast model he is on; but reasoning will not fix the arithmetic (still use a tool) or the sycophancy (still frame neutrally).
5. Fluency check. Every one of these answers read fine. Their being wrong is invisible in the prose, which is the whole point: he must verify, not trust the polish.

Exam-correct outcome: re-supply and verify the TAM figure against the source, re-ask the margin question neutrally in a fresh chat, recompute the table total with a tool, and switch to a reasoning model for the judgment itself – recognising that fluency told him nothing about correctness.

Assessment traps

Practice questions

Key takeaways

• The context window is a shared token budget for instructions, memory, files, chat and reply; on Business it is 54K (Instant) / 256K (Reasoning), on Enterprise 128K / 256K.
• Match reasoning effort to task difficulty; use the lowest effort that gets the result.
• Hallucination is ungrounded generation; ground and verify rather than tweaking temperature.
• Sycophancy is agreement with your stated view; counter it with neutral framing and fresh-chat critique.
• Output is non-deterministic; save outputs you value and never treat agreement as verification.
• The model is unreliable at exact arithmetic and counting; use a computation tool.
• Fluency is not accuracy – confident, polished, fast text is a generator property, not a truth signal.