BackendBeginner

LLM Fundamentals: Tokens, Context Windows & Sampling

Why a token isn't a word, what actually happens when a request would exceed the context window, and what temperature and top_p each control during generation.

Updated 2026-07-243 min read

On this page

Overview

Every LLM request operates on tokens, not words or characters directly, roughly 4 characters or 0.75 words per token in English, and every token in the system prompt, conversation history, and generated output counts against a single shared budget: the context window. A larger context window doesn't uniformly help, providers explicitly document "context rot," where accuracy and recall degrade as token count grows, making what's kept in context as important as how much room is available. Once a response is actually being generated, temperature and top_p are the two levers that shape which token gets picked next, temperature scales how strongly the model favors its most likely tokens, top_p restricts the candidate pool itself to a cumulative-probability threshold before any sampling happens.

Quick Reference

Concept	What it is	Practical takeaway
Token	~4 characters / 0.75 words	Word count is only an approximation of cost/limits
Context window	Total tokens: input + output combined	Everything (system prompt, history, tool defs) counts
Temperature	Randomness across the whole distribution	Low = consistent, high = creative/varied
top_p	Restricts candidates to a probability threshold	Changes which tokens are eligible, not how sharply they're preferred

Syntax

json

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "messages": [{ "role": "user", "content": "Summarize this incident report." }]
}

Examples

json

// Low temperature for a deterministic, analytical task - the same
// input should reliably produce the same classification.
{
  "temperature": 0.0,
  "messages": [{ "role": "user", "content": "Classify this ticket as bug, feature, or question." }]
}

Temperature 0 is not fully deterministic

Even at the lowest temperature setting, provider documentation is explicit that output is not guaranteed to be identical across repeated identical requests. Treat "temperature 0" as "much more consistent," not as a hard determinism guarantee to build a test suite's exact-match assertions on.

Visual Diagram

What counts toward the context window

Common Mistakes

Estimating prompt cost or limits by word count instead of actual token count, which drifts significantly for non-English text or unusual vocabulary.
Assuming a bigger context window is free capacity to dump in irrelevant history, when accuracy measurably degrades as irrelevant tokens accumulate.
Tuning temperature and top_p independently without understanding they act on different parts of the sampling process, then being confused why changing one didn't produce the expected effect on its own.
Building tests that assert byte-for-byte identical output at temperature 0, when providers explicitly don't guarantee full determinism even at that setting.

Performance

Token count directly drives both latency and cost; trimming irrelevant conversation history or tool output is a direct lever on both, not just a cost optimization.
Context window overflow is a request-time failure mode, not a runtime performance issue, catching it before sending (via a token-counting call) avoids a wasted round trip on an oversized request.

Best Practices

Measure prompts in actual tokens (via a token-counting endpoint), not an estimated word count, before relying on limits or cost projections.
Curate what stays in context deliberately as conversations grow, rather than assuming more history is always better.
Default to low temperature for tasks needing consistency (classification, extraction, code) and reserve higher temperature for genuinely creative/generative tasks.
Leave top_p at its default unless a specific, understood need justifies tuning it alongside temperature.

Interview questions

Why does a word count poorly estimate how many tokens a prompt will use?

A token is a commonly-occurring sequence of characters the model was trained on, not a word, common short words are often a single token, while a longer or less common word can split into several tokens ("tokenization" might become " token" plus "ization"). As a rough rule of thumb, one token is about 4 characters or 0.75 words in English, but that ratio shifts with vocabulary, punctuation, and especially non-English text, so a word count is only ever an approximation, not the actual unit the model's context window and pricing are measured in.

A request's input already exceeds the model's context window before generation even starts. What happens, versus a request that only exceeds the limit once output is generated?

If the input alone already exceeds the context window, the API rejects the request upfront with a 400 error, generation never starts at all. If the input fits but input tokens plus the requested max output tokens could exceed the window, current models accept the request and generate as far as they can; if generation actually reaches the window limit before finishing, it stops early with a specific stop reason indicating the context window was exhausted, rather than silently truncating or erroring out mid-response. The distinction matters operationally: the first case is a fixable request-construction bug, the second is a signal to reduce the requested output length or the accumulated conversation history.