Search
4 results for “context-window”
Search results
LLM Fundamentals: Tokens, Context Windows & Sampling
Why a token isn't a word, what actually happens when a request would exceed the context window, and what temperature and top_p each control during generation.
Why does a word count poorly estimate how many tokens a prompt will use?
A token is a commonly-occurring sequence of characters the model was trained on, not a word, common short words are often a single token, while a longer or less common word can split into several tokens ("tokenization" might become " token" plus "ization"). As a rough rule of thumb, one token is about 4 characters or 0.75 words in English, but that ratio shifts with vocabulary, punctuation, and especially non-English text, so a word count is only ever an approximation, not the actual unit the model's context window and pricing are measured in.
A request's input already exceeds the model's context window before generation even starts. What happens, versus a request that only exceeds the limit once output is generated?
If the input alone already exceeds the context window, the API rejects the request upfront with a 400 error, generation never starts at all. If the input fits but input tokens plus the requested max output tokens could exceed the window, current models accept the request and generate as far as they can; if generation actually reaches the window limit before finishing, it stops early with a specific stop reason indicating the context window was exhausted, rather than silently truncating or erroring out mid-response. The distinction matters operationally: the first case is a fixable request-construction bug, the second is a signal to reduce the requested output length or the accumulated conversation history.
What is the practical difference between lowering temperature and lowering top_p, and why would you use them together?
Temperature scales the randomness applied across the entire probability distribution of next-token candidates, low temperature makes the model consistently pick its highest-probability tokens, high temperature flattens the distribution so lower-probability tokens get chosen more often. top_p (nucleus sampling) instead restricts the candidate pool itself to the smallest set of tokens whose cumulative probability reaches the given threshold, then samples only from that trimmed set. They're complementary, not redundant: temperature changes how sharply the model prefers likely tokens, top_p changes which tokens are even eligible to be picked, which is why both are typically left at sensible defaults and only tuned together deliberately for advanced use cases, not adjusted independently without understanding the interaction.