AI Engineering flashcards
12 cards. Answer in your head, flip, and be honest with yourself.
0Still learning1 / 12Know0
Tokens and the context window#
A token is the unit an LLM reads and writes - a word, part of a word or a punctuation mark. In English one token is roughly 3/4 of a word; code, non-Latin scripts and rare words take more tokens per character.
The context window is the maximum number of tokens the model can handle in one request: system prompt + chat history + attached documents + tool definitions + the answer it generates.
Why a frontend engineer cares
| Concern | What happens |
|---|---|
| Cost | APIs bill per token, input and output separately (output is usually several times more expensive) |
| Latency | Time to first token grows with input size; total time grows with output size |
| Limits | A long chat eventually stops fitting - the request fails or old messages are cut |
| Quality | Even inside the limit, details buried in a huge context are followed less reliably |
What it means for the UI
ts
// Every turn re-sends the whole history - the cost of a chat grows with its length
const messages = [
{ role: 'system', content: systemPrompt },
...history, // grows on every turn
{ role: 'user', content: input },
];- Trim or summarize history - keep the last N turns and a running summary of older ones.
- Cap the output with
max_tokens, and handle the "stopped because of the limit" finish reason in the UI ("Continue" button). - Limit attachments - show a size warning before the user pastes a 300-page PDF.
- Count on the server with the provider's tokenizer or token-counting endpoint, not
text.length / 4- the estimate is fine for a progress bar, not for billing.
Tokens are the "bytes" of LLM products: they drive the bill, the latency and the limits. Designing a chat UI without thinking about them is like designing an upload form without a size limit.
space flip · ← → move · 1 still learning · 2 know it