Cancellation, retries and rate limits
LLM requests are slow, expensive and fail more often than a typical REST call. The UI has to handle all three cases explicitly.
Cancellation - the Stop button
ts
const controllerRef = useRef<AbortController | null>(null);
async function send(messages: Message[]) {
controllerRef.current?.abort(); // a new message cancels the old one
const controller = new AbortController();
controllerRef.current = controller;
try {
await streamChat(messages, append, controller.signal);
} catch (e) {
if ((e as Error).name === 'AbortError') return; // user stopped it - not an error
showError(e);
}
}
const stop = () => controllerRef.current?.abort();Aborting the fetch closes the connection; the server should forward that (req.signal) to the upstream request so you stop paying for tokens nobody reads. Keep the partial text and mark it as stopped.
Retries - only where it is safe
| Response | Action |
|---|---|
| 429 Too Many Requests | retry after Retry-After, or exponential backoff |
| 5xx / 529 overloaded | retry with backoff + jitter, a few attempts |
| 400 / 401 / 413 | do not retry - show the error |
| Failure mid-stream | do not silently retry - output would duplicate; offer "Regenerate" |
ts
const delay = (attempt: number) => Math.min(1000 * 2 ** attempt, 15_000) * (0.5 + Math.random());Rate limits on your side
- Disable send while a response is streaming, or queue the message.
- Debounce features that call the model automatically (autocomplete, suggestions).
- Per-user limits on the server - the provider's limit is shared by all your users.
- Show remaining quota when the product has one, instead of a surprise error.
The rule of thumb: retry before the first token, never after. Once the user has seen output, a failure is a UX state ("Response interrupted - Retry"), not something to hide.
updated
did you know it?