Streaming LLM responses
Generating a long answer takes seconds. Streaming shows tokens as the model produces them, so the user sees the first words in a few hundred milliseconds instead of staring at a spinner.
Most LLM APIs stream over Server-Sent Events (SSE): a long-lived HTTP response with Content-Type: text/event-stream where each event is a data: ... line followed by a blank line.
data: {"type":"delta","text":"Hel"}
data: {"type":"delta","text":"lo!"}
data: {"type":"done"}Reading the stream with fetch
EventSource only supports GET, so for a POST with a body use fetch and read response.body:
async function streamChat(messages: Message[], onText: (t: string) => void, signal: AbortSignal) {
const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ messages }), signal });
if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = '';
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
const events = buffer.split('\n\n');
buffer = events.pop()!; // keep the incomplete tail
for (const e of events) {
const data = e.replace(/^data: /, '');
const msg = JSON.parse(data);
if (msg.type === 'delta') onText(msg.text);
}
}
}A chunk can end in the middle of an event (or of a multi-byte character), so buffer until the separator - never JSON.parse a raw chunk.
Rendering
const [text, setText] = useState('');
// append, not replace
streamChat(messages, (t) => setText((prev) => prev + t), controller.signal);- Tokens can arrive faster than you want to re-render; batch updates with
requestAnimationFrameif parsing markdown on every token gets expensive. - Markdown is incomplete mid-stream (an unclosed code fence) - the renderer must tolerate it.
- Keep the scroll pinned to the bottom only while the user has not scrolled up.
Libraries like the Vercel AI SDK wrap all of this, but in an interview you should be able to explain the moving parts: SSE framing,
ReadableStream, buffering and incremental rendering.
updated