AI API Streaming Responses: Make Your App Feel Instant
By Darus S Alfatah · 2026-11-03 · Tutorial · 8 menit
Author note: Cutting time-to-first-token from 3 seconds to 0.8 seconds was felt more by users than switching to a smarter model.
Why Streaming Matters
Large language models produce answers token by token. Without streaming, users wait until the whole answer is done — five to thirty seconds for long output. With streaming, text appears piece by piece as it is generated, making the app feel far faster even though total time is the same. It is one of the cheapest user-experience upgrades in engineering effort.
How Streaming Works in an AI API
On OpenAI-compatible APIs, you send stream: true. The server then replies with Server-Sent Events (SSE): a series of data: lines containing JSON fragments, closed by data: [DONE]. Each fragment carries a text delta, not the full answer.
data: {"choices":[{"delta":{"content":"Hello"}}]}
data: {"choices":[{"delta":{"content":", world"}}]}
data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
data: [DONE]
On the client, the official OpenAI library exposes this as an iterator; you just read the delta and append it to the UI gradually.
Node.js Implementation Example
const stream = await client.chat.completions.create({
model: 'auto',
messages: [{ role: 'user', content: 'Write an opening paragraph.' }],
stream: true
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}
When Streaming Is Best
- Chat UI — users see the answer grow, feels responsive.
- Long documents — start reading before it finishes.
- Voice responses — the text-to-speech pipeline can start earlier.
Conversely, for backend tasks that process full JSON, streaming is optional — the parsed output is only usable once complete.
Pitfalls to Avoid
- Truncated JSON — never parse a partial payload as complete JSON.
- Token usage — request
stream_options.include_usageso token metrics are still recorded. - Dropped connections — handle connections cut mid-stream and offer to continue.
- Proxy timeouts — ensure intermediary servers do not cut slowly flowing responses.
Streaming and Reasoning Models
Reasoning models often "think" for a long time without emitting text. Some send reasoning tokens separately, some hold them back. For a good experience, show a status indicator like "analysing…" instead of leaving a blank screen. On the right gateway, reasoning models in non-stream mode are automatically converted from a stream server-side to avoid timeouts.
Production Tips
- Provide a working "stop" button that truly closes the connection so tokens stop being spent.
- Save the full message to the database after
[DONE], not per fragment. - Measure time to first token, not just total time — that is what users feel.
- Use one API gateway so the streaming format is uniform across models.
Measuring Streaming Performance
Two numbers matter: time to first token (time until the first fragment) and tokens per second. Both describe the experience far better than total duration. Record them per model; the most expensive model does not always start fastest, and for users, a slow start feels worse than a slow finish.
Streaming Across Frameworks
In React, append delta to state with batched updates to avoid excessive renders. In Node backend, forward the Readable stream from the gateway to the client with pipe(). In Python, iterate the generator and write to a websocket. The principle is the same: forward as it arrives, do not buffer everything.
Backpressure
If the client is slow, do not pile data in memory. Use the stream's built-in backpressure; cancel if the client disconnects.
Handling Errors Well
- Detect dropped connections and show a "retry" or "continue" button.
- If the response is empty, explain the likely cause instead of leaving a blank screen.
- Cap retries so you do not burn quota during a prolonged outage.
Streaming vs Non-Streaming
| Aspect | Streaming | Non-stream |
|---|---|---|
| Perceived speed | Faster | Slower |
| Parsing ease | Handle deltas | Direct JSON |
| Best for | Chat UI | Batch pipelines |
The Trend: Streaming and Agents
Going forward, streaming matters more because AI agents produce many intermediate steps — calling tools, reading results, then writing the answer. Showing each step gradually helps users understand what is happening instead of staring at a spinner. So make sure your gateway and intermediary servers do not hold back the data flow, and remember that intermediate status is part of the user experience too.
Conclusion
Streaming is the cheapest way to make an AI app feel instant. Implementation is short, but watch out for truncated JSON, token usage, and dropped connections. Start with BankAI — an OpenAI-compatible endpoint with full streaming support for every model.
Example: a stop button that saves money: Suppose a user hits "stop" halfway through a long answer. If the client closes the SSE connection, token generation stops and you are not billed for tokens never produced. Make sure the UI actually closes the stream, not just hides the text.
Frequently Asked Questions
Can beginners follow this tutorial?
Yes. The steps are explained in order with code examples, suitable for beginners and experienced developers alike.
Is BankAI’s endpoint compatible with the library I already use?
Yes, BankAI is 100% OpenAI-compatible, so libraries such as the OpenAI SDK and LangChain work without code changes beyond the Base URL and API key.
Where do I get an API key to follow this tutorial?
You get the API key (and Secret Token) after registering and completing payment in the BankAI dashboard.