AI API Streaming Responses: Make Your App Feel Instant

By Darus S Alfatah · 2026-11-03 · Tutorial · 8 menit

AI API Streaming Responses: Make Your App Feel Instant

Author note: Cutting time-to-first-token from 3 seconds to 0.8 seconds was felt more by users than switching to a smarter model.

Why Streaming Matters

Large language models produce answers token by token. Without streaming, users wait until the whole answer is done — five to thirty seconds for long output. With streaming, text appears piece by piece as it is generated, making the app feel far faster even though total time is the same. It is one of the cheapest user-experience upgrades in engineering effort.

How Streaming Works in an AI API

On OpenAI-compatible APIs, you send stream: true. The server then replies with Server-Sent Events (SSE): a series of data: lines containing JSON fragments, closed by data: [DONE]. Each fragment carries a text delta, not the full answer.

data: {"choices":[{"delta":{"content":"Hello"}}]}
data: {"choices":[{"delta":{"content":", world"}}]}
data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
data: [DONE]

On the client, the official OpenAI library exposes this as an iterator; you just read the delta and append it to the UI gradually.

Node.js Implementation Example

const stream = await client.chat.completions.create({
  model: 'auto',
  messages: [{ role: 'user', content: 'Write an opening paragraph.' }],
  stream: true
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

When Streaming Is Best

  • Chat UI — users see the answer grow, feels responsive.
  • Long documents — start reading before it finishes.
  • Voice responses — the text-to-speech pipeline can start earlier.

Conversely, for backend tasks that process full JSON, streaming is optional — the parsed output is only usable once complete.

Pitfalls to Avoid

  • Truncated JSON — never parse a partial payload as complete JSON.
  • Token usage — request stream_options.include_usage so token metrics are still recorded.
  • Dropped connections — handle connections cut mid-stream and offer to continue.
  • Proxy timeouts — ensure intermediary servers do not cut slowly flowing responses.

Streaming and Reasoning Models

Reasoning models often "think" for a long time without emitting text. Some send reasoning tokens separately, some hold them back. For a good experience, show a status indicator like "analysing…" instead of leaving a blank screen. On the right gateway, reasoning models in non-stream mode are automatically converted from a stream server-side to avoid timeouts.

Production Tips

  • Provide a working "stop" button that truly closes the connection so tokens stop being spent.
  • Save the full message to the database after [DONE], not per fragment.
  • Measure time to first token, not just total time — that is what users feel.
  • Use one API gateway so the streaming format is uniform across models.

Measuring Streaming Performance

Two numbers matter: time to first token (time until the first fragment) and tokens per second. Both describe the experience far better than total duration. Record them per model; the most expensive model does not always start fastest, and for users, a slow start feels worse than a slow finish.

Streaming Across Frameworks

In React, append delta to state with batched updates to avoid excessive renders. In Node backend, forward the Readable stream from the gateway to the client with pipe(). In Python, iterate the generator and write to a websocket. The principle is the same: forward as it arrives, do not buffer everything.

Backpressure

If the client is slow, do not pile data in memory. Use the stream's built-in backpressure; cancel if the client disconnects.

Handling Errors Well

  • Detect dropped connections and show a "retry" or "continue" button.
  • If the response is empty, explain the likely cause instead of leaving a blank screen.
  • Cap retries so you do not burn quota during a prolonged outage.

Streaming vs Non-Streaming

AspectStreamingNon-stream
Perceived speedFasterSlower
Parsing easeHandle deltasDirect JSON
Best forChat UIBatch pipelines

The Trend: Streaming and Agents

Going forward, streaming matters more because AI agents produce many intermediate steps — calling tools, reading results, then writing the answer. Showing each step gradually helps users understand what is happening instead of staring at a spinner. So make sure your gateway and intermediary servers do not hold back the data flow, and remember that intermediate status is part of the user experience too.

Conclusion

Streaming is the cheapest way to make an AI app feel instant. Implementation is short, but watch out for truncated JSON, token usage, and dropped connections. Start with BankAI — an OpenAI-compatible endpoint with full streaming support for every model.

Example: a stop button that saves money: Suppose a user hits "stop" halfway through a long answer. If the client closes the SSE connection, token generation stops and you are not billed for tokens never produced. Make sure the UI actually closes the stream, not just hides the text.

Frequently Asked Questions

Can beginners follow this tutorial?

Yes. The steps are explained in order with code examples, suitable for beginners and experienced developers alike.

Is BankAI’s endpoint compatible with the library I already use?

Yes, BankAI is 100% OpenAI-compatible, so libraries such as the OpenAI SDK and LangChain work without code changes beyond the Base URL and API key.

Where do I get an API key to follow this tutorial?

You get the API key (and Secret Token) after registering and completing payment in the BankAI dashboard.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

Related articles

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing