PromptItIn logoPromptItIn
AI glossary · Technical

Streaming

Sending model output tokens to the client incrementally as they are generated rather than all at once.

PromptItIn team

A bookshelf with one card pulled out

What it means

Streaming delivers model outputs token-by-token (or in small chunks) over a persistent HTTP connection (typically using Server-Sent Events or WebSockets) rather than waiting for the entire response to be generated before sending.

This dramatically improves perceived latency: users see the first words in under a second rather than waiting 10–30 seconds for a long response to complete. Streaming is now the default mode for all major LLM APIs and consumer products.

Implementing streaming on the client side requires handling partial JSON, managing UI state for incremental text rendering, and gracefully recovering from interrupted streams.

Examples

  • ChatGPT's typing-cursor effect, where words appear word-by-word in real time, is implemented via streaming from OpenAI's API.
  • A code generation tool showing the generated function appearing line by line while the model is still generating later lines.

Inference

The process of running a trained model to generate predictions or responses from new inputs.

Latency

The time delay between sending a request to a model and receiving its first token in response.

API

An Application Programming Interface that lets developers call AI model capabilities programmatically.

Put this into practice

PromptItIn applies techniques like streaming for you: describe what you want and it builds the prompt.

Try PromptItIn freeAdd to Chrome, free

You might also like

How to Use Role in AI Prompts →Tokens and Context Windows Explained →Prompt Engineering Guide 2026: What Actually Works →Data Analysis Prompts →Stable Diffusion →Poe →
← Browse the AI prompt glossary