PromptItIn logoPromptItIn
AI glossary · Technical

Inference

The process of running a trained model to generate predictions or responses from new inputs.

PromptItIn team

A bookshelf with one card pulled out

What it means

Inference is the forward-pass computation that produces a model's output given an input, using fixed (not updated) weights. It is distinct from training, where weights are actively updated via backpropagation.

Inference cost is measured in compute (FLOPs per token), latency (time to first token and time per token), and memory (VRAM to hold model weights plus KV cache).

Inference optimisation is a large engineering discipline: quantisation, speculative decoding, batching, caching, and hardware-specific kernels (flash attention) all reduce cost and latency.

As models are used by millions of users simultaneously, inference efficiency is as important as model quality for production deployment.

Examples

  • An API endpoint receiving a user's chat message, running a forward pass through a 70B parameter model, and returning a streamed response in under 2 seconds.
  • Using speculative decoding to reduce Llama 3 70B inference latency by having a smaller draft model propose token sequences for the large model to verify.

Latency

The time delay between sending a request to a model and receiving its first token in response.

Throughput

The number of tokens or requests an inference system can process per unit of time.

Streaming

Sending model output tokens to the client incrementally as they are generated rather than all at once.

Put this into practice

PromptItIn applies techniques like inference for you: describe what you want and it builds the prompt.

Try PromptItIn freeAdd to Chrome, free

You might also like

How to Write Better AI Prompts (Step-by-Step) →Fine-Tuning AI Models Explained →Prompt Engineering Guide 2026: What Actually Works →Prompt Chaining: How to Get Better Results from AI with Multi-Step Prompts →360 Feedback Request →Stable Diffusion →
← Browse the AI prompt glossary