OpenAI API cost: estimate it, then cut it

How LLM API pricing works

• Input tokens. Everything you send: system prompt, tool definitions, retrieved documents, conversation history and the user’s message. A token is roughly three-quarters of an English word; code, non-English text and unusual formatting use more tokens per character.
• Output tokens. Everything the model generates, priced higher than input. For reasoning models, the hidden reasoning tokens are billed as output too, even though you don’t see them – check the usage field in the response.
• Cached input. Providers discount input tokens that repeat an identical prefix from a recent request. OpenAI applies this automatically above a minimum prompt length; Anthropic’s is opt-in – you add a cache_control marker, either once for automatic placement or at explicit breakpoints – with cache writes priced above normal input and cache reads well below it. Cache lifetimes run from a few minutes to hours depending on provider, model and settings, so the saving depends on traffic patterns.
• Batch processing. Submit requests as a batch for asynchronous processing (completion within a window measured in hours) at a substantial discount. Ideal for anything a user isn’t waiting for.
• Other meters. Embeddings, image and audio inputs, web search and other built-in tools, and fine-tuned model usage each have their own rates.

Two consequences catch teams out. First, everything is re-sent on every call – a chat with 20 turns pays for turn one 20 times. Second, tool definitions count as input: an agent with thirty verbose tools carries thousands of tokens before the user says anything.

How to estimate LLM cost per feature

Estimate per task, not per request, because one user action often triggers several calls. For each call in the task:

1. Count input tokens by part: system prompt, tool schemas, retrieved context, history, user input. Use the provider’s tokenizer or token-counting endpoint on real examples, not guesses.
2. Split input into cacheable and uncached. The stable prefix (system prompt, tools, static instructions) can usually be cached; the per-request tail can’t.
3. Measure output tokens from a prototype on 50+ real inputs, including reasoning tokens. Use the 90th percentile as well as the mean – long tails dominate bills.
4. Multiply by calls per task. For agents, record the distribution of loop iterations; a task averaging four steps with a tail of twenty is priced by the tail.
5. Apply current prices from the pricing page: uncached input × input rate + cached input × cached rate + output × output rate, summed across calls.
6. Multiply by volume, with a margin for retries, guardrail checks and evaluation runs.

Then compare the cost per task with the value per task. If a feature costs more per use than the user pays per month, you’ll want to know before launch, not after. I go through the value side in AI ROI.

Measure before you optimize

Every response includes a usage object with input, cached and output token counts. Log it with the feature name, model, tenant and user on every call, and build a simple dashboard of cost per feature and cost per active customer. Most teams that think they have an “OpenAI bill problem” actually have one feature, one tenant or one runaway loop causing most of it. Find it before touching anything else.

The levers that reduce OpenAI costs

Roughly in the order I try them:

1. Route to the smallest model that passes your evals. Classification, extraction, routing and short rewrites rarely need a frontier model. A cascade – try the small model, escalate to the large one when confidence is low or validation fails – often cuts cost dramatically. You need an evaluation suite to do this safely; without one you’re guessing at the quality cost.
2. Structure prompts for caching. Put stable content first – system prompt, tool definitions, examples, static documents – and variable content last. Don’t inject timestamps or request IDs at the top of the prompt; one changing token near the start defeats the cache for everything after it.
3. Send less context. Retrieve fewer, better chunks with a reranker instead of stuffing twenty passages. Summarize or truncate long conversation history. Trim tool lists per request to the tools relevant to that step.
4. Cap and shape output. Set max output tokens, ask for concise formats, and use structured output so the model returns fields instead of prose. For reasoning models, use the lowest reasoning effort that passes your evals.
5. Batch everything non-interactive. Nightly enrichment, document processing, evaluation runs and backfills belong in the batch API.
6. Cache responses. Exact-match caching for repeated identical requests is safe and cheap. Semantic caching (reusing answers for similar questions) saves more but can return a wrong answer to a subtly different question – use it only where that’s acceptable.
7. Bound agents. Cap iterations and total tokens per run, stop on repeated tool errors, and alert on outliers.
8. Avoid unnecessary calls. Don’t call a model for what a regex, a lookup table or a SQL query can do.

Bigger structural options

• Fine-tune a smaller model for a narrow, high-volume task where a large model works but costs too much. It shifts cost into training and maintenance; worth it only at volume. Check availability first: OpenAI closed self-serve fine-tuning to new customers in 2026 and stops new jobs for existing ones in January 2027, so this increasingly means another provider or an open-weight model. See RAG vs fine-tuning.
• Self-host open-weight models when volume is high and steady, data residency demands it, or a small specialized model is enough. Remember to count GPU idle time, ops effort and on-call – it’s cheaper per token only at sustained utilization.
• Use multiple providers for the same task where models are interchangeable on your evals, and route on price and availability. Keep the abstraction thin; prompt behavior differs between models.

Controls so the bill can’t surprise you

• Project-level budgets and usage alerts in the provider console.
• Per-tenant and per-user quotas in your own code, especially on free plans – an abusive user can generate a large bill overnight.
• Separate API keys or projects per environment and per feature, so spend is attributable.
• Alerts on cost per task drifting up after a deploy – prompt changes often add tokens silently.
• A pricing model for your own product that reflects the usage cost, whether that’s plan limits, credits or usage-based billing.

LLM spend sits next to the rest of your infrastructure bill, and the same habits apply: attribute it, budget it, review it monthly. If cloud costs are also creeping, that’s cloud cost optimization work. If you want help building cost-aware AI features from the start, it’s part of my AI integration work.

Related

• AI integration services for products already in production
• AI ROI: how to measure the return on AI features
• LLM evaluation for product features
• AWS cost optimization without breaking production
• RAG vs fine-tuning: how to choose
• ChatGPT integration services for your product