Back to blog

Tutorial

What Are Cached Tokens? A Complete Guide to Prompt Caching for LLM APIs

OleToken Blog · 7 min read · November 20, 2026

Cached tokens are input tokens whose previously computed prompt state can be reused for an eligible request. Instead of processing the same beginning of a prompt from scratch, a supported LLM API can reuse stored intermediate computation and charge a different input rate.

Prompt caching can reduce processing time and repeated-input cost, but it does not make the entire request free. Providers differ in eligibility, retention, writes, and billing. This guide explains the concepts and uses official OpenAI, Google, and Meta documentation checked on November 20, 2026 for concrete examples.

What prompt caching actually reuses

During inference, a model builds intermediate attention states, often called key-value states. A prompt cache preserves reusable computation for a matching prefix: the unchanged beginning of the rendered input. The API still processes new input and generates a new response.

OpenAI explicitly describes its prompt cache as storing key-value tensors rather than the tokens themselves. That distinction explains the mechanism, but it does not remove the need to review a provider's retention and data-handling terms.

Prompt caching versus response caching

Response caching stores a completed answer in your application and returns it again when appropriate. Prompt caching reuses input computation while producing a new answer. A customer-specific question can therefore share cached instructions without sharing the customer's final response.

Application response caches require their own freshness, authorization, and personalization checks. Do not return one user's answer to another merely because their prompts look similar. Neither cache should become a shortcut around access control.

What are cached input tokens in usage reports?

Cached input tokens are the portion of input recognized as served from an eligible cache. They are typically included within total input, not an additional block of tokens to add to that total. Verify the provider's usage schema before calculating charges.

For example, if total input is 12,000 tokens and 10,000 are cached reads, the remaining 2,000 are not cached reads. Depending on provider behavior, some of those remaining tokens may use an ordinary input rate and others a write rate. Output is accounted for separately.

A cache hit is not guaranteed

Repeated text is only part of eligibility. The selected model, prefix structure, relevant settings, cache lifetime, and routing can affect reuse. Inspect reported usage rather than assuming every second request is fully cached.

A conversation ID is not a guarantee either. Conversation history can preserve a shared prefix, but changing or compacting earlier content can alter the point from which reuse is possible.

How prefixes and breakpoints work

Stable instructions, tool definitions, reference documents, and earlier conversation turns can form a reusable prefix. Place changing user details after stable material where the API's request structure allows it. Avoid adding random values or timestamps at the beginning unnecessarily.

Some providers choose cache boundaries implicitly. Others support explicit breakpoints or cache objects. These mechanisms are not interchangeable: a field valid for one model can be unsupported by another.

OpenAI eligibility example

OpenAI documents a minimum cacheable prefix of 1,024 tokens for GPT-5.6 and later, excluding its hidden system content from that minimum. Supported models can use implicit caching, and documented explicit controls allow choosing breakpoints for eligible content.

Do not pad a short prompt merely to reach a threshold without calculating the extra input cost. A useful cache needs repeated eligible material and enough reuse to justify any write premium.

Calculate cache costs, including writes

GPT-6.2 Sol's short-context Standard rates are $2 per million ordinary input tokens, $0.10 per million cached reads, and $2.50 per million 30-minute cache writes. A twelve-hour write costs $5 per million tokens.

OpenAI states that the write rate is not an additive fee: a token uses the applicable ordinary, read, or write rate. Charging both a full ordinary rate and a full write rate for the same token would overestimate the documented cost.

Worked example: a reusable document prefix

Assume a 10,000-token prefix is written once with 30-minute retention and then fully reused nine times. The write costs $0.025, and the nine reads together cost $0.009. Total prefix cost is $0.034, compared with $0.20 for ten uncached passes at the ordinary rate.

That is an 83% reduction for this prefix under these assumptions. It is not an 83% reduction for the complete application bill. New user input, output, tools, failed requests, and other services are excluded.

Break-even depends on the retention choice

At the same rates, one 30-minute write plus one read costs $0.026 for that prefix, versus $0.04 for two ordinary passes. A twelve-hour write plus one read costs $0.051, so two uses alone would not recover its higher write cost.

Choose retention based on observed reuse timing. Longer retention can be valuable for a frequently reused prefix but wasteful for material seen only once. Model and processing-tier changes may alter the arithmetic.

Provider differences that matter

Gemini 4 Neon: implicit caching only

Google documents implicit caching for Gemini 4 Neon and explicitly says explicit caching is unsupported. Its update guide lists a 4,096-token minimum and Standard cached-input pricing of $0.15 per million tokens versus $3 ordinary input.

That 95% input discount applies to actual cached tokens, not all input by default. Google's pricing also bills output thinking and relevant internal grounding content as documented. Do not transfer an OpenAI cache-write formula to Neon without provider-specific evidence.

Muse Ember: context length sets the tier

Meta lists Muse Ember cached input at $0.004 per million tokens for requests up to 200,000 input tokens and $0.012 above that threshold. Cached tokens still count toward the input length determining the tier.

The higher tier applies to the full request when the threshold is exceeded. A large cached history can therefore affect rates even though its read price is low. Use total request context when selecting the pricing tier.

Common reasons for low cache reuse

Changing the model or tool schema can alter the rendered prefix. Reordering documents, inserting variable content early, or rewriting previous messages can also reduce reuse. Cache expiry and routing behavior create additional limits even when application text appears unchanged.

For GPT-6.2, OpenAI documents that Standard requests outside discounted background windows cannot reuse cache entries created by two-hour or eight-hour background requests. Track these processing routes separately instead of expecting a scheduled job to warm every interactive request.

Debug one variable at a time

Compare otherwise identical requests and record cached usage, writes, model ID, route, and latency. Then vary document order or a setting deliberately. A controlled experiment is more informative than changing several prompt components simultaneously.

Keep an uncached baseline for quality tests. Reusing prompt computation should not become an excuse to leave outdated instructions or obsolete source material in a production workflow.

Monitor token usage and API cost reduction

Track cached reads as a share of eligible input, write volume, ordinary input, output, and total spend per accepted task. A high read ratio can coexist with an expensive bill if output or agent tool loops dominate.

Reconcile estimates with invoices and document the provider-specific fields used. If a unified AI API platform such as OLEToken is your integration layer, verify whether it preserves cache controls, reports usage accurately, and applies the same pricing rules as direct access.

FAQ

Are cached tokens free?

Usually they have a reduced rate rather than zero cost. Rates vary by model, processing tier, and provider. Writes or storage may have separate rules.

Does caching reuse the previous answer?

Prompt caching reuses input computation, not the final answer. Application response caching is a different mechanism with separate correctness and security requirements.

Can I guarantee a cache hit by repeating a prompt?

No. Matching text must also meet the provider's eligibility, lifetime, settings, and routing conditions. Inspect actual cached usage.

Should every application use explicit caching?

Only where it is supported and useful. Some models, including Gemini 4 Neon, support implicit caching only. Choose the documented mechanism and measure its economics.

Next step

Identify one frequently repeated prefix and compare measured costs before and after restructuring it. Review OLEToken if you need a unified API, and confirm cache support for each selected model and route.

Sources