By Zahi Raifer, founder of Orivox
An AI editing service can spend several seconds reading a document, extracting its structure, and asking a model to propose a change. Reusing a previous result looks like an obvious optimization. If two requests contain the same instruction, why pay for the same work twice?
Because an instruction is not the whole input. “Shorten the introduction” means different things on different pages, under different permissions, and after different revisions. A cache can return a technically valid answer that belongs to another customer or an earlier authorization context. The dangerous failure is not a miss. It is an unjustified hit.
Decide what is reusable
Start by naming the object you intend to cache. A parsed public document, a private page summary, a model proposal, and a permission decision have different lifetimes and different security requirements. Putting all four behind a generic “result cache” hides those differences precisely where reviewers need to see them.
A useful cache record describes its artifact type, originating revision, owner scope, creation time, and dependencies. Its value should be interpretable without guessing which request produced it. If an artifact cannot be classified as public, tenant-private, or user-private, do not cache it until that classification exists.
Scope before similarity
In a website-editing workflow driven by chat, a proposed edit depends on more than the conversation text. It may depend on the selected site, the current document, the requested operation, and which resources the current user may inspect. That makes result reuse an authorization design problem before it becomes a performance problem.
Use an explicit namespace that includes the tenant and resource identity. Add the operation and relevant input revision. Include a principal or permission-class identifier when two users within the same tenant can see different material. Never assume that sharing an organization means sharing every page, draft, or attachment.
Authenticate before looking up
A cache lookup should not become a shortcut around access control. Authenticate the caller and check current resource access before retrieving private artifacts. A record that was readable yesterday is not necessarily readable after a role change, a project transfer, or a revoked invitation.
This also applies when the artifact itself contains a previous permission result. Caching authorization requires its own carefully bounded strategy; it should not happen accidentally because a content cache stores an entire handler response. A short time-to-live reduces some exposure, but it does not turn stale permission into valid permission.
Encode dependencies, not vibes
A robust key is a structured description of the conditions under which reuse is valid. Imagine fields for tenant, resource, content revision, operation version, normalized options, model configuration, and authorization epoch. An epoch is simply a version that changes when relevant permissions change.
Serialize these fields unambiguously before hashing them. Plain string concatenation can make different tuples indistinguishable. Preserve meaningful distinctions between absent values, empty strings, and defaults. Normalize only where equivalence is actually guaranteed; lowercasing an identifier or rearranging an ordered list may change its meaning rather than merely tidy its representation.
Reuse computation, not authority
Suppose a cached artifact contains a suggested replacement paragraph and a description of the section it targets. That can save computation. It must not automatically authorize writing the paragraph into the live document. The write path still needs current authorization, current revision checks, and validation of the proposed transformation.
An effective boundary is to cache proposals as inert data. On reuse, run the same deterministic checks that a fresh proposal would face. A cache hit may skip expensive generation, but it should not skip the rules that decide whether the result is safe to apply now.
Separate public and private pools
Shared caches are attractive for genuinely public material: a published documentation page or a stable language reference can serve many callers. But “public” needs evidence. A URL can return different content depending on cookies, authentication headers, region, or account state. The address alone does not prove that responses are interchangeable.
Use separate storage namespaces and explicit admission rules for public artifacts. Do not promote an entry from private to shared merely because several customers requested the same URL. Exclude authenticated fetches from public reuse unless there is a deliberate, reviewed mechanism that establishes content equivalence and public availability.
Invalidation needs an owner
Content-addressed revisions simplify freshness: changed content produces a different key. They do not solve deletion. An old artifact can remain stored after nobody can reach it through the current revision. Account deletion, retention limits, and resource transfers require a way to locate and remove historical entries too.
Maintain a resource-to-artifact index or another reliable ownership map. Give one component responsibility for invalidation events and retries. Record whether deletion completed instead of assuming that publishing an event means every cache consumed it. Content freshness and data retention are separate obligations, even when they touch the same storage.
Protect against late writers as well. A generation started before a permission change can finish after the invalidation message. Compare the captured authorization epoch with the current epoch before admitting its result. Versioned namespaces prevent that old result from becoming a current hit, while retention cleanup still removes the unreachable artifact. For sensitive material, a deletion tombstone can also stop an in-flight task from storing content that has already been scheduled for erasure.
Test deliberate near-misses
Happy-path cache tests establish that repeated requests produce hits. Security tests should establish that almost-identical requests do not. Create two tenants with identical document text and identical instructions, then verify that private artifacts remain isolated. Repeat with two users whose page permissions differ inside one tenant.
Change a document revision, revoke a role, update the operation schema, and delete the source resource. Exercise each change against warm caches, not just empty ones. Include concurrent requests so that a late generation cannot accidentally repopulate an entry that an invalidation event already removed.
Measure the cost of correctness
A global hit rate can reward unsafe sharing. Break metrics down by artifact type and intended scope. Count rejected reuse attempts caused by revision mismatches or authorization changes. Those rejections are not necessarily performance defects; they may show that the boundary is doing its job.
Keep telemetry free of cached document text and credentials. Opaque identifiers, cache-schema versions, and coarse miss reasons usually provide enough information to investigate behavior. Before widening reuse to improve a dashboard, ask which equivalence assumption is changing and whether a test proves that the new assumption is safe.
A cache is a claim that two situations are interchangeable. In an AI workflow, that claim deserves the same scrutiny as any other production contract. Make the dependencies explicit, keep authorization current, and let correctness veto reuse. Saving a model call is useful. Quietly reusing someone else’s context is not an optimization.
Author bio: Zahi Raifer is the founder of Orivox, an AI website editor. He writes about practical engineering boundaries in LLM-powered software.
Author: Zahi Raifer
Founder of Orivox, an AI website editor. Writes about practical engineering boundaries in LLM-powered software.
