Semantic caching when permissions and documents keep changing

The fastest answer can be the wrong artifact

A semantic cache finds a previous answer to a similar question and avoids some retrieval or generation work. That description leaves out the difficult part: whether the previous answer is still valid for this requester, purpose, source state, and policy. Similar language does not establish any of those conditions.

Consider two analysts asking the same question about an incident. One can access the investigation notes; the other can see only a public summary. A shared answer embedding may place their questions at precisely the same location. If the cache treats that match as permission to reuse the first answer, it bypasses the boundary that the retriever was supposed to enforce. Correct retrieval cannot rescue a request that never reaches it.

This is a proposed cache design informed by Structura's explicit Input, Evidence, and Decision records and its preservation of model and rule versions. It is not a claim that every guard described here is implemented in Structura. The design starts with a simple requirement: reuse must preserve the conditions under which the artifact was created, or re-establish those conditions before release.

Decide which computation is being cached

There are several caches inside an AI application. Parsed source text, document embeddings, retrieval results, answer drafts, and final approved answers have different dependencies. An embedding cache may depend on source bytes, preprocessing, and the embedding model. An answer cache additionally depends on the question, selected evidence, generation contract, and release policy.

Separating these layers matters because an authorization change need not force every source to be embedded again, while it must prevent a now-forbidden answer from being returned. Conversely, a parser correction can invalidate an embedding even if user permissions remain unchanged. Treating the whole pipeline as a single cache creates a choice between expensive full invalidation and unsafe reuse.

Traditional HTTP caching provides useful distinctions around freshness, validation, and reuse constraints. A semantic answer cache needs its own application contract beyond those transport rules. The relevant question is not merely whether an entry has expired. It is which facts would need to remain true for this particular artifact to be reusable.

Filter by scope before searching by similarity

The cache lookup should begin with the current request's authorized scope. A tenant identifier alone is often too coarse: a tenant can contain separate teams, purposes, projects, and document permissions. Define a scope using the actual policy semantics, and avoid assuming that equal role names imply equal access. Where safe equivalence classes cannot be established, evaluate source authorization for each candidate at release time.

Within an eligible scope, semantic similarity can nominate candidates. It cannot approve them. Verify the candidate's source manifest, relevant source revisions, generation contract, and freshness conditions. A question that differs only by “before” and “after” can have similar vectors and opposite meanings. Exact constraints such as dates, units, entity identifiers, and requested output type need explicit comparison.

The illustrative contract below keeps semantic matching separate from release. The policy epoch is a coarse invalidation aid, not a replacement for live authorization. A real implementation also has to close the race between validation and release, for example by binding the operation to a policy snapshot whose validity is checked at the boundary.

1type CachedAnswer = {
2  requestConstraints: Record<string, string>;
3  sourceRevisions: Record<string, string>;
4  policyEpoch: string;
5  generationContract: string;
6  createdAt: string;
7  expiresAt: string;
8  answer: string;
9};
10 
11// Similarity nominates a candidate; it does not authorize release.
12// Validate current access and every relevant dependency before returning it.

Use evidence lineage for targeted invalidation

Suppose a cached answer relies on documents A and B. Document C changes. A blanket corpus-version key invalidates the answer, which is conservative but potentially wasteful. A dependency manifest can preserve it if C has no relevance to the answer's validity. That last condition is stronger than saying C was not cited: an authoritative new document might supersede A without having existed when the answer was generated.

A useful manifest therefore needs both positive dependencies and validity assumptions. Positive dependencies identify the evidence used. Validity assumptions can include “these are the current approved procedure revisions” or “the search covered this complete source collection as of this snapshot.” When that collection changes, an absence-based conclusion may need re-evaluation even if its cited sources are untouched.

This is especially important for negative answers. “No matching restriction was found” depends on the search boundary and its completeness. Caching it as a timeless fact turns a temporary search result into an authority claim. Preserve the searched scope, query plan, and observation time, and use shorter or event-driven validity where the cost of missing new information is material.

Revocation must reach derived outputs

A source can disappear for many reasons: correction, retention, a permission change, or removal from the approved corpus. Derived answers should not survive simply because they no longer resemble the original text. Summaries, entity graphs, and cached explanations may disclose the same sensitive information through a different representation.

For an illustrative revocation sequence, first mark the source unavailable to new requests, then invalidate affected answer entries, then remove or restrict derived indexes as required by the retention and access policy. Requests already in flight need an explicit release rule. A worker that validated access before revocation should not automatically be allowed to return the result afterward. The implementation has to choose and document its consistency boundary.

Avoid promising instantaneous global revocation if the system contains disconnected replicas or long-lived local caches. State the propagation model, permitted offline behavior, and maximum accepted staleness. The correct architecture depends on the information involved. Some applications can tolerate delayed freshness checks; others must fail closed when current authorization cannot be established. A cache should express that choice instead of silently making it.

Measure rejected hits as carefully as accepted ones

A high hit rate is not a sufficient success metric. Report candidate matches, eligible matches, released answers, stale rejections, authorization rejections, and constraint mismatches separately. A lower hit rate can be the expected consequence of correctly separating access scopes. Savings should be calculated from the work actually avoided after validation, including the cost of that validation.

A focused test set should pair nearly identical requests with different principals, dates, entities, and source revisions. Include a permission change during an in-flight request, a source that gains a superseding document, and a negative answer whose search corpus expands. Record whether each case returns, recomputes, or refuses, and whether the explanation identifies the correct reason.

Start with exact caching over a narrow, well-defined request contract. Add semantic nomination only after the release checks are reliable, then compare the incremental useful reuse against added mistakes and operational complexity. The strongest outcome may be a hybrid: broad reuse of parsing and embedding work, conservative reuse of retrieval candidates, and very limited reuse of final answers. That is still a valuable cache. It accelerates the system while preserving the meaning of the response.

Keep a sampled audit of accepted hits as well as rejected candidates. Review whether the reused answer still addresses the exact request, whether each citation remains accessible, and whether recomputation would materially change the conclusion. A cache bug can otherwise remain invisible because the avoided computation is also the computation that would have exposed it.