A vector index is a derived view: deletions, model changes, and retrieval integrity

The index is not the source of truth

A vector index can return a convincing neighbor for a document that should no longer participate in an answer. The document may have been deleted, superseded, or moved outside the caller's permissions. Its embedding can remain numerically valid while its use becomes invalid. That is why I think about an index as a derived view with a lifecycle, rather than as the place where a knowledge base finally becomes complete.

This design study follows a hypothetical internal-document collection through updates and an embedding-model change. The objective is a retrieval result that can be connected to an eligible source revision. It does not claim a new retrieval algorithm or measured relevance gain. The difficult work is maintaining meaning across ingestion, indexing, authorization, and answer generation when those steps run at different times.

Give each representation an addressable parent

I would store a stable document identifier, immutable source revision, segment identifier, extraction version, and embedding configuration alongside each vector. The embedding configuration includes the model identity, dimensionality, input formatting, and normalization assumptions. A segment number alone is not a durable citation: re-extraction can change the segmentation while leaving the document title untouched. The parent record needs enough information to resolve the representation back to what was actually read.

A source digest is useful for detecting byte changes, but it is not the entire identity model. Two documents can contain identical text and have different owners or access rules. Conversely, a formatting-only change can alter bytes without changing the passage's meaning. I would keep source identity, content identity, and authorization scope separate so that deduplication cannot accidentally merge records that should be governed independently.

The vector itself also has a type beyond its length. Two models can produce arrays with the same number of elements while assigning incompatible geometric meanings to their coordinates. A database accepting both arrays does not make a mixed similarity search valid. I would enforce an embedding-space identifier in the ingestion contract and query path, and treat a new space as a separately evaluable representation.

Make a deletion visible before cleaning up its derivatives

Deleting the source row does not necessarily remove every derived object at the same moment. A queue may contain a delayed indexing task, an index may be rebuilt from an older export, and a cache may contain an answer generated before the deletion. My first concern is an authoritative eligibility state that prevents those artifacts from being used while asynchronous cleanup catches up. Physical deletion remains a separate operation with its own completion evidence.

For this example, a document revision can become ineligible in the catalog before its vector is removed. The retrieval path checks that state when assembling evidence, and the indexing worker checks it before publishing a delayed result. A stale task must not resurrect a deleted revision merely because its computation completed successfully. The cleanup process records which stores were addressed and where retryable failures remain.

Authorization is another eligibility condition, but a simple filter in an application query should not be advertised as a complete security boundary. Postgres row-security documentation describes policy behavior and important exceptions, including privileged roles and table ownership. I would test the actual serving identity and every path that reads source text. The example predicate below illustrates the intended contract; it does not establish that a deployment enforces it.

1-- Illustrative retrieval contract; authorization must be enforced
2-- by the actual serving identity and policy implementation.
3SELECT e.segment_id, e.source_revision, e.embedding <=> $1 AS distance
4FROM document_embeddings AS e
5JOIN document_revisions AS d
6  ON d.document_id = e.document_id
7 AND d.revision = e.source_revision
8WHERE e.embedding_space = $2
9  AND d.eligible = TRUE
10  AND d.tenant_id = $3
11ORDER BY e.embedding <=> $1
12LIMIT 20;

Migration means two representations and one evaluation question

When changing the embedding model, I would build a separate index from a pinned source snapshot and leave the existing serving path available. The new configuration needs its own manifest: model version, preprocessing, chunking, distance metric, and source coverage. A list of successfully written vectors is insufficient if failures disproportionately exclude large documents, unusual encodings, or a particular category of evidence.

The old and new representations should receive the same labeled queries and the same eligibility scope. That makes it possible to inspect whether the change improves retrieval for the task rather than benefiting from a different corpus. I would report missing-source coverage separately from relevance. An index can appear faster because it quietly contains fewer difficult records. A cutover should not erase the evidence needed to notice that difference.

Approximate search adds another variable. The pgvector documentation distinguishes exact and approximate search and describes how filtering interacts with approximate indexes. I would compare candidate results against an exact baseline over a manageable eligible subset. The benchmark should record extension version and index parameters. This is a proposed evaluation method, not a claim that one index family always wins or that a current feature existed in the original archive period.

Recheck the boundary when an answer is released

A permission or source-state change can occur after retrieval but before an answer is shown. The system needs an explicit policy for that interval. One approach is to pin an eligibility snapshot and define what it permits; another is to reauthorize cited sources immediately before release. Which approach is appropriate depends on revocation requirements and the storage system's consistency model. I would not hide the choice inside an unspecified cache timeout.

The same boundary applies to citations. A citation should resolve to the source revision used to support the claim, while respecting the reader's current access. If the revision is no longer eligible, the interface needs an honest unavailable-source state. Silently redirecting the citation to the newest document can make an earlier answer appear supported by text that did not exist when the answer was created.

Caching can preserve computational work without preserving permission to disclose it. I would bind cached evidence to source revisions and embedding configuration, then validate the request's allowed scope before reuse. This does not require pretending every answer must be regenerated on every request. It requires distinguishing a reusable intermediate artifact from an authorized response, and recording the conditions under which that distinction is enforced.

Test lifecycle failures alongside relevance

A useful test collection would include a source deleted during an indexing job, a tenant moved after retrieval, a delayed event for an obsolete revision, and an embedding created under the wrong configuration. The expected result is not always an empty retrieval set. Sometimes it is an explicit migration failure, a bounded partial result, or an answer that abstains because the remaining evidence no longer supports its conclusion.

For cutover, I would require a source-coverage reconciliation, a relevance comparison on the fixed query set, and a tested rollback that restores a compatible query encoder with its index. Rolling back only the database alias while leaving the new query encoder active can produce structurally valid but meaningless distances. The representation is a pair: the corpus encoder and the query interpretation must remain compatible.

The resulting operating record should identify active configuration, eligible source count, unresolved cleanup work, and sampled retrieval failures. None of those numbers alone proves answer quality. Together with inspectable source revisions, they make the system's behavior explainable when its contents change. That is the standard I would want before calling an embedding pipeline a maintained retrieval system.