Evaluating change impact when the dependency graph is incomplete

The ranker cannot see an edge that was never discovered

A code change affects a consumer that the dependency graph does not contain. The impact report omits that consumer. It is tempting to improve the ranking algorithm, especially when the report already has a neatly ordered list and a score for every visible node. But no reordering of that list can recover an absent relationship.

Change-impact analysis has at least three research problems: discover relevant relationships, prioritize the consequences worth inspecting, and perform checks that can establish what changed. Those parts need separate evidence even when the user experiences them through one interface. Combining them into a single score can hide where the useful information came from.

This article stays with general evaluation principles and illustrative examples. It does not disclose the architecture, scoring mechanisms, or experimental findings of an unreleased project. The central question is how to make a comparison informative: what did the method observe, what decision did it support, and what remains outside the evidence?

Define what each edge actually means

An import edge says that one module depends on another under a particular resolution rule. A call edge may describe a statically resolved invocation. An observed execution edge says something about the workload that produced the observation. A schema relationship describes another kind of dependency. Combining them into a graph is useful only if the edge types and their evidence remain available.

Consider an illustrative service that loads a handler from a configuration string. The source importer might find no ordinary call from the router to the handler. An execution trace can reveal the relationship for one route, but it does not establish coverage of every configuration. A repository contract may declare the intended connection. These are complementary observations, not interchangeable votes that become truth after aggregation.

For every edge, record its source, resolution method, relevant revision, and known scope. Keep unresolved references visible. An unresolved import can be more informative than a silently dropped relationship because it identifies where the report's coverage is weak. The graph should be able to say “unknown here” without needing to invent an edge or pretend that absence establishes independence.

Separate a coverage study from a ranking study

A coverage study asks which known dependencies the extractor recovers under a specified language and repository configuration. Its examples need ground truth that is not generated by the same extractor being tested. Hand-reviewed fixtures, instrumented workloads, and carefully scoped repository cases can contribute different forms of reference evidence, each with declared limits.

A ranking study begins after defining the candidate universe. It asks whether the ordering helps discover relevant consequences sooner than simpler alternatives. Compare against reachability, direct dependents, name-based matching, and a cost-aware baseline where appropriate. If one method receives richer graph evidence than another, the comparison mixes discovery with ranking and cannot isolate the ranker's contribution.

Use separate ablations for edge sources and ranking rules. Remove observed edges while keeping the ranker fixed; then keep the graph fixed while changing the ranker. A null result is useful. It may show that expensive ranking adds little on small graphs, or that a particular evidence source matters more than the scoring formula. Report ties explicitly rather than turning tiny ordering changes into a broad superiority claim.

An observed effect is narrower than a deployment forecast

A test exercises a particular set of inputs under a particular configuration. When a proposed change alters an observed output, that result establishes a contrast in the tested setting. It does not automatically establish a harmful failure. A deliberate interface change may produce a different result while satisfying its revised contract. The acceptance criterion needs to say which differences matter.

Keep the workload description alongside the outcome. Record what paths were exercised, which configurations were included, and which relevant conditions were absent. A narrow fixture can isolate a mechanism precisely; a larger integration test can exercise more interactions. Neither replaces the other. Their conclusions should remain attached to the conditions that produced them.

Use different names for different quantities. Affected describes an observed change under a stated test. Failure requires an acceptance criterion. Operational risk adds consequences, exposure, and uncertainty. A ranking score may help decide what to inspect next, but that does not make it an estimate of the probability or severity of a production incident.

For methods that produce probability estimates, the probability calibration guide distinguishes ranking outputs from probabilities evaluated against observed frequencies. Calibration needs an appropriate held-out population and a declared target. A convincing evaluation should explain whether a score is used only to order checks or whether a stronger interpretation has actually been tested.

Design the experiment around a replayable change

An evaluation unit should bind a base revision, a proposed change, the extraction configuration, the available evidence, and a verification workload. Snapshot those inputs before running candidate methods. The following illustrative manifest is a general study record, not an interface from a particular implementation.

Keep development cases separate from final evaluation cases, with related changes and near-duplicate repository fixtures grouped together. Otherwise tuning on one version of a bug and evaluating on a cosmetically different version can produce misleading confidence. For historical changes, ensure that the analysis sees only information available at the base revision; tests or dependency fixes added after the change can leak the answer.

Verification cost should be recorded alongside discovery quality. A method that finds a consequence earlier may still lose its value if graph construction or instrumentation dominates the workload. Report setup cost, marginal query cost, and check execution separately. These distinctions help a reader decide whether the result applies to a one-off repository inspection or a service that maintains a graph over many changes.

1{
2  "studyId": "synthetic-impact-fixture-07",
3  "baseRevision": "fixture-before",
4  "proposalRevision": "fixture-after",
5  "edgeSources": ["static-imports", "declared-contracts"],
6  "workload": "deterministic-route-fixture",
7  "primaryOutcome": "relevant-consumers-found-within-check-budget",
8  "limitations": ["fixed consumer scope", "no population calibration"]
9}

Make missing evidence change the recommended action

Incomplete coverage should influence how an engineering agent uses the report. A low-ranked consumer with a known path may justify a deferred check. An unresolved dynamic boundary may justify a broader test even when no visible path reaches it. These are different reasons, and the recommendation should retain them.

A useful report can show the selected checks, the evidence behind selection, the unresolved boundaries, and the stopping rule. If the check budget is exhausted, it should identify what remains unverified. Passing selected tests does not establish that all omitted consumers are safe. The conclusion should stay tied to the selected workload and observed outcomes.

The research question is not whether graphs are universally better than simpler methods. It is when particular dependency evidence and ranking choices improve a bounded verification decision. Preserve ties and negative results, and avoid general claims from a few successful examples. A useful evaluation bundle lets another reader inspect the missing relationship, the competing rankings, and the check that changed the conclusion.