RELAPSE

Frozen benchmark · Missed-sibling security fixes

RELAPSE

Recurrence Evaluation for Latent And Patch-missed Sibling Exposures

formally: Missed-Sibling Fix Evaluation Framework (MSFEF) v1.0 · frozen 2026-07-22

A frozen, reproducible benchmark for testing whether a purely structural signal can identify missed-sibling security fixes — where a patch guards some call sites of a function but leaves a structurally similar sibling unguarded, corrected only in a later commit.

01

What RELAPSE is — and what it is not

Security patches are frequently incomplete. A fix guards one code path against a defect — a missing NULL, bounds, or status check — but misses a sibling path with the same structure, corrected in a later commit. RELAPSE asks one narrow, falsifiable question: can a purely structural signal separate genuine missed siblings from look-alikes better than trivial baselines?

It does not claim to have built such a detector. It contributes the apparatus to test the claim rigorously — a frozen detector, a sealed provenance-tracked benchmark, documented evaluation pitfalls with a protocol that avoids them, and an exploratory replication — while staying explicit about what it can show today (engineering soundness) versus what it is built to determine later (a confirmatory result under independent review).

Not included, by design

  • No ground-truth labels and no accuracy claim. The confirmatory result requires two independent reviewers who did not build the detector and never see its scores.
  • No headline performance number. AI-generated or author-generated labels are treated as non-evidence — using them would make the answer untrustworthy by construction.
  • The exploratory and synthetic analyses so far have not produced evidence that the detector outperforms the predefined baselines. This is absence of supporting evidence from non-independent sources — not evidence of no effect. The independent evaluation is built to resolve it either way.

02

The missed-sibling construct

For a function f with multiple call sites, a fix at commit c0 introduces a guard — a conditional on a nearby API call, a NULL/bounds/status check — at some call sites but not at a structurally similar sibling. A later commit c1 adds the analogous guard at the sibling. The episode is the pair (c0, c1) together with the sibling call site and the guard signature.

Construct repair is itself a finding. An initial extractor (v0.3) associated guards to sinks by co-occurrence rather than control-flow, so its "families" mixed unrelated guard kinds — it did not measure what it appeared to. v2.1 repairs this by requiring the guard to dominate the sink call in the AST. All downstream work uses v2.1.

What it looks like (illustrative sketch — not a corpus episode):

/* commit c0 — the incomplete fix: one call site gets a guard */
buf = read_chunk(src_a);
+ if (buf == NULL) return ERR;  /* guard added here... */
process(buf);
...
buf2 = read_chunk(src_b);        /* ...but this sibling call site */
process(buf2);                   /* is left unguarded */

/* commit c1 — later: someone notices and fixes the missed sibling */
buf2 = read_chunk(src_b);
+ if (buf2 == NULL) return ERR;
process(buf2);

The episode is the pair (c0, c1). The detector's job — the open question — is to look at c0 alone and rank the unguarded sibling above the thousands of look-alike call sites that are fine.

03

Six sealed components

Everything needed to reproduce, extend, or contest the benchmark ships together — nothing depends on a live service or a mutable dataset.

DETECTOR

Frozen detector

7 hashed Python files (extractor, v2.1 guard→sink logic, miner, oracle, evaluator); imports hash-verified.

CORPUS

Sealed corpus

50 deduplicated episodes, 24 yielding C repos, fingerprint 4ad5278e…, full per-episode provenance.

HARNESS

Evaluation harness

LORO; genuine/plausible AP + P@{3,5,10}; three baselines; bootstrap CIs. run_benchmark.py scores any method.

LABELING

Labeling kit

Blind packet plus integrity-enforcing resolver — κ, adjudication, no partial verdicts, provenance metadata.

GOVERNANCE

Governance tooling

Isolated mining, hash-verified imports, stray-file detection, manifest fingerprinting.

DOCS

Docs & fixtures

Pre-registration, compliance audit, adopter guide, reviewer protocol, regression fixture, citation/license.

04

The sealed corpus

50
Dedup episodes (HO01–HO50)
24
Yielding C repos (~40 mined)
200
Observable commits / repo
8%
Max share, top repo

Provenance, not vibes. Each episode records its repository, both commits, connected-component id, and the extractor/guard/oracle hashes used. The canonical manifest is fingerprinted (4ad5278e44def34…), and every superseded fingerprint is preserved, giving an auditable seal history.

Registered stopping rule. Stop at the first repository whose import brings the deduplicated total to ≥50; never trim to exactly 50. The rule fired at libarchive, yielding exactly 50 episodes. Connected-component deduplication collapses multi-commit fixes so episode counts aren't inflated.

05

Evaluation protocol

A method is only credible if it beats the boring explanations. The frozen ranker must outrank all three trivial baselines, with uncertainty attached to every number.

Detector under test
Frozen linear ranker: 3·[named-API] + 2·guard_jaccard + log(1+family_size) + 0.4·n_uncovered − 5·[generic]. Coefficients frozen; never tuned on held-out data.
Protocol
LORO — leave-one-repository-out.
Metrics
Average Precision and P@3 / P@5 / P@10, reported separately for a strict genuine and a looser plausible-or-genuine target.
Baselines
Family-size, specificity-only (named-API first), and random / base-rate — the trivial explanations most likely to masquerade as a result.
Uncertainty
Bootstrap 95% confidence intervals. At n=50 only pooled AP-with-CI is interpretable; per-repository P@k is treated as noise, not a headline.

06

Three evaluation pitfalls it is built to avoid

These are general to structural-signal evaluations, and the protocol is reusable beyond this detector. Each was measured, not hypothesized.

6.1 · Labeling circularity

If labels come from the detector's authors — or an automated labeler keying on the same structural features — labels and ranker correlate by construction. Measured directly: a trivial specificity-only rule reproduced non-independent labels at ~85% and beat the ranker (AP 0.88 vs 0.68). The "encouraging" number was an artifact of the label source. Mitigation: independent reviewers only; author/AI labels are non-evidence.

6.2 · Commit multiplicity

Git history can represent one logical fix as several linked commits, inflating episode counts and correlating duplicates. Mitigation: connected-component deduplication before sealing; permanent IDs never regenerated.

6.3 · Baseline-comparison failure

On a corpus where ~84% of items are positive, everything scores ≈0.85 — reporting AP without strong trivial baselines invites over-reading. Mitigation: mandatory family-size, specificity, and base-rate baselines with CIs; only pooled AP-vs-baseline is interpretable.

07

Status & the open question

Pre-registered success criteria

Primary: at least one genuine missed-sibling episode confirmed by independent review. Secondary: ranker pooled AP above both the specificity and family-size baselines, with non-overlapping bootstrap CIs. Two reviewers, independent, not the detector's authors, blind to scores and to each other. The evaluator is then run exactly once.

Next study — MSFEF-C: label-source contamination. The field increasingly labels security datasets with LLMs. MSFEF-C measures, on this frozen corpus, how much LLM-generated labels inflate the detector's measured score versus blind independent human labels — and whether the inflation is explained by the labeler and the detector reading the same structural features. Design: three LLM arms (three vendors, frozen prompt, temperature 0), author labels, and the two-reviewer human gold standard; hypotheses and analysis plan fixed in advance. Status: pre-registration FROZEN — publicly timestamped by commit 6828822 (2026-07-25, SHA-256 b8b517e6…), before any label collection.

Become a reviewer

The confirmatory result is blocked on exactly one resource: two independent human reviewers. Everything else — corpus, protocol, tooling — is frozen and waiting.

Who qualifies: you did not build the detector, you never see its scores, and you can read C diffs. You label 50 episode packets (commit pairs with context) as genuine / plausible / coincidental / unrelated using the written protocol — roughly a day of careful work.

What you get: named credit in the published result (or anonymity if you prefer), and co-authorship consideration on the confirmatory write-up. The verdict is reported whichever way it comes out.

Interested? Write to kfoughali@dzlaws.org with a sentence about your background. The labeling kit and reviewer protocol ship in the archived artifact.

08

Get it, run it, reproduce it

1 · Get the artifact

Download the frozen v1.0 artifact from the permanent Zenodo archive — citable by DOI, hash-verifiable, never changes: doi.org/10.5281/zenodo.21539681. Docs, fixtures & pre-registration also on github.com/kimkou2024/relapse-msfef.

2 · Run the pipeline

Python 3 standard library only. Unpack and run the one-minute synthetic check:

tar xzf missed_sibling_framework_v1.0.tar.gz && cd missed_sibling_framework_v1.0
python3 evaluation/run_benchmark.py fixtures/heldout_SYNTHETIC_DRYRUN.csv "demo"
# expect: [genuine] base=0.480 | AP ranker=0.472 CI(0.32,0.66) | specificity=0.442 | family=0.476

Verify your download

Deposited files match these SHA-256 sums:

README.md764e122047b580ce34ddb2d153bcdc0c2da0f2aaad172b8fef6aac4bfb856f3b
LICENSE685bb0941f6b880fd7ce98bc9a85da4fef27ca36b5b0b8ee95d2d1a1f7984aa5
CITATION.cff828065bf2e0cd69fede094a63151613f39c4dfe44aa8e4d1e1b852bd2fd0e949
CHANGELOG.md4416a80ff785cebad54a97b4c2c279f42863990dcebcace44a0a30cbb566b279
PAPER_DRAFT.md7ad21a3b1994463647993dbc9a50bf307ebefb85473f1bdd7adbdc45704d72ab
ARTIFACT_FROZEN_v1.0.md9bea79e1150d16189b982dbbfa0bc03e944f2f57896f208653cd19630a70e654
REPRODUCIBILITY.mddf80dcc4b6bb0c92dd7dd369391ed3a9051e67fd5740749decf8fb3958ab9fa6

Verify integrity before you trust a run: recompute the sealed corpus fingerprint and the frozen-code SHA-256 hashes. Any change to frozen_code/ is a new version — v1.0 is immutable.

corpus fingerprint: 4ad5278e44def34d725a87741b9d1ee2981327754e5572617d8cf42d17b39551

08 · b

Run it here, in your browser

A client-side companion to the frozen run_benchmark.py. Paste a labeled features CSV with columns event_id,named_api,guard_jaccard,family_size,n_uncovered,generic,label — it scores the frozen ranker against the specificity and family-size baselines and reports whether the ranker beats both trivial baselines. No install.

Exploration only. The authoritative, hash-verified numbers come from the frozen Python reference; this in-page tool uses the documented ranker and standard AP / P@k.

Load the synthetic demo, or paste your own labeled CSV, then Run.

09

Cite & license

Citation (CFF)

Foughali, Karim (2026). Missed-Sibling Fix Evaluation Framework (MSFEF) v1.0.
corpus-fingerprint: 4ad5278e…
DOI: 10.5281/zenodo.21539681

Zenodo archive: doi.org/10.5281/zenodo.21539681
Source & pre-registration: github.com/kimkou2024/relapse-msfef

License & use. Code: MIT, © 2026 Karim Foughali.