Frozen benchmark · Missed-sibling security fixes
RELAPSE
formally: Missed-Sibling Fix Evaluation Framework (MSFEF) v1.0 · frozen 2026-07-22
A frozen, reproducible benchmark for testing whether a purely structural signal can identify missed-sibling security fixes — where a patch guards some call sites of a function but leaves a structurally similar sibling unguarded, corrected only in a later commit.
01
What RELAPSE is — and what it is not
Security patches are frequently incomplete. A fix guards one code path against a defect — a missing NULL, bounds, or status check — but misses a sibling path with the same structure, corrected in a later commit. RELAPSE asks one narrow, falsifiable question: can a purely structural signal separate genuine missed siblings from look-alikes better than trivial baselines?
It does not claim to have built such a detector. It contributes the apparatus to test the claim rigorously — a frozen detector, a sealed provenance-tracked benchmark, documented evaluation pitfalls with a protocol that avoids them, and an exploratory replication — while staying explicit about what it can show today (engineering soundness) versus what it is built to determine later (a confirmatory result under independent review).
Not included, by design
- No ground-truth labels and no accuracy claim. The confirmatory result requires two independent reviewers who did not build the detector and never see its scores.
- No headline performance number. AI-generated or author-generated labels are treated as non-evidence — using them would make the answer untrustworthy by construction.
- The exploratory and synthetic analyses so far have not produced evidence that the detector outperforms the predefined baselines. This is absence of supporting evidence from non-independent sources — not evidence of no effect. The independent evaluation is built to resolve it either way.
02
The missed-sibling construct
For a function f with multiple call sites, a fix at commit c0 introduces a guard — a conditional on a nearby API call, a NULL/bounds/status check — at some call sites but not at a structurally similar sibling. A later commit c1 adds the analogous guard at the sibling. The episode is the pair (c0, c1) together with the sibling call site and the guard signature.
Construct repair is itself a finding. An initial extractor (v0.3) associated guards to sinks by co-occurrence rather than control-flow, so its "families" mixed unrelated guard kinds — it did not measure what it appeared to. v2.1 repairs this by requiring the guard to dominate the sink call in the AST. All downstream work uses v2.1.
What it looks like (illustrative sketch — not a corpus episode):
/* commit c0 — the incomplete fix: one call site gets a guard */ buf = read_chunk(src_a); + if (buf == NULL) return ERR; /* guard added here... */ process(buf); ... buf2 = read_chunk(src_b); /* ...but this sibling call site */ process(buf2); /* is left unguarded */ /* commit c1 — later: someone notices and fixes the missed sibling */ buf2 = read_chunk(src_b); + if (buf2 == NULL) return ERR; process(buf2);
The episode is the pair (c0, c1). The detector's job — the open question — is to look at c0 alone and rank the unguarded sibling above the thousands of look-alike call sites that are fine.
03
Six sealed components
Everything needed to reproduce, extend, or contest the benchmark ships together — nothing depends on a live service or a mutable dataset.
Frozen detector
7 hashed Python files (extractor, v2.1 guard→sink logic, miner, oracle, evaluator); imports hash-verified.
Sealed corpus
50 deduplicated episodes, 24 yielding C repos, fingerprint 4ad5278e…, full per-episode provenance.
Evaluation harness
LORO; genuine/plausible AP + P@{3,5,10}; three baselines; bootstrap CIs. run_benchmark.py scores any method.
Labeling kit
Blind packet plus integrity-enforcing resolver — κ, adjudication, no partial verdicts, provenance metadata.
Governance tooling
Isolated mining, hash-verified imports, stray-file detection, manifest fingerprinting.
Docs & fixtures
Pre-registration, compliance audit, adopter guide, reviewer protocol, regression fixture, citation/license.
04
The sealed corpus
Provenance, not vibes. Each episode records its repository, both commits, connected-component id, and the extractor/guard/oracle hashes used. The canonical manifest is fingerprinted (4ad5278e44def34…), and every superseded fingerprint is preserved, giving an auditable seal history.
Registered stopping rule. Stop at the first repository whose import brings the deduplicated total to ≥50; never trim to exactly 50. The rule fired at libarchive, yielding exactly 50 episodes. Connected-component deduplication collapses multi-commit fixes so episode counts aren't inflated.
05
Evaluation protocol
A method is only credible if it beats the boring explanations. The frozen ranker must outrank all three trivial baselines, with uncertainty attached to every number.
06
Three evaluation pitfalls it is built to avoid
These are general to structural-signal evaluations, and the protocol is reusable beyond this detector. Each was measured, not hypothesized.
6.1 · Labeling circularity
If labels come from the detector's authors — or an automated labeler keying on the same structural features — labels and ranker correlate by construction. Measured directly: a trivial specificity-only rule reproduced non-independent labels at ~85% and beat the ranker (AP 0.88 vs 0.68). The "encouraging" number was an artifact of the label source. Mitigation: independent reviewers only; author/AI labels are non-evidence.
6.2 · Commit multiplicity
Git history can represent one logical fix as several linked commits, inflating episode counts and correlating duplicates. Mitigation: connected-component deduplication before sealing; permanent IDs never regenerated.
6.3 · Baseline-comparison failure
On a corpus where ~84% of items are positive, everything scores ≈0.85 — reporting AP without strong trivial baselines invites over-reading. Mitigation: mandatory family-size, specificity, and base-rate baselines with CIs; only pooled AP-vs-baseline is interpretable.
07
Status & the open question
- Engineering, methodology & governance: COMPLETE — frozen and immutable at v1.0 (2026-07-22).
- Exploratory external replication: done, separate — PostgreSQL, QEMU, redis, sqlite (7 "silver" candidates, ≈0.9% of observable commits, mostly generic). Not confirmatory.
- Confirmatory evaluation: PENDING independent human review, by design.
Pre-registered success criteria
Primary: at least one genuine missed-sibling episode confirmed by independent review. Secondary: ranker pooled AP above both the specificity and family-size baselines, with non-overlapping bootstrap CIs. Two reviewers, independent, not the detector's authors, blind to scores and to each other. The evaluator is then run exactly once.
Next study — MSFEF-C: label-source contamination. The field increasingly labels security datasets with LLMs. MSFEF-C measures, on this frozen corpus, how much LLM-generated labels inflate the detector's measured score versus blind independent human labels — and whether the inflation is explained by the labeler and the detector reading the same structural features. Design: three LLM arms (three vendors, frozen prompt, temperature 0), author labels, and the two-reviewer human gold standard; hypotheses and analysis plan fixed in advance. Status: pre-registration FROZEN — publicly timestamped by commit 6828822 (2026-07-25, SHA-256 b8b517e6…), before any label collection.
Become a reviewer
The confirmatory result is blocked on exactly one resource: two independent human reviewers. Everything else — corpus, protocol, tooling — is frozen and waiting.
Who qualifies: you did not build the detector, you never see its scores, and you can read C diffs. You label 50 episode packets (commit pairs with context) as genuine / plausible / coincidental / unrelated using the written protocol — roughly a day of careful work.
What you get: named credit in the published result (or anonymity if you prefer), and co-authorship consideration on the confirmatory write-up. The verdict is reported whichever way it comes out.
Interested? Write to kfoughali@dzlaws.org with a sentence about your background. The labeling kit and reviewer protocol ship in the archived artifact.
08
Get it, run it, reproduce it
1 · Get the artifact
Download the frozen v1.0 artifact from the permanent Zenodo archive — citable by DOI, hash-verifiable, never changes: doi.org/10.5281/zenodo.21539681. Docs, fixtures & pre-registration also on github.com/kimkou2024/relapse-msfef.
2 · Run the pipeline
Python 3 standard library only. Unpack and run the one-minute synthetic check:
tar xzf missed_sibling_framework_v1.0.tar.gz && cd missed_sibling_framework_v1.0
python3 evaluation/run_benchmark.py fixtures/heldout_SYNTHETIC_DRYRUN.csv "demo"
# expect: [genuine] base=0.480 | AP ranker=0.472 CI(0.32,0.66) | specificity=0.442 | family=0.476
Verify your download
Deposited files match these SHA-256 sums:
| README.md | 764e122047b580ce34ddb2d153bcdc0c2da0f2aaad172b8fef6aac4bfb856f3b |
| LICENSE | 685bb0941f6b880fd7ce98bc9a85da4fef27ca36b5b0b8ee95d2d1a1f7984aa5 |
| CITATION.cff | 828065bf2e0cd69fede094a63151613f39c4dfe44aa8e4d1e1b852bd2fd0e949 |
| CHANGELOG.md | 4416a80ff785cebad54a97b4c2c279f42863990dcebcace44a0a30cbb566b279 |
| PAPER_DRAFT.md | 7ad21a3b1994463647993dbc9a50bf307ebefb85473f1bdd7adbdc45704d72ab |
| ARTIFACT_FROZEN_v1.0.md | 9bea79e1150d16189b982dbbfa0bc03e944f2f57896f208653cd19630a70e654 |
| REPRODUCIBILITY.md | df80dcc4b6bb0c92dd7dd369391ed3a9051e67fd5740749decf8fb3958ab9fa6 |
Verify integrity before you trust a run: recompute the sealed corpus fingerprint and the frozen-code SHA-256 hashes. Any change to frozen_code/ is a new version — v1.0 is immutable.
corpus fingerprint: 4ad5278e44def34d725a87741b9d1ee2981327754e5572617d8cf42d17b39551
08 · b
Run it here, in your browser
A client-side companion to the frozen run_benchmark.py. Paste a labeled features CSV with columns event_id,named_api,guard_jaccard,family_size,n_uncovered,generic,label — it scores the frozen ranker against the specificity and family-size baselines and reports whether the ranker beats both trivial baselines. No install.
Exploration only. The authoritative, hash-verified numbers come from the frozen Python reference; this in-page tool uses the documented ranker and standard AP / P@k.
Load the synthetic demo, or paste your own labeled CSV, then Run.
09
Cite & license
Citation (CFF)
Foughali, Karim (2026). Missed-Sibling Fix Evaluation Framework (MSFEF) v1.0.
corpus-fingerprint: 4ad5278e…
DOI: 10.5281/zenodo.21539681
Zenodo archive: doi.org/10.5281/zenodo.21539681
Source & pre-registration: github.com/kimkou2024/relapse-msfef
License & use. Code: MIT, © 2026 Karim Foughali.