Evidence Before Erasure
A benchmark of how well automated collection policies preserve evidence from a live AI incident before it disappears. Periodic snapshots are compared with a budget-matched, event-driven policy on a real trace. The main contrast came back null, and the report says so.
Track 2 of the Apart x CeSIA AI Incident Response Research Sprint, built by a team of five. The pipeline is Python with the standard library only, developed as frozen stages that were each audited by someone who did not write the code.
The question: when AI agents leave evidence on a page that can change or vanish, does a naive time-interval snapshotter lose materially more critical evidence than a policy that reacts to change notifications under the same request and storage budget? The test trace is a real one, AI agents sharing answers through a dormant wiki during evaluations.
The project implements a state-reconstruction engine, a shared observer and storage layer, four collection policies and an evaluation benchmark with human-labelled evidence. Each stage was specified in a contract before it was written and then checked independently.
The full 791-row experiment sweep was authorised and failed on a 7,200-second timeout before finishing its first row. A reduced primary contrast was run directly instead: every phase scored 0 of 23 held-out critical items, a clean null ten percentage points short of the pre-set threshold. The submitted report states that result, how it was obtained outside the normal authorisation gate, and which rows remain unfinished.
Limits stated in the repository: one incident family, a purposive benchmark rather than a representative sample, and a simulation of a shared feed rather than a measurement of historical HTTP visibility.
Project goal
Measure, under an equal budget, how much critical evidence different collection policies preserve from a live incident, and report the result honestly whichever way it falls.
Key features
- Four collection policies compared under matched request and storage budgets
- State-at-time reconstruction engine and shared observer/storage layer
- Human-labelled benchmark with development and held-out splits
- Stage-by-stage frozen contracts with independent audits
- Standard-library-only Python, unit tested
- Null primary result reported with full process disclosure
Project details
Event
Apart x CeSIA AI Incident Response Sprint
Type
Evaluation Research (team of five)
Primary
Null: 0 of 23 held-out items
Status
Submitted; sweep incomplete
Technologies
Links