Reproducibility
A publication contract for agent-systems evidence
Evidence before narrative
Agent evaluations are unusually sensitive to prompt wording, provider behavior, tool availability, runtime policy, judge variance, and infrastructure failures. A score without those conditions is not a reusable result. Protocore publications therefore ship an evidence package alongside the article.
Claims ledger
Every quantitative claim receives a stable public identifier and a ledger entry containing:
| Field | Purpose |
|---|---|
| Definition | States exactly what is counted and what counts as success. |
| Scope | Names the task set, sample size, repetitions, and exclusions. |
| Configuration | Records model, runtime, tool policy, and relevant limits. |
| Provenance | Links the claim to sanitized data and the code that derived it. |
| Uncertainty | Reports variation or confidence intervals where the design supports them. |
| Limitations | Prevents a local result from becoming an unconditional product claim. |
| Date | Makes drift visible instead of silently replacing an earlier observation. |
Reproducible figures and tables
Figures and tables are generated from versioned, publication-safe inputs. Manual editing of rendered values is not allowed. Analysis scripts must fail when an expected column, sample, or configuration is absent rather than filling gaps with assumptions.
Three kinds of outcome
Public evaluations distinguish at least three layers:
- Platform outcome — did the run start, stream, and terminate as designed?
- Behavioral outcome — did the agent follow the relevant tool and runtime policy?
- Task outcome — is the resulting artifact or answer objectively valid?
An LLM judge may add a fourth perspective, but it does not replace the other three. Judge prompts, sampling settings, and calibration belong in the evidence package.
Public-safety boundary
Publication artifacts are produced as a clean-room export. They exclude credentials, private endpoints, personal or customer identifiers, production traces, private account identifiers, and deployment details. Synthetic examples replace private operational data when the scientific question does not require it.
The site build checks both source and generated output. PDFs and retained TeX are included in the same verification boundary as HTML.
Build contract
Quarto is the single article source. A paper renders to scholarly HTML and PDF, retains TeX when needed for submission, and packages its reviewed source. The public nginx image receives only the generated publish/ context; private data cannot be copied into it from the repository root.
The scholarly HTML edition is the primary accessible publication artifact: it provides semantic headings, text alternatives for figures, keyboard-operable tables, and reflow at narrow widths. Current LaTeX PDFs remain selectable and searchable but are not tagged for assistive technology. Each publication presents the HTML alternative next to its PDF download until a stable tagged-PDF path can meet the same reproducibility and submission requirements.
See the repository README for the exact build and verification commands.