Tier-1 Reproducibility Survey
200 bioRxiv genomics preprints (Oct–Dec 2025) were scored for computational reproducibility readiness — whether a third party could, in principle, re-run the analysis from what the authors published. No code was executed in this stage; we read the manuscript and linked artifacts only.
What this is NOT
This is not a reproducibility success rate. Tier-1 produces an upper bound (“ceiling”) on readiness from documentation alone. A paper scored “partially ready” here may still fail when code is actually run (Tier-2).
At a glance — how many papers at each readiness level?
Each paper is placed in one bucket based on the strongest environment-capture claim in the manuscript. Think of this as “how much reproducibility groundwork the paper itself documents.”
Readiness ladder (papers)
Three buckets sum to 100%. Most papers (67.0%) name tool versions but do not ship a closed environment.
Documentation depth per paper
152 papers (76.0%) have at least one analysis step partially documented (verdict RI). 48 papers (24.0%) have no step reaching even partial readiness.
The version axis alone caps half of all computational steps. Environment capture is the binding constraint on this corpus.
How the Tier-1 check works
GenomicsChain Tier-1 is a static, human-reviewed assessment aligned with
NASEM (2019) reproducibility scope and ACM Artifact Badging v1.1 verdict tiers.
Each paper becomes a structured report.json with a sign-off field. The field is
in place; none of the published reports is signed yet.
Stage-2 funnel (what happens next)
- 29 papers (14.5%) claim a strong enough environment pin to enter code verification.
- 17 papers will be randomly audited from the remainder (10% stratified sample).
- Total Stage-2 load: 46 papers — only then can we say “reproduced” vs “not.”
Standards & checkpoints
Four axes scored on every computational step
| Axis | Score 0 | Score 1 | Score 2 (Stage-1 ceiling = 1 for Version) |
|---|---|---|---|
| Data | Missing or inaccessible | ID present but gated / incomplete | Public or legitimate controlled-access path documented |
| Code | Not shared | Reconstructable from description | Exact public artifact linked |
| Version (environment) |
Unknown / unpinned | Partial pins (tool names, floating reqs) | Closed environment — lockfile or container digest (verified in Stage 2) |
| Params | Not stated | Partially stated | All parameters for the step documented |
Step verdicts (plain language)
RI — Partially readyEvery axis ≥ 1. Artifacts exist but gaps remain. Most common at Stage-1.
NI — Needs investigationAt least one axis is 0 but recoverable in principle (missing link, missing seed, etc.).
X — BlockedUnrecoverable from published materials (no code, proprietary black box, etc.).
R★ — Declared closureAll axes at 2 in declared environment — but unexecuted. Requires Stage-2 code run; 0 steps at Stage-1.
References
The Tier-1 readiness metric is not a new private score: its scope, verdict tiers and graduated levels are taken from the standards below. Roles are stated so a reader can check what each source does and does not license us to claim.
- National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi:10.17226/25303 Scope anchor. “Reproducibility” on this page means the NASEM sense — same input data, code and methods → consistent results. Replicability (new data) and scientific validity are out of scope.
- Association for Computing Machinery (2020). Artifact Review and Badging — Version 1.1 (24 August 2020). acm.org/publications/policies/artifact-review-and-badging-current Verdict mapping. R★ ≈ Artifacts Available + Evaluated-Functional; RI ≈ Available but not turnkey; NI ≈ below Evaluated; X ≈ not Available. ACM’s “exact reproduction not required, within an acceptable tolerance” is why stochastic steps are scored by distributional match. Version 1.1 matters: pre-2020 definitions had reproducibility and replicability the other way round.
- Essawy, B. T., Goodall, J. L., Voce, D., Morsy, M. M., Sadler, J. M., Choi, Y. D., Tarboton, D. G., & Malik, T. (2020). A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling & Software, 134, 104753. doi:10.1016/j.envsoft.2020.104753 Graduated tiers (repeatable / runnable / reproducible / replicable). Our RI (“partially ready”) corresponds to their runnable.
- Martín del Pico, E., Gelpí, J. L., & Capella-Gutiérrez, S. (2024). FAIRsoft — a practical implementation of FAIR principles for research software. Bioinformatics, 40(8), btae464. doi:10.1093/bioinformatics/btae464 Scale precedent — automated indicator scoring applied to tens of thousands of bioinformatics tool entries, which is why a Tier-1-at-scale survey is feasible. Boundary: FAIRsoft scores software FAIRness, not paper-level re-execution. A high FAIRsoft score is not a readiness score.
- Barker, M., Chue Hong, N. P., Katz, D. S., et al. (2022). Introducing the FAIR Principles for research software. Scientific Data, 9, 622. doi:10.1038/s41597-022-01710-x FAIR4RS is a motivation, not a verification rubric — it does not supply the re-execution axis this survey scores.
- Nüst, D., & Eglen, S. J. (2021). CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. F1000Research, 10, 253. doi:10.12688/f1000research.51738.2 Closest existing service, and the model for the “verification, not validation” boundary: we check whether published materials support re-running the analysis, not whether the science is correct. CODECHECK is manual and per-paper; this survey is AI-assisted, not yet human-signed-off, and sample-wide.
- Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024). CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv:2409.11363. arxiv.org/abs/2409.11363 Why human sign-off is mandatory. The best agent in that benchmark solved about a fifth of the hardest tier, so an unsupervised AI score would measure the agent, not the paper.
- Plesser, H. E. (2018). Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in Neuroinformatics, 11, 76. doi:10.3389/fninf.2017.00076 Terminology history — why this page states which definition it uses instead of assuming one.
- Kanwal, S., Khan, F. Z., Lonie, A., & Sinnott, R. O. (2017). Investigating reproducibility and tracking provenance — a genomic workflow case study. BMC Bioinformatics, 18, 337. doi:10.1186/s12859-017-1747-0 Depth precedent in genomics — a single workflow examined in detail. This survey trades that depth for breadth (N = 200), which is the gap it is designed to fill.
Standards alignment does not transfer their authority to our numbers. Tier-1 scores are contestable and supersedable; the rubric (v0.2.5), the sampling pre-registration and the 13-type gap dictionary are GenomicsChain’s own and are versioned separately.
Environment capture claims (detail)
Finer breakdown of what authors claim about pinning their software environment, from weakest (left) to strongest (right).
Capture claim distribution
Read: bar length = number of papers. Takeaway: 67.0% stop at “versions named in text”; only 14.5% claim a pin strong enough for code verification. Zero papers claim an immutable container digest.
Stage-2 funnel
Read: green = papers entering code check; grey = stops at Stage-1; mint = random audit sample from the rest. Takeaway: 14.5% advance; 85.5% terminate without code execution.
| Claim type | Papers | Share | Meaning |
|---|---|---|---|
| No pinning stated | 10 | 5.0% | The paper does not describe how to rebuild the software environment. |
| Floating versions (≥ / latest) | 27 | 13.5% | Versions use open ranges or “latest” — results may drift over time. |
| Exact tool names, environment not closed | 134 | 67.0% | Specific tool versions appear in text, but there is no single installable, closed environment. |
| Release snapshot (e.g. Bioconductor date) | 8 | 4.0% | Authors cite a dated release (e.g. Bioconductor version) plus interpreter — stronger, still unverified here. |
| Lockfile / conda env claimed | 21 | 10.5% | Authors point to a lockfile or equivalent — eligible for code verification (Stage 2). |
| Container with digest | 0 | 0.0% | Authors cite an immutable container digest — strongest claim; none observed in this sample. |
Where documentation breaks down (step level)
4,005 computational steps across 200 papers. Each step is scored on four axes; the lowest axis caps the verdict.
Mean axis score (0–2)
Read: higher = better documentation on that axis. Takeaway: Version (environment) averages 0.57 — lowest of the four. At Stage-1, Version cannot reach 2 without code verification anyway.
Binding axis — what limits each step?
Read: for each step, the axis with the lowest score is the bottleneck (ties split evenly). Takeaway: Version alone caps ~49.9% of steps — environment capture is the corpus-wide constraint, not data sharing.
Verdicts & computational coverage
Step verdicts across all papers
Read: RI = partially ready, NI = needs investigation, X = blocked. R is always 0 at Stage-1. Takeaway: 54.8% of steps have a recoverable gap; 6.0% are blocked.
How much of each paper is computational?
Read: S/(S+W) = share of enumerated steps that are computational vs wet-lab (out of scope). Median 83% — most papers mix computation with lab work.
Common documentation gaps
Gap types from the 13-type rubric dictionary — how many papers exhibit each gap (a paper can have multiple). Shares are computed over n = 199 papers: 1 of the 200 analyzed reports is excluded from gap tallies as malformed, so percentages in this table differ slightly from the paper-level figures elsewhere on this page.
| Gap | Papers | Share |
|---|---|---|
| Version not recorded | 194 | 97.5% |
| Sample / cohort mapping missing | 181 | 91.0% |
| Custom database or script not shared | 171 | 85.9% |
| Stochastic step without seed | 152 | 76.4% |
| Sub-pipeline outsourced | 137 | 68.8% |
| Dead or broken link | 131 | 65.8% |
| Wet-lab step (out of scope) | 121 | 60.8% |
| Analysis code not deposited | 120 | 60.3% |
Phase 1 file bootstrap · Generated 2026-08-05T10:30:13Z · API: /api/v1/tier1_survey/stats · Chain block #145
Canonical proof path (meeting anchor submit) is Phase 2. This preview reads from
data/tier1_survey/collection.