Scientific reasoning for
trustworthy ML research.
Most trackers show you numbers. experiment-audit checks whether your claim about those numbers actually holds — missing evidence, contradictions, and confidence that isn't assumed.
Claim to report, in one pass
Claim — “95% accuracy on CIFAR-10”
Six rules. One trace from claim to verdict.
No dashboard, no vibes — every stage in the pipeline hands the next one exactly what it found, so the final report can point back to the evidence behind it.
Claims
A statement about a result — “model-x reaches 95% accuracy on CIFAR-10.” Scoped to a subject, category, and dataset.
Evidence
Metrics, configs, logs, and prior runs — whatever's meant to back the claim up, traced to its source.
Reasoning
Six rules run in sequence: missing evidence, scope match, contradiction, confidence, judgment, recommendation.
Contradictions
Cross-checked against every other claim and evidence item on file — silent conflicts don't stay silent.
Confidence
A computed score from evidence quality, quantity, and what's missing — never an assumed number.
Scientific report
A structured verdict — supported, partially supported, or unsupported — with the reasoning shown, not just stated.
One claim. Watch the verdict get built.
This is the actual six-rule pipeline, condensed to one example — a claim that looks fine until it's checked against a run already on file.
Claim
“▌”
Reasoning engine
Waiting on evidence and contradiction checks.
Everything a claim needs to earn its confidence.
Five capabilities in the reasoning engine itself, five ways to reach it — from a Python import to an agent calling it over MCP.
Scientific reasoning
A six-rule pipeline that runs claims against evidence end-to-end, or rule-by-rule when you need to inspect one step.
Claim verification
Checks whether a claim has any supporting evidence trace at all — no assumption gets a free pass.
Contradiction detection
Flags conflicts across claims and evidence before they make it into a paper or a PR description.
Confidence scoring
Driven by evidence quality and quantity, not a fixed heuristic dressed up as a number.
Scientific reports
Markdown, JSON, or plain text — every finding traces back to the evidence that produced it.
Python API
experiment_audit.reasoning embeds directly in your own tooling, notebooks, or CI checks.
CLI
experiment-audit reasoning run and schema — pipe claims in, get a report out.
Claude Code skill
The reasoning discipline itself, installable as a skill — triggers automatically on review-style prompts.
MCP integration
Eight audit tools exposed over MCP, available to any agent that speaks the protocol.
Weights & Biases
Read-only run, sweep, and metric access behind the MCP tools — no write path, ever.
The reasoning discipline, installed as a skill.
How to phrase findings, weigh contradictory evidence, and write structured reviewer-style feedback — shipped as a Claude Code skill, with all eight MCP audit tools available the moment WANDB_API_KEY is set.
Your prompt
“Is this ablation confounded?”
Claude
calls the skill
experiment-audit
checks the claim
Report back
verdict + confidence
Install the skill
Or wire the MCP server directly
Two independent layers, one repository.
The reasoning engine takes claims and evidence directly — no dependency on W&B, FastMCP, or any backend. The MCP layer feeds it real run data. They can be used separately or together.
the scientific reasoning engine
MCP layer over Weights & Biases
swappable data access
reasoning run · reasoning schema
reasoning/ has zero network calls, ever — server.py and backends/ talk to Weights & Biases through a read-only key, validated fail-fast on server start.
Built for the moment before you write the claim down.
ML engineers
sanity-check a result before it ships in a report or a PR description
AI researchers
catch confounded ablations and out-of-scope comparisons before submission
Graduate students
get reviewer-style feedback on a results section before your advisor does
Research labs
a shared, deterministic check for scientific claims across a team's experiments