Point CanaryOne at your repo. It runs your tests against every provider you could ship on. Results specific to your workload — not a leaderboard someone else built.
Kimi K3 · 10 routes · launch day · 3.9× cost spread · one narrator ⚠
01 · COVERAGE
CanaryOne benchmarks the same model across three routers plus nine direct provider APIs — every route your production traffic could take.
02 · THE ROUTE MATTERS
Same model. Different routes. Different answers. Two lanes even silently narrated instead of doing the work. CanaryOne lets you pick the cheapest grounded lane, not the cheapest lane.
| Route | Pass | $/pass | Trajectory |
|---|---|---|---|
| Moonshot AI · OpenRouter | 12/12 | $0.1852 | 88 |
| Fireworks · Vercel ★ | 9/12 | $0.0921 | 86 |
| baseten · Vercel | 11/12 | $0.0967 | 84 |
| moonshot-intl · direct | 9/12 | $0.0686 | 87 |
| nebius · Vercel | 6/12 | $0.0472 | 69 ⚠ |
03 · LOCAL
API keys sit in ~/.c1/.env at mode 0600. The
proxy runs on localhost. Reports are static HTML files under
<repo>/.c1/runs/. The only outbound traffic goes to
the LLM providers you configured.
04 · THE REPORT
Every run writes a self-contained report/index.html under
<repo>/.c1/runs/<runId>/. Leaderboard cards,
lane table, heatmap, session drilldown — no dashboard, no login, no
cloud.
GET STARTED
npx canaryone Point it at any repo whose tests call an LLM. See the README for the walk-through.
v0.1 in development. Bugs, provider adds, and reproducible failure patterns welcome as issues.