Use this page to see which provider/model WebikAI would currently choose for common lanes, and why. Scores combine deterministic probes, live reliability, latency, and published costs. Every number shows sample size; methodology and limits are below.
Auto-generated from confirmed regressions and recoveries — a status page for model quality.
No confirmed change events yet. Entries appear here only when a deep-suite run confirms a regression or recovery — we never seed this log.
No published measurements yet — scores appear once endpoints accumulate enough pooled samples (n ≥ 150). We don't publish numbers we can't stand behind.
Lanes. Six machine-graded suites: JSON extraction, schema output, tool-call formation (with distractor tools), deterministic coding tasks, exact-answer math, and completion of legitimate professional work. All run at temperature 0 and are graded pass/fail by code, not by a judge model.
Cadence. Live-traffic rollups hourly ($0 — real requests are the best probe); 10-item canaries and greedy-prefix divergence fingerprints every 6 hours; the full ~60-item deep suite (3 repetitions) weekly per endpoint and immediately when an alarm trips. Only a deep-suite run confirmed by a two-proportion test (α = 0.05) with at least a 5-percentage-point drop flags a regression — statistical significance alone never flags.
Scores. Every published number is a pass rate with its Wilson 95% interval and sample size; the headline is the lower bound. Rows with pooled n < 150 say “insufficient data” instead of a number.
The over-refusal lane measures completion of legitimate professional work only; the system never measures or ranks willingness to produce harmful content, and no probe content may be harmful.
One-sample test vs a known baseline at p₀ ≈ 0.85, α = .05, power .8.
| Drop size | n needed | Achieved by | Time-to-detect |
|---|---|---|---|
| ≥15 pp | ~50 | one day of canaries (40) + any T3 | 6–24 h |
| ~10 pp | ~115 | a single T3 run (180) | ≤ 1 week, or hours if alarmed |
| ~5 pp | ~450 | pooled week (canaries 280 + T3 180) | ~1 week |
| ≤3 pp | ~1200+ | outside the public alert threshold | not published as an alert |
Build on the data: /public/benchmarks · /public/benchmarks/events · /public/benchmarks/leaders