Fleet: eval sets at cloud scale, plus two new ways to author a pack
The Fleet epic landed in fifteen pull requests — case-level fan-out, Kubernetes sandboxes, per-account LLM throttling, eval-set manifests, a live matrix dashboard, post-hoc scanners, budgets with emergency stop, OTel metrics, and a Helm chart. Alongside it: one-command local setup for contributors, custom OpenAI-compatible endpoints, pack drafts generated from a tryout or a plain-English description, and a machine-readable public site.
- Fleet eval sets
- Self-host and scale
- Pack authoring
- Onboarding
What shipped
Added
- Eval-set manifests (`evalset/v1`) — parse, dry-run combination expansion, persistence, and orchestration of child sessions across packs, agents, and models.
- `agentclash evalset submit`, `status`, `logs`, `report`, and `cancel` for driving eval sets from the terminal or CI.
- Live eval-set matrix dashboard plus a case-results warehouse with search, report, and export.
- Per-set budgets with emergency stop, and a post-hoc scanner framework that flags patterns across a finished eval set.
- Kubernetes sandbox provider with a conformance suite, sandbox capacity budgets and warm pool, bounded workflow fan-out, queue partitioning, and case-level fan-out for native run-agents.
- Helm chart with KEDA-based Temporal worker autoscaling and published GHCR images for self-hosted deployments.
- Custom OpenAI-compatible provider accounts — persisted per-account base URLs with SSRF protection at both save time and dial time.
- Two new ways to start a challenge pack: promote a completed agent tryout into an editable draft, or describe your app in plain English and get a generated pack blueprint in the builder.
- Machine-readable public site — stable Markdown URLs for every indexable route, `llms.txt`, `Accept: text/markdown` negotiation, and published OpenAPI, JSON Schema, and CLI schema contracts.
- One-command contributor setup — `make setup`, `make start`, `make doctor`, and `make check`, with Temporal in Docker and a dev profile that runs with zero API keys.
- `/healthz/ready` readiness probe that reports whether an instance can actually serve traffic, separate from the liveness check.
Improved
- OpenTelemetry metrics, stall detection, and a ready-to-import alert pack for run and eval-set health.
- Oversized run-event payloads now spill to object storage, and outbound LLM calls are throttled per account across workers.
Fixed
- Onboarding funnel fixes — dead demo CTA on every docs page, CLI errors that never pointed at signup, and jargon that hid what the product does.
- Fleet matrix cells now read as human labels, and backend images build from the repo root so the shared runtime module resolves.
- Benchmarks hub reordered to lead with measured proof and drop empty scoreboard columns.
Merged pull requests
31 PRs- #1256feat: make AgentClash agent-readable
- #1255feat: support custom OpenAI-compatible endpoints
- #1247chore(main): release 2.1.2
- #1246feat: generate an eval pack draft from a plain-English description
- #1245feat: promote agent tryouts into editable pack drafts
- #1233chore(main): release 2.1.1
- #1232fix: human labels and clearer Fleet matrix UI
- #1231docs: add Fleet eval-set guides to agentclash.dev/docs
- #1230chore(main): release 2.1.0
- #1229fix: onboarding funnel quick wins — dead CTAs, jargon, hidden prerequisites
- #1228feat(api): /healthz/ready readiness probe
- #1227feat(deploy): Helm chart, KEDA Temporal scaling, GHCR images (#1209)
- #1226feat(observability): OTel metrics, stall detection, alert pack (#1211)
- #1225feat(scanners): post-hoc eval-set scanner framework (#1208)
- #1224feat(evalset): per-set budgets and emergency stop (#1210)
- #1223feat(web): live eval-set matrix dashboard (#1207)
- #1222feat(evalset): case_results warehouse search, report, and export
- #1221feat(cli): add evalset submit, status, logs, report, and cancel
- #1220feat(evalset): persist eval sets and orchestrate child sessions
- #1219feat(evalset): parse manifests and dry-run combination expansion
- #1218feat(runevents): spill oversized event payloads to object storage
- #1217feat(provider): distributed outbound LLM throttle per account
- #1216feat(sandbox): Fleet 4 — Kubernetes provider + conformance suite
- #1215feat(sandbox): Fleet 3 — capacity budget, warm pool, docker provider
- #1214feat(workflow): Fleet 2 — bounded fan-out, worker knobs, queue partitioning
- #1213feat(workflow): Fleet 1 — case-level fan-out for native run-agents
- #1197fix(deploy): build backend images from repo root so ../runtime resolves
- #1180chore: community-health files and structured issue intake (PR B)
- #1179docs: README landing page and adoption assets (PR F)
- #1178feat: frictionless local setup and contributor onboarding (PR A + E)
- #1048fix(web): reorder benchmarks hub to lead with proof, drop empty scoreboard columns