Fleet

Fleet overview

Run packs × agents × models × repeats as one eval set with a live matrix, warehouse exports, budgets, and transcript scanners.

Fleet is AgentClash's cloud-scale eval plane. You declare one manifest (packs × agents × models × repeats). AgentClash expands it into combinations, runs them with bounded concurrency and isolated sandboxes, then gives you a live matrix, queryable case results, optional budgets, and post-run transcript scanners.

Use Fleet when a single run create or eval session is not enough: nightly sweeps, multi-agent bakeoffs, multi-pack regression, or self-hosted fleets that need worker autoscaling.

What you get

Surface	Role
Eval-set manifest (`evalset/v1`)	Declarative packs × agents × models × repeats
CLI `agentclash evalset`	Submit, follow, logs, report, cancel
Web matrix	Live cells per combination under workspace → Eval sets
Case warehouse	Search, report, CSV/JSONL export of per-case outcomes
Budgets	Soft USD caps and workspace emergency stop
Scanners	Optional post-completion transcript analysis
Self-host scale	Helm chart, KEDA queues, Kubernetes sandboxes, metrics

Prerequisites

Before you submit an eval set:

A workspace with API access (agentclash auth login, workspace selected).
Challenge pack versions available in the workspace (or catalog slugs).
Agent deployments wired with provider credentials / BYOK.
A sandbox provider configured for native packs (e2b, docker, or kubernetes on self-host).

Hosted users typically do step 1–3 on agentclash.dev. Self-hosters also need Temporal, Postgres, workers, and a sandbox path; see Self-host at scale.

Shortest path

bash

1export AGENTCLASH_API_URL="https://api.agentclash.dev"
2
3agentclash auth login --device
4agentclash workspace use <workspace-id>
5
6agentclash evalset init
7# edit agentclash.evalset.yaml — packs, deployments, repeats, budget
8
9agentclash evalset submit agentclash.evalset.yaml --dry-run
10agentclash evalset submit agentclash.evalset.yaml --yes --follow
11agentclash evalset report <eval-set-id> --format csv

Open the set in the product UI: Workspace → Eval sets → your set for the live matrix and case explorer.

How it relates to runs and eval sessions

A run is still the unit of execution and scoring.
An eval session groups repeated runs of one pack lineup.
An eval set is the Fleet layer: many packs × agents × models × repeats, orchestrated as child sessions/runs under one set ID and matrix_key identity.

See Runs and evals for the noun map, then continue with Eval-set manifests.

Docs in this section

Eval-set manifests — YAML fields, expansion, limits
CLI evalset workflow — submit through report
Matrix and warehouse — UI + export APIs
Budgets and scanners — spend caps and findings
Self-host at scale — Helm, sandboxes, queues, observability

PreviousEval workflows & gates NextEval-set manifests