Fleet

Eval-set manifests

Author evalset/v1 YAML for packs × agents × models × repeats, dry-run expansion, and combination caps.

An eval-set manifest is a YAML (or JSON) document with schema: evalset/v1. It is the only input agentclash evalset submit needs beyond workspace auth.

Generate a starter file:

bash

1agentclash evalset init
2# or: agentclash evalset init my-sweep.yaml

Minimal example

yaml

1schema: evalset/v1
2name: nightly-coding-sweep
3
4packs:
5  - catalog/code-review
6
7agents:
8  - deployment: claude-opus-default
9  - deployment: gpt-default
10
11models: []          # empty = harness default model axis (one implicit slot)
12repeats: 3
13seeds:
14  strategy: auto
15limits:
16  max_concurrent_runs: 20
17  budget_usd: 50
18case_fanout: true
19# scanners:
20#   - reward-hacking
21#   - tool-misuse

Fields

Field	Required	Meaning
`schema`	yes	Must be `evalset/v1`
`name`	yes	Human-readable set name
`packs`	yes	Pack refs: workspace version UUID, pack slug, or `catalog/<slug>`
`agents`	yes	List of `{ deployment, label? }`. `deployment` is a deployment UUID or exact name
`models`	no	Optional model override axis. Empty → one implicit slot (harness defaults)
`repeats`	no	Positive integer (default `1`)
`seeds.strategy`	no	`auto` (default) or `explicit`
`seeds.seeds`	when explicit	Positive ints, length must equal `repeats`, unique
`limits.max_concurrent_runs`	no	Soft concurrency hint for orchestration
`limits.budget_usd`	no	Soft USD budget (see Budgets and scanners)
`case_fanout`	no	Prefer per-case Temporal activities for native packs
`scanners`	no	Built-in scanner names to run after the set completes

Agent labels

Optional label overrides how the agent appears in matrix_key. Labels must stay unique after normalization (duplicate labels fail expansion).

yaml

1agents:
2  - deployment: 11111111-1111-1111-1111-111111111111
3    label: opus
4  - deployment: 22222222-2222-2222-2222-222222222222
5    label: gpt

Expansion

The cartesian product is:

text

|packs| × |agents| × |models or 1| × repeats

Each cell gets a stable matrix_key used by the matrix UI, logs filter, warehouse rows, and CLI reports.

Dry-run without creating anything:

bash

agentclash evalset submit sweep.yaml --dry-run

Or call POST /v1/eval-sets/expand with the manifest JSON and optional max_combinations.

Caps

Default expansion cap: 2000 combinations.
Server hard ceiling: 2000 (MaxAllowedCombos). Clients cannot raise it.
Duplicate matrix_key values are rejected (usually colliding agent labels).

Resolving pack and agent refs

On submit, the CLI resolves:

Pack: workspace UUID, slug, or catalog/<slug> (catalog instantiate only on definitive not-found).
Agent: deployment UUID or exact deployment name in the workspace.

Prefer UUIDs in CI for stability. Use names locally when iterating.

Relation to challenge packs

Manifests reference challenge packs; they do not replace pack YAML. Case inputs, tools, sandboxes, and scoring still come from the pack. For pack authoring see Challenge pack reference.

PreviousFleet overview NextCLI evalset workflow