K
Start free

Fleet

CLI evalset workflow

Submit, follow, log, report, and cancel Fleet eval sets with agentclash evalset.

The agentclash evalset command group is the primary operator surface for Fleet.

bash
1export AGENTCLASH_API_URL="https://api.agentclash.dev"
2# or point at your self-hosted API
3
4agentclash auth login --device
5agentclash workspace use <workspace-id>

Commands

CommandPurpose
evalset init [path]Write a commented evalset/v1 starter manifest
evalset submit <file>Expand, optionally confirm, create the set
evalset listRecent sets in the workspace
evalset status <id>Roll-up status (--watch until terminal)
evalset logs <id>Multiplexed run-event streams (-f to follow)
evalset report <id>Scorecard table / json / csv
evalset cancel <id>Request cancellation

Global flags that matter: --json, --workspace, --api-url, --non-interactive.

Submit loop

bash
1# Expand only (no create)
2agentclash evalset submit sweep.yaml --dry-run
3
4# Create after confirmation
5agentclash evalset submit sweep.yaml
6
7# CI / agent-friendly: skip prompt and poll to completion
8agentclash evalset submit sweep.yaml --yes --follow --json

Useful submit flags:

  • --yes — skip the post-expansion confirmation
  • --dry-run — expand only
  • --follow — poll until terminal
  • --poll-interval / --timeout — follow timing

Exit codes: non-zero when the followed set ends failed or cancelled.

Status and logs

bash
1agentclash evalset status <eval-set-id>
2agentclash evalset status <eval-set-id> --watch --json
3
4agentclash evalset logs <eval-set-id> -f
5agentclash evalset logs <eval-set-id> -f --combination '<matrix_key>'

logs fans out across child runs. Session fetch failures and status-probe errors surface as errors (they do not silently look like success).

Report

bash
1agentclash evalset report <eval-set-id>
2agentclash evalset report <eval-set-id> --format csv > scorecard.csv
3agentclash evalset report <eval-set-id> --format json

For full case-level warehouse export (searchable rows, filters), use the web case explorer or the warehouse HTTP APIs documented in Matrix and warehouse.

Cancel

bash
agentclash evalset cancel <eval-set-id>

Cancellation asks Temporal to stop the parent workflow, then transitions the set when delivery succeeds (or the workflow is already gone). Transient Temporal errors do not mark the set cancelled while work may still be running.

Machine-readable automation

bash
1agentclash evalset submit sweep.yaml --yes --follow --json
2agentclash evalset status <id> --json --query '.eval_set.status'

Schema introspection for the whole CLI (including evalset) is available via agentclash schema --json with no auth.

Next

PreviousEval-set manifestsNextMatrix and warehouse