# CLI evalset workflow

Submit, follow, log, report, and cancel Fleet eval sets with agentclash evalset.

Source: https://www.agentclash.dev/docs/fleet/cli-evalset
Markdown export: https://www.agentclash.dev/md/docs/fleet/cli-evalset

The `agentclash evalset` command group is the primary operator surface for Fleet.

```bash
export AGENTCLASH_API_URL="https://api.agentclash.dev"
# or point at your self-hosted API

agentclash auth login --device
agentclash workspace use <workspace-id>
```

## Commands

| Command | Purpose |
|---|---|
| `evalset init [path]` | Write a commented `evalset/v1` starter manifest |
| `evalset submit <file>` | Expand, optionally confirm, create the set |
| `evalset list` | Recent sets in the workspace |
| `evalset status <id>` | Roll-up status (`--watch` until terminal) |
| `evalset logs <id>` | Multiplexed run-event streams (`-f` to follow) |
| `evalset report <id>` | Scorecard table / json / csv |
| `evalset cancel <id>` | Request cancellation |

Global flags that matter: `--json`, `--workspace`, `--api-url`, `--non-interactive`.

## Submit loop

```bash
# Expand only (no create)
agentclash evalset submit sweep.yaml --dry-run

# Create after confirmation
agentclash evalset submit sweep.yaml

# CI / agent-friendly: skip prompt and poll to completion
agentclash evalset submit sweep.yaml --yes --follow --json
```

Useful submit flags:

- `--yes` — skip the post-expansion confirmation
- `--dry-run` — expand only
- `--follow` — poll until terminal
- `--poll-interval` / `--timeout` — follow timing

Exit codes: non-zero when the followed set ends **failed** or **cancelled**.

## Status and logs

```bash
agentclash evalset status <eval-set-id>
agentclash evalset status <eval-set-id> --watch --json

agentclash evalset logs <eval-set-id> -f
agentclash evalset logs <eval-set-id> -f --combination '<matrix_key>'
```

`logs` fans out across child runs. Session fetch failures and status-probe errors surface as errors (they do not silently look like success).

## Report

```bash
agentclash evalset report <eval-set-id>
agentclash evalset report <eval-set-id> --format csv > scorecard.csv
agentclash evalset report <eval-set-id> --format json
```

For full case-level warehouse export (searchable rows, filters), use the web case explorer or the warehouse HTTP APIs documented in [Matrix and warehouse](https://www.agentclash.dev/md/docs/fleet/matrix-and-warehouse).

## Cancel

```bash
agentclash evalset cancel <eval-set-id>
```

Cancellation asks Temporal to stop the parent workflow, then transitions the set when delivery succeeds (or the workflow is already gone). Transient Temporal errors do not mark the set cancelled while work may still be running.

## Machine-readable automation

```bash
agentclash evalset submit sweep.yaml --yes --follow --json
agentclash evalset status <id> --json --query '.eval_set.status'
```

Schema introspection for the whole CLI (including evalset) is available via `agentclash schema --json` with no auth.

## Next

- [Matrix and warehouse](https://www.agentclash.dev/md/docs/fleet/matrix-and-warehouse)
- [CLI reference](https://www.agentclash.dev/md/docs/reference/cli) (generated command dump)