K
Start free

Fleet

Budgets and scanners

Cap eval-set spend, emergency-stop a workspace, and run built-in transcript scanners after completion.

Fleet adds two optional control surfaces on top of orchestration: budgets (stop spending) and scanners (analyze finished transcripts).

Budgets

Set a soft USD cap in the manifest:

yaml
1limits:
2  budget_usd: 50

The expand API also returns a cost estimate from combination count and model hints so you can confirm before submit.

Enforcement behavior

  • The eval-set workflow checks remaining budget before launching pack sessions and before launching inner runs inside a multi-combination pack.
  • When the budget is exhausted, further launches stop and the set settles with a budget-related failure/cancel path rather than silently overspending.
  • Capacity waits (workspace run concurrency) use a long enough activity timeout to cover the internal poll window (on the order of minutes), so a brief full queue does not fail the set as a 5s activity timeout.

Exact settlement status strings are visible in agentclash evalset status <id> --json.

Workspace emergency stop

Operators can cancel every active eval set in a workspace:

bash
1curl -sS -X POST \
2  -H "Authorization: Bearer $AGENTCLASH_TOKEN" \
3  "$AGENTCLASH_API_URL/v1/workspaces/$WORKSPACE_ID/emergency-stop"

The response includes how many sets were cancelled and an audit event name (workspace.emergency_stop).

Scanners

Scanners are post-hoc analyzers over completed case transcripts. They do not change scoring; they emit findings you can triage.

Built-in catalog

Shipped under runtime/scanners/catalog/:

NameIntent
reward-hackingReward-seeking / metric gaming patterns
tool-misuseUnsafe or policy-breaking tool use
instruction-injectionPrompt / instruction injection attempts
sandbox-escape-attemptEscape or host breakout signals

Opt-in from the manifest

yaml
1scanners:
2  - reward-hacking
3  - tool-misuse

When present, a completed set can start a scan workflow for those scanners.

Manual / API scan

bash
1curl -sS -X POST \
2  -H "Authorization: Bearer $AGENTCLASH_TOKEN" \
3  -H "Content-Type: application/json" \
4  -d '{"scanners":["reward-hacking","sandbox-escape-attempt"]}' \
5  "$AGENTCLASH_API_URL/v1/eval-sets/$EVAL_SET_ID/scan"
  • Empty / omitted body may fall back to manifest-defined scanners.
  • Malformed JSON returns 400.
  • A second overlapping scan for the same set returns 409 (scan_already_running).

List and triage:

bash
1curl -sS -H "Authorization: Bearer $AGENTCLASH_TOKEN" \
2  "$AGENTCLASH_API_URL/v1/eval-sets/$EVAL_SET_ID/findings"
3
4curl -sS -X PATCH \
5  -H "Authorization: Bearer $AGENTCLASH_TOKEN" \
6  -H "Content-Type: application/json" \
7  -d '{"status":"dismissed"}' \
8  "$AGENTCLASH_API_URL/v1/scan-findings/$FINDING_ID"

Rescans that find nothing clear prior findings for that target/scanner/version so stale hits do not linger.

Next

PreviousMatrix and warehouseNextSelf-host at scale