Datasets overview

Dataset evals run a pinned version of labeled examples through a challenge pack and deployment, then compare candidate runs against recorded baselines. They complement ad-hoc eval runs when you need reproducible regression signal tied to curated examples.

Lifecycle

1. Create or import a dataset in the workspace (web UI or API).
2. Pin a version with examples linked to challenge cases.
3. Run a dataset eval — AgentClash executes each example and aggregates pass/fail.
4. Record a baseline from a green eval run.
5. Gate CI with agentclash dataset test (see Dataset CI Gates).

1export AGENTCLASH_API_URL="https://api.agentclash.dev"
2export AGENTCLASH_TOKEN="<token>"
3export AGENTCLASH_WORKSPACE="<workspace-id>"
4
5# Start an eval against a pinned version
6agentclash dataset eval <datasetId> \
7  --version <dataset-version-id> \
8  --pack <challenge-pack-version-id> \
9  --challenge support \
10  --deployment <deployment-id>

Record a baseline

A baseline snapshots the per-example outcomes of one completed, green eval run so later candidates can be compared against it. Recording a baseline is an API call (no CLI subcommand yet): POST /v1/workspaces/{workspaceID}/datasets/{datasetID}/baselines with the eval run_id and an optional label. The response returns a baseline id — pass that to agentclash dataset test --baseline <id> to gate candidates against it.

1curl -sS -X POST \
2  "$AGENTCLASH_API_URL/v1/workspaces/$AGENTCLASH_WORKSPACE/datasets/<datasetId>/baselines" \
3  -H "Authorization: Bearer $AGENTCLASH_TOKEN" \
4  -H "Content-Type: application/json" \
5  -d '{"run_id": "<eval-run-id>", "label": "support v1 baseline"}'

Baselines and regression suites

Syncing a pinned dataset version into a regression suite copies each example's provenance — its dataset-example ID and trace metadata — into the workspace regression suite, so manifest-based release gates can reference the same curated examples. (Under the hood these land in the workspace_regression_cases table keyed by dataset_example_id.)

1agentclash dataset sync-regression-suite <datasetId> \
2  --version <dataset-version-id> \
3  --pack <challenge-pack-version-id> \
4  --challenge support \
5  --suite-name "Support dataset regression"

Re-running sync is idempotent: cases that already exist for an example are skipped, so you can safely re-sync after adding new examples.

CI gate output formats

agentclash dataset test supports human text, JSON gate payloads, and JUnit XML for CI parsers. Use --max-regressions 0 to fail on any newly failing example versus the baseline.

See also

• Synthetic dataset generation — Fast Self-Instruct and Agentic Self-Instruct in workspaces
• Dataset CI Gates — GitHub Actions recipe and API surface
• CI/CD workload recipes — pick realistic agent workloads
• Interpret results — read scorecards and timelines