# Runs and Evals

The product language around runs and evals is easy to blur. The current codebase makes one distinction especially important.

Source: https://www.agentclash.dev/docs/concepts/runs-and-evals
Markdown export: https://www.agentclash.dev/md/docs/concepts/runs-and-evals

A **run** is the concrete execution object you create, stream, rank, compare, and inspect in AgentClash today.

In the current user-facing product surface, `run` is the first-class noun:

- `agentclash run create`
- `agentclash run list`
- `agentclash run ranking`
- `agentclash compare gate --baseline <RUN_ID> --candidate <RUN_ID>`

The workflow-first surface (`agentclash eval start`, `agentclash baseline set`,
`agentclash eval scorecard`) wraps these resource commands with name-based
selectors and a bookmarked baseline so day-to-day evaluation does not require
juggling raw run IDs. The resource commands above remain the canonical
ID-centric path for CI and automation.

A run is not just one model token stream. It is the container for a scored evaluation attempt inside a workspace, including the challenge pack version, selected agent deployments, lifecycle timestamps, and ranking output.

The word **eval** is broader. People use it to mean “the experiment I am trying to run” or “the graded set of results I care about.” That is reasonable, but if you are reading the code or the CLI, you should anchor on this:

- **Run** = the concrete resource you create and query.
- **Eval** = the broader exercise or outcome you are trying to measure.

You will also see **eval sessions** in the CLI and API: an eval session groups several repeated runs of the same configuration and aggregates their scores together (created by `agentclash eval start --repetitions`, backed by the `/v1/eval-sessions` endpoint). It is still built on runs underneath.

**Eval sets** (Fleet) sit one layer above sessions: one manifest expands packs × agents × models × repeats into many sessions/runs under a shared set ID and `matrix_key`. Use `agentclash evalset` and the workspace Eval sets UI for that path. See [Fleet overview](https://www.agentclash.dev/md/docs/fleet).

The day-to-day single-comparison workflow still revolves around runs and ranked run results. Keep that noun map in your head and the CLI/API stay easier to follow.

## Practical rule of thumb

Use **run** when you are talking about a real resource ID. Use **eval** when you are talking about the experiment design or the larger testing loop. Use **eval set** when you are sweeping a cartesian product of packs and agents.

## See also

- [Hosted Quickstart](https://www.agentclash.dev/md/docs/getting-started/quickstart)
- [Fleet overview](https://www.agentclash.dev/md/docs/fleet)
- [CLI Reference](https://www.agentclash.dev/md/docs/reference/cli)
- [Datasets overview](https://www.agentclash.dev/md/docs/guides/datasets-overview)
- [Multi-turn packs](https://www.agentclash.dev/md/docs/challenge-packs/multi-turn)
- [Security evaluation](https://www.agentclash.dev/md/docs/guides/security-evaluation)