Concepts

Runs and Evals

The product language around runs and evals is easy to blur. The current codebase makes one distinction especially important.

A run is the concrete execution object you create, stream, rank, compare, and inspect in AgentClash today.

In the current user-facing product surface, run is the first-class noun:

agentclash run create
agentclash run list
agentclash run ranking
agentclash compare gate --baseline <RUN_ID> --candidate <RUN_ID>

The workflow-first surface (agentclash eval start, agentclash baseline set, agentclash eval scorecard) wraps these resource commands with name-based selectors and a bookmarked baseline so day-to-day evaluation does not require juggling raw run IDs. The resource commands above remain the canonical ID-centric path for CI and automation.

A run is not just one model token stream. It is the container for a scored evaluation attempt inside a workspace, including the challenge pack version, selected agent deployments, lifecycle timestamps, and ranking output.

The word eval is broader. People use it to mean “the experiment I am trying to run” or “the graded set of results I care about.” That is reasonable, but if you are reading the code or the CLI, you should anchor on this:

Run = the concrete resource you create and query.
Eval = the broader exercise or outcome you are trying to measure.

You will also see eval sessions in the CLI and API: an eval session groups several repeated runs of the same configuration and aggregates their scores together (created by agentclash eval start --repetitions, backed by the /v1/eval-sessions endpoint). It is still built on runs underneath.

Eval sets (Fleet) sit one layer above sessions: one manifest expands packs × agents × models × repeats into many sessions/runs under a shared set ID and matrix_key. Use agentclash evalset and the workspace Eval sets UI for that path. See Fleet overview.

The day-to-day single-comparison workflow still revolves around runs and ranked run results. Keep that noun map in your head and the CLI/API stay easier to follow.

Practical rule of thumb

Use run when you are talking about a real resource ID. Use eval when you are talking about the experiment design or the larger testing loop. Use eval set when you are sweeping a cartesian product of packs and agents.