# What is agent evaluation?

Agent evaluation measures whether an AI agent completes a real task correctly under constraint. Unlike prompt tests, it scores the whole trajectory: tools, artifacts, cost, latency, and evidence quality.

Source: https://www.agentclash.dev/glossary/agent-evaluation
Markdown export: https://www.agentclash.dev/md/glossary/agent-evaluation

## How agent evaluation differs

Prompt eval checks text from one call. Agent evaluation reruns multi-step work in a sandbox and preserves replay when something fails.

- OpenTelemetry-compatible trace import
- Pinned datasets and golden test cases
- Baseline versus candidate regression checks
- Replay trails for tool calls, outputs, and artifacts
- Scorecards for correctness, cost, latency, and evidence
- CI gates for prompt, model, RAG, and tool changes

## Typical eval workflow

### 1. Import the evidence

Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.

### 2. Pin the baseline

Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.

### 3. Replay the evidence

Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.

### 4. Gate the release

Compare candidate and baseline runs, then fail CI before a regression reaches users.

## Go deeper

Read the platform overview, then author a challenge pack for your first repeatable eval.

- [Agent evals](https://www.agentclash.dev/agent-evals): Real-task agent evals with replay evidence and CI gates.
- [LLM agent evaluation](https://www.agentclash.dev/llm-agent-evaluation): Evaluate LLM agents on full trajectories, not one-shot answers.
- [Compare tools](https://www.agentclash.dev/compare): See how AgentClash differs from prompt-eval platforms.
- [Agent evaluation platform](https://www.agentclash.dev/platform/agent-evaluation): Product overview for real-task eval.
- [Glossary index](https://www.agentclash.dev/glossary): More AgentClash terms.
- [Datasets overview](https://www.agentclash.dev/docs/guides/datasets-overview): Import examples, record baselines, sync regression suites, and gate CI.
- [Dataset CI gates](https://www.agentclash.dev/docs/guides/dataset-ci-gates): Fail builds when a candidate regresses against a pinned baseline.

## Agent evaluation FAQ

### Is agent evaluation the same as LLM benchmarking?

Benchmarks compare models on fixed tasks. Agent evaluation also covers your prompts, tools, harness, and release gates on workloads you own.

### What outputs does an agent evaluation produce?

A scorecard, replay of the trajectory, artifacts from the run, and a pass or fail against validators and gates you define.

### Where should teams start?

Promote one escaped failure into a challenge pack, establish a baseline run, then compare the next candidate in CI or a benchmark eval.