# What is a challenge pack?

Challenge packs are AgentClash's unit of repeatable agent evaluation. Encode the task once so every model, prompt, or harness change reruns the same workload with the same constraints.

Source: https://www.agentclash.dev/glossary/challenge-pack
Markdown export: https://www.agentclash.dev/md/glossary/challenge-pack

## What packs contain

Inputs, tool policy, sandbox resources, validators, judges, artifacts, and pass conditions that together define a fair eval or regression test.

- OpenTelemetry-compatible trace import
- Pinned datasets and golden test cases
- Baseline versus candidate regression checks
- Replay trails for tool calls, outputs, and artifacts
- Scorecards for correctness, cost, latency, and evidence
- CI gates for prompt, model, RAG, and tool changes

## From pack to gate

### 1. Import the evidence

Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.

### 2. Pin the baseline

Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.

### 3. Replay the evidence

Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.

### 4. Gate the release

Compare candidate and baseline runs, then fail CI before a regression reaches users.

## Authoring resources

Use the challenge pack docs and authoring guide to publish your first pack.

- [Agent evals](https://www.agentclash.dev/agent-evals): Real-task agent evals with replay evidence and CI gates.
- [LLM agent evaluation](https://www.agentclash.dev/llm-agent-evaluation): Evaluate LLM agents on full trajectories, not one-shot answers.
- [Compare tools](https://www.agentclash.dev/compare): See how AgentClash differs from prompt-eval platforms.
- [Challenge packs feature](https://www.agentclash.dev/features/challenge-packs): Feature overview for pack-based eval.
- [Challenge pack docs](https://www.agentclash.dev/docs/challenge-packs): Reference hub for pack authors.
- [Benchmarks hub](https://www.agentclash.dev/benchmarks): Public eval runs on frozen packs.
- [Dataset CI gates](https://www.agentclash.dev/docs/guides/dataset-ci-gates): Fail builds when a candidate regresses against a pinned baseline.
- [CI/CD agent gates](https://www.agentclash.dev/docs/guides/ci-cd-agent-gates): Block pull requests when agent behavior gets worse.

## Challenge pack FAQ

### How is a challenge pack versioned?

Packs are versioned YAML bundles in your workspace. Pin a version for benchmarks and CI so comparisons stay reproducible.

### Can one pack power both benchmarks and CI?

Yes. The same frozen pack can back a public benchmark eval and an internal release gate once your team trusts the scoring rules.

### Where are examples?

See example packs in the repository and the challenge pack reference docs for field-by-field authoring.