Feature
Challenge packs for repeatable agent evaluation
Challenge packs are AgentClash's unit of agent evaluation: a real task, tool policy, scoring rules, and artifacts encoded once so every model or harness change reruns the same workload.
Candidate
Baseline
Control
replay timeline
ci verdict
Correctness improved, latency within budget, and required artifacts were preserved for review.
agentclash run create --follow
What challenge packs encode
Built for reviewable agent decisions
Inputs, sandbox resources, allowed tools, validators, judges, and pass conditions — everything needed for a fair, repeatable agent eval.
Sandboxed real-tool execution
Head-to-head runs with fair constraints
Scorecards for correctness, cost, latency, and tool strategy
Replay trails for every important action
Challenge packs that turn failures into reusable tests
CI gates for baseline versus candidate decisions
Workflow
Pack lifecycle
Package the task
Describe the workload as a challenge pack with inputs, tools, scoring rules, and artifacts.
Race the agents
Run every candidate against the same task with the same constraints.
Replay the evidence
Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence after the run.
Gate the release
Compare candidate and baseline runs, then fail CI before a regression reaches users.
Author packs with docs
Bring your first workload into the loop
Read the challenge pack docs and authoring guide, then promote escaped failures into packs your whole team can run.
FAQ
Challenge packs FAQ
What is a challenge pack?
A challenge pack is a versioned agent evaluation workload with inputs, tool policy, scoring rules, and expected artifacts.
Can challenge packs run locally and in CI?
Yes. The same pack can power exploratory races, hosted runs, and pull request gates.
How do teams create challenge packs?
Start from a real failure or release risk, encode it as YAML, and iterate with replay evidence until the scoring rules match what reviewers expect.