# Open source AI agent evaluation for real tasks

AgentClash is MIT-licensed and self-hostable. Import traces or curated datasets, keep replay evidence, and turn failures into reusable CI gates without a black-box vendor loop.

Source: https://www.agentclash.dev/open-source-ai-agent-evaluation
Markdown export: https://www.agentclash.dev/md/open-source-ai-agent-evaluation

## What open-source regression testing should include

Open source matters when your agent test stack needs to run inside your repo, your CI, and your sandbox policy, not just inside someone else's hosted UI.

- OpenTelemetry-compatible trace import
- Pinned datasets and golden test cases
- Baseline versus candidate regression checks
- Replay trails for tool calls, outputs, and artifacts
- Scorecards for correctness, cost, latency, and evidence
- CI gates for prompt, model, RAG, and tool changes

## From first eval to team-wide gate

### 1. Import the evidence

Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.

### 2. Pin the baseline

Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.

### 3. Replay the evidence

Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.

### 4. Gate the release

Compare candidate and baseline runs, then fail CI before a regression reaches users.

## Start with the repo

Clone AgentClash, boot the local stack, and wire the same challenge packs into hosted runs or pull request gates.

- [Agent evals](https://www.agentclash.dev/agent-evals): Real-task agent evals with replay evidence and CI gates.
- [LLM agent evaluation](https://www.agentclash.dev/llm-agent-evaluation): Evaluate LLM agents on full trajectories, not one-shot answers.
- [Compare tools](https://www.agentclash.dev/compare): See how AgentClash differs from prompt-eval platforms.
- [Self-host guide](https://www.agentclash.dev/docs/getting-started/self-host): Boot Postgres, Temporal, API, and worker locally or in your cluster.
- [Quickstart](https://www.agentclash.dev/docs/getting-started/quickstart): Validate the CLI and get to your first runnable command.
- [Write a challenge pack](https://www.agentclash.dev/docs/guides/write-a-challenge-pack): Turn a real task into a repeatable agent evaluation.
- [CI/CD agent gates](https://www.agentclash.dev/docs/guides/ci-cd-agent-gates): Fail a pull request when an agent regresses.

## Questions OSS teams ask before adopting

### Is AgentClash actually open source?

Yes. AgentClash is MIT-licensed. You can self-host the API server, worker, and web app, or use the hosted product when that is faster for your team.

### Can we run evals entirely on our own infrastructure?

Yes. AgentClash supports local stacks, self-hosted deployments, and sandbox providers you control. Challenge packs and scorecards work the same whether the run is local or hosted.

### How does open source help with agent regression testing?

You can inspect scoring rules, pack definitions, replay artifacts, and CI gate manifests in git. That makes agent eval auditable instead of a dashboard-only black box.