Open source
Open source AI agent evaluation for real tasks
AgentClash is MIT-licensed and self-hostable. Import traces or curated datasets, keep replay evidence, and turn failures into reusable CI gates without a black-box vendor loop.
Candidate
Baseline
Control
replay timeline
ci verdict
Correctness improved, latency within budget, and required artifacts were preserved for review.
agentclash run create --follow
What open-source regression testing should include
Built for reviewable agent decisions
Open source matters when your agent test stack needs to run inside your repo, your CI, and your sandbox policy, not just inside someone else's hosted UI.
OpenTelemetry-compatible trace import
Pinned datasets and golden test cases
Baseline versus candidate regression checks
Replay trails for tool calls, outputs, and artifacts
Scorecards for correctness, cost, latency, and evidence
CI gates for prompt, model, RAG, and tool changes
Workflow
From first eval to team-wide gate
Import the evidence
Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.
Pin the baseline
Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.
Replay the evidence
Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.
Gate the release
Compare candidate and baseline runs, then fail CI before a regression reaches users.
Start with the repo
Bring your first workload into the loop
Clone AgentClash, boot the local stack, and wire the same challenge packs into hosted runs or pull request gates.
Agent evals
Real-task agent evals with replay evidence and CI gates.
LLM agent evaluation
Evaluate LLM agents on full trajectories, not one-shot answers.
Compare tools
See how AgentClash differs from prompt-eval platforms.
Self-host guide
Boot Postgres, Temporal, API, and worker locally or in your cluster.
Quickstart
Validate the CLI and get to your first runnable command.
Write a challenge pack
Turn a real task into a repeatable agent evaluation.
CI/CD agent gates
Fail a pull request when an agent regresses.
FAQ
Questions OSS teams ask before adopting
Is AgentClash actually open source?
Yes. AgentClash is MIT-licensed. You can self-host the API server, worker, and web app, or use the hosted product when that is faster for your team.
Can we run evals entirely on our own infrastructure?
Yes. AgentClash supports local stacks, self-hosted deployments, and sandbox providers you control. Challenge packs and scorecards work the same whether the run is local or hosted.
How does open source help with agent regression testing?
You can inspect scoring rules, pack definitions, replay artifacts, and CI gate manifests in git. That makes agent eval auditable instead of a dashboard-only black box.