Feature
Synthetic dataset generation inside AgentClash
Generate eval-ready examples from pinned seeds without leaving your workspace. Choose fast prompt-only expansion or Agentic Self-Instruct with weak-vs-strong judge filtering.
Candidate
Baseline
Control
replay timeline
ci verdict
Correctness improved, latency within budget, and required artifacts were preserved for review.
agentclash run create --follow
Generation strategies
Built for reviewable agent decisions
Fast Self-Instruct adds volume quickly. Agentic Self-Instruct runs weak and strong solver rollouts with acceptance policies tuned to the useful difficulty zone.
OpenTelemetry-compatible trace import
Pinned datasets and golden test cases
Baseline versus candidate regression checks
Replay trails for tool calls, outputs, and artifacts
Scorecards for correctness, cost, latency, and evidence
CI gates for prompt, model, RAG, and tool changes
Workflow
From generation to gates
Import the evidence
Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.
Pin the baseline
Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.
Replay the evidence
Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.
Gate the release
Compare candidate and baseline runs, then fail CI before a regression reaches users.
Run your first job
Bring your first workload into the loop
Open a dataset, start synthetic generation, then baseline and gate the accepted rows.
Agent evals
Real-task agent evals with replay evidence and CI gates.
LLM agent evaluation
Evaluate LLM agents on full trajectories, not one-shot answers.
Compare tools
See how AgentClash differs from prompt-eval platforms.
Synthetic generation guide
Docs for UI and CLI generation jobs.
DataSmith platform page
Offline SDK for training export.
Datasets overview
Import examples, record baselines, sync regression suites, and gate CI.
Dataset CI gates
Fail builds when a candidate regresses against a pinned baseline.
CI/CD agent gates
Block pull requests when agent behavior gets worse.
FAQ
Synthetic generation FAQ
Where do I start generation in AgentClash?
Open Workspaces, Datasets, your dataset, then Synthetic generation in the UI or use agentclash dataset generate from the CLI.
What happens to rejected examples?
Rejected rows are stored with reason codes and solver attempts so you can review why the judge declined them.
Can I export for fine-tuning from AgentClash?
AgentClash optimizes for eval formats. For SFT, DPO, and Hugging Face export, use the DataSmith Python SDK on the same seeds.