Synthetic data
Synthetic data generation for AI agents that teaches, not noise
Bulk self-instruct produces volume without signal. AgentClash and DataSmith run a weak-vs-strong judge loop that accepts examples only when a strong solver succeeds and a weak solver struggles: the useful difficulty zone for fine-tuning and eval.
Candidate
Baseline
Control
replay timeline
ci verdict
Correctness improved, latency within budget, and required artifacts were preserved for review.
agentclash run create --follow
Why weak-vs-strong beats prompt-only generation
Built for reviewable agent decisions
Meta FAIR Autodata showed that agentic self-instruct with solver separation beats standard synthetic pipelines across CS, legal, and math settings. DataSmith implements the practical loop; AgentClash runs it hosted for regression-ready datasets.
OpenTelemetry-compatible trace import
Pinned datasets and golden test cases
Baseline versus candidate regression checks
Replay trails for tool calls, outputs, and artifacts
Scorecards for correctness, cost, latency, and evidence
CI gates for prompt, model, RAG, and tool changes
Workflow
From seeds to export or CI gates
Import the evidence
Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.
Pin the baseline
Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.
Replay the evidence
Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.
Gate the release
Compare candidate and baseline runs, then fail CI before a regression reaches users.
Start generating
Bring your first workload into the loop
Use DataSmith locally for training export, or AgentClash workspaces for eval baselines and CI gates on the same examples.
Agent evals
Real-task agent evals with replay evidence and CI gates.
LLM agent evaluation
Evaluate LLM agents on full trajectories, not one-shot answers.
Compare tools
See how AgentClash differs from prompt-eval platforms.
DataSmith platform page
SDK + hosted generation overview.
Introducing DataSmith
Launch blog with pipeline details.
Synthetic generation docs
Run Agentic Self-Instruct in AgentClash.
Datasets overview
Import examples, record baselines, sync regression suites, and gate CI.
Dataset CI gates
Fail builds when a candidate regresses against a pinned baseline.
FAQ
Synthetic data generation FAQ
What is synthetic data generation for AI agents?
It is the process of creating labeled training or eval examples for agent workloads using LLM-driven pipelines instead of manual curation alone.
How does Agentic Self-Instruct improve data quality?
A challenger proposes examples, weak and strong solvers attempt them, and a judge accepts rows only when the score gap indicates the example is learnable but not trivial.
Should I use DataSmith or AgentClash for generation?
Use DataSmith for offline SFT, DPO, and Hugging Face export. Use AgentClash when generated examples should feed dataset evals, baselines, and CI regression gates.