# Agentic Self-Instruct: weak-vs-strong synthetic data

Agentic Self-Instruct is a four-role loop for synthetic dataset generation: challenger proposes, weak and strong solvers attempt, judge filters. DataSmith ships it as a Python SDK; AgentClash runs it hosted for eval-ready datasets.

Source: https://www.agentclash.dev/agentic-self-instruct
Markdown export: https://www.agentclash.dev/md/agentic-self-instruct

## The four roles

Challenger generates candidates from seeds. Weak solver represents the model you want to improve. Strong solver represents a reference path. Judge scores quality and separation before acceptance.

- OpenTelemetry-compatible trace import
- Pinned datasets and golden test cases
- Baseline versus candidate regression checks
- Replay trails for tool calls, outputs, and artifacts
- Scorecards for correctness, cost, latency, and evidence
- CI gates for prompt, model, RAG, and tool changes

## Research to production

### 1. Import the evidence

Start from OpenTelemetry traces, curated datasets, support transcripts, or a real failure your team already saw.

### 2. Pin the baseline

Record the current accepted behavior so every prompt, model, RAG, or tool change has a fair comparison point.

### 3. Replay the evidence

Inspect tool calls, outputs, artifacts, latency, cost, and judge evidence when a candidate gets worse.

### 4. Gate the release

Compare candidate and baseline runs, then fail CI before a regression reaches users.

## Implement the loop

Read the synthetic generation guide, then try DataSmith locally or Agentic Self-Instruct in a workspace dataset.

- [Agent evals](https://www.agentclash.dev/agent-evals): Real-task agent evals with replay evidence and CI gates.
- [LLM agent evaluation](https://www.agentclash.dev/llm-agent-evaluation): Evaluate LLM agents on full trajectories, not one-shot answers.
- [Compare tools](https://www.agentclash.dev/compare): See how AgentClash differs from prompt-eval platforms.
- [DataSmith GitHub](https://github.com/Atharva-Kanherkar/datasmith): Open-source Python SDK and CLI.
- [Synthetic data generation](https://www.agentclash.dev/synthetic-data-generation-agents): Broader SEO hub for agent dataset generation.
- [Glossary: Agentic Self-Instruct](https://www.agentclash.dev/glossary/agentic-self-instruct): Short definition and when to use it.
- [Datasets overview](https://www.agentclash.dev/docs/guides/datasets-overview): Import examples, record baselines, sync regression suites, and gate CI.

## Agentic Self-Instruct FAQ

### What is Agentic Self-Instruct?

An agentic loop where a challenger generates examples and weak/strong solvers plus a judge determine whether each row is in the useful difficulty zone for training.

### How is it related to Meta FAIR Autodata?

Autodata formalized weak-vs-strong agentic self-instruct for synthetic data. DataSmith implements the practical inner loop; it is not an official Meta release.

### Can I tune acceptance thresholds?

Yes. Both DataSmith and AgentClash expose strong pass rate, weak fail rate, and minimum score gap controls.