# AgentClash Changelog Product updates shipped in ten-day periods since April 15, 2026. Source: https://www.agentclash.dev/changelog ## Aug 07 – Aug 15, 2026 Fleet: eval sets at cloud scale, plus two new ways to author a pack The Fleet epic landed in fifteen pull requests — case-level fan-out, Kubernetes sandboxes, per-account LLM throttling, eval-set manifests, a live matrix dashboard, post-hoc scanners, budgets with emergency stop, OTel metrics, and a Helm chart. Alongside it: one-command local setup for contributors, custom OpenAI-compatible endpoints, pack drafts generated from a tryout or a plain-English description, and a machine-readable public site. Period page: https://www.agentclash.dev/changelog/2026-08-07 - **Added**: Eval-set manifests (`evalset/v1`) — parse, dry-run combination expansion, persistence, and orchestration of child sessions across packs, agents, and models. - **Added**: `agentclash evalset submit`, `status`, `logs`, `report`, and `cancel` for driving eval sets from the terminal or CI. - **Added**: Live eval-set matrix dashboard plus a case-results warehouse with search, report, and export. - **Added**: Per-set budgets with emergency stop, and a post-hoc scanner framework that flags patterns across a finished eval set. - **Added**: Kubernetes sandbox provider with a conformance suite, sandbox capacity budgets and warm pool, bounded workflow fan-out, queue partitioning, and case-level fan-out for native run-agents. - **Added**: Helm chart with KEDA-based Temporal worker autoscaling and published GHCR images for self-hosted deployments. - **Added**: Custom OpenAI-compatible provider accounts — persisted per-account base URLs with SSRF protection at both save time and dial time. - **Added**: Two new ways to start a challenge pack: promote a completed agent tryout into an editable draft, or describe your app in plain English and get a generated pack blueprint in the builder. - **Added**: Machine-readable public site — stable Markdown URLs for every indexable route, `llms.txt`, `Accept: text/markdown` negotiation, and published OpenAPI, JSON Schema, and CLI schema contracts. - **Added**: One-command contributor setup — `make setup`, `make start`, `make doctor`, and `make check`, with Temporal in Docker and a dev profile that runs with zero API keys. - **Added**: `/healthz/ready` readiness probe that reports whether an instance can actually serve traffic, separate from the liveness check. - **Improved**: OpenTelemetry metrics, stall detection, and a ready-to-import alert pack for run and eval-set health. - **Improved**: Oversized run-event payloads now spill to object storage, and outbound LLM calls are throttled per account across workers. - **Fixed**: Onboarding funnel fixes — dead demo CTA on every docs page, CLI errors that never pointed at signup, and jargon that hid what the product does. - **Fixed**: Fleet matrix cells now read as human labels, and backend images build from the repo root so the shared runtime module resolves. - **Fixed**: Benchmarks hub reordered to lead with measured proof and drop empty scoreboard columns. ## Jul 02 – Jul 12, 2026 A shared runtime module, local Docker execution, and a big cleanup Agent execution moved out of the backend into a standalone `runtime` module that the API server, worker, and CLI all import — the foundation for running evals locally. Local runs gained a Docker sandbox provider and bring-your-own-key credential resolution. A 34k-line removal pass retired dead and demoted surfaces, and CLI 2.0.0 shipped. Period page: https://www.agentclash.dev/changelog/2026-07-02 - **Added**: Standalone `runtime` Go module — shared runner contracts, execution harness, and frozen execution-context model imported by backend, worker, and CLI alike. - **Added**: Local execution store — SQLite execution/result persistence and filesystem artifact storage, plus a CLI wrapper proving the CLI can drive the runtime without backend internals. - **Added**: Docker sandbox provider for local runs, so an eval can execute on your own machine instead of a hosted sandbox. - **Added**: Bring-your-own-key credential resolution for local runs — provider keys read from environment, config file, or OS keychain. - **Improved**: CLI 2.0.0 released: freemium pricing model, challenge-pack catalog commands, `evaltest` local eval runs, and the eval SDK's pytest and promote-failures adapters. - **Improved**: Removed eight clusters of dead or demoted product surfaces across backend, runtime, CLI, and web — 34,712 lines deleted, no database tables dropped. - **Improved**: Workspace navigation dropped the retired Playgrounds and Knowledge Sources sections. ## Jun 22 – Jul 01, 2026 DataSmith launch: weak-vs-strong synthetic data for agents DataSmith ships as an open-source Python SDK for high-signal synthetic dataset generation with web-grounded seed construction, OTLP trace ingestion, and SFT/DPO export. AgentClash adds platform marketing, docs, and SEO surfaces for the hosted Agentic Self-Instruct loop. Period page: https://www.agentclash.dev/changelog/2026-06-22 - **Added**: DataSmith — open-source Python SDK for weak-vs-strong Agentic Self-Instruct synthetic data generation, inspired by Meta FAIR Autodata. - **Added**: /platform/datasmith landing page for the SDK + hosted generation story with FAQ and JSON-LD product schema. - **Added**: Synthetic dataset generation docs guide covering Fast Self-Instruct, Agentic Self-Instruct, CLI, and DataSmith export. - **Added**: Launch blog: Introducing DataSmith with pipeline overview, trace ingestion, and AgentClash integration table. - **Added**: SEO landing pages for synthetic data generation, Agentic Self-Instruct, trace-to-dataset workflows, and glossary definition. - **Improved**: Datasets overview and sitemap updated with DataSmith and synthetic generation discovery links. ## Jun 02 – Jun 11, 2026 Portable agent skills and one-command host install Twenty-five portable Agent Skills shipped with docs, an embedded CLI snapshot, and a standalone agent-skills bundle repo. Install or doctor skills on Claude, Codex, Cursor, OpenClaw, Hermes, and OpenCode with one command, or export an offline bundle as a directory or tar.gz. Period page: https://www.agentclash.dev/changelog/2026-06-02 - **Added**: 25 portable Agent Skills — hub, quickstart, compare-and-triage, eval runner, challenge-pack authoring, harness setup, multi-turn operator, dataset workflows, CI release gate, security evaluation, and workspace admin. - **Added**: `agentclash integration install|doctor` for Claude, Codex, Cursor, OpenClaw, Hermes, and OpenCode — copies the embedded snapshot into each host's skills directory. - **Added**: `agentclash skills export` — offline install bundles as a flat directory or tar.gz, with optional host-specific layout. - **Added**: Standalone agent-skills distribution repo with manifest verification and sync from canonical docs source. - **Added**: `agentclash schema` — machine-readable CLI command tree for agents and tooling. - **Improved**: Shell completion now validates its `--shell` argument instead of silently accepting invalid values. - **Improved**: IndexNow pings and sitemap image entries for faster search-engine discovery. - **Added**: First public coding-agent benchmark report with frozen Expression Evaluator Arena pack, monthly blog summary, and reproducible scorecard export workflow. - **Added**: /benchmarks hub summarizes the measured GPT generations comparison with replay-backed scoreboard and methodology appendix. ## May 25 – Jun 01, 2026 Datasets, /try demos, multi-turn transcripts, and Hermes harness The datasets platform shipped end-to-end — import, generation, trace ingest, eval gates, and full workspace UI. /try CLI demos, multi-turn transcript exports, Hermes harness support, and PostHog analytics rounded out the release window. Period page: https://www.agentclash.dev/changelog/2026-05-25 - **Added**: Datasets foundation — interop import/export, version snapshots, eval runs, baselines, CI gate verdicts, and JUnit output. - **Added**: Self-Instruct synthetic dataset generation with async jobs and CLI parity. - **Added**: Production trace ingest, trace candidates, and promote-to-example workflow. - **Added**: Full dataset management UI in the workspace — import mapping, eval wait, dataset test flow, and generation history. - **Added**: /try interactive terminal demos with E2B sandbox, free-trial gateway, and Kimi/Qwen/OpenCode integrations. - **Added**: OpenAI Responses API as a first-class execution mode. - **Added**: Multi-turn conversation transcript API with replay rendering, scorecard view, and PDF export. - **Added**: Vibe eval draft persistence for in-progress eval configuration. - **Added**: Hermes agent harness support in E2B templates and harness execution workflow. - **Added**: PostHog usage analytics across backend, CLI, and web. - **Improved**: SEO and AEO discoverability expanded across public marketing and docs surfaces. - **Improved**: Landing page bar promoting /try CLI demos. - **Fixed**: Dataset generation provider validation returns clear 400 errors instead of opaque failures. ## May 15 – May 24, 2026 Security packs, multi-turn eval, and replay polish Security-family challenge packs, stress-run CLI, and vault boundary harnesses established AgentClash as a security eval surface. Multi-turn evaluation with human takeover and case templating extended the engine beyond single-shot runs. Period page: https://www.agentclash.dev/changelog/2026-05-15 - **Security**: SecurityPolicy schema and SecurityScore dimension for security-family challenge packs. - **Security**: Canonical secret-hygiene and prompt-injection challenge packs shipped. - **Security**: CLI `stress-run` subcommand with Anthropic Messages provider and --no-system-guard leak surfacing. - **Security**: agent-vault-stress harness with real Vault SDK, function calling, campaign mode, and bundled HTTP mock. - **Security**: Infisical and HashiCorp Vault boundary packs for vault-framed canary leak testing. - **Added**: Multi-turn evaluation — user simulators, hybrid executor, human takeover, calibration reviews, and post-run arena. - **Added**: Case template renderer and bundle validation for code-execution challenge packs. - **Added**: Multi-turn conversation events with transcript helpers in the event model. - **Improved**: Replay trace output upgraded with IDE-level syntax highlighting and rich text rendering. - **Improved**: Planted secrets from security packs wired into sandbox provisioning. ## May 05 – May 14, 2026 CI intelligence, prompt eval, and GitHub-native workflows Failure clusters, regression provenance, and CI setup generators connected the eval loop to GitHub. Prompt eval CLI commands, the PR comment bot, and E2B harness runners closed the gap between local runs and production gates. Period page: https://www.agentclash.dev/changelog/2026-05-05 - **Added**: Failure cluster rollups, identity keys, taxonomy classification, and trend charts in the web UI. - **Added**: Regression provenance, validation signals, proposed-case queue, and remediation hints. - **Added**: CI setup generators with workspace page and one-click setup pull request creation. - **Added**: Prompt eval CLI commands — config validation, remote preflight, compile, follow, and GitHub Action mode. - **Added**: AgentClash PR comment bot with links back to failure review in the UI. - **Added**: E2B harness runners for Claude Code and OpenClaw agent execution. - **Added**: Production failure capture and explicit validation runs for proposed regression cases. - **Improved**: CLI preserves CI curation metadata and surfaces failure taxonomy in workflow output. - **Improved**: Evaluator validity signals exposed in scorecards for clearer trust boundaries. - **Fixed**: Dozens of CI, regression, and failure-review edge cases hardened across API and web. ## Apr 25 – May 04, 2026 Peer standings, CLI distribution, and a redesigned public site Live peer standings injected into running agents, the CLI shipped through npm with production defaults, and the marketing site got a full redesign — /why, pricing, docs foundation, and broadcast-style run views. Period page: https://www.agentclash.dev/changelog/2026-04-25 - **Added**: Live peer standings — injection at step boundaries, newswire formatting, and token split between agents vs comparison context. - **Added**: CLI `--peer-standings` flags and UI toggle for live comparison standings during runs. - **Added**: Workflow-first eval commands in the CLI. - **Added**: Dedicated /why manifesto page and pricing section with tier cards and trial CTA. - **Added**: Released CLI binaries now default to https://api.agentclash.dev. - **Improved**: AgentClash-branded auth with liquid-glass login, starfield hero, and 3D clash mark on landing. - **Improved**: Workspace navigation made instant with client-side prefetching. - **Improved**: Run detail and replay pages restyled with instrument-panel aesthetics. - **Improved**: Playground comparison workspace refactored for clearer side-by-side evals. ## Apr 15 – Apr 24, 2026 Scoring depth, failure review, and the first regression suite UI The evaluation engine gained deterministic validators, LLM judge dimensions, and behavioral scoring. Failure review and regression suites landed in the workspace UI, alongside CLI distribution hardening and the xAI provider adapter. Period page: https://www.agentclash.dev/changelog/2026-04-15 - **Added**: Sandbox code-execution validator and math-equivalence scoring for deterministic checks. - **Added**: LLM judge dimensions with n-wise comparisons, persisted results, and scorecard rationale cards in the UI. - **Added**: Behavioral spec scoring — signal extraction, scorecards, and dimension contribution metadata. - **Added**: Text-generation metrics and token F1 validator with evidence linked back to run events. - **Added**: Failure review API and workspace Failures page with filters, detail drawer, and run links. - **Added**: Regression suite CRUD, failure promotion flow, and full regression UI (suites, cases, promotion dialog). - **Added**: xAI provider adapter wired into the execution engine. - **Added**: Run replay step detail now surfaces model output and tool results. - **Added**: Workspace invite emails via Resend. - **Improved**: Run-agent scorecard redesigned with an inspector-style layout. - **Improved**: Cross-platform CLI install scripts hardened for macOS, Linux, and Windows. - **Improved**: Authenticated users redirect from the landing page straight to the dashboard.