# CI/CD Agent Gates

Use a repo-tracked AgentClash CI manifest to define which agent revision, workload, baseline, and gate a pull request should run.

Source: https://www.agentclash.dev/docs/guides/ci-cd-agent-gates
Markdown export: https://www.agentclash.dev/md/docs/guides/ci-cd-agent-gates

AgentClash CI should gate an agent revision, not only a prompt diff.

Prompt-focused tools can usually watch `prompts/**` and rerun a prompt eval. AgentClash's main product model is richer: an agent change can touch instructions, workflow code, tool bindings, provider models, runtime limits, output schemas, guardrails, or retrieval configuration. The CI contract therefore needs to name the candidate agent build, deployment settings, challenge workload, baseline, and gate policy explicitly.

## The manifest is the contract

Create a repo-tracked manifest:

```bash
agentclash ci init .agentclash/ci.yaml
agentclash ci validate .agentclash/ci.yaml
agentclash ci validate .agentclash/ci.yaml --remote --json
agentclash ci baseline --manifest .agentclash/ci.yaml --json
agentclash ci should-run --changed-file prompts/system.md --json
```

The generated manifest has this shape:

```yaml
version: 1
trigger:
  paths:
    - .agentclash/agent.json
    - prompts/**
    - tools/**
  labels:
    - agentclash/eval
candidate:
  build:
    agent_build_id: 00000000-0000-0000-0000-000000000001
    spec_file: .agentclash/agent.json
  deployment:
    name: pr-candidate
    runtime_profile_id: 00000000-0000-0000-0000-000000000002
    provider_account_id: 00000000-0000-0000-0000-000000000003
    model: gpt-5.5
evaluation:
  challenge_pack_version_id: 00000000-0000-0000-0000-000000000005
  input_set_id: 00000000-0000-0000-0000-000000000006
  # For deterministic voice eval packs, set: mode: text-sim
  regression_suites:
    - 00000000-0000-0000-0000-000000000007
baseline:
  run_id: 00000000-0000-0000-0000-000000000008
  refresh: manual
  max_age_days: 30
gate:
  fail_on: regression
regressions:
  promote_failures: proposed
```

The IDs in the generated file are placeholders. Replace them with workspace resources before using the manifest for a real gate.

Local validation is always offline. Add `--remote` when you want the CLI to call the AgentClash API and verify that the manifest's agent build, runtime profile, provider account, challenge pack version, input set, regression suites or cases, and baseline are visible from the selected workspace. Because this makes real authenticated API calls, set `AGENTCLASH_API_URL`, `AGENTCLASH_TOKEN`, and `AGENTCLASH_WORKSPACE` in CI and expect normal API latency, rate limits, and token scoping rules. JSON output includes a `remote.checks[]` entry per referenced field, so CI can report whether a failure came from the local manifest contract or from an API/resource check.

## What each section means

- `trigger` says which repository paths and optional labels should cause the workflow to run.
- `candidate.build` names the existing AgentClash build and the source-backed build-version spec to test.
- `candidate.deployment` names the runtime resources used for the candidate deployment.
- `evaluation` names the workload: challenge pack version, optional input set, and optional regression suites or cases. For voice packs, set the optional `evaluation.mode: text-sim` to request a deterministic text-simulated voice eval (`text-sim` is the only mode supported today; `audio-sim`, `live-call`, and `replay-import` are reserved for future use).
- `baseline` names the locked reference run or deployment, plus explicit refresh and staleness rules.
- `gate` names the release-gate failure threshold.
- `regressions` controls whether failed cases should only be reported, proposed for promotion, or eventually auto-promoted on main.

The important distinction is:

```text
agent build/deployment = thing under test
challenge pack/regression suite = workload used to test it
release gate = decision policy
```

If you are deciding what the workload should contain, use [CI/CD Workload Recipes](https://www.agentclash.dev/md/docs/guides/ci-cd-workload-recipes) for coding, research, support/ops, and long-horizon agent patterns.

## Baseline strategy and refresh

For pull request gates, prefer `baseline.run_id`. It pins the exact accepted mainline run, so every reviewer can see what changed when the baseline moves. Add `baseline.run_agent_id` only when the locked run has multiple participants and the gate must compare against one specific agent lane.

Use `baseline.deployment_id` only when the team intentionally wants a moving selector. `agentclash ci baseline` resolves it to the newest completed run in the workspace that matches the manifest workload and includes that deployment. The command prints the exact resolved `run_id` and `run_agent_id` so downstream automation still compares against concrete IDs.

Use `baseline.max_age_days` when a stale baseline should block the gate. The resolver checks the chosen run's `finished_at` or `created_at` timestamp and fails instead of silently comparing against old behavior.

Refreshes are explicit:

```yaml
baseline:
  run_id: 00000000-0000-0000-0000-000000000008
  refresh: manual
  max_age_days: 30
```

- `manual`: after a successful mainline run, update `baseline.run_id` in a reviewed change.
- `propose`: automation may propose the new baseline, but a human still reviews the manifest change.
- `auto_on_main`: a protected mainline workflow may update the manifest with an auditable commit after the gate passes.

Resolve the baseline before running a gate:

```bash
agentclash ci baseline \
  --manifest .agentclash/ci.yaml \
  --json
```

The JSON includes `strategy`, `source`, `baseline.run_id`, optional `baseline.run_agent_id`, `refresh.mode`, and `refresh.next_action`.

## Decide whether CI should run

Use `agentclash ci should-run` when you want AgentClash to explain whether a pull request touches the agent contract. A matching path or label produces `should_run: true`; unrelated docs-only changes produce `should_run: false`.

```bash
agentclash ci should-run \
  --manifest .agentclash/ci.yaml \
  --changed-file prompts/system.md
```

Labels can force the gate even when paths do not match:

```bash
agentclash ci should-run \
  --manifest .agentclash/ci.yaml \
  --changed-file docs/readme.md \
  --labels agentclash/eval \
  --json
```

In GitHub Actions, `ci should-run` reads pull request labels from `GITHUB_EVENT_PATH` automatically when `--labels` is omitted, and the bundled action passes that behavior through. Use `--github-event <path>` only when testing a saved event payload locally.

For local or GitHub Actions diffing, pass refs explicitly:

```bash
agentclash ci should-run \
  --manifest .agentclash/ci.yaml \
  --base origin/main \
  --head HEAD \
  --json
```

## GitHub Actions sketch

The manifest is the single source of truth for the candidate revision, workload, baseline, and gate. A pull request workflow can validate it, decide whether it should run for the changed files, then let `agentclash ci run` create the candidate build version, deployment, run, and release-gate evaluation.

Use the reusable AgentClash action when you want the standard GitHub integration without rewriting the shell glue:

```yaml
name: AgentClash gate

on:
  pull_request:
    paths:
      - ".agentclash/**"
      - "prompts/**"
      - "tools/**"

jobs:
  agentclash:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: "22"

      - name: Run AgentClash CI gate
        id: agentclash
        uses: agentclash/agentclash/.github/actions/agentclash-ci@main
        with:
          token: ${{ secrets.AGENTCLASH_TOKEN }}
          workspace: ${{ secrets.AGENTCLASH_WORKSPACE }}
          manifest: .agentclash/ci.yaml

      - name: Upload AgentClash gate artifacts
        if: always() && steps.agentclash.outputs['should-run'] == 'true'
        uses: actions/upload-artifact@v4
        with:
          name: agentclash-ci
          path: |
            ${{ steps.agentclash.outputs.result-file }}
            ${{ steps.agentclash.outputs.artifact-dir }}/*.json
```

The action installs the published `agentclash` npm package by default, runs `ci validate --remote`, runs `ci should-run`, auto-detects pull request labels from the GitHub event payload, skips unrelated changes, runs `ci run` when matched, posts or updates a sticky structured PR comment when pull request context is available, and exposes `should-run`, `skip-reason`, `run-id`, `gate-verdict`, `exit-code`, `result-file`, and `artifact-dir` outputs. It preserves the CLI exit code, so a blocking gate fails the workflow normally. Grant `pull-requests: write` when you want GitHub-hosted PR comments; commenting is best-effort and permission failures do not override the AgentClash result. When run metadata is available, the comment links reviewers directly to the AgentClash candidate run, baseline run, comparison, failures, scorecard, replay, and regression cases. If setup fails before a candidate run is created, the sticky comment reports the errored setup state and points reviewers at the GitHub Actions log.

`agentclash ci run` exits nonzero when the gate verdict should block CI, when the candidate run times out, or when the manifest/API setup is invalid. In GitHub Actions, it automatically attaches repository, pull request, branch, default branch, commit, workflow, event, and workflow-run URL metadata to the AgentClash run. It also appends a reviewer-friendly Markdown section when the `$GITHUB_STEP_SUMMARY` environment variable is set, while the bundled action turns the same run evidence into the PR comment. Pass `--summary-file <path>` for another Markdown destination, or `--github-step-summary=false` to disable the automatic GitHub summary.

`--artifact-dir` writes stable JSON files intended for `actions/upload-artifact`: `result.json` for the final CLI envelope, `run.json` for run creation/completion payloads, `scorecard.json` for candidate scorecard evidence, `comparison.json` for baseline/candidate comparison evidence, and `gate.json` for the release-gate verdict and policy metadata. The summary and artifacts include the challenge pack version, baseline, candidate, policy, verdict, top evidence lines, regression candidate promotion outcomes, and AgentClash links when the API returns them. Use `--ci-repository`, `--ci-pull-request`, `--ci-branch`, `--ci-default-branch`, `--ci-commit`, and the other `--ci-*` flags when running from another CI system or a custom wrapper.

## Regression promotion policy

Do not auto-promote every PR failure by default. A bad run, flaky dependency, or weak evaluator could pollute the regression suite. Use this conservative progression:

```yaml
regressions:
  promote_failures: disabled
```

Report failures only. When the gate fails, the CLI records that promotion was skipped and does not call failure-listing or promotion endpoints.

```yaml
regressions:
  promote_failures: proposed
```

Create reviewable candidates after a failing gate. The CLI lists the candidate run's failure-review items, checks each target suite from `evaluation.regression_suites`, skips any challenge identity that already has a non-archived/non-rejected case, then calls the promote-failure API with `status: proposed`.

Proposed cases appear in the regression suite UI without entering future runs. A reviewer can accept them by changing status to `active`, or reject/archive them if the failure is noisy, duplicated, or not worth keeping.

```yaml
regressions:
  promote_failures: auto_on_main
```

Create active cases only from protected default-branch runs. The CLI refuses pull request events, `refs/pull/*`, missing default branch metadata, and non-default branches. GitHub Actions usually supplies the default branch through the event payload; custom CI wrappers should pass `--ci-default-branch main`.

All modes preserve the original gate exit code. Promotion errors are reported in the human output, JSON `regression_promotions.errors`, GitHub step summary, and artifact `result.json`, but a blocking regression still exits with the gate failure code.

## Current limits

- `agentclash ci validate` validates the manifest shape locally; pass `--remote` for API-backed resource checks.
- `agentclash ci should-run` only decides whether a gate should run; `agentclash ci run` performs the orchestration.
- `agentclash ci run` creates a one-off candidate deployment for the manifest build version; cleanup/retention policy is still a follow-up.
- GitHub Check Runs with rich annotations are still follow-up work; use the sticky PR comment, GitHub step summary, and uploaded JSON artifacts today.
- Regression candidate promotion requires `evaluation.regression_suites`; without at least one target suite, `ci run` reports promotion as blocked.

## See also

- [Agents and Deployments](https://www.agentclash.dev/md/docs/concepts/agents-and-deployments)
- [Challenge Packs and Inputs](https://www.agentclash.dev/md/docs/concepts/challenge-packs-and-inputs)
- [Eval Workflows and Gates](https://www.agentclash.dev/md/docs/challenge-packs/eval-workflows-and-gates)
- [CI/CD Workload Recipes](https://www.agentclash.dev/md/docs/guides/ci-cd-workload-recipes)
- [Datasets overview](https://www.agentclash.dev/md/docs/guides/datasets-overview)
- [Multi-turn packs](https://www.agentclash.dev/md/docs/challenge-packs/multi-turn)
- [Security evaluation](https://www.agentclash.dev/md/docs/guides/security-evaluation)