Automatically generating Playwright E2E specs with Codex CLI and feeding failure logs back to let it self-correct

E2E testing is a lot of work. From the Selenium era to Playwright, no matter what you use, the moment a single selector changes your tests cascade into failures and the entire afternoon disappears fixing them. I was skeptical at first — "AI writes tests for me? I'll just have to rewrite them anyway" — but after running Codex CLI with Playwright MCP for a few days, my opinion shifted a bit.

The key insight is it doesn't just generate code. Codex CLI writes code inside a sandbox, runs npx playwright test directly, feeds the failure log back as input for the next decision, and repeats the cycle of fixing and re-running. Define the conditions and rules for this loop in a single AGENTS.md file, and the agent keeps running until everything passes. In the meantime, you can do something else.

This post covers everything from setting up the Playwright MCP server to structuring the AGENTS.md-based loop, along with the failure points you'll actually hit when running it for real.

How Codex CLI Handles E2E Testing

The Structure of the Agent Loop

Codex CLI is a coding agent that runs in the terminal. Which model it uses varies by release according to the OpenAI Codex documentation, so rather than naming a specific model, let's focus on its nature: an agent that can read and write files and execute shell commands inside an isolated sandbox. The important point is that it's not a simple code completion tool. It reads files, writes code, executes terminal commands, and feeds the results back as its own input.

In an E2E scenario, the loop flows roughly like this:

In this loop, the Playwright MCP server acts as the bridge. The @playwright/mcp package runs as an MCP (Model Context Protocol) server, letting the agent control a headless browser through accessibility tree snapshots. One thing worth noting here: the accessibility tree gives the agent the semantic structure of the DOM, not the rendered visual pixel state. So saying "the agent sees the UI with its eyes" is an overstatement. Visual regressions like color, alignment, and overflow won't be caught by this loop alone — you'd need to add screenshot comparison or a separate visual regression tool.

AGENTS.md — The File That Controls the Loop

AGENTS.md is a project-level guidance file format originally proposed by the OpenAI Codex team. It was later publicly documented at agents.md, and a convention emerged where various agent tools reference the same filename. There's no confirmed neutral foundation managing this format, so let's treat it as simply "a shared format that has become a de facto convention."

Codex CLI reads this file when starting a task and loads rules like retry conditions, test commands, and check gates into context. This is where the shape of the loop gets defined.

Setup: From Connecting the MCP Server to Generating the First Spec

Step 1: Register the MCP Server in the Codex CLI Config

Codex CLI's config file is either ~/.codex/config.toml in the home directory (global) or .codex/config.toml in the project root. For the exact path and schema, it's safest to check codex --help and the official documentation for the version you have installed. Below is a conceptual example.

# ~/.codex/config.toml (conceptual example)
[mcp_servers.playwright]
command = "npx"
args = ["@playwright/mcp@0.0.x"]  # pin to the specific version you intend to use

Leaving it as @latest means every time the server releases a new version, the tool signatures change and your prompts start failing out of nowhere. In a team repo, pin to a specific version and upgrade intentionally.

Step 2: Write AGENTS.md

This is the core. I made the mistake of writing "always run all tests" at first and ended up drowning in unrelated failures. Daniel Vaughan's post Why 'Always Run Tests' in AGENTS.md Makes Things Worse makes the same point.

# AGENTS.md
 
## Testing
- After any change, always run `npx playwright test --reporter=line`
- Run the full suite only when changing shared modules; otherwise run only the modified specs
- On failure: analyze the failure log, fix the selector or assertion, then re-run
- If the test runner returns a non-zero exit code, treat it as failure and continue to the next turn
- If the same spec fails 3 times in a row, stop auto-fixing and report to a human
 
## Code Style
- Test files live in the `tests/e2e/` directory
- Use the Page Object Model pattern; refer to existing files in the `pages/` directory
- Prefer accessibility-based selectors: `getByRole`, `getByLabel`, etc.
 
## Assertion Quality Rules
- No trivial assertions like expect(true).toBe(true)
- Each test must verify at least one of: text actually visible in the UI, URL, or state
 
## Checks (all must pass before done)
- npx playwright test
- npx tsc --noEmit
- npx eslint src/

Pay close attention to the exit code rule. Playwright's test runner conventionally exits with a non-zero code on test failure, and in practice treating pass=0, failure=non-0 is the safest approach. Conditioning on a specific number like "retry only on exit code 2" can cause normal test failures to get mixed up with argument parsing errors or interrupts and slip through.

Step 3: Run Codex CLI

codex "Write a Playwright E2E spec in tests/e2e/auth.spec.ts for the flow
       from login through dashboard navigation to logout,
       and keep fixing it until all Checks in AGENTS.md pass."

From this point the agent operates autonomously. It queries the UI structure via the Playwright MCP accessibility tree, writes the spec, runs the tests, and on failure reads the log to fix selectors or assertions.

Points Easy to Miss in the Retry Loop

Automatic Retry on Exit Code and the Escape Condition

The agent's retry logic itself is simple. If the test command exits with a non-zero code, the failure output is added to the next turn's context and it tries again. Because it only checks the exit code to determine pass or fail, using a reporter like --reporter=line that leaves a clear failure summary in stderr is helpful for debugging.

The problem is the escape condition. Without any safeguard, the following failure patterns emerge:

• Repeatedly failing on the same selector, repeating the same fix
• Quietly weakening assertions until they pass
• Burning through all session tokens/context before asking for human intervention

That's why it's much better in practice to add explicit guards to AGENTS.md: "stop after N consecutive failures" and "do not delete assertions or replace them with unconditionally truthy checks." Here's a flow with explicit human intervention points:

'Passing' and 'Meaningful' Are Different Problems

There's a risk to address right here. The agent is optimized to make the exit code become 0, not to judge whether a test is meaningful. When stuck in repeated failures, it may quietly soften assertions or escape into something effectively meaningless like expect(page).toBeTruthy(). This deserves emphasis now, not at the end. A loop that passes is no guarantee the test will catch regressions.

There are two countermeasures. One is to include assertion quality rules in AGENTS.md as shown above. The other is for a human to scan the expect lines in each spec during the review stage. Both are necessary.

Strengthening Gates with Hooks

Codex CLI's hooks (before/after tool calls, at session end) let you enforce npx tsc --noEmit or lint pass as hard conditions. Hook names and schemas have changed between versions, so check the docs for the CLI you're using before wiring them up. The point is: going beyond "tests pass" as your sole signal — including types, lint, and static analysis in the gate — makes it harder for the agent to manufacture a passing run by cutting corners.

Layering in a Monorepo

In a single repo, one root AGENTS.md is enough, but in a monorepo each service has different test commands. Codex CLI and many other agent tools follow the convention that placing an AGENTS.md in a subdirectory causes it to be referenced alongside the root file when working in that path. There's no officially confirmed behavior for special filenames like AGENTS.override.md merging automatically, so let's stick to verified patterns only.

/
├── AGENTS.md                       # shared rules
├── packages/
│   ├── payments/
│   │   └── AGENTS.md               # payments-specific test commands
│   └── notifications/
│       └── AGENTS.md               # notifications-specific rules

# packages/payments/AGENTS.md
 
## Testing
- When this directory changes: run `make test-payments`
- No need for the full suite; run only the payments service specs
- Rules from the root AGENTS.md apply as-is; this file overrides only the test command

Merge and precedence rules vary by CLI version. If you want to confirm exactly which instructions ended up in context, check codex --help for supported debug options (log level flags, session log files, etc.) first.

What Running It Actually Reveals

Pros and Cons in the Same Grain

The Most Common Mistake: 'Always Run Everything'

Writing "always run all tests" in AGENTS.md means even a one-line change triggers a full E2E run, and the agent gets stuck wrestling with unrelated failures. Specifying selective execution of only the changed specs is far better.

Pin the MCP Server Version

@playwright/mcp@latest looks convenient, but when the tool signatures change the entire prompt breaks unexpectedly. In a team repo, always pin the version and only bump it after reviewing the release notes.

Relationship to Playwright's Own Agent Features

Playwright has also expanded its agent-friendly tooling in recent releases. Since the exact names and feature boundaries of sub-components differ by version, check the Playwright official documentation for the version you're using. At a high level: "if Playwright provides features for auto-generating or self-healing tests, they can be attached before or after the Codex CLI loop."

The natural setup is a two-layer safety net: let the tool handle shallow regressions it can fix on its own, and pass everything else into the Codex CLI loop.

Things to Watch When Moving to CI

To run Codex CLI on a PR trigger, the CLI won't be present on the GitHub Actions runner — an install step is mandatory. The following is a conceptual example; the exact package name, installation method, and flags must follow the official docs for your CLI version.

# .github/workflows/codex-e2e.yml (conceptual example)
name: Codex E2E Generation
on:
  pull_request:
    types: [opened, synchronize]
 
jobs:
  generate-e2e:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Install Codex CLI
        run: npm install -g @openai/codex   # verify actual package name in official docs
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      - name: Run Codex E2E generation
        run: |
          codex "Write E2E specs for the changed files and make all Checks in AGENTS.md pass"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

If you've heard that parallel execution is possible with a subcommand like codex cloud exec, verify whether that subcommand actually exists, and check its billing and isolation policy in the CLI's codex --help and official docs before documenting it. Writing commands that don't exist will cause teammates to fail when trying to reproduce your setup.

Final Thoughts

To summarize: the value of this loop isn't "it writes tests for you" — it's closer to "it builds a feedback circuit so humans don't have to re-read failure logs." But when you actually run this circuit, you find three places where human hands are still needed.

• Escape conditions and assertion quality rules in AGENTS.md. Without these, the agent will tear down assertions to manufacture a passing run.
• Selector strategy. The loop converges faster when the app's markup already exposes stable selectors like getByRole and getByLabel cleanly. Screens with poor accessibility attributes will push the agent into brute-force XPath.
• Review. A human needs to scan the expect lines in passing specs at least once. Without this step, you quietly arrive at a state where CI is green but regressions keep shipping.

So think of this tool not as "a machine that writes E2E tests for you," but as a tool that removes the tedious parts — grinding through selector rewrites and re-reading failure logs. Those tedious parts are exactly what used to eat entire afternoons, which makes them worth removing.

References

• Codex CLI Official Documentation (OpenAI Developers)
• Playwright Official Documentation
• Playwright MCP Repository (@playwright/mcp)
• agents.md Format Introduction
• Model Context Protocol Official Site
• Why 'Always Run Tests' in AGENTS.md Makes Things Worse — Daniel Vaughan
• End-to-End Testing with Codex CLI and Playwright — Daniel Vaughan
• Test-Driven Development with Codex CLI — Daniel Vaughan
• Loop Engineering with Codex CLI — Daniel Vaughan