Planting specs and unit tests together into legacy functions without tests — a workflow with Codex CLI, AGENTS.md, and a dry-run verification loop

Anyone who has inherited a legacy codebase knows the feeling. You run git blame but still can't tell why the code looks the way it does, there's not a single test, and the team has reached a silent consensus of "just don't touch it." I looked the other way at first, too — until right after a deployment, that very function started behaving strangely. I had no choice but to confront it. The workflow I'll describe here — extracting a spec first, setting boundaries with AGENTS.md, and validating with a dry-run hook loop — is what finally brought that function to a state where it could be touched. It wasn't perfect, but at least it built a safety net that would tell us "what broke" in the next change.

What this post covers is how to use Codex CLI to analyze a legacy function with no tests, generate a spec document and unit tests at the same time, and raise real coverage without false positives. I'll focus on three pillars: a Spec-First approach that locks in the spec before anything else, using AGENTS.md to set concrete boundaries on agent behavior, and a dry-run verification loop that prevents the "looks done but actually wrong" state.

Codex CLI is a terminal-based coding agent released by OpenAI in 2025 (check the official release notes for the exact launch date). It handles not just code generation but also review, refactoring, and test writing — and it's especially useful when dealing with legacy functions that have no documentation and no tests.

AGENTS.md — The Contract for This Workflow

AGENTS.md is an agent instruction file that holds project overview, build/test commands, code style, architectural constraints, and security considerations. It operates in three layers — global → project root → subdirectory — so you can apply different test strategies per module.

It's worth establishing upfront why AGENTS.md is an independent pillar in this workflow, just as important as spec and test generation. The spec answers "what does the code do"; AGENTS.md answers "what is the agent allowed to do." These two documents answer different questions. With a spec but no AGENTS.md, an agent will reference the spec but tends to write tests in whatever way it finds convenient — excessive mocking, verifying internal implementation details, and so on.

There is one strong principle here: AGENTS.md should be written by hand by the team. In the blog post Over-Mocked Tests and Coding Agents by Daniel Vaughan, developer-written AGENTS.md is compared to LLM-auto-generated AGENTS.md, with a strong recommendation in favor of the former — check the original source for specific figures. The point is that this file should be close to "a contract the team has agreed upon," and the moment you let AI write that contract on your behalf, the contract loses its meaning.

Why Extract the Spec First — and the Pitfall

If you hand an agent a legacy function with no tests and just say "write tests," the agent has no idea what the function is supposed to do and generates tests solely from the current implementation. This cements even buggy behavior as "correct."

The classic technique for building a safety net around legacy code is Characterization Testing, as described by Michael Feathers in Working Effectively with Legacy Code. You feed in the current inputs, record the actual outputs, and use those outputs as the validation baseline.

There's an apparent contradiction here. Earlier I said "don't cement bugs," yet Characterization Testing is precisely the technique of fixing the current behavior as-is. How you handle that tension determines whether this workflow succeeds or fails. The approach I use splits it into two stages.

The key point is that the spec draft the agent produces is not ground truth — it's a review artifact. Someone needs to scan it and flag items that look like obvious bugs. Those items get tests that lock the current behavior (otherwise you won't detect them during refactoring), plus a separate issue. Skip this review step and use the spec directly as an anchor, and what you have isn't Spec-First — it's just "Characterization Tests auto-generated by AI."

Walking Through It — rate_limiter.py as an Example

Let's follow the workflow using a legacy rate_limiter.py implemented with the token bucket algorithm.

Step 1: Extract the Spec Draft

First, instruct Codex CLI to analyze the function's current behavior in natural language and save it as a spec file. The command format below (putting natural language directly inside quotes) may require a subcommand like codex exec depending on your Codex CLI version — check the codex --help output for the version you have installed.

codex "Analyze all functions in rate_limiter.py, write a natural language spec describing the current behavior, and save it to spec/rate_limiter.md. Do not speculate — only describe behavior observed in the code."

The generated spec/rate_limiter.md will look roughly like this.

# rate_limiter.py Behavior Spec
 
## RateLimiter.__init__(capacity, refill_rate)
- capacity: maximum number of tokens in the bucket
- refill_rate: number of tokens added per second
- On initialization, the bucket starts full at capacity
 
## RateLimiter.consume(tokens=1)
- If tokens are sufficient, deducts the requested amount and returns True
- If tokens are insufficient, returns False with no change to bucket state
- On each call, calculates elapsed time and refills tokens (up to capacity)
 
## Edge Cases
- Calling with tokens=0 always returns True
- Initializing with capacity=0 means consume always returns False

The team reviews this draft, marks anything suspicious (e.g., whether tokens=0 always returning True is actually intentional), and finalizes it. From this point on, even if the implementation changes, tests validate against this spec.

Step 2: Specify Test Boundaries in AGENTS.md

This is the most important step. Vague instructions don't work. "Write good tests" is meaningless — you need to nail down exactly what may and may not be mocked.

# AGENTS.md
 
## Test Policy
 
### Mocking Rules
- Only HTTP API calls and database connections may be mocked
- Internal logic, internal helper functions, and pure calculation functions must use real objects
- Time dependencies should be controlled with freezegun or unittest.mock.patch,
  but the patch target must be the module path where it is used (e.g., rate_limiter.time.time)
- Mocking the RateLimiter class itself is prohibited
 
### Test Structure
- Each test verifies a single behavior (maintain the AAA pattern)
- Must cover all edge cases in spec/rate_limiter.md
- Run tests with: pytest tests/ -v --tb=short
 
### Prohibited
- Writing empty tests that only contain pass with no assert
- Writing tests that directly verify implementation details (internal variable names, etc.)

The Daniel Vaughan blog post cited earlier includes an example from the browser-use open-source repository where adding the single-line instruction Never mock anything in tests, always use real objects! led to a significant reduction in mocking commits from the agent. Check the original post for the exact numbers and baseline. The lesson to draw from this example isn't the precise figure — it's the principle that one concrete instruction beats several vague paragraphs.

Step 3: Generate Unit Tests from the Spec

codex "Read spec/rate_limiter.md and write unit tests for rate_limiter.py in tests/test_rate_limiter.py following the test policy in AGENTS.md. All edge cases in the spec must be covered."

Don't accept the generated tests as-is. Time mocking in particular often gets the path wrong. Here is a reviewed example.

# tests/test_rate_limiter.py
from unittest.mock import patch
import pytest
from rate_limiter import RateLimiter
 
 
class TestRateLimiterInit:
    def test_bucket_full_on_init(self):
        # Fix refill at 0 to verify initial token count in isolation
        limiter = RateLimiter(capacity=3, refill_rate=0)
        assert limiter.consume(tokens=3) is True
        # Confirming that the capacity+1 request fails proves the initial value is exactly capacity
        assert limiter.consume(tokens=1) is False
 
    def test_zero_capacity_always_rejects(self):
        limiter = RateLimiter(capacity=0, refill_rate=0)
        assert limiter.consume() is False
 
 
class TestConsume:
    def test_sufficient_tokens_returns_true(self):
        limiter = RateLimiter(capacity=5, refill_rate=0)
        assert limiter.consume(tokens=3) is True
 
    def test_insufficient_tokens_returns_false(self):
        limiter = RateLimiter(capacity=2, refill_rate=0)
        assert limiter.consume(tokens=3) is False
 
    def test_zero_tokens_always_true(self):
        limiter = RateLimiter(capacity=0, refill_rate=0)
        assert limiter.consume(tokens=0) is True
 
    def test_bucket_unchanged_on_rejection(self):
        limiter = RateLimiter(capacity=2, refill_rate=0)
        limiter.consume(tokens=3)  # failure case
        assert limiter.consume(tokens=2) is True
 
    def test_token_refill_over_time(self):
        # Must patch time.time at the path used by the rate_limiter module
        with patch('rate_limiter.time.time') as mock_time:
            mock_time.return_value = 0.0  # fix return value before init is called
            limiter = RateLimiter(capacity=10, refill_rate=5)
            limiter.consume(tokens=10)  # drain the bucket
 
            mock_time.return_value = 2.0  # 2 seconds elapsed
            # 2 seconds × 5 tokens/second = 10 tokens refilled
            assert limiter.consume(tokens=10) is True

Two things were fixed. First, test_bucket_full_on_init originally called consume() ten times and only checked pass/fail — if a refill sneaks in between, the test can pass even when the initial state is wrong. Fixing refill_rate=0 and then confirming that the capacity+1 request fails is more accurate. Second, patch('time.time') only patches the time module itself and often fails to change the name referenced inside rate_limiter. You need patch('rate_limiter.time.time') to target the call site. Also, return_value must be set before the RateLimiter is constructed so that the time.time() call inside __init__ gets back 0.0.

Step 4: Dry-Run Verification Loop

This is the last safety device in the workflow. Two layers of validation are attached as hooks so that a "looks done but actually wrong" state can never flow into a commit.

Treat the hook configuration file below as a conceptual example only. The actual hook names, configuration format, and supported values must be verified in the documentation for the Codex CLI version you are using.

# Conceptual example - check your Codex CLI version docs for actual key names and values
[hooks.pre_tool_use]
command = "pytest tests/ -q --tb=short"
on_failure = "abort"
 
[hooks.stop]
command = "pytest tests/ -v --cov=. --cov-fail-under=80"
on_failure = "reject"

One practical caveat: running the full pytest suite on every file save becomes very slow — tens of seconds to minutes per save as the test suite grows. In practice, a reasonable split is to run only the tests related to the changed file in the pre-hook (e.g., pytest tests/test_rate_limiter.py -q) and run the full suite only in the stop hook. Otherwise the agent spends all its time in a "save → wait → save → wait" loop.

Trade-offs — Things That Need to Be Said Honestly

Excessive Mocking Lies to Your Coverage Numbers

In discussion around the research by Hora & Robbes, referenced in Daniel Vaughan's blog post cited earlier, the tendency for agent-generated tests to have higher mocking rates than human-written tests is repeatedly noted. For the exact dataset size, comparison methodology, and figures, it is safer to go directly to the original paper — I came across this discussion through a blog post and have not read the paper firsthand, so I will not repeat specific numbers here.

The direction is what matters. If coverage numbers rise while tests only validate wiring, real regressions won't be caught during refactoring, and harmless refactors will break tests. That is exactly what you want to avoid in legacy code.

Common Mistakes

Mistake 1: Letting AI Write AGENTS.md

This file is a contract that the team must discuss and write themselves. You can use an AI-generated draft as a reference, but committing it as-is defeats the purpose.

Mistake 2: Using Flags Based on Old Documentation

Codex CLI is evolving rapidly, so options from old blog posts (e.g., automation flags like --full-auto) may have been removed or renamed in the current version. Check the official changelog each time for specifics on which flags changed in which version. I've been caught off guard by this myself — pasted in an old example only to find the option no longer existed.

Mistake 3: Mixing Old and New Configuration

If approval policies or sandbox settings were reorganized in a newer version and you mix old table names with new keys, values will be silently ignored or cause conflicts. Keep the config file consistent with the format in the docs for the version you are using.

Mistake 4: Letting Untrusted Project Settings Load Without Review

When opening an unfamiliar repository, if hooks and rules are loaded implicitly, arbitrary commands could execute in your local environment. Be sure to check your trusted project list settings.

Non-Deterministic Behavior

Run the same prompt twice and you get different tests. This non-determinism makes setting regression baselines difficult. Experimental tools using behavioral fingerprinting to address this have started appearing recently, but my judgment is that they are not yet ready for production pipelines. The practical approach is to have a person review the generated tests once, commit the ones that pass, and use them for regression validation only against future changes.

Which Function Should You Pick for Your First Try

Trying to run this workflow on all 100 legacy functions at once means you'll never start. For your first attempt, pick a function that meets these three criteria.

• A function changed at least twice in the last 6 months: High change frequency means it will keep needing to be touched, so the safety net pays back fastest.
• A function with little external I/O and deterministic inputs/outputs: Defer functions entangled with time, randomness, network, or DB. Pure calculation functions have the highest success rate at first.
• A function that has appeared in the issue tracker at least once due to a bug: In this case, when extracting the spec, the team will naturally have plenty of items to flag as "obvious bug," and the review step attaches organically.

The first function I picked wrong didn't meet any of these three criteria — it was simply "the scariest-looking" one. I grabbed a function entangled with time, cache, DB, and an external API as my first attempt and lost several days. The function from the deployment incident I mentioned in the introduction happened to meet the second criterion well enough that coverage was added using this workflow, and in the subsequent refactoring the root cause could finally be isolated. Your first attempt should not be the flashiest function — it should be the one with the fastest payback.

References

• Michael Feathers, Working Effectively with Legacy Code — the original source on Characterization Testing
• Test-Driven Development with Codex CLI: AGENTS.md Test Gates and Hook-Based Verification (blog)
• Over-Mocked Tests and Coding Agents (blog, includes discussion of Hora & Robbes research)
• Agent Instruction Files: AGENTS.md, CLAUDE.md, Cross-Tool Portability (blog)
• AgentAssay: Statistical Validation of Non-Deterministic Agent Workflows (blog)
• Modernizing your Codebase with Codex — OpenAI Cookbook
• Release notes and --help output from the official Codex CLI repository — always check here for version-specific flags and hook names