Debugging AI-generated code with checklists

Learn how to build systematic debugging checklists that catch common failure patterns in AI-generated code before deployment to production systems.

You push a fix that looked clean in the diff, merge it, and thirty minutes later someone pings: "The migration fails in staging because the script assumes UTC timestamps but our test DB uses local time." The AI agent got the core logic right but missed a boundary condition you would have caught in code review. This pattern—correct on first inspection, wrong under real conditions—is the central debugging challenge with AI-generated code.

When an LLM generates 200 lines in 30 seconds, you inherit code with no authorship memory. The model didn't "decide" to skip null checks because it judged the risk acceptable—it generated that pattern because similar code in its training data often omitted them. You're debugging a statistical artifact, not a decision trail. Traditional debugging workflows assume a human wrote every line with intent and can reconstruct their reasoning. AI-generated code has no such memory.

Checklists bridge that gap. Not compliance theater, but systematic audits that catch entire categories of defects before they ship. This guide walks through building checklists that actually work in agent-driven workflows, grounded in failure patterns observed across production systems.

Why standard debugging falls short

AI-generated code breaks in three predictable ways that slip past normal review:

Assumption drift: The model infers requirements from your prompt and surrounding code, but those inferences quietly diverge from reality. You ask for "pagination support" and get cursor-based pagination when your API contract requires offset-based. The code compiles, tests pass with mocked data, but the real endpoint returns 400.

Partial implementations: AI agents generate the happy path thoroughly but treat error handling as an afterthought. The code works beautifully when everything goes right and fails silently—or catastrophically—when inputs are malformed. A function that averages an array of numbers crashes on empty input because the model never considered len(arr) == 0.

Context misalignment: The model doesn't know which parts of your codebase are stable contracts versus prototypes. It might refactor a public interface downstream teams depend on, or carefully preserve a deprecated pattern you were trying to phase out. When you write code yourself, you carry this context. When an agent writes it, you don't.

These failures share a signature: they look reasonable at every local scope but break in unexpected interactions between modules. A human reviewer scanning for syntax correctness and aesthetic issues will miss them. Checklists force you to look for specific failure modes in a defined order.

The compile-first gate as your baseline

Before you write custom validation rules, make sure you're actually running the compile/lint/test suite as a hard gate. This sounds trivial but most AI coding tools treat test failures as suggestions.

In Goatfied's agent loop (plan → constrain → edit → validate → retry), the validate step is blocking: if cargo build or npm run typecheck fails, the diff doesn't proceed to retry—it gets constrained by the error and re-edited. This catches entire categories of bugs (type mismatches, import errors, unused variables) without custom rules.


# Example: validation gate in .goatfied/config.yml

validate:

  build: "cargo build --all-features"

  test: "cargo test --lib"

  lint: "cargo clippy -- -D warnings"

  # If any fail, agent re-enters edit phase with errors as constraints

Three gates form the compile-first pipeline:

Gate 1: Compilation as a boolean filter. If your language compiles, enforce it. A model that generates async functions without proper await handling or mismatched type signatures should never reach a reviewer's queue. Non-compiling code triggers an immediate retry with the error message fed back to the agent.

Gate 2: Linting as correctness hygiene. Linters catch more than style—they find unused variables that signal incomplete logic, missing null checks, and import cycles. An AI-generated TypeScript function that reads user.profile.email without checking if profile exists will fail @typescript-eslint/no-unsafe-member-access if you've enabled strict null checks.

Gate 3: Existing test suite as regression detector. The AI doesn't know which integration test covers the code path it just modified. Run the full suite (or at minimum, tests tagged to the modified modules). A passing suite doesn't prove correctness, but a failing suite proves the change is unsafe. This catches about 30% of logical errors: methods renamed without updating call sites, changed function signatures breaking downstream consumers.

Your checklist should start here and only add rules for issues that slip through static analysis. If your linter already flags unwrapped Result types in Rust, don't write a separate checklist item for error handling.

Core failure patterns that need explicit checks

Even with strong automated gates, six categories of bugs consistently escape. Structure your checklist around these:

Signature drift and orphaned callers

The agent renames a function parameter or changes a return type but doesn't propagate the change to all call sites. Compiled languages catch some of this; dynamic languages and cross-module calls often don't.

Check: Search the codebase for the old function name or parameter. If you changed processOrder(orderId) to processOrder(request), grep for processOrder( and verify every match now uses the new signature.


# After AI edit, diff function signatures and grep for old patterns

git diff main --stat | grep -E '\.(ts|py)$' | xargs grep -n 'oldFunctionName'

Missing null/undefined guards in new branches

AI often adds a new conditional path—if (config.featureFlag)—without checking whether config itself can be null. The happy-path test passes; production crashes when config is undefined.

Check: For every new if or switch statement, trace back to the source of the tested variable. Does the schema or type definition allow null? If so, does the new code handle it? Scan new conditionals for object property access (obj.prop) without a prior null check on obj.

Schema and API contract mismatches

The agent drafts a database query or API call based on outdated documentation or an incorrect assumption. The code compiles, the test mocks return happy data, but the real endpoint returns 400 or the query returns empty.

Check: Cross-reference new query/API code against the actual schema or OpenAPI spec. If the AI added a field user.email, confirm that field exists in your current schema version. Run a local schema validator or contract test—in Goatfied's workflow, the validate step can invoke a lint rule that checks Prisma queries against schema.prisma or API calls against a local OpenAPI file.

Boundary condition coverage gaps

For every loop, does the checklist force you to test zero iterations, one iteration, and maximum iterations? For string handling, verify empty strings, whitespace-only input, and strings exceeding expected lengths.

Check: Diff test files against code files. If you added 20 lines to billing.ts, did you add or modify a test in billing.test.ts? Use coverage diffing tools like jest --coverage --changedSince=main to surface new uncovered lines. Any new throw, return null, or else branch should have at least one test exercising it.

Example integration test pattern:


it('retries on transient database errors', async () => {

  const mockDb = {

    query: jest.fn()

      .mockRejectedValueOnce(new Error('Connection timeout'))

      .mockResolvedValueOnce({ rows: [] })

  };

  const result = await fetchUser(mockDb, 'user-123');

  expect(mockDb.query).toHaveBeenCalledTimes(2);

  expect(result).toBeDefined();

});

If the AI didn't implement retry logic, this test fails and you know to add it.

Error propagation paths

AI-generated error handling frequently swallows exceptions or logs without retrying/failing appropriately. Trace the error flow: does a failed API call bubble up, or does it return null and continue? Does the code distinguish between retryable network errors and permanent 4xx failures?

Check: Look for try/catch blocks with only console.error() in the catch clause when calling code expects a return value. Verify error paths return structured JSON with appropriate HTTP status codes. If your checklist item is "confirm error paths return structured JSON," and the diff returns plain text, the next edit gets that failure injected as a requirement in Goatfied's constraint system.

Copy-paste drift in similar functions

The agent duplicates a block of code (e.g., authentication logic) across two handlers. It tweaks one copy but not the other, creating subtle inconsistencies—one handler logs errors, the other swallows them.

Check: Run a duplicate-code detector (jscpd, pylint --disable=all --enable=duplicate-code) on the diff. Flag blocks >10 lines that appear twice with minor differences. Either extract a shared helper or document why the duplication is intentional and test both paths independently.

Partitioning checklists by change type

A diff that adds a new API endpoint needs different scrutiny than a refactor that extracts a helper function. Applying every rule to every diff creates false positives and slows the loop.

Partition your checklist by change classifier. Before validation, tag the diff:

New public surface (API route, exported function, CLI command): Check auth, rate limiting, input validation, error response format
Database migration: Check reversibility, index strategy, null handling in existing rows
Async boundary (new goroutine, tokio spawn, JS Promise): Check cancellation, error propagation across the boundary, resource cleanup
Dependency update: Check for breaking changes in CHANGELOG, verify tests cover the integrated behavior

The agent can self-classify with a prompt, but for reliability, use mechanical signals: does the diff touch routes.rs or add a @app.route decorator? Does it create a new .sql file in migrations/? Mechanical checks avoid the agent misclassifying its own work.

Here's a minimal working example for a backend API endpoint:


## Generated endpoint checklist



- [ ] Input validation: tested null, empty, malformed, and boundary values

- [ ] Auth/authz: verified both authenticated and unauthenticated requests

- [ ] Database queries: checked for N+1 patterns, confirmed indexes exist

- [ ] Error responses: return correct HTTP status codes and structured errors

- [ ] Logging: sensitive data (tokens, PII) not logged at INFO level

- [ ] Backwards compatibility: existing clients can still parse response

The power is in making each item testable. "Error handling is present" is useless. "Tested 400 response for invalid UUID format" is actionable.

Trading off depth for iteration speed

Every additional checklist item adds latency to the plan → edit → validate loop. If validation takes thirty seconds because you're running five static analyzers and a custom script that checks for hardcoded secrets, you slow the agent's ability to iterate when validation fails.

The tradeoff: shallow checklists with fast feedback let the agent try more solutions. Deep checklists catch more on the first attempt but penalize retries.

In practice, tiered validation works best:

1. Fast gate (< 5 seconds): Compile, type-check, fast unit tests, basic linting

2. Standard gate (< 30 seconds): Full test suite, integration tests for touched modules

3. Pre-merge gate (< 3 minutes): E2E tests, security scans, deployment dry-run

The agent loop runs tiers 1 and 2 on every retry. Tier 3 runs once before the PR is marked ready. This keeps the edit-validate cycle tight while still catching expensive-to-fix issues before merge.

A minimal CI pipeline looks like:


name: AI Code Validation

on: [pull_request]

jobs:

  validate:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v3

      - run: npm ci

      - run: tsc --noEmit

      - run: npm run lint

      - run: npm test -- --coverage --coverageThreshold='{"lines":70}'

      - run: |

          git diff origin/main --name-only | wc -l | awk '{if ($1 > 10) exit 1}'

The last step fails if more than 10 files changed—a forcing function to keep PRs small. Adjust the threshold for your codebase, but enforce something or AI will generate unbounded diffs.

Checklists as diff-scoped constraints, not repo-wide audits

The biggest design mistake: asking the agent to validate the entire codebase state rather than just its changes. "Does the application handle rate limiting correctly?" is an audit. "Does the new endpoint in this diff apply the standard rate-limit decorator?" is a scoped check.

Goatfied's constraint system helps here: when validation fails, the error becomes a constraint on the next edit attempt. The agent doesn't re-audit—it fixes the specific violation. Scoped checklists make the agent loop converge faster because retries have a clear target, not an open-ended "improve code quality" prompt.

Break the checklist into two stages:

Automated (CI gate):

Compile, lint, type-check
Test suite (including coverage diff)
Schema/contract validation
Dependency conflict check (npm ls, pip check)
Duplicate-code scan

Human spot-check (2–5 min):

Signature drift: grep for old function names
New conditionals: null/undefined guards
Copy-paste logic: duplication report review
Test coverage: any new error paths untested?

The agent retries if validation fails, so most PRs that reach human review have already passed the mechanical checks.

When to let the human decide

Some checklist items can't be mechanically verified and shouldn't be auto-passed by the agent. These need a human gate:

Does this change preserve backward compatibility with the deployed API version?
Is the new feature flag name consistent with our existing naming scheme?
Does the error message expose information useful for debugging without leaking internal state?

Mark these as manual review required in your workflow config. The agent completes its loop, but the PR doesn't auto-merge. A staff engineer confirms the subjective parts before deploy.

This is especially important for diffs that touch authentication, billing logic, or data retention policies—domains where a missed edge case has disproportionate impact.

Post-deploy monitoring for AI-generated changes

Once merged, AI-generated code needs telemetry that surfaces behavioral drift from expected patterns.

Metric 1: Error rate by commit. Tag errors in your logging/APM system with the commit SHA that introduced the code. If a commit authored or co-authored by AI sees a 5x spike in exceptions compared to baseline, investigate immediately.

Metric 2: Performance regression detection. AI-generated code sometimes introduces O(n²) loops or redundant database queries. Compare p95 latency for endpoints modified in the last deploy. If a previously 50ms endpoint now takes 300ms, the new code likely has an efficiency bug.

Metric 3: Unexpected state transitions. If your application models workflows (order processing, user onboarding), track state transition counts. An AI change that accidentally skips a validation step might let records move from pending to completed without intermediate states. Graph transition frequencies pre- and post-deploy.

Metric 4: Diff size and churn correlation. Large diffs (>200 lines) from AI agents have higher defect rates. Track PR size vs. post-deploy incidents. If you see a pattern, configure your agent to prefer smaller, more incremental changes (Goatfied's default is to keep diffs under 100 lines and prompt the agent to split larger refactors).

Growing the checklist over time

Checklists decay. The items that catch bugs in month one become rote busywork by month six, while new failure patterns emerge that aren't covered.

When a bug escapes to production, run a quick postmortem: which checklist item would have caught it? If none, add a new item. Examples teams have added after real incidents:

"Verify new environment variables are documented in .env.example"
"Check that new database migrations include a rollback script"
"Scan for hardcoded localhost URLs in new integration code"

Keep the list short—10 items max. If it grows beyond that, automate the common ones and keep only the judgment-heavy checks for humans. Treat your checklist as a living document tied to your CI metrics. If you deploy ten AI-generated PRs and none fail the "boundary condition coverage" check but three fail in production due to race conditions, that's a signal. Add a concurrency checklist section, remove or demote the boundary item.

Goatfied users running self-hosted instances often instrument this directly: log which checklist items caught issues during validation versus which issues escaped to staging. After 30-50 PRs, re-weight the checklist. Items with high false-positive rates get refined or removed. Items that caught zero issues but also cost near-zero time to verify can stay as cheap insurance.

Practical constraints

Checklists add friction. They slow down fast iteration, especially on prototypes where breakage is acceptable. Use them selectively:

Enable always: Production PRs, customer-facing features, schema changes
Disable or lighten: Internal tools, proof-of-concept branches, hotfixes under time pressure (but add a follow-up ticket)

The goal isn't zero defects. It's to catch the predictable, boring failures before they consume engineer time in production debugging or rollback meetings. Better to have a tight eight-item list that actually gets used than a comprehensive 30-item list that people rubber-stamp.

One forcing function: if your checklist takes longer to complete than manually reviewing the code would have taken, you've over-indexed. The goal is structured verification that's faster and more reliable than ad-hoc review, not bureaucracy that adds toil without proportional safety gains.

The most effective teams encode these checks into their CI/CD pipeline and pull request templates. Add a .goatfied/review-checklist.md to your repo with the core human review items. Link it in your PR template. Fail CI if any of the tripwire gates fail—don't make compilation/linting/testing optional just because an AI generated the code. Use commit message conventions to tag AI contributions (e.g., [ai-assisted]), then filter your incident postmortems by tag. Over time, you'll identify which types of changes need stricter pre-merge review.

This isn't about distrusting AI—it's about treating generated code with the same rigor you'd apply to any external contribution. A well-configured agent loop with strong gates, a sharp human checklist, and tight post-deploy telemetry turns AI from a liability into a reliable contributor.