Loop / Harness Engineering: The Maker/Checker Split, Three Ways
Stop trusting agent loops that mark their own homework. False completeness is a structural failure, and production agents need a real maker/checker split before “done” means anything.
The most dangerous agent failure is not the one that crashes loudly. It is the one that smiles, says “done,” and quietly ships broken work into production.
Every engineer has learned to fear the red build. But the real nightmare is the green checkmark from an AI agent that never finished the job.
Summary: This article looks at one of the quietest failure modes in AI agent systems: the moment an agent says “done” before the work is actually complete. It explores why self-checking breaks down, what false completeness looks like in real builds, and why production agents need a stronger verification layer before teams can trust them with serious work. Loop Engineering is a subset of Harness Engineering.
Before we get into it, please consider subscribing to my Substack newsletter. Please share this article, like it, and restack it. It really helps.
Thank you for reading and helping these ideas reach more builders.
The False Completeness Problem in AI Agent Verification
You ship an agent loop. The agent runs. It reports “done.” You open the repo. Half the functions have placeholder bodies. Two files never got written. The code does not compile.
The agent was not lying, exactly. It genuinely believed it was done. That is the problem.
This failure mode has a name: false completeness. It is what happens when the same model that produces the work also evaluates whether it is finished. The maker declares victory. Nobody checks. The output goes out broken.
What False Completeness Looks Like in Practice
False completeness is not hallucination in the casual sense. The agent is not making up facts about the outside world. It is making a confident but wrong judgment about the state of its own output. Concretely:
Half-built code: functions with stub bodies, classes with missing methods, tests that were planned but never written
Placeholders:
# TODO: implement this,...,passleft in production pathsNon-compiling repos: missing imports, undefined symbols, syntax errors introduced during refactoring that the agent never ran
Incomplete file sets: the agent said “I created all five files” but one is missing or empty
In every case, the agent ran its own self-check and passed itself. The check was structurally incapable of catching the error because the checker was the maker.
The failure mode the author calls false completeness: drawing on the framing in selmakcby/loop-engineering: occurs when half-built code, placeholders, or non-compiling output is confidently declared finished by the maker.
Why AI Agent Self-Verification Is a Structural Problem
The instinct is to fix the prompt. Add “double-check your work.” Add “verify every file was created.” Add “confirm the code compiles before reporting done.”
It does not work. Not reliably.
The problem is not that AI agents have bad instructions. The problem is that the agent uses the same weights, context window, and reasoning process to produce and evaluate the output. Any blind spot in the production step is also a blind spot in the evaluation step. The maker has already “decided” what done looks like. The checker inherits that decision.
This structural failure has empirical support. A March 2026 study found that language models systematically fail to flag high-risk or low-correctness actions when evaluating output they themselves generated, compared to evaluating identical output presented as external work [Khullar et al., arXiv:2603.04582, March 2026]. The effect is strongest precisely where reliable self-monitoring is most needed: on incorrect code and unsafe tool-use actions. An earlier result found that the same self-preference amplifies across self-refinement cycles; the more an LLM refines its own output, the more it tends to favor what it has already produced [Xu et al., arXiv:2402.11436, ACL 2024].
This is the key insight from loop-engineering practice [selmakcby/loop-engineering, 2026]:
“Never let the AI verify its own done. Maker and checker must be different things.”
That is the foundational rule. It is categorical. There is no prompt that bypasses it.
The Maker/Checker Split Doctrine
Self-verification produces false completeness. The only reliable fix is architectural separation between maker and checker. This doctrine is the load-bearing idea behind every well-designed agentic AI system.
The maker generates output. The checker evaluates that output. Those two roles must be held by different things. If they are the same thing (same model call, same context, same reasoning process), you do not have a checker. You have the maker running twice.
“Different things” turns out to have three distinct implementations, each with a different cost and quality profile:
A separate model call: the highest-quality checker, and the most expensive. A different model, potentially on a different cost tier, evaluates the maker’s output against a quality threshold. You get the full reasoning power of a language model applied to checking, including subjective quality judgments that code cannot make.
A platform-managed grader: the middle path. You write a rubric in plain markdown. An isolated, Anthropic-managed grader evaluates the maker’s output against your criteria. This approach carries lower operational burden than running your own evaluator; it also delivers better quality than code-only checks.
A programmatic hook: the cheapest option with genuine architectural separation. Python code intercepts the agent loop before or after execution. The checker is your code, not another model call. Zero additional model tokens. Limited to what deterministic code can verify, which is often exactly what you need.
Parts 2, 3, and 4 implement each of these strategies concretely. Part 5 gives you the decision framework for choosing between them.
The rest of this article is about implementation. But before the code, internalize the doctrine:
The maker/checker split is not a prompt pattern. It is an architectural constraint. Until maker and checker are structurally separate, your agent loop has no reliable termination condition.
Implementation 1: Evaluator-Optimizer Pattern (Separate Model Checker)
What does “a different model” actually mean in practice? Not a different system prompt. Not a different temperature setting. A completely independent model invocation that was not present when the output was generated and has no stake in calling it finished.
The highest-quality implementation of the maker/checker split runs the checker as a separate model call. Not a separate prompt to the same call. A separate, independent invocation, potentially on a completely different model. This is how production-grade agentic AI teams enforce output quality at scale.
This is the evaluator-optimizer pattern, implemented in evaluator_optimizer.py from lastmile-ai/mcp-agent (PyPI: mcp-agent, v0.2.6 as of December 2025; 8,300+ GitHub stars as of June 2026, repository: https://github.com/lastmile-ai/mcp-agent) [lastmile-ai/mcp-agent GitHub, June 2026].
Evaluator-Optimizer Architecture: How the Maker/Checker Loop Works
The pattern has two roles:
Optimizer (the maker): generates or refines the output
Evaluator (the checker): scores the optimizer’s output against a quality threshold
These roles run on separate model calls. The critical architectural feature: they can run on separate model tiers. The optimizer might use a powerful model for generation; the evaluator might use a faster, cheaper model for scoring. Or the reverse, if evaluation requires more nuance. The point is that cost is controlled independently for each role.
The loop is:
1. Optimizer generates output
2. Evaluator scores output → QualityRating + needs_improvement flag
3. If threshold met OR needs_improvement is False → stop
4. Otherwise → pass evaluator feedback back to optimizer → repeat
The QualityRating Enum: Scoring Thresholds Explained
The evaluator’s score is a QualityRating value (think of it as a four-point rubric scale, not a floating-point score). The exact enum values, in ascending order [lastmile-ai/mcp-agent source, 2026]:
POOR → FAIR → GOOD → EXCELLENT
(see https://github.com/lastmile-ai/mcp-agent/blob/main/src/mcp_agent/workflows/evaluator_optimizer/evaluator_optimizer.py)
These map to integer values 0-3 respectively (POOR = 0, FAIR = 1, GOOD = 2, EXCELLENT = 3). They are not strings. They are not paraphrases. The enum has exactly four members. When you configure the loop, you set a min_rating threshold; the loop continues until the evaluator’s score meets or exceeds that threshold.
Loop Stopping Conditions: Threshold, Convergence, and Hard Cap
The loop has three independent stopping conditions. Any one of them halts the loop [lastmile-ai/mcp-agent source, 2026]:
Threshold met: the evaluator returns a
QualityRatingat or abovemin_ratingNo improvement needed: the evaluator returns
needs_improvement = FalseIteration cap reached: the loop has run
max_refinementsiterations without meeting the quality threshold.
The second condition matters because an evaluator might rate output as FAIR but still signal that further iterations will not help: the output has converged. Stopping on needs_improvement prevents runaway loops even when the threshold has not been met. The third condition is the hard ceiling. The loop halts at the cap even if neither quality condition has been satisfied.
Why Separate Model Tiers Matter for Cost and Quality
Running the evaluator as a separate model call unlocks two things.
Cost control. If evaluation is cheaper than generation, you can run a small, fast model as the evaluator and a large model as the optimizer. Most refinement loops spend the majority of tokens on generation; the evaluator call is a small fraction of total cost. The 2026 industry pattern confirms this tiering: frontier-class models for generation, flash/mini-class models for evaluation scoring, with flash-tier judges substantially cheaper and fast enough for production use after calibration against a frontier reference. [1]
Subjective quality. A model evaluator can reason about things that code cannot: “Is this explanation clear to a junior developer?” “Does this code follow the project’s style?” “Is this summary accurate given the source?” Those judgments require language understanding. A shell script cannot make them.
This is the key trade-off compared to cheaper implementations: if you need to verify subjective quality (style, clarity, appropriateness, nuance), a model evaluator is the only checker that works. Code checkers cannot evaluate what “good” means in a subjective domain.
When to Use the Evaluator-Optimizer Pattern
Reach for the evaluator-optimizer pattern when:
The quality criterion is subjective and requires reasoning (clarity, correctness of explanation, appropriateness of tone)
You need to refine output across multiple passes, with each pass informed by evaluator feedback
Token cost per iteration is acceptable given the quality requirement
You are already in an MCP-native orchestration environment and want
mcp-agent‘s multi-agent infrastructure
Skip this pattern when:
The check is mechanical and deterministic (did the file get created? does the code compile? did all tests pass?)
Cost per iteration is a hard constraint
You can express the quality criterion in a rubric that a platform-managed grader can apply
Python Implementation: mcp-agent Evaluator-Optimizer Code
The evaluator_optimizer.py pattern in mcp-agent wires the loop like this (illustrative structure):
# Illustrative structure from lastmile-ai/mcp-agent evaluator_optimizer.py
from mcp_agent.workflows.evaluator_optimizer.evaluator_optimizer import ( # ①
EvaluatorOptimizerLLM,
QualityRating,
)
workflow = EvaluatorOptimizerLLM(
optimizer=optimizer_agent, # maker: generates/refines output
evaluator=evaluator_agent, # checker: separate model call
min_rating=QualityRating.GOOD, # loop until GOOD or EXCELLENT
max_refinements=5, # ②
)
result = await workflow.generate_str(message=”Write a clear explanation of backpressure.”)
# Loop runs until evaluator rates output >= GOOD, or needs_improvement is False, or max_refinements hit① Import EvaluatorOptimizerLLM and QualityRating from the full submodule path: mcp_agent.workflows.evaluator_optimizer.evaluator_optimizer, not the package root. [lastmile-ai/mcp-agent source, 2026]
② max_refinements caps the total number of optimizer iterations; the loop halts when this ceiling is reached, even if neither quality-stopping condition has been met.
The evaluator runs against the optimizer’s output and returns structured feedback including the QualityRating and whether the output still needs improvement. The optimizer receives that feedback and refines its output. The loop continues until the stopping condition is met.
The evaluator and optimizer are independent agent configurations: separate model selections, separate system prompts, separate tool sets. This is what makes the separation architectural rather than cosmetic.
Implementation 2: CMA Outcomes (Platform-Managed Maker/Checker)
What if you want the quality of a model checker but do not want to wire up and operate the evaluator yourself? That is the tradeoff the CMA Outcomes system is designed for.
The Evaluator-Optimizer pattern puts you in charge of running the evaluator model. You configure it, you pay for it, you maintain it. For many workloads, that operational overhead is unnecessary. Anthropic can manage the grader for you, and your AI agents get the same architectural separation guarantee without the plumbing.
That is what the CMA Outcomes system does. CMA stands for Claude Managed Agents (the Claude deployment API) [Anthropic CMA Docs, 2026]. The Outcomes system is the platform’s built-in maker/checker split: you write a rubric in plain markdown, and an isolated, Anthropic-managed grader evaluates your agent’s output against it [Anthropic CMA Outcomes Docs, 2026]. This approach was introduced with the anthropics/launch-your-agent reference implementation [GitHub, June 2026], launched June 9, 2026.
What an Isolated Grader Guarantees
The grader is not the same model that produced the output. That is the architectural guarantee the system provides [Anthropic CMA Outcomes Docs, 2026].
When your deployed agent finishes a task, the Outcomes system routes the output to a separate grader process managed by the platform: not your application code, not the same model session, not the same context [Anthropic CMA Outcomes Docs, 2026]. The grader reads your rubric and scores the output. If the output does not satisfy the rubric, the system triggers another iteration.
This is architectural separation enforced at the platform level. You do not have to implement the separation yourself. You just have to write the rubric.
Writing a Quality Rubric for Agent Evaluation
The rubric is plain markdown. Write it in plain English. The grader is a language model; it can reason about criteria written in natural language. You do not need a structured format, a scoring schema, or a special syntax [Anthropic CMA Outcomes Docs, 2026].
A rubric might look like:
## Output Quality Criteria
- The response must directly answer the user’s question without preamble.
- All code examples must be syntactically valid Python 3.11.
- The explanation must not use jargon without defining it first.
- The response must not include placeholder text such as “TODO” or “...”.
- The summary must accurately reflect the content of the source document.The grader evaluates the agent’s output against each criterion and determines whether the output is satisfactory. If it is not, the system requests another iteration, passes the grader’s feedback back to the agent, and tries again [Anthropic CMA Outcomes Docs, 2026].
CMA Outcomes Result States
When the CMA Outcomes system finishes evaluating an agent run, it returns one of five result states. These are exact enum values; do not paraphrase them [Anthropic CMA Outcomes Docs, 2026]:
satisfied: The grader determined the output meets all rubric criteria.needs_revision: The grader found deficiencies; another iteration was triggered.max_iterations_reached: The loop hit the iteration ceiling without reachingsatisfied.failed: The agent encountered an error that prevented normal completion.interrupted: The run was cancelled externally (timeout, user action, system event).
Your application code should handle all five states. The happy path is satisfied. The three non-error, non-happy-path states (needs_revision during iterations, max_iterations_reached on ceiling hit, interrupted on external cancellation) are normal operational outcomes. Only failed indicates a structural problem: typically a rubric that fundamentally mismatches or contradicts the agent’s task description.
Controlling Agent Iteration Limits
The CMA Outcomes system is designed to loop; each needs_revision result triggers another agent iteration with grader feedback. Two configuration parameters control the loop ceiling [Anthropic CMA Outcomes Docs, 2026]:
Default
max_iterations: 3 (the loop will run at most 3 times without explicit configuration)Maximum allowed
max_iterations: 20 (the platform ceiling; you cannot exceed this)
In practice, well-written rubrics often lead the agent to converge within a handful of iterations, though the exact number depends on task complexity and rubric specificity. A well-written rubric tends to converge without hitting the ceiling. If you are regularly hitting max_iterations_reached, the rubric is probably too strict, the agent is not capable of satisfying it, or both.
Deployment Configuration
The CMA Outcomes system is configured at deployment time via the Claude deployment API:
POST /v1/deployments (see https://platform.claude.com/docs/en/managed-agents/define-outcomes)
The outcomes configuration field carries your rubric and iteration settings:
{
“name”: “my-agent-deployment”,
“model”: “claude-opus-4-8” (see https://www.anthropic.com/news/claude-opus-4-8),
“system_prompt”: “...”,
“outcomes”: {
“rubric”: “## Criteria\n\n- The response must ...\n- All code must ...”,
“max_iterations”: 5
}
}The outcomes field is the key. Without it, the platform runs the agent once and returns the result. With it, the platform runs the grader loop automatically.
When to Use CMA Outcomes for Maker/Checker Verification
Reach for CMA Outcomes when:
You are already deploying agents through Anthropic’s managed deployment API
Your quality criteria can be expressed as a markdown rubric, written in plain English and checked by a language model [Anthropic CMA Outcomes Docs, 2026]
You want the platform to manage the grader lifecycle rather than running your own evaluator model
Cost is a consideration but you still need richer checking than deterministic code can provide
The CMA Outcomes system sits between the Evaluator-Optimizer pattern and the programmatic hook on both the cost and quality spectrum. The grader is a model, so it can evaluate subjective criteria. But you do not manage the grader directly; the platform does. That reduces operational overhead while maintaining the architectural separation guarantee.
Skip CMA Outcomes when:
The check is purely mechanical (code execution, file existence, test results): a programmatic hook is cheaper and more reliable
You need to inspect or control the grader’s reasoning (the grader is managed; you see the result state, not the reasoning trace)
You are not using Anthropic’s managed deployment API
Implementation 3: PreToolUse Defer Hook (Zero-Cost Code Checker)
What if the check you need is not a matter of opinion? Not “is this explanation clear?” but “does this code compile?” Not subjective quality but hard facts: did the file exist, do the tests pass, does the schema validate?
The first two implementations both spend model tokens on the checker. The Evaluator-Optimizer runs a separate model call on every iteration. CMA Outcomes runs a platform-managed grader. Both deliver quality through language model reasoning, and both pay for it in tokens.
Sometimes you do not need a language model to verify the output. You need a compiler. A test runner. A filesystem check. A schema validator. Deterministic code that returns pass or fail. This is the verification path most agentic AI teams underuse, and it is often the right one.
For that class of verification, the PreToolUse hook in the Claude Agent SDK delivers genuine architectural separation at zero additional model cost.
The Claude Agent SDK Hooks System
The Claude Agent SDK (anthropics/claude-agent-sdk-python) provides a Hooks system that injects Python functions at specific points in the agent execution loop [Claude Agent SDK Python Docs, 2026]. A hook is not a prompt. It is not a model call. It is a Python function that runs synchronously in the loop, can inspect the agent’s state, and can intervene in its execution.
The hook type for the maker/checker pattern is the PreToolUse hook [Claude Agent SDK Python Docs, 2026]. It fires before a tool call executes: before the filesystem write occurs, before the API call is made, before the test runs. You can see what the agent is about to do and stop it.
How PreToolUse Works
When the agent decides to call a tool, the PreToolUse hook fires first. Your Python function receives the pending tool call and returns a decision [Claude Agent SDK Python Docs, 2026]:
Allow: the tool call executes normally
Deny: the tool call is rejected, and the rejection reason is passed back to the model
Defer: execution is suspended entirely
The defer path is the maker/checker pattern:
# Returning this from a PreToolUse hook suspends the tool call
return {
“hookSpecificOutput”: {
“hookEventName”: “PreToolUse”,
“permissionDecision”: “defer”
}
}Returning permissionDecision: "defer" does three things [Claude Agent SDK Python Docs, 2026]:
Halts the tool call: the tool does not execute
Sets
stop_reason='tool_deferred'on the result: the loop knows why it stoppedSurfaces the pending call in
ResultMessage.deferred_tool_use: your code has the full call details
Reading the Deferred Tool Call
After the hook fires and defers the call, control returns to your Python code. You read the deferred call from the result:
# Illustrative pattern: Claude Agent SDK
result = await client.run(agent=agent, task=task) # ①
if result.stop_reason == ‘tool_deferred’: # ②
deferred = result.deferred_tool_use
# Your checker runs here: pure Python, zero model tokens
if my_checker_passes(deferred): # ③
# Resume: let the tool call execute
result = await client.resume(result)
else:
# Reject: the tool call should not execute
result = await client.reject(result, reason=”Checker failed: ...”)① client.run() starts the agent task; the loop runs until a stopping condition fires, including a PreToolUse defer. ② stop_reason == 'tool_deferred' is the signal that the hook intercepted a tool call; all other stop reasons indicate normal or error termination. ③ my_checker_passes() is your Python verification logic (compile check, test runner, schema validator, or any deterministic code) and runs at zero additional model cost.
Your checker is Python. It can run real tests, compile code, validate schemas, check file existence, run linters, and query databases. Whatever deterministic validation your domain requires.
Zero additional model tokens. The model is not involved in the check.
Why “Checker is Python” Is an Architectural Property
The point is not just that Python is cheap to run. The point is that Python is a different kind of thing than a model call.
A model checker inherits the same reasoning process as the maker; it can evaluate subjective quality, but it can also share the maker’s blind spots. A Python checker has no blind spots. If you run pytest and 3 tests fail, you get exactly that result. The checker is not making inferences about whether the tests might pass; it ran them.
This is the strongest form of the maker/checker separation for mechanical checks:
Different execution layer (not a model call at all)
Deterministic (same input always produces the same result)
Auditable (you can inspect the checker code)
No cost increase per iteration
The tradeoff: code can only verify what code can express. “Does this code compile?” is a mechanical check. “Is this explanation clear to a new developer?” is not. For subjective quality, you need a model checker.
Bonus Pattern: Mechanical Stop Hook
The PreToolUse hook runs inside an agent loop built on the Claude Agent SDK. But there is an even simpler variant that requires no SDK at all: the Mechanical Stop Hook for Claude Code.
Claude Code supports hook scripts wired to lifecycle events in .claude/settings.json [Claude Code Hooks Reference, 2026]. The Stop event fires when Claude Code is about to report that it is done. You can intercept it (the equivalent of running your tests before a git push, but automated):
{
“hooks”: {
“Stop”: [
{
“matcher”: “”,
“hooks”: [
{
“type”: “command”,
“command”: “/path/to/check_done.sh”
}
]
}
]
}
}Your shell script runs real tests: pytest, cargo test, npm test, whatever the project requires. If the tests pass, the script exits with code 0, and Claude Code terminates normally. If the tests fail, the script blocks termination via one of two mechanisms [Claude Code Hooks Reference, 2026]:
Exit code 2: Claude Code re-enters the loop
JSON response:
{"decision": "block", "reason": "3 tests failed in auth module"}. Claude Code re-enters the loop, and the reason string goes into context
This is the absolute cost floor of the maker/checker spectrum. The check is a shell script. It runs real tests. Zero model tokens. No platform dependency. No SDK required.
It is also the most mechanical: the checker is a script, not a model. It can run tests and check exit codes. It cannot evaluate prose quality or make judgment calls.
When to Use PreToolUse Defer or Mechanical Stop Hooks
Reach for the PreToolUse Defer Hook when:
The check is deterministic and can be expressed in Python code
Token cost per iteration is a hard constraint
You are building on the Claude Agent SDK and need to intercept specific tool calls
The verification is mechanical: compilation, testing, schema validation, file existence
Reach for the Mechanical Stop Hook when:
You are using Claude Code (not a custom SDK loop)
The check is a test suite or shell-script-expressible condition
You want the absolute minimum additional cost (zero tokens, no platform dependency)
The check is a terminal condition: “was the whole task actually completed?” rather than “is this individual tool call safe?”
Neither pattern evaluates subjective quality. If “good enough” cannot be expressed in code, you need a model checker. But if it can (and more often than you might expect, it can), these patterns deliver genuine architectural separation at the lowest possible cost.
AI Agent Verification Decision Matrix: Matching Checker to Verification Need
Three implementations. Each is a genuine maker/checker split. Each delivers architectural separation between the model that produces output and the process that verifies it. But they differ dramatically in cost, quality ceiling, and what kinds of checks they support.
This is the core question every agentic AI team faces: which checker fits the verification you actually need?
Here is the full comparison, including the anti-pattern you should not use:
Evaluator-Optimizer: Checker type: separate model call. Cost: highest (every iteration). Quality ceiling: highest: can evaluate subjective quality. Best for: style, clarity, nuance, rubric-needing judgment.
CMA Outcomes: Checker type: Anthropic-managed grader. Cost: medium (platform overhead). Quality ceiling: good; rubric-constrained model reasoning. Best for: platform-deployed agents, markdown-expressible criteria.
PreToolUseDefer Hook: Checker type: Python code. Cost: zero additional model cost. Quality ceiling: limited to what code can verify. Best for: compilation, testing, schema validation, file existence.Mechanical Stop Hook: Checker type: shell script. Cost: zero tokens. Quality ceiling: limited to what a script can verify. Best for: test suites, terminal completion checks, Claude Code loops.
Self-check (don’t): Checker type: same model. Cost: low (single call). Quality ceiling: false completeness; structurally unreliable. Best for: nothing. This is the anti-pattern.
Sources: Evaluator-Optimizer: lastmile-ai/mcp-agent, GitHub 2026; CMA Outcomes: Anthropic CMA Outcomes Docs, 2026; PreToolUse Defer Hook: Anthropic Claude Agent SDK Hooks Docs, 2026; Mechanical Stop Hook and false completeness anti-pattern: selmakcby/loop-engineering, GitHub 2026.
If you are a paid subscriber, thank you. Your support makes this work possible.
If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.
Free subscribers typically receive access to the full versions of paid articles after one to two months.
The Three Forms of Maker/Checker Architectural Separation
The comparison above shows cost and quality, but the more important dimension is what kind of separation each strategy provides. There are three distinct forms:
Different model (Evaluator-Optimizer): A separate model call with its own context, its own system prompt, and potentially its own model tier. The evaluator is not the optimizer; it was not involved in producing the output, so it has no investment in the output being “done.” It can see things the optimizer cannot see because it is looking from the outside. lastmile-ai/mcp-agent, GitHub 2026
Different managed process (CMA Outcomes): The grader is not a model call you make; it is an Anthropic-managed process that the platform routes output to. You write the rubric; the platform manages the grader lifecycle. The separation is enforced at the infrastructure level: the grader is isolated from the agent that produced the output, so it is not influenced by the agent’s reasoning. Anthropic CMA Outcomes Docs, 2026
Different execution layer (PreToolUse Defer Hook, Mechanical Stop Hook). The checker is not a model at all. It is Python code or a shell script: a fundamentally different category of thing. It cannot share the model’s reasoning, its biases, or its blind spots. It runs deterministic logic. Same input, same output, every time. Anthropic Claude Agent SDK Hooks Docs, 2026; selmakcby/loop-engineering, GitHub 2026
All three forms satisfy the core doctrine. What changes is what the checker can evaluate, and what it costs.
How to Choose Your AI Agent Checker Strategy
The choice between strategies comes down to what you are actually verifying.
Subjective quality: use a model checker
If the criterion requires language understanding (”is this explanation clear?”, “does this code follow our conventions?”, “is this summary accurate to the source?”), you need a model checker. Only a model can reason about subjective quality. AI agents cannot reliably evaluate their own prose any more than they can reliably evaluate their own code.
Between the two model checker options: if you are running on Anthropic’s managed deployment platform, CMA Outcomes reduces operational burden Anthropic CMA Outcomes Docs, 2026. If you need fine-grained control over the evaluator’s system prompt, model selection, and feedback structure, use the Evaluator-Optimizer pattern from mcp-agent lastmile-ai/mcp-agent, GitHub 2026.
Mechanical correctness: use a code checker
If the criterion is deterministic (”does the code compile?”, “do all tests pass?”, “do all expected files exist?”, “does the output match this schema?”), you do not need a model checker. Code is more reliable, faster, and free.
Between the two code checker options: if you are in a Claude Agent SDK loop and need to intercept specific tool calls, use the PreToolUse Defer Hook Anthropic Claude Agent SDK Hooks Docs, 2026. If you are using Claude Code and need a terminal completion check, use the Mechanical Stop Hook selmakcby/loop-engineering, GitHub 2026.
Cost-constrained paths: use a defer hook or stop hook
If token cost is a hard constraint (long loops, high-volume workloads, cost-sensitive pipelines), avoid model checkers for checks that code can make. A test runner that costs zero additional tokens is always cheaper than an evaluator that costs another model call.
In practice, code-based checkers cost less per invocation than model-based checkers because they require no additional inference calls; exact savings depend on model selection, task complexity, and call volume.
The Core Maker/Checker Doctrine, Restated
You have seen three implementations. You have the decision framework. The underlying principle is the same whether you are building agentic AI pipelines with mcp-agent, deploying through CMA, or wiring hooks into a Claude Agent SDK loop:
Never let the AI verify its own done. Maker and checker must be different things. selmakcby/loop-engineering, GitHub 2026
“Different” is now precise. It means one of three things:
A different model (the evaluator is not the optimizer)
A different managed process (the grader is platform-managed, not the agent)
A different execution layer (the checker is code, not a model)
None of these is a prompt. The doctrine is not “use a better prompt for self-checking.” The doctrine is “eliminate self-checking by design.” An agent loop with no checker, or with a self-check, has no reliable termination condition. False completeness is not a bug you patch. It is a structural void you fill by adding a real checker. selmakcby/loop-engineering, GitHub 2026
Choose the checker that fits the verification. Ship loops that actually finish.
Loop Engineering Is a Subset of Harness Engineering
Loop engineering matters because the loop is where the agent’s behavior becomes real. The loop decides when the model runs, what tools it can call, what feedback it receives, when it retries, and when it claims the task is finished. That makes loop engineering one of the most important disciplines inside production agent design.
But loop engineering is not the whole system. It is a subset of harness engineering.
Harness engineering is the larger discipline around the model: context assembly, memory, tool permissions, guardrails, observability, audit trails, human review, recovery paths, and verification gates. The loop is one part of that harness. It is the execution cycle. The harness is the operating environment that makes the cycle safe, observable, and reliable.
False completeness is a perfect example of why the distinction matters. You cannot solve it by making the loop more confident, more verbose, or more reflective. You solve it by adding a harness-level constraint: the maker does not get to verify its own done.
That is the larger lesson. Loop engineering gives the agent motion. Harness engineering gives that motion boundaries, evidence, and accountability. If you want agents you can trust with production work, you need both, but the loop must live inside the harness, not replace it.
Related Articles
If this article sparked ideas you want to follow further, these pieces cover the patterns directly:
Why Single-Agent AI is Dead: Inside Anthropic’s New Blueprint for Long-Running Agents: Introduces the adversarial three-agent GAN-style architecture (Planner, Generator, Evaluator) that is the multi-agent implementation of the maker/checker doctrine: the Evaluator role is structurally identical to the checker described here. Directly complements the evaluator-optimizer pattern covered in Implementation 1.
Architecting Production-Grade Agents through LLM Orchestration and Agentic Loops: The implementation companion for readers who want to go deeper into production agent loop architecture: covering checkpointing, tool routing, and the infrastructure choices that determine whether a loop can self-correct. Provides the broader agentic context for the maker/checker split strategies covered here.
Harness Engineering vs Context Engineering: The Model is the CPU, the Harness is the OS. Establishes the architectural framing that underpins the maker/checker doctrine: the harness is the operating system the model runs inside, and verification gates are harness-level responsibilities. Explains why prompt-level self-checking cannot substitute for structural separation between maker and checker.
Giving Your AI a Computer: An Introduction to the Claude Agent SDK Covers the Claude Agent SDK that powers Implementation 3 in this article: the PreToolUse hook system, tool execution model, and agent loop lifecycle. Essential background for readers who want to understand how the zero-cost programmatic checker integrates with the SDK’s execution model.
The Eleven Patterns Behind Every Production Agentic System (And Where JSON Schemas Actually Earn Their Keep) Catalogs the recurring structural patterns in production agentic systems, including output validation and quality-gate patterns that are the natural extension of the maker/checker decision matrix presented here. Readers who have internalized the three-way split will find the full pattern library here.
References
If this helped you, please consider subscribing to my Substack newsletter.
Likes, comments, and shares really do make a difference. They help grow the channel, support the work, and get these ideas in front of more builders who are trying to understand AI agents, harness engineering, and production AI systems.
Thank you for reading and for helping the work reach more people.
About the Author — Claude Certified Architect
Rick Hightower is a former Senior Distinguished Engineer at a Fortune 100 company, focusing on delivering ML / AI insights to front-line applications, and a practitioner building multi-agent production systems. Follow him on SubStack and Medium for more hands-on agent engineering content. You can also book him to speak and train your team: Check out Rick Hightower’s SpeakerHub.
Rick Hightower helps companies become AI-first through practical mentoring, executive and team training, and custom AI solution development. He is a former Senior Distinguished Engineer at a Fortune 100 company, where he focused on bringing ML and AI insights into real front-line business applications.
Subscribe to Rick’s newsletter to see videos and guides.
Rick is a Claude Certified Architect, AI systems practitioner, and builder of production multi-agent systems. He is currently working on authoring a book on Harness Engineering with Manning Publishing. He created Skilz, a universal agent skill installer supporting 30+ coding agents, including Claude Code, Gemini, Copilot, and Cursor, and co-founded one of the largest agentic skill marketplaces.
Today, Rick and the Spillwave team work with leaders and teams who want to move beyond AI experiments and build real AI capability inside their companies. He helps organizations adopt AI safely, train their people, redesign workflows, and build practical AI systems that create measurable business value.
Ready to make your company AI-first? Connect with Rick on LinkedIn, Substack, or Medium, book him to speak or train your team, or visit Spillwave to explore mentoring, training, and custom AI solutions for your organization.