Loop engineering is verifier engineering

An agent is a while loop. Six lines of Python. Akshay Pachaar put the code in his article, a version of the Daily Dose of DS piece by his co-founder Avi Chawla, and said the quiet part out loud: nobody is competing on the while statement. So when half your timeline announces that loop engineering is the skill of 2026, the skeptic has a fair question ready. Is this just a for-loop with a content strategy?

It's a fair question, and the honest answer is worse than the skeptic expects. The loop really is trivial. The pattern really is old: Anthropic documented evaluator-optimizer in December 2024, eighteen months before it went viral under a new name. And when METR measured whether elaborate loops beat a naive one, Claude Code beat plain ReAct in 50.7% of bootstrap samples. A coin flip. Codex lost to its baseline. Quadrupling the token budget from 8M to 32M "barely changed" the result.

So the deflation is correct. The loop is not the product. But that is where the interesting question starts, not where it ends. Because every article in this genre agrees on the one word that carries the weight, and agreeing on it turns out to be worth almost nothing.

Verify.

The loop is trivial. That is the most interesting thing about it.

METR's result is the sharp one: two of the most heavily engineered agent harnesses on earth could not beat act, observe, repeat to statistical significance. A 27-paper synthesis from Oxford, published a week ago, lands independently in the same place: "additional scaffolding does not consistently improve reliability."

Four results from three studies. If harness cleverness were the active ingredient, these experiments were built to find it.

50.7%Claude Code vs plain ReAct

Bootstrap samples where the elaborate harness beat act-observe-repeat. A coin flip.

METR, Feb 2026

14.5%Codex vs Triframe

Same test, other direction. The engineered loop lost to the baseline.

METR, Feb 2026

4.9 ptsspread across four harnesses

Terminal-Bench 2.1, top five: 83.8% down to 78.9%, across Claude Code, Codex, Terminus 2 and Cursor CLI.

tbench.ai, Jun 2026

~0bought by 4x the tokens

Raising the budget from 8M to 32M left the measurement “barely changed.”

METR, Feb 2026

Now hold that next to the thing that does move: grounded feedback. Huang et al. tested self-correction without external feedback and every cell went down or sideways. GPT-3.5 on CommonSenseQA fell from 75.8% to 38.1% after being asked to check its own work. That is a model talking itself out of correct answers, at three to five times the token cost. Their diagnosis is the load-bearing sentence in the literature: earlier papers showed self-correction working, but "the improvements in these studies result from using oracles to guide the self-correction process, and the improvements vanish when oracle labels are not available."

Accuracy after each round of self-correction, with no external feedback. Six benchmark-and-model pairs. Not one of them goes up.

Reflexion looks like the counterexample and is the confirmation. Its wins come from executed unit tests and binary environment signals, not introspection. On MBPP, same technique and same model, it went from 80.1% to 77.1%. Everyone quotes the 91% HumanEval number. Nobody quotes the regression sitting one row down in the same table.

The boxes are identical. The only difference is what the arrow on the way back is carrying.

Closed over its own opinion

Costs 3-5x the tokens to get worse. Huang et al: 75.8% to 38.1%.

Closed over reality

Compounds. Every Reflexion win runs on an executed test.

One caveat, and it cuts at my own argument: Huang and Reflexion are 2023 papers on 2023 models, and nobody has published a clean replication on frontier reasoning models. The mechanism is echoed by 2026 work. The numbers are stale. As of mid-2026 that is the biggest hole in the anti-self-critique case, and anyone who tells you otherwise is quoting a three-year-old table at you like it's a law of physics.

Everyone already crowned the verifier. It didn't help.

Here is what surprised me when I read all eleven instead of skimming them.

They don't bury the verifier. They crown it. Anatoli Kopadze, whose piece has 16.4 million views, makes it the final block of his taxonomy and writes: "This is the one block that decides whether the loop helps you or just spends your money. Everything else is plumbing. This is the part that makes it real." Khairallah AL-Awady: "The verifier is the point of the whole thing." Addy Osmani: the model that wrote the code is "way too nice grading its own homework." Akshay: "done should mean the tests pass, not the agent feeling good about its work."

I had a whole section drafted about how the industry buries the verifier under plumbing. It's false. Kopadze got to my thesis eleven days before I did, in almost my words.

So this is not a case of anyone missing the obvious. It is worse. It's universal agreement that changes nothing, and the proof is sitting inside the most-read article itself. Kopadze's four-box test gates the whole decision to build a loop on box two: something must be able to automatically reject bad output. Then his hands-on exercise, the thing he tells 16.4 million readers to paste into Claude, is a prompt that instructs the model to "score the result 1-10 on each criterion. Be brutally honest." That is the agent grading its own homework. It fails his own box two, one screen after he wrote it.

He isn't being lazy. He is doing the thing the word lets you do. "Add a verifier" sounds like a design and is actually a noun. You can nod at it, crown it, put it in bold, and still ship self-grading, because nothing in the slogan tells you what a verifier has to be.

A verifier needs two independences. Yours probably has one.

Here is the distinction I could not find anywhere in 43 million views, and it falls straight out of the primary docs.

Independent context. The judge is not the maker. A fresh model with its own window, uncontaminated by the reasoning that produced the work. Anthropic's Prithvi Rajasekaran has the cleanest statement of why: "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work, even when, to a human observer, the quality is obviously mediocre." His fix is a lever, not a vibe: "Tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work."

Independent evidence. The judge can go look for itself. It runs the tests. It reads the files. It inspects the artifact rather than the story told about the artifact.

Those are not the same property, and almost every loop people ship has the first without the second.

Take /goal, the most recommended primitive in this corpus, the one people cite as proof the maker/checker split is solved. Read the docs. The evaluator is a fresh model, defaulting to Haiku, which is real independence of context. And then: "It does not call tools, so it can only judge what Claude has already surfaced in the conversation." And: "It doesn't run commands or read files independently, so write the condition as something Claude's own output can demonstrate."

Sit with that. Your verifier cannot run your tests. It reads Claude's account of having run your tests. /goal all tests pass works for a specific and fragile reason the docs are admirably honest about: Claude runs the tests, the output lands in the transcript, the evaluator reads the transcript. The gate is downstream of the maker's self-report. Independent context, borrowed evidence.

That is a design constraint, not a bug, and Anthropic's docs are more honest about it than any article celebrating the feature. Managed Agents' grader is the other design: it inspects artifacts in a separate window, which the docs say exists "to avoid being influenced by the main agent's implementation choices." Both independences. And a test suite has both, holds no opinions, and costs nothing per run.

Independent context: is the judge a different mind? Independent evidence: can it go and look for itself?

contextevidence

Self-critique

“Now review your work.” The maker grades its own homework and gives itself an A.

contextevidence

A reviewer with tools, same thread

Sees the reasoning that produced the work, so it inherits its blind spots.

contextevidence

/goal, an LLM judge

Fresh model, but it reads the transcript. It cannot run your tests. It reads the claim that they ran.

contextevidence

A test suite. A grader that opens the artifact.

Its own window, its own evidence, no opinion to flatter.

The convergence is what convinced me. The loudest critics of multi-agent architecture are Cognition, who published "Don't Build Multi-Agents" in June 2025. In April 2026 Walden Yan revised it, and of the three patterns he now endorses, the one with numbers attached is a dedicated code-review agent: an average of two bugs per PR, roughly 58% of them severe. The detail that matters is that it works better when the reviewer and the coder do not share initial context.

Anthropic ran it forward. Cognition ran it backward. Both landed on the same object.

Loops work exactly where checking is cheaper than doing

This is why code got loops first, and it isn't because programmers are early adopters.

Where a check is cheap, objective, and mechanical, the loop compounds: tests pass or fail, the type checker is not negotiable, Lighthouse returns a number, the build compiles or it doesn't. Where "done" is a judgment call, the loop does not fail loudly. It produces confident, plausible, endless output and calls it finished.

One axis. On the left, something that is not the agent can reject the work for free. On the right, only a human can, and the loop cannot tell the difference.

a machine can fail itonly a person can

• Type check
  tsc exits non-zero
• Lint pass
  eslint exits non-zero
• Failing test
  the suite is red or green
• Lighthouse budget
  the score is a number
• Dependency bump
  CI passes or it doesn't
• Flaky test repro
  it reproduces or it doesn't
• SEO pages
  traffic, eventually
• API design
  someone senior has a feeling
• Prose
  taste
• Architecture
  you find out in a year

Kopadze's four-box test is genuinely the best framework in the corpus. Two of its boxes are economics and capability: does the task repeat at least weekly, can the agent do the work end to end. The other two are the same box asked twice. Something must be able to automatically reject bad output, and "done" must be objective rather than a judgment call. Those are one requirement wearing two hats, and it is the only one that matters.

Which brings me to the most interesting artifact I found. Jason Zhou reports a production loop shipping "20-40 high-quality pages every day" of SEO content, "already driving traffic to the company without me looking at it." He presents this as the triumph. Read it again and ask the only question that matters: what rejects a bad page? Traffic does. Eventually. The verifier is a metric, the metric is the target, and Goodhart has been waiting at the end of that sentence the whole time.

I'm not dunking on him, and the reason matters. His artifact system is the most concrete engineering in the corpus, his inner-loop/outer-loop split is a better taxonomy than most, and his sitemaps are consistent with the volume he claims. That is exactly why it's the perfect example. A serious engineer, running a real system that does what he says it does, with a verifier made of a number that can be satisfied without the work being good. The loop is not failing. That's the point. It will hit 20-40 pages a day forever, and nothing in it can ever tell him whether that was worth doing.

The trap runs the other way too. When the verifier is the target, agents game it: delete the failing test, special-case the assert, satisfy the letter. A gate you never spot-check is a gate that rots. And yes, this is the honest hole in my own argument. Who verifies the verifier? Nothing here makes that go away. What it does is move the problem somewhere you can put your hands on it. A rotting test suite is a thing you can read, version, and fix on a Tuesday. "The model seemed confident" is not.

The loop discourse is a loop with no verifier

I read eleven articles about agent loops to write this. Roughly 43 million views between them. Then I did what none of them did to each other. I checked.

Views as of 14 July 2026. First-hand Repackaged

@AnatoliKopadze

16.4Mpromotes a product, undisclosed

@0xCodez

8.4Mdigest of Osmani's essay

@ClaudeDevs

5.9Mthe team that built it

@0xwhrrari

2.6M

@addyosmani

2.3Mthe origin essay

@RLanceMartin

2.2Mran the experiments

@ArchiveExplorer

1.9Mthe one that got Ralph right

@akshay_pachaar

1.7M

@addyosmani

900K“Own the Outer Loop”

@jasonzhou1993

500Kruns them in production

@eng_khairallah1

350Ksame roadmap, 20 steps

The single most-viewed explainer, at 16.4 million views, turns into an ad for a Telegram bot at the line "for 99% of everyday ones, there is already a ready, dead-simple solution," and never turns back. I counted: everything from that line to the end is 31% of the body by word count. The link carries an attribution code, t.me/mira?start=social_x_200626_howtostart, which is how a partner counts arrivals. The piece never says what the relationship is. It out-viewed every first-hand account in the corpus combined.

Two articles, with 24.8 million views between them, describe Geoffrey Huntley's "Ralph Wiggum loop" as a failure mode where an agent quits early and bills you in silence. Huntley invented Ralph on purpose, advocates it, and documented its real failure modes himself. It is while :; do cat PROMPT.md | claude-code ; done. He reported, in July 2025, an engineer clearing a $50k contract deliverable for $297 with it.

Geoffrey Huntley invented Ralph on purpose and advocates it. Everything below calls it a failure mode.

Jul 2025ghuntley.com/ralphthe primary source

while :; do cat PROMPT.md | claude-code ; done

“Ralph can replace the majority of outsourcing at most companies for greenfield projects.”

↓ never consulted by anything below ↓

1. 8 Jun 2026AlphaSignal newsletter

   “Engineer Geoffrey Huntley documented this failure mode.”

   ↓ copied, not checked ↓

2. 9 Jun 2026A 14-step roadmap8.4M views

   Same wording. Same wrong claim. Plus a 50% break-even statistic that traces to no source at all.

   ↓ copied, not checked ↓

3. 20 Jun 2026The most-viewed explainer16.4M views

   “Loops do not crash, they bill you in silence.” Same inversion. Same invented statistic.

That error has an alibi: an AlphaSignal newsletter from June 8 describes Ralph the same wrong way, and both articles are downstream of it. The invented statistic they also share, a 50% accept-rate break-even, traces to no source at all. I can't tell you who copied whom. What I can tell you is that a wrong claim about how to build loops rode 24.8 million views without anything in the chain stopping it, and not one of the eleven describes Ralph as what its author says it is. The closest, at 1.9 million views, links Huntley's repo and gets his fresh-context pattern right, then lists "Ralph Wiggum loops" in its own table of anti-patterns.

A digest of Addy Osmani's essay out-performed the essay by nearly four to one, and its author cites Addy throughout. This isn't plagiarism, it's distribution: a well-made summary beat the thing it summarized, and a month later someone shipped the same roadmap with the step count raised from fourteen to twenty.

Even the best sources thin out under a check. The most rigorous post in the corpus, by an Anthropic engineer with a genuine track record in evaluation, has a headline of "~6x more improvement" on a metric where lower wins. Six times what? The post doesn't say, and I couldn't reconstruct it.

Here's the pattern, and it's more damning than the version I first wrote. The numbers that survive a check are the borrowed ones: Anthropic's own 8x merge rate, an arXiv paper's 520-of-17,022, a published RCT's 50-versus-67. The numbers the authors generated themselves are the ones that don't: the 6x, the 50% break-even, the six-day rewrite that names no engineer and no language. Relaying someone else's measurement is the only thing this genre does reliably, and relaying is not checking, which is how a wrong claim about Ralph rode 24.8 million views with a citation attached the whole way.

That is the argument, delivered by the subject. This genre is a content loop with engagement as its reward function, and engagement is exactly the verifier this post has been describing: independent context, no independent evidence. A fresh judge, reading the transcript, with no ability to run the tests. So it rewards the remix over the source, the confident roadmap over the careful experiment, the funnel over the engineer.

The most useful piece I read is the Claude Code team's own, and it's the only one that opens by admitting the discourse is mush: "If you spend some time on X trying to pin down what a loop actually is, you'll come across multiple different answers." Then it does what no roadmap does. It sorts loops by what you hand off.

The Claude Code team’s taxonomy. Each rung down hands the verifier more of your judgment.

Loop	You hand off	It stops when	What is checking the work
Turn-baseda prompt	the typing	Claude judges it is done	You, in the chair, reading every diff
Goal-based/goal	the stop condition	an evaluator model says the condition holds	A fresh model reading the transcript
Time-based/loop, /schedule	the trigger	you cancel it, or the work lands	Whatever gate you wrote, running unattended
Proactiveroutines, workflows	the prompt	each task exits on its own goal	The whole gate. You are asleep.

That's the real taxonomy, and it makes the thesis obvious. The ladder is not a progression in sophistication. It's a progression in how much of your judgment you've handed to something that cannot be embarrassed. You cannot climb a rung your verifier can't hold.

You can loop the work. You cannot loop the verdict.

The skeptic's case has been walked back by its own authors, and I'd rather tell you that than win cheap. METR marked their famous "19% slower" study out of date themselves; the continuation puts both cohorts' confidence intervals across zero. What survived should worry you more. Those developers predicted a 24% speedup, experienced a 19% slowdown, and afterwards still believed they'd been sped up by 20%. The measurement moved. The metacognitive gap didn't. We are unreliable narrators of our own productivity, and a loop that runs while you sleep is a machine for widening exactly that gap.

Which is why Addy's follow-up is the most important thing anyone wrote this cycle and got a fifth of the attention of the roadmaps. He splits it into Quality, Verdict, and Answerability. "The model may write the line, but the Verdict is mine." Comprehension debt is what accrues when the loop ships code faster than you read it. Cognitive surrender is what happens when you stop having an opinion because the loop seems to have one.

I should tell you how this post was made, because it would be absurd not to. I wrote it inside one of these loops: a goal with a stop condition, sub-agents fanning out across eleven sources, a fleet of reviewer agents grading the draft against a rubric. It worked. It found the Ralph inversion and the duplicated statistic.

So here's the one thing to do tomorrow. Go read your loop's stop condition and ask which of the two independences it actually has. Write the gate as a command, not a criterion: if it can't reduce to a non-zero exit code, it isn't a verifier yet, it's a hope. Use /goal only for conditions the transcript can prove, and stop pretending it gates anything else. And if the check genuinely needs judgment, give the judge its own window and its own tools, because one without the other is theater. The dumbest verifier that touches reality beats the cleverest one that doesn't.

And then there's the part I have to tell you, because the draft of this post failed its own thesis twice.

A research agent told me the Telegram article was "roughly 40% promotion." I put 40% in the draft. Nobody had measured it. A fact-checking agent later put it at about a third, so I stopped taking numbers from agents and counted it myself: 31%. For a few hours, the paragraph indicting this genre for unverified numbers ran on an unverified number I'd inherited from an agent and never questioned. It was fluent, it was plausible, and I printed it because it sounded like the number I wanted.

The second was worse, because the agent was right. It told me Akshay's article retells his co-founder's newsletter piece. I read "uncredited" and started drafting an accusation. The word "co-founder" was sitting in the same sentence. They run the newsletter together; it's their own material. The agent reported it accurately and I misread it into the story I already liked.

The third one is the reason this section exists. An earlier draft of the passage above debunked Jason Zhou. I'd pulled his sitemap, found 94 pages where his claim implied hundreds, and written that the only falsifiable number in his post doesn't survive contact with his own website. It took thirty seconds and it felt great. A review agent then pointed out that I'd tested his blog against a claim he never made about blogs, and that the second sitemap I'd waved away on a technicality holds 931 pages timestamped across 25 days, or roughly 37 a day, sitting right inside the range he claimed. I pulled it myself. The agent was right. I had run a check, gotten the answer I wanted, and stopped.

That is the whole disease in one paragraph, and I had it. A verifier I controlled, pointed at a target I chose, returning the answer I wanted, dressed up as diligence. If a reviewer with its own context and its own tools hadn't gone and looked, I'd have published a false accusation against a named engineer in an essay about how nobody checks anything.

The loop didn't verify anything. It generated candidates. I verified, badly, and something with independent evidence caught me. Your evals were always the moat, and that distinction is the whole job now: you can automate the work, you can automate the checking of the work, and you still cannot automate the part where someone decides what good means and puts their name on it.

Design the loop. Then go build the thing that can tell it no.

Glossary

Terms

agent harness

The code around a language model that runs it in a loop: it feeds in context, calls tools, checks results and decides when to stop. The harness, not just the model, determines how reliable an agent is.

bootstrap sampling

A statistics method that repeatedly redraws random samples, with replacement, from the data you have, to see how stable a result is. It shows whether a win could just be luck.

cognitive surrender

Adopting an AI's answer as your own without checking it. Unlike handing off a chore, the person gives up the judgment itself, and often grows more confident even when the answer is wrong.

CommonSenseQA

A benchmark of multiple-choice questions that need everyday common-sense knowledge to answer, widely used to test language models' reasoning.

comprehension debt

The growing gap between the code a team ships and the code it actually understands. It builds when AI writes code faster than people read it, and is paid later in slow debugging.

Evals

Repeatable tests for an AI system: feed it inputs, score the outputs against a standard, and compare runs. They show whether a change actually helped or hurt.

Evaluator-optimizer

A loop where one model call produces an answer and another critiques it, and the feedback goes back for a revision. It only works when the critic has something solid to check against.

Fan-out

Splitting a task into parts that run in parallel, such as several sub-agents each researching a different source, then gathering the results. It is fast but needs a rule for when to stop.

Goodhart's law

When a measure becomes a target, it stops being a good measure. People and AI systems end up optimizing the number itself instead of the thing it was meant to track.

Graders

The scoring logic inside an eval: code that checks results, a model applying a rubric, or a human reviewer. Consistent graders are what let you compare runs fairly.

HumanEval

A coding benchmark of hand-written programming problems where a model writes a function and unit tests check it. It is a common score for comparing code-generating models.

Maker/checker

A control where one party produces the work and a different party reviews it before it counts. In AI loops, a separate checker catches what the maker misses.

MBPP

Mostly Basic Programming Problems: a benchmark of entry-level Python tasks used to measure how well a model writes working code from a plain description.

Multi-agent

A setup where several AI agents, often a lead plus helpers, split a task and report back. It can cover more ground at the cost of more compute and coordination.

Oracle (feedback)

An outside source of ground truth, such as a test result or a known correct answer, that tells a model whether it was right. Without one, models asked to review their own work rarely improve.

Ralph Wiggum loop

A technique where a coding agent is run in a bare loop and fed the same task again and again until the work is done. It is named after a Simpsons character and was created by Geoffrey Huntley.

Randomized controlled trial

A study that randomly assigns people to a treatment group or a comparison group, so differences in outcome can be credited to the treatment rather than to who chose it.

ReAct

A pattern where a language model alternates between reasoning in text and taking actions such as calling a tool, then reads the result before continuing. Many agent loops build on it.

Reflexion

A method where an agent writes a short note on why an attempt failed and reads it on the next try, learning from mistakes without any retraining.

Reward function

The rule that scores how well an AI system did, which training or search then tries to push upward. A poorly chosen score gets exploited instead of satisfied.

Token (LLM)

The chunk of text, often a word or part of a word, that a language model reads and writes. Context limits and pricing are counted in tokens, so they measure how much text a model handles.

Tools

Claude

Anthropic's family of large language models and the assistant built on them, available in apps and through an API.

Claude Code

Anthropic's agentic coding tool that reads a codebase, edits files and runs commands, in the terminal, IDEs, a desktop app and the web.

Codex

OpenAI's coding agent, which runs in the terminal and is also offered in IDEs and the cloud.

GPT-3.5

An older OpenAI language model family that understands and generates natural language or code, offered through the OpenAI API for chat and other text tasks.

Haiku

The fastest model tier in Anthropic's Claude family, used where low latency and low cost matter more than maximum capability.

Lighthouse

An open-source tool from Google that audits web pages for performance, accessibility, SEO and other quality measures.

Managed Agents

Anthropic's hosted agent harness. It runs Claude as an autonomous agent in managed infrastructure, with tools, sandbox and session state handled for you.

Telegram

A cloud-based messaging app for mobile and desktop, with a bot API that lets programs send and receive messages.