I've been running a lot of gauntlet loops lately. If you haven't met the pattern, it's Matt Shumer's coinage-- his own write-up is the source to read, and Claude of Duty is the demonstration that made it famous: a Call of Duty-style FPS in Three.js, built from a single published prompt. The shape: a lead agent gets an ambitious goal and a concrete reference for quality, decomposes the work into separately judgeable pieces, builders build, and fresh-context critics compare the actual artifact against the reference-- blind A/B where possible-- looping on the largest gap until the thing meets the bar or you hit your cost ceiling. Shumer's own emphasis is that the bar is the most important part: "make it amazing" fails, actual Call of Duty screenshots work. (There's also a broader survey of the loop-engineering family if you want the landscape.) It's a genuinely good pattern, and the separation of builder from critic fixes the "grading your own homework" problem I've ranted about before.
Read the "Honest assessment" section of the Claude of Duty README, though, before you take it as proof of anything. Eleven adversarial critics scored the frames 3.59 → 4.14 → 4.05 → 5.05 out of 10 against real Call of Duty, and in a blind A/B every critic in every round picked the real frame. That plateau reads to me as a measurement ceiling more than an intelligence one, and the README hands you the evidence: three straight rounds of critics called the weapon "untextured" when it was specular-dominated with a diffuse term that measured L=26 against a shipped L=67. The loop burned three laps iterating on a misread. Better critics wouldn't have caught that. A light meter would have.
Which matches where my own loops spend their time: the space between the two roles. Critics squint at an artifact trying to decide how to compare it to the reference, and builders get feedback like "the pacing feels off in the second section" and have to guess what, mechanically, to change. The loop's speed limit isn't intelligence. It's measurement.
Racing teams sorted this out a long time ago. They hired people who build test rigs, staging environments, and dashboards, and then let the driver drive. So my gauntlet loops now have a third role, and I've been calling the variant what it obviously is:
The pit crew loop
Builder, critic, and pit crew. The pit crew doesn't drive and doesn't judge the driving. Its entire job is instrumentation and tooling-- making the next lap faster and the judging more accurate. And it works the whole race, not just the garage: setup before lap one, then a pit stop after every critique lap, scoped by whatever that lap just surfaced:
-
Turn the reference into executable checks. Before the builders start, the pit crew's first task is converting the quality bar into acceptance and e2e tests wherever it can. "Matches the reference" becomes a suite that runs in seconds, and a whole class of critic deliberation just evaporates-- the artifact passes or it doesn't, and critics save their judgment for the parts that genuinely need taste.
-
Build purpose-specific harnesses. Replay rigs, fixture generators, a screenshot bot that produces the blind A/B pair automatically, a script that spins the artifact up in the exact configuration critics need to see. Whatever removes manual steps between "builder finished" and "critic judging real output."
-
Put numbers on the vague stuff. Latency budgets, bundle sizes, phase timings, error rates. Numbers aren't the whole story, obviously. But "the largest gap"-- the thing the gauntlet iterates on-- should get picked by measurement wherever it can be, instead of by whichever critic wrote the most confident paragraph.
-
Give the critics instruments too. The judging half of "faster laps, better judging" is the half people skip. A measured report delivered next to the frames, a comparison plate that puts this lap beside the last one, a capture mode the critique keeps wishing it had-- tooling the critic is pit crew work of the first rank, because a critic with a light meter stops writing three rounds of "the weapon looks untextured" at a specular problem.
And one standing rule that does most of the work: any note a critic gives twice becomes a check the pit crew automates. Complaints are instrument requests in disguise. The second time a critic writes "the export is missing the header row again," that sentence is done being feedback and starts being a test. That's what the pit stops are for-- each one takes the lap's complaints and fumbles and turns them into the next lap's instruments.
That rule only reaches the mechanical notes, though, and the taste-shaped ones are the actual bottleneck. "The pacing feels off" never becomes an assertion no matter how long you stare at it. So those get promoted in the other direction: a note that keeps coming back and refuses to automate goes into the reference as a new exemplar-- one more thing in the blind A/B pile, chosen because critics kept reaching for it anyway. Three buckets, then. Automate it, exemplify it, or admit it was a one-off and let it go.
This keeps working on me in real life
When I put a pipeline in charge of producing documents that another system would parse, the unlock wasn't a better writer agent-- it was a little harness that renders the finished PDF and then parses it back out the way the consuming system would. Critics stopped debating font vibes and started reading parse output. In Multicam Toolbox, per-run metrics with phase timing are what turn "the tool feels slow on long episodes" into "the diarization phase is 80% of the wall clock, go stare at that." Same shape both times: the instrument did more for iteration speed than any amount of added cleverness.
The failure modes, because there are two good ones
First: the pit crew can gold-plate. An agent asked to build tooling will, with complete sincerity, construct a beautiful dashboard nobody requested-- the gauntlet's cost boundary has to apply to instruments too, and "does this measurement change what the builder does next lap" is the test. Second: builders optimize what you measure, with all the monkey-paw energy that implies. The fresh-context integration critic at the end stays essential precisely because it judges the whole artifact and not the scoreboard. Instruments narrow the argument; they don't end it.
So why does this need its own seat?
Because of where the tokens go. In an agent loop they get spent on judgment or on measurement, and only one of those compounds. Judgment evaporates when the session ends-- the critic who understood exactly why lap eleven was worse than lap nine is gone, and lap twelve starts from a paragraph. A test, a harness, a metric: still there on lap forty, still there next month when a completely different loop runs against the same project. Every other role's output is consumed by the lap it was produced for. The pit crew's output is the only thing that makes every future lap cheaper, and work like that never happens when it's something builders are supposed to squeeze in on the side.
Full disclosure on how much of this is actually wired up: the pit crew lap has been run by hand so far. I do the instrumentation pass myself, ahead of the pipeline, and then let the build/critique cycle go. Making it a real phase kind in Ripline is the next job, along with a head-to-head worth the name-- same goal, same reference, one pipeline with a pit crew lap and one without. Right now my evidence that any of this helps is that it feels faster, which is precisely the sort of claim a pit crew exists to kill. I also have no idea yet where the crew's budget should cap out.
If you try a pit crew loop, or you've got a better name for it, email me. Especially if you've got a better name. I've been staring at this one long enough to love it, which is exactly when someone should stop me.