Get notified when Mark publishes

Receive a notification each time a new blog post is published. You will receive a confirmation email to approve your email. Unsubscribe anytime.

Mark Smith

Mark Smith

· 11 min read

Why My AI DevOps Team Runs on a Ralph Loop

My AI DevOps board looked frozen for twelve hours. The fix was not smarter orchestration, but a Ralph loop with real hard stops.

Why My AI DevOps Team Runs on a Ralph Loop
Share

A Ralph loop lets AI agents pull their own work continuously while the human supervises the loop, sets constraints, and approves irreversible steps.

Key takeaway

  • A continuous AI agent loop works best when agents pull owned work from live state while the human supervises above the loop.
  • My DevOps board looked frozen for twelve hours, but the agents were busy. The problem was a mis-tuned loop and a dashboard that hid the work.
  • Three hard stops make unattended agent loops practical: an iteration cooldown, no-progress detection, and a hard dollar budget.

I run my business on a fleet of AI agents on a Linux virtual machine in Azure, orchestrated by an open-source platform called OpenCLAW. One slice of that fleet is my DevOps department: ten agents that build, test, ship, and operate the software the rest of the business depends on. They have a manager. They have a backlog. They have a release process. They just happen to be software.

One morning, the board had not moved in twelve hours.

This is the story of why that happened, why my first explanation was wrong, and how a deceptively simple pattern, the Ralph loop, got ten agents driving their own work around the clock.

Meet the team: Tara and the CCDC#

My DevOps department runs as ten agents. Nine are specialists, each owning one phase of the delivery lifecycle: planning, design, code, test, release, deploy, security, monitoring, and knowledge. The tenth is Tara, the engineering manager. Tara does not write code. She triages, prioritises, and assigns, the way a good engineering manager does.

They coordinate through a single GitHub Projects board I call the Cloverbase Control and DevOps Centre (CCDC). Every piece of work is a card with a status, a phase, a priority, and an owner. The board is the single source of truth for what the team is doing. If it is not on the board, it is not happening.

That last sentence turned out to matter more than I realised.

The symptom: a frozen board#

I checked the CCDC one morning and the comment history on the active work showed a dead zone from roughly 7:00pm to 5:00am. Ten hours, then twelve, with nothing visibly progressing. No cards moving columns. No epics closing. From the outside, the team looked asleep.

The obvious conclusion, and the one I reached first, was that the agents were idling. My orchestration looked like theatre: agents going through the motions of being "engaged" without doing real work.

I was wrong. The only reason I found out is that I make a habit of debugging against live state rather than my own mental model.

I almost fixed the wrong thing#

Before rebuilding anything, I looked at the virtual machine, the GitHub API, and the orchestration logs. Not at what I assumed was happening. At what was happening.

Tara's hourly orchestration had been running every single hour, all night, and reporting itself engaged. The specialists had been shipping genuine pull requests with passing tests. The work was real. My "it is all theatre" theory was flatly wrong.

There were three real causes, and none of them was an idle agent:

  1. The sweep only advanced cards marked "Ready", and the board almost never has a "Ready" card. Work tends to live in "In Progress" or "In Review", and that work was not actively driven. So most sweeps concluded "board attended, nothing actionable" while real work in progress sat untouched.
  2. Tara could not read the board mid-task. Through a credential and tooling quirk, the agent process that ran the hourly review did not hold the token it needed to query the board on the fly. When she improvised a query, it failed, and she reported "no board access". She was, in effect, managing a team while blindfolded.
  3. The big epics never closed. One parent epic had over 300 comments. Child slices were completing underneath it constantly, but because the parent stayed open, the board looked frozen even as work shipped beneath the surface.

The lesson I keep relearning: when a system says "nothing is happening", it usually means the wrong thing is happening out of sight, and your dashboard is not telling you. If I had trusted my first instinct, I would have rebuilt a system that was fundamentally sound.

The pattern: the Ralph loop and the Mayor#

Once I understood the real problem, the fix pointed straight at a pattern that keeps surfacing in the agent-engineering community: the Ralph loop, named and popularised by Geoffrey Huntley.

The Ralph loop is almost insultingly simple. Read the plan, pick the next task, implement it, test it, commit it, clear the context, and repeat. State lives in text and Git, not in a fragile, ever-growing context window that degrades over hours. It rests on four principles: keep the context constrained, build in backpressure, let the agent act autonomously, and keep human oversight above the loop.

Two newer ideas layer on top. The first is multi-agent supervision: a continuous orchestration loop overseeing worker agents in a "Mayor and patrol" shape, where one role keeps order and assigns, and the workers go and do. The second is the /goal primitive now showing up in tools like Codex and Claude Code: run until a validator confirms the end state is reached, with the success criteria, constraints, iteration ceiling, and budget specified up front.

Here is the realisation that reframed the whole problem for me: Tara already was the Mayor, and the specialists already were the patrol. I had not built the wrong architecture. I had built the right one and mis-tuned it.

The operating split is simple:

RoleJob
MayorTara triages and assigns work. She does not do it.
PatrolThe specialists pull and drive their own queues to done.
GatesRelease, deploy, and security stage everything up to the irreversible step, then stop and wait for me.

What we built: the patrol layer#

Two changes fixed the operating model, one small and one structural.

The small one: I re-tuned Tara to drive all open work on every sweep, instead of waiting for the perpetually empty "Ready" column. I also stopped her improvising board queries by embedding the entire board snapshot directly into her instructions each cycle. No more managing blindfolded.

The structural one is the part I am proud of. I gave each specialist its own wake-and-work loop. Instead of idling until Tara pokes them, the delivery agents now wake on a schedule, look at the board for cards assigned to them, and drive each one to its done condition: branch, pull request, checks green, merge.

Tara stays the Mayor. She owns triage and assignment. The specialists own execution. They no longer wait to be told to work; the work is already theirs.

Three of them are deliberately gated: release, deploy, and security. These agents build and stage everything right up to the irreversible step, such as cutting a release, deploying to production, or applying a security change, and then they stop. They post a "ready for approval" note and wait. I approve those from my phone. The loop is autonomous, but it never walks through a one-way door without me.

For the earlier build story, see Building a DevOps team from AI agents on OpenCLAW. This post is the operating-model correction that came after the team was already real.

The three hard stops (non-negotiable)#

Every serious write-up on agent loops converges on the same three guardrails. I built all three, because an unattended loop without them is how you wake up to a surprise.

Hard stopWhat it preventsHow it works
Re-wake cooldownRunaway iterationNo agent can be re-triggered more often than a set interval.
No-progress detectionRepeated work on the same blockerIf a card sees several wakes with no movement, it goes on a multi-hour hold.
Hard dollar budgetSurprise spendA cost check runs before any wake and skips the cycle when the day's spend is near the cap.

On top of those, a per-item claim lock stops two agents grabbing the same card, and a kill switch, a single file I can create, pauses every wake instantly. Autonomy with an off switch.

This is the same philosophy I used when building safety controls into my Microsoft 365 tooling. The details are different, but the design principle is the same: let the system act, but make irreversible or costly action explicit. I wrote more about that in 23 Safety Rules I Built Into a Microsoft 365 CLI for AI Agents.

From hourly to every five minutes#

At first the patrol rode Tara's hourly cycle. That worked, but an hour is a long time for a freshly triaged card to sit before someone picks it up. So I moved the wake loop onto its own timer that fires every five minutes, deployed through a narrow, audited helper so the change itself stayed disciplined, and then retired the old hourly hook so there is a single, unambiguous source of cadence.

The clever part is that most of those five-minute ticks do almost nothing. If every agent the loop would wake is still inside its cooldown, the tick scans the board, finds nobody eligible, logs its cost, and exits in a couple of seconds. The actual work rate is governed by the cooldowns, not the timer.

The five-minute cadence is not there to work the agents harder. It is there so new work gets noticed fast.

The outcome#

The board moves around the clock now. Tara triages continuously. The specialists pull and drive their own queues every few minutes. Gated work waits politely for my approval. Because the hard stops are real, I can leave the whole thing running and go to sleep.

My own role changed completely. I went from asking "why is nothing happening?" to supervising above the loop. I watch the board, get pinged for approvals and genuine blockers, and hold the kill switch. I am no longer a step inside the machine, handing out tasks one at a time. I set the goal, the evidence of done, and the constraints, and then I get out of the way.

That, to me, is the whole point of loop engineering done properly. The bottleneck in most AI automation is not the model. It is the human standing in the middle of the loop, dispatching work item by item, convinced they are orchestrating when they are actually the thing everything is waiting on.

What I'd tell you if you're building this#

  • Debug against live state, not your mental model. My first diagnosis was confident and wrong. "Nothing is happening" is usually "the wrong thing is happening out of sight."
  • Put the human above the loop. If you are the dispatcher, you are the bottleneck. Set the goal and guardrails, then step back.
  • Build the three hard stops first: iteration cooldown, no-progress hold, and dollar budget. Do that before you let anything run unattended.
  • Let each agent own its queue. A supervisor that assigns plus workers that pull beats a supervisor that micromanages every step.
  • Gate the irreversible steps. Stage everything, stop before the one-way door, and make the human the approver.

If you are connecting agents into Microsoft 365 work, the same pattern applies. The toolchain matters because agents need clean programmatic access to enterprise systems, which is why I wrote Why I'm Building My Own Microsoft 365 CLI for AI Agents and later Connecting AI Agents to Microsoft Planner. The loop only works when the work system is readable, writable, and safe enough for agents to touch.


Mark Smith is Principal AI Strategist at Cloverbase. To discuss this article or work with me, contact me at Cloverbase.

Mark Smith

Mark Smith

Principal AI Strategist · Microsoft MVP

Helping people build practical AI skill in the Intelligence Age.

Discussion

Comments

Loading the discussion for this post.

Loading

Leave a comment

Your email stays private. If it matches a Gravatar account, your public avatar can appear after the comment is approved.

Used only for reply notifications and optional Gravatar matching. Never displayed publicly.

Max 2000 characters0/2000

More from nz365guy

Artificial Intelligence

Building a DevOps team from AI agents on OpenCLAW

Ten AI agents, each owning a phase of the DevOps infinity loop, coordinated by an engineering-manager agent. How I built a development team that plans, builds...

· 15 min read
Artificial Intelligence

Building an org directory for AI agents on OpenCLAW

AI agents in multi-agent systems do not know who each other are. Here is how to build an org directory that gives every agent persistent awareness of the full...

· 8 min read