Agents make engineering hard again

I think we're nearing the end of the "prompt demo" phase of AI. The door is now firmly ajar on the engineering phase, and the exciting news for engineers is that these shiny new agentic systems are bringing all the hard bits back again.

The great prompt illusion

There's been a lot of focus on the prompt as the product up until this point. It makes sense — the barrier is low, the wow factor high; the worm on that hook is juicy. The next step on the journey to getting that prompt into production? Cavernous.

That prompt looks so nicely defined sitting all alone in its well-styled textarea, but in the real world it needs a runtime, model access, tools, state, orchestration, continued evaluation, deploying, monitoring. The list goes on, and it's all very predictable from an engineering standpoint.

An agent? It takes a village (of engineers)

And now agents. Now that single prompt has autonomy. An agent requires state — likely a LOT of it. It has dependencies, credentials (probably a lot of these as well, looking at you MCP), significant side effects, multiple users, costs… the list goes on. They're powerful and have the potential for real, impactful business consequences, but they're almost a perfectly designed thought experiment aimed at stressing all the assumptions of modern-day infrastructure. So this brand new, shiny technology means we now need to go back to ensuring we're doing all those fundamentals again.

Return of the stateful process

Engineering for scale is hard, coupled with the fact that the bar for "scale" keeps moving. We keep inventing new terms just to get a handle on it: enterprise scale became web scale, web scale became hyperscale, hyperscale became planet scale…

We've worked hard and learned a lot. Small, stateless, short-lived, horizontally scalable, easily replaceable processes are where we've been aiming. Containers, Kubernetes, "serverless", edge compute — these are the tools and approaches we've been using to get there. Agents significantly complicate this.

They are often long-running processes with large amounts of working context, mutable workspaces, local codebases, browser sessions, authenticated tools, and an ever-growing backlog of memory. They communicate over long-running streams. They accumulate state. They make decisions based on context that often doesn't fit neatly into a request-response lifecycle. That is to say: almost the exact opposite of what we have been aiming for.

So what's the problem?

The problem is not that long-running, stateful systems are impossible. We have been building them forever (we used to mostly default to building them). The problem is that agents make stateful systems feel easy to create and hard to operate.

With their very particular flavour of requirements, agents in production require that we ask a lot of those familiar questions again: What if this long process gets halfway through and dies? Can it safely resume? Can we adequately explain why it made a specific decision or tool call? How do we consider cost control? Can its credentials be revoked quickly? Can we accurately discern the difference between a model error, a tool error, a permissions issue, a bad prompt, stale context, or plain vendor outage? None of these are AI-specific problems — but then again, agents aren't remotely interested in quacking like the ducks our infrastructure has been evolving to handle.

And the new problems ARE new

As well as making us reconsider those old familiar problems, agents also present genuinely new ones: non-deterministic execution, prompt injection, tool misuse and call failures, hallucinated actions, highly variable latency, wildly unpredictable costs, evaluation difficulty. On the face of it, some of these challenges seem diametrically opposed to many of the objectives we've been striving toward for years.

The fundamentals matter so much more now

Security, observability, testing, governance, and — my personal favourite — resilience now, more than ever, must become first-class product features for these systems. With agents acting autonomously on behalf of real-world users or businesses, we must understand what they're doing, why they're doing it, what has been affected, and, critically, how to stop and recover them. The exciting bit? There's some real engineering complexity in solving the approach to many of these.

The production agent readiness checklist

We've spent a lot of time considering the kinds of questions an enterprise-grade platform for running production agents needs to answer. A first draft of an agent readiness checklist looks something like:

1. Can you trace every model call, tool call, skill execution, and state change?
2. Are you able to replay and introspect each failed or anomalous run?
3. Can the process resume if it encounters a fatal condition?
4. Can you apply constraints on fundamentals such as runtime, cost, and tool access?
5. How does the system behave with vendor degradations or outages?
6. Can behaviour be accurately assessed before and after deployments?
7. Are there realistic checkpoints and intervention gates ahead of critical actions?

Our aim at Rightbrain is to continue to address these challenges and provide a world-leading runtime for our customers' production agentic tooling — so they can focus their time exclusively on the real business value within their existing systems, without having to think about the hard parts.

Adapted from the original essay: Agents Make Engineering Hard Again.