AI Agents Are Just Backend Microservices

Some recent reflections from my work: the essence of an AI agent is converging more and more toward mature software engineering. It isn't something brand-new or esoteric; it can be pulled back into the software-engineering frameworks we already know well.

Almost every component can find its counterpart in a traditional backend system. It's just that the code-driven business logic has partly turned into LLM-driven logic, replacing the if/else branch: the AI can choose which path to take on its own. The upside is that the LLM's own inference ability can accept and produce far more flexible, varied input and output. The downside is that it's too slow and the output is not stable enough.

So from this angle, the essence of an AI Agent is a variant of a backend microservice, suited to specific business scenarios.

If you need to build a stable, high-concurrency, low-latency, strongly consistent payment interface, then without a doubt you should still go with a traditional backend service, because what you need there is determinism, performance, and idempotency, not letting the LLM "think about whether the user wants to pay."

But if the business is more open, more fuzzy, and relies more on natural-language input (customer service, a research assistant, internal knowledge Q&A, coding assistance, and so on, where natural language is the input), then the LLM's inference ability becomes extremely valuable. It can turn a service from "can only handle predefined parameters" into "can understand the user's intent and dynamically organize the execution path."

Since both are essentially services, they can of course be mixed. A traditional microservice can be exposed to an AI Agent as an MCP tool, so the Agent can call it when needed and get a stable, verifiable result: creating a ticket, querying Databricks data, and so on. Conversely, an AI Agent can be packaged as a traditional backend service, exposing an HTTP API, a gRPC API, or a message-queue interface to the outside. Other systems don't need to know whether it's internally implemented with LLM inference, a rules engine, or plain ordinary code.

Back to Agent development. We mainly use Google's ADK framework, which I've studied quite a bit. One very interesting perspective is that you can draw a systematic correspondence between the various concepts in ADK and traditional backend development. Here's the whole map at a glance:

A2A = Service Discovery

A2A (Agent-to-Agent) in ADK is essentially service discovery. In traditional microservices, one service needs to know another service's address, capabilities, interface, and calling method. A2A solves the same kind of problem: how one Agent discovers another Agent and hands a task off to it. For example, a Planner Agent can hand "query the data" to a Data Agent, "generate the report" to a Writer Agent, and "execute the operation" to an Ops Agent. This is no different from service orchestration in microservices. The only difference is that traditional services pass structured RPC between each other, while Agents pass natural-language tasks and context.

Multi-Agent = Microservice Decomposition

This is a general concept; I think Multi-Agent is very much like microservice decomposition. One big Agent tends to become harder and harder to maintain. Its prompt gets bigger and bigger, its tools more and more numerous, its context messier and messier, and its behavior less and less stable. This problem looks very much like a traditional monolithic service: at first putting all the logic in one service is convenient, but as business complexity rises, the boundaries get blurry, the risk of changes grows, and the system gets harder and harder to test.

So the more reasonable approach is to decompose. The Planner Agent is responsible for planning, the Research Agent for retrieval, the Coding Agent for writing code, the Reviewer Agent for checking, and the Executor Agent for execution. Each Agent has a single responsibility, exposes only limited capability, and maintains only its own context and tool set.

This is almost identical to the bounded context in microservices. The purpose of decomposing Agents is not to make the architecture look complex, but to reduce the cognitive load on a single Agent, make responsibility boundaries clearer, and make the system more testable, more observable, and easier to evolve.

Tool / function call = SDK

A Tool can be a local function, an HTTP API, a database query, an MCP Server, or even another Agent. But to the LLM, they're all just "callable capabilities" wrapped locally.

In the traditional backend we design an API schema; in an Agent it's a tool schema. A good Tool definition is not just a parameter list, but a complete interface contract: when it should be called, what the parameters mean, and what the return value represents. So Tool Engineering is really API Engineering.

MCP = REST API / gRPC

MCP is also a general concept. The limitation of a plain tool is that maybe the tool you wrote for Agent A can't easily be used by Agent B, because of reasons like incompatible format. MCP (Model Context Protocol), as its name suggests, is a model-context protocol that tries to unify the communication rules between the model and external tools. Of course, in actual conversation, when many people say "MCP" they mean a server that has implemented MCP and can be called by an agent. MCP lets different Agents access databases, file systems, and business APIs in a consistent way, just as, in the traditional backend, REST APIs define fixed fields and calling conventions and everyone simply follows them.

Writing this far, we actually discover that there are very many ways to give an AI agent extra capabilities. You can spin up a new agent, use A2A as the interface, and build a multi-agent system. You can keep a single Agent and, by connecting to MCP external services, add extra functionality. You can also package an external service as a Tool for the Agent to call directly, similar to writing the dependency into your own codebase as an SDK. Many people feel this part is currently very chaotic, but I think that even in the traditional world these different solutions are all viable, so in the end you need some architectural design to decide which solution is most reasonable for each situation.

At present the AI Agent field may lack a commonly accepted architectural design theory that tells you when it's appropriate to use a tool, and when to use MCP or a sub-agent. There are also voices saying MCP will be packaged into a tool's CLI and replaced, and so on. How far it will eventually evolve I can't say either; maybe some approaches will indeed die off, and maybe they'll coexist.

Having covered the macro view, let's talk about some concepts inside a single agent.

Prompt = Business Logic?

In the traditional backend, business rules are written in code: what condition triggers what action, in what order it executes, all fixed down by the developer with deterministic syntax. But for an AI Agent, a considerable part of this logic is moved out of the code into a natural-language-based prompt, directly telling the LLM what to do.

To emphasize again: compared with the deterministic semantics of traditional code, with a Prompt the same intent (phrased a different way, with the wording tweaked, or an example added) may make the model behave differently. And code can't do anything with even a bit of uncertainty in it, like having it write an essay, whereas the LLM's inference ability plus a suitable prompt can write almost any essay, so there may still be some essential difference here.

Context = Working Memory

Context is essentially the Agent's working memory. When a traditional program runs it has memory that holds the current function call stack, local variables, intermediate state, and dependency objects. The Agent is the same; it's just that its "memory" is the tokens placed into the model's context window. The system prompt, user input, conversation history, tool descriptions, retrieval results, intermediate reasoning summary, and tool response together make up the context for the current model call.

So the context window can be likened to a very expensive, very limited, and very short-lived working memory.

Skill = Reusable Function

A Skill is essentially a reusable capability module, similar to a function, library, or business component in traditional software. A Skill can encapsulate a set way of executing a particular task. For example, "analyze SQL query performance" can be a Skill, and "generate the periodic report" can be a Skill; it may contain a prompt, tools, examples, constraints, an output format, and even a small workflow. But from an engineering angle, it just abstracts out a chunk of reusable capability, to avoid making the model understand the task from scratch every time.

This is the same intent as, when writing code, extracting repeated or similar logic into a function with some parameters.

RAG = Database

RAG (Retrieval-Augmented Generation) lets the AI first search an external database and, based on the extra knowledge obtained from it, give a better answer, especially useful when there are too many documents to fit into the context window.

A traditional database basically relies on a query language like SQL to be queried. Since an AI Agent's input is mostly natural language, it also needs a way to query directly with natural language. The most common approach, and the process most people think of by default when they mention RAG, is to turn the knowledge into vectors and put them into a vector database, and at the same time turn the natural-language prompt generated by the user or the AI into a vector too, then do a vector-to-vector comparison, ultimately achieving natural-language-based search.

There's a large amount of engineering practice in the middle here: how to index the data in the vector database, how many dimensions the text is turned into as a vector, how to chunk and store the text if there's too much of it, and so on. Later this further extends into not relying only on vector search, but doing a weighted hybrid search that mixes multiple retrieval methods, and other more complex approaches.

Structured Output = Schema / Protobuf

Structured Output is essentially a Schema, or more like the protobuf of the Agent world. Once an Agent has to enter a production system, it can't just output a piece of natural language. Natural language is suitable for people to read, but not suitable as input for a downstream system. What the downstream system needs is a stable structure: JSON, SQL, a workflow plan, an action list, a classification result, or some kind of business schema.

In the traditional backend, communication between services needs protobuf, JSON schema, Avro, or a database schema. Their role is to define fields, types, nested structure, and compatibility. The Agent system is the same. If the Agent's output is unstable, the downstream system can't parse it reliably.

Session = Request Context

The Session in ADK holds the context of the current interaction, the user input, intermediate results, tool-call history, and possibly execution information related to traces and logs. The way ADK implements it is also very simple: it's just key-value pairs that live in memory and only serve the current request.

A traditional HTTP request usually finishes in a few hundred milliseconds, but an Agent Session may last several minutes or even tens of minutes. The user may keep asking follow-ups, the Agent may call tools multiple times, may produce intermediate results, and may keep context across multiple steps. Although the lifecycle is longer, it's still essentially the context of a single request.

Memory = A Purpose-Built Cache & Database

This counts as a relatively new concept. People always have the fantasy of "AI growing up" and "the more it's used, the smarter it gets," and the approach is roughly that the Agent, like a human, "remembers" many things through continuous interaction and thereby performs better and better.

But from the current engineering implementation, Memory is essentially just a cache, database, profile store, vector DB, or knowledge base. What it solves is not a magic problem, but a data-storage and retrieval problem. User preferences, historical tasks, long-term context, business knowledge: these things already exist in traditional systems. A recommendation system stores user preferences, a CRM stores customer history, a search system maintains an index, and a knowledge base stores organizational documents. Agent Memory just lets the LLM use this information at the right time through semantic retrieval.

So the Memory Service is more like a "database with embeddings." What's really hard is not storing the information, but deciding what information is worth storing, when it should be retrieved, whether the retrieved results are relevant, how to handle expired information, how to modify wrong memories, how the user can delete them, and who should be trusted when multiple memories conflict.

Workflow = State Machine / Business Logic

Writing an AI Agent in ADK is essentially drawing a flow chart (workflow) and defining what each Node concretely does; here a node is just like a class or function in traditional code, and the business logic strings these classes/functions together. For example, an ordering system goes through creating the order, locking inventory, initiating payment, confirming payment, creating the fulfillment order, and notifying the user, each step with state transitions, failure handling, and compensation logic.

The Agent system is the same. A Research Agent may need to first understand the question, then search for material, then summarize the result, then self-check, and finally output the answer. A Data Agent may need to first understand the user's question, find the relevant table, generate SQL, execute the query, check the result, and finally explain the conclusion. These flows are all essentially workflow orchestration.

ADK supports things like Sequential Agent, Loop Agent, and Parallel Agent (note: the "agent" here is ADK internal terminology, more like the meaning of a node, not an independent AI Agent), which offer multiple ways of connecting and splicing these nodes into a complete workflow.

The only difference is that the nodes of a traditional workflow are usually deterministic code, while some nodes of an Agent workflow have become LLM inference. So the focus of an Agent Workflow is not "letting the model freely improvise," but clearly distinguishing which steps allow reasoning, which steps must be validated, which steps must be confirmed by a human, and which steps must be executed by deterministic code.

Eval = Testing System and Profiling

Eval is essentially the (performance) testing system of the Agent world. The traditional backend has unit tests, integration tests, and regression tests, used to ensure code changes won't break existing behavior. Agents also need a similar system. It's just that testing an Agent is more complex, because its output isn't always completely deterministic. The same question, under different model versions, different context, and different temperature settings, may produce different results.

So Agent Eval is more like a combination of a testing system and a quality-evaluation system. It doesn't just look at whether the program throws an exception, but has to judge the quality of the Agent's behavior: whether the tool calls are correct, whether the output conforms to the schema, whether hallucination appears, whether the user's task is completed, whether safety boundaries are respected, and whether human confirmation is requested at critical steps.

Eval also has very many technical details. For instance, testing in traditional software development is often done before release and then you're done, whereas for an agent there's a corresponding "offline eval," and beyond that there's also an "online eval" that keeps running after going live. Doing eval is like collecting data beforehand, labeling the correct answers, building a regression set, designing the specific metrics and scoring in the eval's grader, and figuring out how to compare different versions, or even the regression of different modules. Because of the innate nature of the LLM generating natural language, it's inevitably hard to do a natural-language comparison. For example, when comparing a real customer-service reply vs. the agent's reply to a user's question, maybe the concrete wording is truncated differently but both solve the problem, so you may end up bringing in another LLM as a judge to determine whether the agent's answer is valid. And the correctness of the LLM-as-a-judge itself is worth separately doing yet another eval for.

Closing Thoughts

Writing to this point, and based on my own work experience, I think that making an AI Agent truly reliable is not always about a stronger model or a more clever prompt, but about the things we're already familiar with in traditional software engineering: service decomposition, interface design, state management, observability, testing, and validation. What the LLM brings is just replacing some of the "deterministic nodes" with "inference nodes"; the skeleton of the whole system hasn't changed.

So rather than treating the AI Agent as something completely new and different, it's better to see it as an upgrade at the input and output layer of backend engineering. It lets the system consume fuzzy natural language and dynamically organize the execution path, but the price is sacrificing determinism, speed, and stability. The architectural design of an AI Agent is essentially about drawing clear boundaries: which steps are handed to inference, which steps must be guarded by deterministic code, and which steps need a human to make the final call.

This field looks today like it has many concepts and chaotic solutions, but this chaos itself is nothing new. Microservices went through the same thing back in the day. I think the eventual outcome is: first, the AI Agent can't overturn software engineering; it just adds a bit of something new on top of traditional software engineering. Second, it may be that as future foundation models become exaggeratedly powerful, to the point where you no longer need a lot of engineered solutions and constraints, determinism and stability can also be achieved (I briefly tried Fable 5, and I feel it still can't do it). The model will keep iterating and upgrading; which road it eventually goes down I can't say. I can only say that if you want to use the LLM / AI Agent most efficiently, you can still keep driving with the mindset of software engineering.

Reference: Original discussion on 1point3acres

← Back to all posts