Notes from the trenches on building production-grade, multi-agent AI systems, where the frontier meets the fundamentals of good software engineering.
Measuring agent quality is two jobs, not one: prove a change is good enough to ship, then catch a system you haven't touched decaying anyway. Automated vs. human eval as a division of labor, the pre-release gate (offline regression, shadow, canary, A/B), and monitoring degradation and distribution drift once real users — and a model you don't control — start moving under you.
Read the article →Evals are the test suite for a probabilistic system. Why outcome and trajectory both matter, why consistency (pass^k) is a different axis than raw capability (pass@k), how to grade with rules and a calibrated LLM-as-judge without the biases, and how evals graduate from capability checks into a regression gate that runs on every commit.
Read the article →These aren't three competing techniques; they're three stages of one pipeline: retrieve wide, fuse, rerank narrow. How semantic search, hybrid search (BM25 + dense, fused with RRF), and cross-encoder reranking fit together, and the two places they pay off hardest inside an agent: semantic caching and runtime tool selection.
Read the article →A guardrail is just input validation and output sanitization wrapped around an inference layer. How input and output guardrails work, the four ways to implement them (rules, classifiers, LLM-as-judge, guard models), and why the synchronous-versus-asynchronous decision quietly shapes your agent's latency, safety, and UX.
Read the article →Memory feels solved with one agent and a markdown file, then falls apart the moment you add a second agent or a hundred facts a day. How agents share state, resolve conflicting facts, and make knowledge discoverable without hand-wiring aren't AI problems; they're distributed systems problems, and the teams treating them that way ship agents that work.
Read the article →The more you build with agents, the more they rhyme with mature software engineering. A concept-by-concept mapping (A2A to service discovery, MCP to REST, RAG to the database, Eval to the test suite), and why making agents reliable is an architecture problem, not a model problem.
Read the article →