Thesis
I've spent the last four months running a 5-agent system in production for vendor management. This is a field report — what worked, what broke, and the patterns that emerged from operating agents at scale in real business processes.
The short version: multi-agent orchestration is harder than building individual agents, but the payoff is exponential when you get the coordination right.
The Architecture
The system has five agents coordinated by a central orchestrator:
Orchestrator: Routes incoming tasks, manages state, handles escalation logic. Doesn't do any domain work itself — pure coordination.
Procurement Agent: Handles vendor lookups, PO generation, price comparison, and basic negotiation within pre-set parameters.
Reporting Agent: Aggregates data across systems, generates weekly/monthly summaries, identifies anomalies.
Comms Agent: Drafts and sends stakeholder updates, follow-up reminders, and status notifications through Slack and email.
Monitoring Agent: Tracks agent performance, flags confidence drops, triggers human escalation.
Lesson 1: The Handoff Problem
The biggest source of failures wasn't individual agent performance — it was handoffs between agents. When the Procurement Agent needed the Reporting Agent to verify a budget number before approving a PO, the data format mismatch caused 23% of handoffs to fail in the first week.
The fix: strict interface contracts. Every agent-to-agent communication now goes through a typed schema. The Orchestrator validates the payload before routing. Failures dropped to under 2%.
The lesson is counterintuitive: the smarter your agents are individually, the more rigid your inter-agent protocols need to be. Flexibility inside, structure between.
Lesson 2: Confidence Scoring Changes Everything
Every agent action is tagged with a confidence score from 0.0 to 1.0. Below 0.7, the agent flags for human review. Below 0.4, it halts entirely.
This sounds simple but it transformed the system's reliability. In the first month, agents operated at full autonomy 67% of the time. By month three — after tuning thresholds based on observed error patterns — it hit 83%. Not because the agents got smarter, but because the boundary between autonomy and escalation got more precise.
The monitoring agent graphs these confidence distributions in real time. When I see a cluster of scores dropping toward 0.7 in the Procurement Agent, I know something changed in the vendor data — usually a new vendor format the agent hasn't seen before.
Lesson 3: The Orchestrator Is the Product
Here's the insight I didn't expect: the individual agents are almost commodity components. The Procurement Agent is basically a Claude wrapper with domain-specific tools. What makes the system valuable is the Orchestrator's decision logic — how it routes, prioritizes, parallelizes, and handles failures.
The orchestrator contains ~40% of the total system logic. It decides: can this task be parallelized? Which agent is the bottleneck? Should I retry a failed handoff or escalate immediately? When two agents give conflicting data, which one wins?
If I were starting over, I'd spend 60% of the design time on orchestration patterns and 40% on individual agents. Most teams do the opposite.
Lesson 4: You Need Agent Observability
Traditional logging isn't sufficient for multi-agent systems. You need to see the decision chain — not just what happened, but why each agent chose its path.
We built a trace view that shows: task received → orchestrator routing decision → agent execution → confidence score → handoff or escalation → outcome. Each step is clickable and shows the agent's reasoning.
This saved us during a production incident where the Comms Agent was sending duplicate notifications. The trace showed that the Orchestrator was routing the same task twice because a race condition in the state management layer. Without the trace view, we'd have been debugging blind.
Ideas to take with you
- Agent-to-agent handoffs are the primary failure point — use strict typed schemas
- Confidence scoring with tuned thresholds enables progressive autonomy (67% → 83%)
- The orchestrator is the most valuable component — invest 60% of design time there
- Agent observability requires decision-chain tracing, not just traditional logging
- Multi-agent ROI is exponential but only after coordination patterns stabilize (~month 2-3)
References
- Anthropic: Tool Use and Agent Patterns — Anthropic Docs
- LangChain: Multi-Agent Architectures — LangChain Blog
- Andrew Ng: Agentic Design Patterns — DeepLearning.AI
Thanks for reading.
Keep reading ↗