Designing a Cost-Aware Distributed Agent Control Plane

Core thesis: Adding more agents or more machines does not automatically solve the problem. What is actually missing is a control system that knows which work deserves deep reasoning, which work should be answered immediately, which work should be distributed, and which work should never spawn a subagent in the first place.

Where It Started: When “Simple Work” Began Taking as Long as Architecture Work

Modern AI coding agents can create subagents, divide work in parallel, read code, run commands, edit files, and consolidate results. But real-world use reveals another side of that capability: simple questions that once took almost no time now take anywhere from tens of seconds to several minutes.

This does not mean the model has become less intelligent. The opposite is true. Model capability has grown to the point where the harness grants more freedom for planning, tool use, repeated verification, and subagent creation. The system becomes more capable, but coordination, context, and reasoning costs grow with it.

I call this cost the Harness Tax — time spent preparing context, selecting tools, planning, decomposing work, waiting for results, verifying, and summarizing, rather than doing the essential work itself.

Clear examples include questions such as:

• Is this method name appropriate?
• Could this header be duplicated?
• Should the alias be plan, review, or analyze?
• Where is the implementation of this interface?

These tasks may need only one file read or one or two grep calls. But a harness without a clear budget may choose a frontier model, enable extended reasoning, create an explorer and a reviewer, and only then synthesize the answer.

The right question is therefore not merely “Can an agent perform this task?” It is:

“How much intelligence, time, token budget, tool activity, and worker capacity does this task justify?”

The Truth About Multiple Machines: They Help, but They Solve Different Bottlenecks

When Codex or Claude Code is used with cloud models, model reasoning happens on the provider's infrastructure. Our machines are responsible for local execution such as:

• reading and writing files
• Git, worktrees, and branches
• builds, tests, and linting
• Docker and integration environments
• browser automation
• MCP servers and local tools

A second or third machine can significantly help with heavy build/test workloads, multiple browser-test suites, local GPU work, or isolated worker environments. But it does not directly accelerate a request that is waiting for model reasoning in the cloud.

This is why distributed agents must be designed around two separate planes:

1. Reasoning plane — selects the model, reasoning effort, context, and number of agents
2. Execution plane — selects the machine, environment, capabilities, and resource quota

Sending everything to multiple machines without a policy is not good scale-out. It only expands inefficiency from one machine to three.

distributed-agent-control-plane.svg

The Required Architecture: An Agent Control Plane

A system that addresses this problem contains the following essential components.

1. Agent Gateway

The gateway receives intent from a user, IDE, or CI system and converts it into a task contract with a clear scope, such as repository, base commit, allowed files, expected output, and acceptance criteria.

The gateway should not forward a prompt directly to a model. It must first classify the task as:

• fast answer
• repository lookup
• bounded code change
• diagnosis
• review
• architecture/design
• parallelizable workload

2. Cost-Aware Model Router

The router selects a model according to the actual need instead of making a frontier model the default for every task.

An example policy:

The important principle is that a task must demonstrate complexity before receiving more reasoning budget rather than receiving the maximum budget from the start.

3. Scheduler and Capability Registry

The scheduler chooses a worker according to capabilities such as:

workers:
  machine-1:
    capabilities: [dotnet, sql-server, docker]
    max_jobs: 2

  machine-2:
    capabilities: [node, angular, playwright]
    max_jobs: 3

  machine-3:
    capabilities: [gpu, local-llm, embeddings]
    max_jobs: 1

A Playwright task should not be sent to a machine without browser dependencies, and a .NET integration test should not be sent to a worker without a SQL Server test environment. Worker selection must therefore account for capability, queue depth, lease, timeout, and resource pressure.

4. Isolated Worker Runtime

Each worker should operate in its own Git worktree, branch, or container. Multiple agents should never write to the same working directory because that creates race conditions, context drift, and patch conflicts.

Essential practices include:

• every task references a base commit SHA
• write scope is limited by module or file
• workers are not allowed to merge their own work
• results are returned as a patch with test evidence
• work is cancelled when its lease or heartbeat expires

5. Result Store and Aggregator

Worker output must be a structured contract, not a message saying “done.”

{
  "taskId": "AGM-20260718-001",
  "workerId": "machine-2",
  "baseCommit": "a92f30c",
  "status": "completed",
  "summary": "Updated request resolver",
  "evidence": [
    "src/Extensions/HttpRequestValueExtensions.cs:42"
  ],
  "branch": "agent/AGM-20260718-001/machine-2",
  "tests": {
    "passed": 47,
    "failed": 0
  },
  "metrics": {
    "durationMs": 84213,
    "toolCalls": 11,
    "inputTokens": 18342,
    "outputTokens": 2150
  }
}

The aggregator is responsible for checking evidence, running the required verification, resolving conflicts, and synthesizing the final answer. It should never trust a worker's summary without a patch, test, or file reference to support it.

The Main Problems in Current Harnesses

Context Inflation

As a session grows longer, more conversation history, tool schemas, and prior results are sent back to the model. A short question may therefore carry tens of thousands of tokens from earlier work.

Model Inheritance

If subagents inherit the primary model, a simple exploration task may be sent to an expensive, high-reasoning model by accident. The thinking configuration may also be inherited from the main session.

Unbounded Fan-Out

Parallel decomposition helps when work units are independent. But when a task is small or highly dependent, coordination costs exceed the benefit. Allowing subagents to create more subagents without a depth limit further increases token usage, latency, and tracking difficulty.

Verification Without Marginal Value

Agents often ask, “Can I verify more?” instead of asking, “Could this additional verification change the answer?” The result is repeated reading, planning, and review even after sufficient evidence already exists.

Policies That Should Be Embedded in the System

Do not spawn a subagent when the request can be answered
without repository inspection or with at most two tool calls.

Spawn subagents only when:
1. There are at least two independent work units.
2. Each unit has a bounded output contract.
3. Parallel execution saves more time than coordination costs.
4. Workers do not write to the same files or working tree.

Do not allow recursive delegation unless the operation
explicitly permits it.

In addition, every operation type should have its own execution budget, for example:

agent-execution-budget-and-state.svg

Observability: See What an Agent Is Doing Without Exposing Chain of Thought

The system should expose operational state, not private reasoning:

CREATED → QUEUED → ASSIGNED → RUNNING → VERIFYING → COMPLETED
                              ├─ WAITING_FOR_TOOL
                              ├─ BLOCKED
                              ├─ RETRYING
                              └─ TIMED_OUT / CANCELLED / FAILED

Each state transition should emit an event containing:

• task ID, agent ID, and parent agent ID
• model and reasoning effort
• start time, duration, and time-to-first-token
• tool name, command duration, and exit code
• input, output, and cache tokens
• retry count and timeout
• files touched and patch reference
• test result and final exit reason

OpenTelemetry is well suited to this problem because it can combine metrics, logs, and traces with Grafana, Loki, and Tempo. That makes it possible to answer questions current harnesses leave unclear, such as:

• Is the agent still working or stuck?
• Was the time spent in the model or a tool?
• Which agent is producing an abnormal number of tokens?
• Which tasks retry repeatedly?
• Which subagent stopped sending a heartbeat?
• Which fan-out saved less time than it cost?

Metrics That Should Be Measured

Do not measure only the number of completed tasks. Measure the economics of the decisions as well:

• Time to First Useful Result — time until the first usable piece of evidence appears
• Total Wall-Clock Time — time from task receipt until verification completes
• Coordination Ratio — orchestration time compared with actual execution time
• Token per Accepted Change — tokens consumed per accepted patch
• Rework Rate — proportion of tasks returned by the verifier for correction
• Parallel Efficiency — time saved divided by the cost of additional agents
• Wrong-Model Rate — tasks escalated or downgraded after execution begins
• Idle Worker Time — workers left idle because the scheduler chose the wrong capability or a dependency was not ready

A Practical Roadmap

Phase 1 — Policy Before Distribution

• add task classification
• define reasoning, tool, and agent budgets
• limit subagent depth
• add explicit stop conditions

Phase 2 — Structured Telemetry

• define the task state machine
• emit heartbeat and tool events
• expose tokens, latency, and retries per agent
• build a dashboard comparing the fast path with the deep path

Phase 3 — Isolated Local Workers

• use a separate Git worktree for each agent
• enforce the output contract
• add a verifier before merge
• test multiple workers on one machine first

Phase 4 — Remote Worker Pool

• add a queue and capability registry
• distribute work to a second or third machine
• add leases, cancellation, and an artifact store
• scale according to bottlenecks measured in production

Phase 5 — Learning Router

• use historical telemetry to improve model routing
• predict task duration and probability of rework
• apply graduated autonomy based on task-category statistics

Conclusion

The problem in the multi-agent era is not a shortage of intelligent models or machines to run them. It is the absence of a system that decides how much intelligence and how many resources each task deserves.

Adding machines without governance can turn six overthinking agents on one machine into eighteen overthinking agents across three machines.

The most important principle is therefore:

Scale the control plane before scaling the worker pool.

This is the transition from “using AI to write code” to designing an AI Platform and Software Delivery System that measures cost, supports auditability, and governs autonomy with discipline.