Agent Runtime Design

A running collection of observations on agent runtime design. The entries are independent, but they point at the same judgment: protocol-layer design determines the quality of agent behavior.

Veto boundary — multi-agent collaboration is not relaying orders. The orchestrator decomposes tasks and assigns intent, but every receiving agent needs an explicit veto boundary: under what conditions it may refuse an upstream instruction, and when it must comply. Orchestration without veto boundaries has no brakes when an error propagates. (note)

Natural language as tool state — when an agent writes bash, it adds comments and uses && echo ok || echo failed to translate exit codes into words. It looks redundant; it is actually converting state into the representation a model consumes most easily. When designing tool use, let tool input carry an intent description and tool output return natural-language state, rather than bare data alone. (note)

Tool-call role design — Claude putting tool_result under role: user is not confusion: that "user" carries external observations handed back by the client / environment, not a human speaking. Codex / the Responses API instead model tool calls and results as independent events. The two protocol shapes affect context operations (trimming, replay, caching) differently. (note)

Context-system layering (six responsibilities) — feeding "context" to the model as one undifferentiated whole is where complexity starts running away. Split the responsibility: the System Prompt holds only stable identity, a few hard boundaries and the output protocol (no transient state, no bad-case exceptions); Dynamic Context holds this turn's confirmed facts, state and allowed space; State/Code/Engine own state transitions and deterministic rulings; Tool/Retrieval fetches external facts on demand, labeled with source and freshness (not everything possibly relevant poured in at once); the LLM only understands the real conversation, handles ambiguity and phrases things naturally (it should not keep re-guessing deterministic facts the system already knows); Frontend/Guard owns the last gate of user confirmation and action execution (a high-risk action is never entrusted to one line of prompt). Each representation carries a different kind of complexity: boolean is a gate, enum is a rail, schema is a structural boundary, natural language handles the open world — the test being whether a rule can tolerate probabilistic execution. (source)

Conversational air — the opposite extreme of structuring is starving the model: once everything becomes booleans/enums/short fields, the model no longer knows the user has been waiting for a while, that the same question was asked last turn, or whether this reply should absorb a complaint or confirm a plan — producing replies where "every field is right but it sounds like a stranger". Delete verbatim lines like "my parents are coming along this time, let's not rush it" and a structurally correct itinerary can still feel terrible. The principle: stop deterministic information from masquerading as prose that needs understanding, while keeping the context the model needs to understand a real conversation — not "the less natural language the better", but "the fewer unowned implicit rules the better". It is the boundary condition to "natural language as tool state", from the other side: state can be translated into language; context cannot be compressed into fields. (source)

Latency as answer quality — real-time agents live inside a narrow user decision window: a suggestion that arrives before the decision forms is help; arriving after, it is only an interruption. Measured case: adding RAG to a real-time support copilot improved every offline metric, moved end-to-end latency from ~2.6s to ~5.5s, and post-launch adoption went to zero — the model did not get dumber, the content did not get worse, the system threw no errors; the product failed. Corollaries: a capability (such as retrieval) need not sit on the real-time critical path — it can retreat into offline assets, few-shot material or non-real-time flows; and evaluation must put end-to-end latency and real adoption into the same picture, watching three scales at once — components and contracts, end-to-end chains, live product outcomes. A correct single node only proves that single node correct; it cannot testify for the whole harness. (source)

The engineering-practice counterpart is agent-iteration-harness: once iteration runs into multiple rounds, runtime design questions (who sees which context, who holds a veto, how regressions get caught) surface as swings in output quality.