Context engineering and production agents

Why would an agent spend 40 times its normal budget on one question?

Take an ops agent: a small helper that engineers ask questions like “find out why checkout is slow”. It reads the logs and reports back. On the developer’s laptop it worked every time.

In production a spend alert fires: the agent is costing 40 times its normal rate. The trace shows that it has not crashed. It is stuck in a loop: it reads a 600-token log, then reads the same log again, and then again.

In Agents, your agent answered a question in three steps and read 1,737 characters in total. Here is what this one sends the model on each of eight steps, measured by this lesson’s Python file:

Pause and predict The last call is 5,616 tokens. How many input tokens did the whole run bill?

The loop has no bug: it does exactly what its code tells it to do. What is missing is everything a production system wraps around the loop: a budget, a way to keep the context small, tools designed for a caller that guesses, a gate in front of actions that cannot be undone, and a trace, so that you can see any of this happening. Without the trace, nobody would know that the agent was reading the same log over and over.

The window is a budget

A common first suggestion is to switch to a model with a million-token window, so that the agent never runs out of room.

A bigger window does not solve the problem, for three reasons. First, every token is re-sent and re-billed on every step, so the cost grows with the square of the run length, and the latency grows with it. Second, models use long contexts less reliably than short ones. Third, anything a tool returns, including text written by an attacker, lands in the same window as your instructions.

The model is a fixed function of its context, so the engineering left to you has two halves. Context engineering means deciding what goes into the window on every turn. Harness engineering means deciding what the loop is allowed to do, spend and touch. Every curation step can drop something the model needed, and every guard adds latency or a human: you are trading capability for predictability. Tools like Claude Code and Codex are made of both halves.

Context engineering

Deciding, for every model call, which tokens are in the window: what to include, what to shorten, what to leave out and fetch later.

For example: Instead of pasting a 40-line log into the history, keep its one error line and a file reference the model can ask for.

Here is one context window, drawn to scale, at step 1 and at step 8 of the same run:

Six kinds of text compete for that space: the system prompt, the tool definitions (every tool’s name, description and schema is sent on every call), retrieved documents, the conversation history, tool results, and any notes the agent keeps for itself. Every token of it costs three things:

• Money. Input tokens are billed on every call. A token that sits in the history for 20 steps is paid for 20 times.
• Latency. The whole input is prefilled before the first output token appears.
• Attention. Liu et al. (2023), “Lost in the Middle”, found that models use information best at the start or end of the input and markedly worse in the middle. Newer models find a single fact almost anywhere, but still get worse when they must combine many facts spread across a long input.

Five ways to keep the window small

As you saw in Agents, the model has no memory between calls. Each of these techniques is your code deciding what text goes back into the window on the next call.

Tools are an API whose client guesses

A tool definition is an API contract. The caller is a model that has never read your source, cannot ask a colleague, and picks a tool by reading one paragraph of description. Design for that caller:

• Few tools, clearly separated. Every definition costs tokens on every call. One search_orders(customer, status) tool is better than a list_orders tool that leaves the model to filter 5,000 rows inside its context.
• Descriptions written like documentation for a new hire: precise names, typed arguments, units, an example. The description is a prompt.
• Token-efficient results. Return only the fields that matter, paginate, filter on the server, and when you must truncate, include a reference to the full result.
• Errors the model can act on. A bare 400 Bad Request tells the model nothing it can fix. Say what was wrong, what was expected, and what to do next. The error message is the model’s only debugger.
• Side effects that are safe to repeat (idempotent). The loop will retry, so let the caller pass a request id: “create the ticket” sent twice still creates one ticket.

MCP (Model Context Protocol)

A standard plug for tools. Write a server once that exposes your system’s capabilities, and any compatible LLM application can connect to it.

For example: A “GitHub server” offers tools such as creating an issue. Your editor’s assistant and a chat app can both use it without custom glue.

MCP standardises how tools are connected. It does not decide whether a tool is well designed or safe: a tool that arrives over MCP still spends context on its definition, still returns text that lands in your window, and still needs the permission checks below.

Security: the model cannot tell instructions from data

You saw prompt injection turn an answer into BANANA. In direct injection the user types the attack. In indirect injection the attack waits inside something the agent reads on behalf of an innocent user, such as a web page, an email or a tool result. Indirect injection is the dangerous one, because reading untrusted content is an agent’s job.

Everything else is ordinary security engineering: least privilege (a read-only token for a read-only task, a tool list per task); sandboxed code execution with no network or credentials by default; human approval for irreversible actions (sending, paying, deleting, deploying); and validating outputs as well as inputs, for example a generated SQL statement or URL against an allow-list.

Reliability and observability

An agent is a distributed system whose flakiest dependency is also the one making the decisions. Give it what you give any such system: budgets on steps, tokens and money, each stopping the run with its own reason (a token budget is a rate limiter whose unit is tokens or rupees instead of requests: Design a rate limiter); loop detection; timeouts on every tool; retries with backoff, only for calls that are safe to repeat; and a fixed fallback when a budget runs out (a human, or the partial result, never silence).

Trace and span

A trace is the full record of one run. A span is one timed unit of work inside it: one model call, or one tool call, with its inputs, outputs, token counts and cost.

For example: Span 3: model call, 184 tokens in (136 from cache), 28 out, 772 ms. Span 4: tool call calculator, 5 ms.

An agent trace is a distributed trace whose spans are model calls and tool calls. Observability in System Design

With traces you can answer “why did this run cost four dollars?” by looking, and sample runs for human review, which is where new golden items come from. Agent evals score the outcome (did the task succeed?) and the trajectory, the path the agent took: did it use the expected tools, stay under budget, avoid forbidden actions, and ask for approval when it should have? A right answer reached by reading a file the agent should not have touched is a failure your final-answer scorer will never see.

Let's see the numbers

Why the bill grows with the square of the run. Take a fixed prefix of 2,000 tokens (system prompt and tools). Each step appends 1,000 tokens: 200 the model wrote, 800 a tool returned. The model is stateless, so call i is sent the prefix plus the i − 1 steps before it:

The prefix is sent 10 times, so 20,000 tokens. The appended steps are re-sent 0 + 1 + 2 + … + 9 = 45 times, so 45,000 more. The window never held more than 11,000 tokens, and you paid for 65,000.

Pause and predict Keep the same prefix and the same growth per step, but run 20 steps instead of 10. What happens to the billed input?

What prompt caching changes. Each call’s input begins with the whole previous call’s input, unchanged. Providers can keep the KV cache for that prefix and bill the repeated part at a fraction of the price.

Take example prices: $3 per million input tokens, $15 per million output tokens, cached reads at 10% of the input price and cache writes at 125%. In the 10-step run above, 54,000 of the 65,000 input tokens are cache hits.

• Without caching the run costs $0.225.
• With caching it costs $0.087, which is 61% less.

The window did not change by one token: call 10 is still 11,000 tokens long. Caching lowers the price of the repeated tokens, and the window stays exactly as large. It also only works while the prefix is byte-for-byte identical. Rewrite the history, reorder the tools, or put a timestamp in the system prompt, and everything after the change is a cache miss. Entries also expire when nobody reads them. Anthropic keeps a cached prefix for 5 minutes by default, restarting the clock on each read, and offers a 1-hour lifetime whose cache writes cost twice the input price. OpenAI’s newest models keep one for at least 30 minutes, with a 12-hour option, and several of its earlier models can keep one for up to 24 hours. Real providers also skip short prompts: below a minimum, between 512 and 4,096 tokens depending on the model, nothing is cached and no error says so. Both Anthropic and OpenAI now cache a growing conversation automatically and also accept breakpoints you mark yourself, and on Anthropic and OpenAI’s newest models writing to the cache costs more than ordinary input (1.25 times for the short lifetime). The numbers here assume every shared prefix is cacheable, so treat them as the best case.

The real run. This is the stuck ops agent from the start of the lesson, reproduced by python phase6-engineering/agent_budget.py: 8 steps, tokens estimated as characters ÷ 4, the same example prices. Three strategies, each costed with and without caching:

Pause and predict Three strategies: resend everything (naive), truncate each tool result to 100 tokens, or compact the history at 3,500 characters. With caching on, which run is cheapest?

The math

The first term is linear and the second quadratic. You have three levers and the techniques above map onto them: shrink g (truncate, offload, better tools), shrink F (fewer tool definitions, retrieval on demand), or stop the second term from accumulating (compaction, sub-agents, which restart the sum).

Latency has the same structure: time to first token grows with the fresh input that must be prefilled, and the rest grows with output tokens, because decoding is sequential. An agent’s wall time is the sum over all its calls, plus every tool. Ten steps of 3 seconds each make the user wait 30 seconds.

What you should remember

1. The window is a budget. Every token in it costs money, latency and attention on every call. Keep it small with compaction, truncated and offloaded tool results, retrieval on demand, sub-agents and notes in files; each can lose information, and that is the trade.
2. Billed input = n·F + g·n(n−1)/2. Re-sending the history makes the bill quadratic in run length. Prompt caching discounts the prefix and does not shrink it, and rewriting history forfeits the cache.
3. Tools are an API for a caller that guesses: few, precisely described, typed, token-efficient, with actionable errors and idempotent side effects. MCP standardises how tools are connected; it says nothing about whether they are safe.
4. Private data + untrusted content + an outbound channel is the configuration to avoid. There is no complete defence against prompt injection, so use least privilege, sandboxes, approval for irreversible actions, budgets with reasons, and a trace of every span.

Where this appears in a real LLM

In this lesson

• A scripted stand-in model with three bad habits
• Tokens estimated as characters ÷ 4, example prices, placeholder latencies
• A four-tool registry and a one-type schema
• An approval callback that returns true or false

In a real system

• A real model whose mistakes are varied and unrepeatable
• Token counts and cache hits reported by the API, reconciled against an invoice
• Dozens of tools with JSON schemas, many arriving over MCP
• Permission systems, sandboxes, audit logs, and people on call for the agent

The same design in the coding agent you use

A coding agent such as Claude Code or Codex is the loop from Agents with this lesson wrapped around it. From their public documentation:

This explains things you have seen at the keyboard. A long session gets vaguer about early decisions (they were compacted away). A fresh session with a good instruction file often beats continuing a long one. A sub-agent’s exploration does not clutter your main conversation, and sometimes returns a report missing the detail you wanted. A tool that dumps 5,000 lines makes the rest of the session worse. Each of these follows from what is, and is not, in the context window.

For the ops agent at the start, a reasonable fix is a step budget, a 100-token cap on tool results with the rest offloaded to a file, and a p99 cost alert that pages a human long before spending reaches 40 times normal. All three live in the code around the loop, and the loop itself does not change.