After building the small agent loop I wrote about here, the next problem was cost — every step re-sent the entire conversation history, and tool descriptions that were originally written for a human reading documentation, not a model deciding whether to call them. Three changes made the biggest difference, and none of them were about clever wording.
Tool Descriptions Were Doing Too Much Talking
My first tool descriptions read like API docs — full sentences, examples, edge cases, all included on every single call regardless of whether the model ever used that tool. For a system with a dozen tools, that's a lot of tokens spent describing tools that go unused in nine out of ten requests.
{
"name": "search_docs",
"description": "Search internal documentation by keyword. Query strings, keywords",
"parameters": {
"type": "object",
"properties": { "query": { "type": "string" } },
"required": ["query"]
}
}
Cutting descriptions down to the minimum the model actually needs to decide whether to call a tool — not how to use every option — shrank the tool-spec section of every request by more than half, with no measurable drop in the model correctly choosing between tools.
Structured Output Beats Free-Text Parsing
The second change was asking for a fixed schema instead of prose whenever the output needed to be parsed downstream. Before, a response like "the ticket priority should probably be high, since it mentions an outage" needed a second, fragile parsing step. A schema removes the ambiguity entirely and, in practice, shortens the model's own output too, since it isn't spending tokens on hedging language:
{
"priority": "high",
"confidence": 0.82,
"reason": "mentions an active outage"
}
Constraining the shape of the answer turned out to make answers more consistent, not less — free text tends to wander, and a schema gives the model less room to pad.
Conversation History Doesn't Need to Grow Forever
The biggest single win was the least glamorous: not resending the full conversation on every agent step. Early on, step five of a six-step tool-calling loop was resending every tool result from steps one through four, verbatim, because that was the simplest thing to implement.
def trim_history(messages: list, keep_last_n_tool_results: int = 2) -> list:
system_and_user = [m for m in messages if m["role"] in ("system", "user")]
tool_and_assistant = [m for m in messages if m["role"] in ("assistant", "tool")]
return system_and_user + tool_and_assistant[-keep_last_n_tool_results * 2:]
Keeping only the most recent tool results, and summarizing older ones into a single short note instead of dropping them entirely, cut the average request size for later steps in a run by roughly 60%, without the model losing track of what it had already tried.
Measuring, Not Guessing
None of these changes were obvious wins on paper — I only trusted them after logging actual token counts per request before and after, over a batch of the same 50 representative test prompts. The tool-description trim and the history trim together accounted for almost all the savings; the output-schema change barely moved total tokens but noticeably improved correctness, which made it worth keeping regardless of cost.
What Didn't Help
I spent a while trying to shave tokens off the system prompt itself with terser phrasing, and it made basically no measurable difference — the system prompt is sent once and is tiny compared to tool specs and growing history. That effort would have been better spent on the history-trimming problem from the start; lesson noted for next time I optimize something before measuring where the cost actually lives.