Field note: what your LLM calls actually cost per turn from Julia (measured) + a tiny TokenMeter

Tooling
1

Field note: what your LLM calls actually cost per turn when you drive them from Julia — and a tiny helper to see it

I’ve been building an agent loop in Julia that calls an LLM each step (PromptingTools-style: a system prompt, a growing message history, and a set of tool/function schemas). The token bill was ~2x what I estimated from “prompt length”, so I instrumented every request/response and wrote down where the tokens actually went. Sharing the numbers in case they save someone a surprise invoice.

Measured over a few hundred turns of a working agent (not a benchmark — a real task loop):

  • Tool/function schemas are re-sent and re-billed on every single turn. They’re not “set once”. In my loop the tool definitions alone were ~3,200 input tokens per turn. If you have 8–10 tools with verbose JSON-schema descriptions, that’s the single biggest line item and it’s invisible in your own prompt string.
  • Retrieved / injected context dominated input: ~34% of every input payload was retrieved context (docs, prior results) I’d stuffed in “just in case”. Trimming it to what the current step needs was the biggest single saving.
  • Input:output ratio was ~1.9:1. I’d assumed generation was the cost driver; it wasn’t. ~two-thirds of spend was input I was re-sending each turn (history + tools + context). Output was the cheap part.

The practical lesson: the thing to optimize is not “shorter answers”, it’s what you re-send every turn. Cache/trim tool schemas, prune history aggressively, and only inject the context the current step needs.

A minimal helper I now wrap every call in — it just accumulates the usage the API already returns, so you can see the running total and the per-turn breakdown instead of guessing:

Base.@kwdef mutable struct TokenMeter
    turns::Int = 0
    prompt::Int = 0      # input tokens (history + tools + context)
    completion::Int = 0  # output tokens
end

# `usage` is whatever your client returns per call (a NamedTuple/Dict with
# prompt_tokens / completion_tokens). PromptingTools exposes msg.tokens.
function record!(m::TokenMeter, usage)
    m.turns += 1
    m.prompt     += get(usage, :prompt_tokens, usage[1])
    m.completion += get(usage, :completion_tokens, usage[2])
    return m
end

function report(m::TokenMeter)
    ratio = m.completion == 0 ? Inf : round(m.prompt / m.completion, digits=2)
    (; m.turns,
       avg_in  = m.turns == 0 ? 0 : m.prompt ÷ m.turns,
       avg_out = m.turns == 0 ? 0 : m.completion ÷ m.turns,
       in_out_ratio = ratio)
end

Wrap your call site, record!(meter, msg.tokens) after each response, and print report(meter) at the end of a run. The avg_in climbing turn over turn is the signal your history/context isn’t being trimmed.

Curious how others here handle this in Julia agent loops — do you cache tool schemas out of the per-turn payload, or is everyone just eating the re-billing? And has anyone found the history-trim vs. context-quality tradeoff worth automating?

(Disclosure: I work on tooling at a small AI studio; no product pitch here, just numbers I wish I’d measured earlier. Happy to share the raw instrumentation approach if useful.)

2

TokenMeter apparently counts tokens instead of cost, but I’ve read a few times before that output tokens are generally charged at higher rates than input tokens. When you said “two-thirds of spend”, were you comparing token count as well or did you mean that inputs costed twice as much money? Any chance you also estimated cost per input token versus output token?