An agentic request is a chain of dependent calls, so latency compounds. Budget it explicitly or it will be decided for you by whichever component is slowest that day.
Write the budget down
For a request with a two-second target, an honest allocation might be: 80ms retrieval, 120ms reranking, 900ms first token, streaming thereafter, 300ms of tool calls, and the remainder as headroom. Once written, the budget makes tradeoffs concrete — adding a second reranking pass is no longer free, it is 150ms of somebody else’s allocation.
Stream, and stream something meaningful
Time to first token matters more than total duration for anything a person is watching. Streaming turns a four-second wait into a four-second read. Where the first token is genuinely slow — a long retrieval phase — stream the intermediate state instead: what is being searched, what was found. Honest progress beats a spinner.
Parallelise what does not depend
Agent loops are often written sequentially out of habit. Independent retrievals, independent tool calls and independent validations can run concurrently. The dependency graph is usually much wider than the code implies, and this is free latency.
Cache the stable prefix
Prompt caching cuts time to first token substantially when the prefix is long and stable, which describes most production prompts. The requirement is byte-identical prefixes, so put system instructions and tool definitions first and keep volatile content at the end.
Set timeouts per stage and degrade deliberately
A single global timeout produces the worst outcome: everything fails together after a long wait. Per-stage timeouts with defined fallbacks — skip reranking, return fewer sources, answer without the optional enrichment — turn a failure into a slightly worse answer delivered on time.
Watch the tail
Optimising p50 while p99 climbs makes the system feel worse even as the average improves, because users remember the bad request. Alert on p95 and p99, and treat a widening gap between them as a queueing or retry problem rather than a model problem.
What to take away
Budget by stage, protect time to first token, parallelise the independent parts, degrade on a plan, and measure the tail. Perceived performance is a design decision.