Inside Agentic
Tool Use

Modern LLM APIs are deceptively simple on the surface — you POST some JSON, you get text back. But when those APIs support tool use, what actually happens between request and response spans multiple subsystems: authentication layers, context management, GPU-accelerated autoregressive decoding, sandboxed function dispatch, and multi-turn feedback loops. This post walks through every stage of that journey, with the math and engineering that make it work.

The Request Leaves Your Machine

It starts with a POST. The client serializes a messages array, an optional system prompt, a tools list, and generation hyperparameters into a JSON body, then fires it over TLS 1.3 to the API gateway.

// Minimal agentic request
{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 4096,
  "system": "You are a research assistant...",
  "messages": [{ "role": "user", "content": "What is the weather in Tokyo?" }],
  "tools": [{
    "name": "get_weather",
    "description": "Returns current weather for a city.",
    "input_schema": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] }
  }]
}

The wire protocol is HTTP/2, meaning the connection is multiplexed — several concurrent requests share a single TCP stream. TLS terminates at an edge load balancer, well before the request reaches compute. The plaintext JSON then travels over the internal network to the API gateway proper.

Gatekeeping Before a Single Token is Processed

The gateway's first job is to decide whether this request should proceed. This happens in strict order and fast — it must complete in single-digit milliseconds or it degrades perceived time-to-first-token.

API Key Authentication

The x-api-key header carries a long random token, typically at least 128 bits of entropy encoded in base-64. The gateway computes a cryptographic hash of the presented key and does a constant-time comparison against stored hashes — the actual secrets are never stored plaintext. Constant-time comparison is critical because otherwise an attacker can probe which key bytes match by measuring tiny response-time differences.

Schema Validation & Input Sanitisation

The JSON body is validated against a schema before it touches downstream services. This catches malformed requests early: missing required fields, invalid tool definitions, wrong content block types, impossible sampling parameters, and messages that exceed the model's context window.

Token counting happens here too. The serialized context must fit within the model's context window $C$. If $\text{tokens}(\text{prompt}) + \text{max\_tokens} > C$, the gateway returns a 400 immediately rather than burning capacity on a request that cannot be served.

Rate Limiting: Token Bucket

Limits are enforced on requests-per-minute (RPM) and tokens-per-minute (TPM) simultaneously. A common implementation uses a token bucket per API key in Redis or another low-latency shared store:

If the bucket has insufficient tokens for the estimated request cost, the gateway responds 429 Too Many Requests with a Retry-After header. More sophisticated systems combine token buckets with sliding-window logs so bursty but legitimate users are treated fairly across multiple regions.

Assembling the Context Window

Once authenticated and validated, the request moves into context assembly. This is where the clean API abstraction gets flattened into a single linear token sequence the model can consume.

The messages array is serialized using a model-specific conversation template. A simplified version looks like:

<!-- BOS -->[BOS] [SYSTEM] You are a helpful assistant... [/SYSTEM]
[HUMAN] What is the weather in Tokyo? [/HUMAN]
[ASSISTANT] ← model generates from here

The tools array is serialized into this same token stream, usually after the system prompt, describing each tool's name, description, and JSON Schema. There is no separate router network deciding whether to call a tool; the tool decision emerges from the same autoregressive forward pass that generates normal text.

KV-Cache Priming

For long static system prompts — a common pattern in agentic deployments — providers cache the key-value state computed during prefill. On subsequent requests sharing the same prefix, cached KV state is reused, dramatically reducing compute and latency for the static portion:

Tokenization: Text to Integer IDs

BPE starts with a vocabulary of individual bytes (256 entries), then iteratively merges the most frequent adjacent pair into a new token, repeating until the vocabulary reaches target size $V$. Common English words map to one or a few tokens; rare words and code are split into subword units.

Each integer token ID $t_i$ is mapped to a dense vector via the embedding matrix $\mathbf{E} \in \mathbb{R}^{V \times d}$, then position is encoded via RoPE:

Inference: The Forward Pass at Scale

At each of $L$ layers, the model applies multi-head self-attention followed by a feedforward network. The attention mechanism computes a weighted sum over all previous token representations:

KV-Cache During Generation

Autoregressive generation produces one token per forward pass. Without caching, each new token would require recomputing attention over the entire context from scratch — $O(n^2)$ per token for a sequence of length $n$. The KV-cache stores the key and value matrices for prior tokens, reducing each generation step to attend over existing state instead of rebuilding it.

This memory pressure is why serving stacks care about grouped-query attention, multi-query attention, cache paging, prefix reuse, and eviction policies. A long tool session can be bottlenecked by KV-cache residency before raw FLOPs become the limiting factor.

Sampling & Temperature

Each forward pass produces logits $\mathbf{z} \in \mathbb{R}^V$. Temperature $T$ controls sharpness before sampling:

Model Parallelism

From Tokens to Tool Calls

When the model decides a tool should be called, it emits a structured tool_use content block. The serving infrastructure detects this mid-stream — stop_reason: "tool_use" is the signal.

// Model's raw response at tool_use stop
{
  "role": "assistant",
  "content": [
    { "type": "text", "text": "I'll look up the weather in Tokyo." },
    { "type": "tool_use", "id": "toolu_01A2B3C4", "name": "get_weather", "input": { "city": "Tokyo" } }
  ],
  "stop_reason": "tool_use"
}

The Execution Model

In client-side tool use, the API returns at stop_reason: "tool_use". Your code executes the tool, then submits a new message containing a tool_result block. The model never directly calls external systems; it emits structured intent, and the host runtime decides what to run.

Sandboxing & Security

For server-side tool execution, sandboxing is non-negotiable. Common defenses include seccomp-bpf or gVisor syscall filtering, Firecracker microVM isolation, network egress allowlists, CPU and memory quotas, wall-clock timeouts, and strict secret-scoping so a tool can only access credentials it actually needs.

Parallel Execution

When multiple tool calls are independent (no data dependency), parallel dispatch gives:

Feeding Results Back: The Agentic Loop

The tool result is injected as a tool_result block attributed to the original tool_use_id. The full updated messages array — the user turn, the assistant's tool call, and the tool observation — is submitted as a new API request. This is the agentic loop.

Each iteration adds tokens to the context:

This growth has two practical effects. First, cost: every later inference call prices and processes the larger accumulated context. Second, reliability: if raw tool outputs are dumped into the transcript without summarization, the model may attend to stale or irrelevant details in later turns.

Loop Termination

The loop terminates when the model returns stop_reason: "end_turn", or when a guardrail fires: maximum iteration count, context-window exhaustion, repeated tool failures, or human-approval checkpoints for high-stakes actions. Good orchestrators do not merely let the model call tools forever; they bound the process and summarize failure modes back into the context.

The Final Response: Streaming to the Client

Once the model emits stop_reason: "end_turn", the response streams via Server-Sent Events. Tokens are emitted as sampled — no buffering of the full response.

Speculative Decoding

A small draft model generates $k$ candidate tokens quickly; the large target model verifies them in one batched forward pass. Accepted tokens are emitted for the cost of ~one forward pass:

Observability: What Gets Measured

After the final response is transmitted, the infrastructure emits telemetry for billing, capacity planning, quality debugging, and safety monitoring. Agentic sessions especially need traceability because the visible answer is the product of several model calls, tool calls, retries, and branching control-flow decisions.

Distributed Tracing

For multi-service pipelines, OpenTelemetry-style tracing is essential. A root trace span is created at the gateway; child spans record validation, inference prefill, token generation, each tool execution, retries, and response streaming. Correlating these by a shared trace_id makes it possible to reconstruct the causal timeline of a complex agentic session and explain why one run was slow or wrong.

The Full Picture

What feels like a single API call is an orchestrated sequence of distributed systems decisions. The agentic loop multiplies this complexity: each iteration re-traverses the full stack with a growing context window and accumulated tool results. Understanding each layer — from TLS handshakes and rate-limit algorithms to KV-cache eviction and speculative decoding — is what separates someone who uses an agentic framework from someone who can build, debug, and optimise one.

• Minimise context growth. Tool results can be large. Summarize aggressively before injecting them back. Every token costs compute on every subsequent call.
• Exploit prompt caching. Put static system prompt and tool schemas at the start of context, never the end. Caching only works for prefixes.
• Parallelize independent tool calls. Concurrently executing $n$ tools reduces latency by factor $\approx n$.
• Design for failure. Tools fail. Networks partition. Rate limits fire. Orchestrators must handle partial failures without feeding poisoned state to the model.
• Trace everything. Agentic sessions are non-deterministic and hard to reproduce. OpenTelemetry spans on every call make debugging tractable.

The stack will continue to evolve — longer contexts, faster inference, richer tool ecosystems — but the fundamental structure of the agentic loop is likely to remain: plan, call, observe, reason, repeat. Getting the plumbing right is what makes the rest possible.