Appendix A: Glossary¶
The vocabulary of production AI systems spans distributed systems, machine learning, and platform engineering. This glossary defines terms as they are used in this book, with cross-references to the chapters where each concept is covered in depth.
A¶
Action. A function call with parameters generated by an agent. Actions
are typed -- the agent knows what actions are available and what parameters
each expects. An action might be lookup(entity: "france", attribute:
"capital") or transfer(from: "checking", to: "savings", amount: 500).
See Ch 24, Ch 25.
Adapter. The per-provider code that translates the gateway's canonical request and response format to and from a provider's wire format. Adapters are where every provider's peculiarities are quarantined. See Ch 18.
ADR (Architecture Decision Record). A document capturing a significant architectural decision, its context, alternatives considered, and rationale. ADRs provide a historical record that explains why the system looks the way it does. See Ch 32.
ANN (Approximate Nearest Neighbor). Algorithms that find vectors close to a query vector without computing distance to every stored vector. HNSW is the dominant ANN algorithm in production vector databases. ANN trades exactness for speed: it might miss the true nearest neighbor but finds very close ones in O(log n) time. See Ch 16.
Approval workflow. A pattern where high-stakes operations are recorded as pending and require explicit human approval before completion. The model can initiate but not finalize. Sometimes called human-in-the-loop. See Ch 24.
Argument sanitization. Cleaning tool arguments to remove or block potentially dangerous content such as injection payloads, control characters, or excessive length. Sanitization happens after validation. See Ch 24.
Arithmetic intensity. The ratio of compute operations to memory operations. High intensity means compute-bound; low intensity means memory-bound. Prefill has high intensity; decode has low intensity. See Ch 10.
Attention. The mechanism that lets each token in a sequence interact with every other token. For a sequence of length n, attention computes an n-by-n matrix of pairwise relationships. This is where the O(n^2) complexity comes from. See Ch 10.
Audit trail. An append-only log of every tool operation, recording who requested it, what arguments were supplied, what the outcome was, and when it happened. The foundation for forensics, compliance, and anomaly detection. See Ch 24, Ch 29.
B¶
Backpressure. The mechanism for consumers to signal they cannot keep up. When a model provider rate-limits you, backpressure pauses consumption rather than building an unbounded queue of requests that will all fail. See Ch 6.
Batch size. The number of sequences processed simultaneously by an inference server. Larger batches amortize the cost of reading model weights across more sequences, improving throughput at the cost of latency. See Ch 10.
BM25 (Best Matching 25). A ranking function that scores documents against a query based on term frequency (TF), inverse document frequency (IDF), and document length normalization. BM25 has two tunable parameters: k1 (controls TF saturation, typically 1.2) and b (controls length normalization, typically 0.75). Unlike raw TF-IDF, BM25 saturates so that additional occurrences of a term add diminishing value. See Ch 15, Ch 16.
Brownout. Degraded provider performance without elevated error rates. The dominant failure mode in practice, and the one most systems handle worst. Latency increases significantly while error rates stay flat. See Ch 1, Ch 18.
C¶
Canonical schema. The gateway's own request and response shape. Consumers code against this, never against a provider's schema. If a provider's schema leaks into consumer code, you have built a proxy, not a gateway. See Ch 18.
CAP theorem. During a network partition, a distributed system must choose between consistency (reject requests to maintain correctness) and availability (serve requests with potentially stale data). AI systems add a twist: the "partition" is often a third-party provider, and you cannot choose to fix it. You can only choose how to fail. See Ch 1.
Checkpoint. A saved state of the plan and goal tracking, enabling backtracking. When an approach fails, the agent can restore a checkpoint and try a different path rather than starting from scratch. See Ch 25.
Chunking. The process of splitting a document into smaller segments for indexing and retrieval. Chunk size affects both retrieval precision (smaller chunks match more specifically) and context coherence (larger chunks preserve more context). Common strategies include fixed-size, sentence-based, and semantic chunking. See Ch 14, Ch 15.
Circuit breaker. A pattern that stops calling a failing dependency to prevent cascade failures. Three states: closed (normal operation), open (requests rejected immediately), and half-open (testing if recovery has occurred). For AI workloads, circuit breakers must trip on latency, not just errors -- a provider at 40-second latency is effectively failed even if no errors appear. See Ch 1, Ch 18.
Connection pooling. Reusing a set of persistent connections to a service rather than establishing new connections per request. Critical for AI workloads where requests hold connections for tens of seconds. Pool exhaustion is often the first scaling bottleneck. See Ch 2, Ch 18.
Consistency model. A contract about what reads see after writes. Eventual consistency guarantees convergence but allows stale reads. Strong consistency guarantees reads see the latest write but may fail during partitions. Causal consistency preserves ordering within causal chains. See Ch 1.
Consumer group. A set of Kafka consumers that cooperate to consume a topic. The broker assigns partitions to consumers in the group, ensuring each partition has exactly one owner. When a consumer joins or leaves, the broker rebalances partitions across the remaining consumers. See Ch 6.
Context window. The maximum number of tokens a model can process in a single request, including both input and output. Context windows range from 4K tokens for small models to 200K+ tokens for frontier models. Context usage directly affects cost and latency. See Ch 11, Ch 13.
Continuous batching. A batching strategy that allows sequences to join and leave the inference batch independently. When a sequence completes generation, its slot is immediately filled by a waiting request. Also called iteration-level batching. More complex than static batching but provides better GPU utilization. See Ch 10.
Cosine similarity. A measure of angle between two vectors, ranging from -1 (opposite directions) through 0 (orthogonal) to 1 (same direction). For normalized vectors (magnitude 1), cosine similarity equals the dot product. Retrieval systems typically normalize embeddings at index time to make similarity computation a single dot product. See Ch 12, Ch 16.
Cost attribution. Tracking AI spend by tenant, workload, or feature. Because AI requests have highly variable costs (a 100-token request and a 10,000-token request cost 100x different), cost attribution requires token-level accounting, not just request counting. See Ch 23.
D¶
Dead letter queue (DLQ). A destination for messages that cannot be processed after repeated attempts. Rather than block the partition forever, unprocessable messages are moved to a separate topic for manual inspection and potential replay. See Ch 6.
Decode. The phase of inference where the model generates output tokens one at a time. Each token requires reading the model weights and the entire KV cache, making this phase memory-bound. Decode is dramatically slower per token than prefill. See Ch 10, Ch 13.
Embedding. A fixed-length vector representation of text. Two texts with similar meaning produce vectors that are close in the embedding space. The dimensions are learned during training and do not have human-interpretable meaning. Common embedding dimensions range from 384 to 1536. See Ch 12, Ch 16.
Eval (Evaluation). A test suite measuring model or system behavior against expected outputs. Evals range from simple string matching to LLM-as-judge scoring. The eval suite is what makes claims testable; without it, assertions about quality are opinions. See Ch 21.
Eventual consistency. A consistency model guaranteeing that, if no new writes occur, all replicas will eventually converge to the same value. Reads may return stale data temporarily. Suitable for user-facing state where 200ms of staleness is acceptable. See Ch 1.
Exactly-once delivery. A message delivery guarantee ensuring each message is processed exactly one time, even with retries. Achieved through idempotency keys combined with transactional commits. See Ch 6, Ch 8.
Exponential backoff. A retry strategy where wait time doubles after each
failure. With jitter (random variation), backoff prevents synchronized retries
that create load spikes. The formula: delay = min(cap, base * 2^attempt) *
random(0.5, 1.5). See Ch 1, Ch 18.
F¶
Fail-open. A policy where, if a check cannot complete, the request is allowed through. Budget checks should generally fail open because blocking all traffic is worse than temporary overspend. Contrast with fail-closed. See Ch 18.
Fail-closed. A policy where, if a check cannot complete, the request is rejected. PII redaction should fail closed because allowing potentially sensitive data through is worse than temporary unavailability. Contrast with fail-open. See Ch 18, Ch 22.
Frontier model. A capability tier representing the most capable models available. Used in this book instead of specific model names because names and capabilities change quarterly. Frontier models are 10-100x more expensive than mid-tier models but may be required for complex reasoning. See terminology in STYLE-GUIDE.md.
Function calling. Synonym for tool calling in some provider APIs. The capability for a model to output structured function invocations alongside or instead of text. See Ch 24.
G¶
Gateway. A service that terminates all model traffic from internal consumers, applies policy, dispatches to one or more providers, and emits telemetry. Single ingress, many egresses. Not a proxy -- a gateway transforms requests, enforces budgets, and provides observability. See Ch 18.
Goal drift. When an agent's recent actions no longer relate to the original goal. Drift detection compares recent action signatures against goal keywords and flags divergence. A drifting agent needs intervention, not more iterations. See Ch 25.
Golden set. A curated collection of test cases with known correct answers used to evaluate model or system behavior. Golden sets should be versioned, reviewed, and updated as edge cases emerge. See Ch 21.
H¶
Handoff. The transfer of control from one agent to another, including context and partial results. Handoffs enable specialization -- each agent handles tasks within its competence. See Ch 27.
Heartbeat interval. How often a Kafka consumer sends heartbeats to the broker. Default is 3 seconds. In traditional consumers, heartbeats are sent by a background thread while the main thread processes messages. See Ch 6.
Histogram. A metric type that counts observations in buckets. Latency distribution, token count distribution. Histograms enable percentile calculation without storing every value. For LLM workloads, bucket boundaries must extend to 60+ seconds. See Ch 9.
HNSW (Hierarchical Navigable Small World). The dominant approximate nearest neighbor algorithm in production vector databases. HNSW builds a hierarchy where upper layers have fewer nodes and longer edges. Search enters at the top and descends, reducing the search space at each layer. Build-time parameters: M (connections per node), efConstruction. Query-time parameter: ef. See Ch 16.
Human-in-the-loop. A pattern where high-stakes operations require human approval before execution. Synonym for approval workflow. See Ch 24.
Hybrid search. Combining vector similarity search with keyword-based search (typically BM25) and fusing the results. Hybrid search handles both semantic queries ("how do we handle authentication") and exact queries ("PostgresAdapter timeout") better than either method alone. See Ch 16.
I¶
Idempotency. The property that processing a request multiple times produces the same result as processing it once. Critical for AI workloads because the cost of duplicate processing is high. Implemented via idempotency keys. See Ch 6, Ch 24.
Idempotency key. A client-supplied identifier that uniquely identifies a logical operation. If the same key is seen twice, the second execution returns the result of the first without re-executing. Prevents duplicate side effects from retries. See Ch 6, Ch 24.
Incident. An event that degrades or threatens to degrade service quality, requiring a response. Incidents are classified by severity, with SEV-1 being customer-impacting. AI-specific incidents include model degradation, cost overruns, and safety violations. See Ch 33.
Inverted index. A data structure mapping terms to the documents containing them. Given a query term, the index returns all matching documents without scanning the full corpus. BM25 and other keyword methods rely on inverted indexes for sub-linear lookup. See Ch 15, Ch 16.
Isolation. Preventing tenants from affecting each other. Resource isolation limits how much capacity one tenant can consume. Data isolation prevents cross-tenant data access. Failure isolation prevents one tenant's errors from cascading. See Ch 31.
J¶
Jailbreak. An attempt to override a model's safety guidelines and elicit harmful or unintended behavior. Jailbreaks exploit the model's instruction- following capability to circumvent its training. Defense requires layered controls: input filtering, output filtering, and behavioral monitoring. See Ch 22.
Jitter. Random variation added to retry delays to prevent synchronized
retry spikes. Full jitter uses random(0, backoff) rather than fixed backoff.
Without jitter, retries from multiple clients align and create periodic load
spikes that can cause cascading failures. See Ch 1, Ch 18.
K¶
Kafka. A distributed event streaming platform. Kafka provides durable, ordered message storage with consumer groups for parallel processing. For AI workloads, default timeouts must be increased because processing times are 100x longer than traditional consumers expect. See Ch 6.
KV cache. Memory storing previously computed key-value pairs during autoregressive generation to avoid recomputation. Grows linearly with sequence length. For long contexts, the KV cache dominates GPU memory usage. A 7B model with 4K context requires ~2GB of KV cache per sequence. See Ch 10.
L¶
Latency-aware circuit breaker. A circuit breaker that trips on slow calls, not just failed ones. Critical for AI workloads where brownouts (slow but successful responses) are the dominant failure mode. Configure with a slow-call threshold in absolute milliseconds, not relative to baseline. See Ch 18.
LLM gateway. Synonym for gateway in the context of language models. See Gateway.
Load balancing. Distributing requests across multiple backend instances. For AI workloads, round-robin fails because requests have variable duration. Use least-connections or weighted algorithms. Long-lived streaming connections require sticky routing or graceful handoff. See Ch 3.
Log correlation. The practice of including trace ID and span ID in every log entry. When you find a problematic trace, you can query all logs for that trace ID across all services. See Ch 9.
Loop detection. Identification of repetitive action patterns that indicate an agent is stuck. A simple detector tracks a sliding window of recent actions; if a pattern repeats, the agent is looping. More sophisticated detectors track "progress" and flag lack of forward movement. See Ch 25.
M¶
Max poll interval. The maximum time between Kafka poll() calls. Unlike session timeout (which is about heartbeats), this is about the processing loop itself. If a consumer does not call poll() within this interval, it is considered stuck. Default is 5 minutes. See Ch 6.
MCP (Model Context Protocol). A protocol for exposing tools and resources to LLM applications in a standardized way. MCP separates tool definition from tool implementation, enabling tool sharing across applications. See Ch 26.
Mean Reciprocal Rank (MRR). The average of 1/rank for the first relevant document across a set of queries. If the first relevant document appears at position 1, the reciprocal rank is 1.0. If it appears at position 3, the reciprocal rank is 0.33. MRR measures how quickly users find what they need. See Ch 16, Ch 17.
Metric. A numeric measurement aggregated over time. Request count, error rate, latency percentile, token consumption. Metrics are how you know the system is healthy or sick without reading every log line. For AI workloads, token-based metrics are essential -- request counts alone are misleading. See Ch 9.
Mid-tier model. A capability tier representing models balanced between capability and cost. Used instead of specific model names. Mid-tier models are typically 10x cheaper than frontier models with 80-90% of the capability for most tasks. See terminology in STYLE-GUIDE.md.
MTTD (Mean Time to Detect). The average time between an incident starting and being detected. For AI systems, detection requires AI-specific metrics: latency percentiles over 30 seconds, token consumption anomalies, and quality degradation measured by evals. See Ch 33.
MTTR (Mean Time to Recover). The average time between incident detection and service restoration. Reducing MTTR requires runbooks, automation, and clear escalation paths. For AI systems, "recovery" may mean failing over to a different provider rather than fixing the original. See Ch 33.
Multi-agent system. A system where multiple specialized agents collaborate to accomplish goals. Agents may work sequentially (pipeline), in parallel (swarm), or hierarchically (orchestrator delegates to workers). See Ch 27.
Multi-provider routing. Directing requests to different providers based on cost, capability, latency, or availability. A fallback provider handles requests when the primary is degraded. Never add a provider to a route without eval coverage. See Ch 19.
Multi-tenant. A system serving multiple tenants (customers, organizations) from shared infrastructure. Multi-tenant AI systems require isolation (one tenant cannot affect another), fairness (no tenant monopolizes resources), and attribution (cost and usage tracked per tenant). See Ch 31.
N¶
Noisy neighbor. A tenant whose workload degrades performance for other tenants on shared infrastructure. In AI systems, a tenant making expensive requests (long prompts, high output) can exhaust shared provider quotas. Fix with per-tenant rate limiting and resource isolation. See Ch 31.
O¶
Observation. The result of an action in an agent loop, including success/failure status, the actual result (if any), and error details (if failed). Observations are the ground truth that reasoning builds on. See Ch 25.
Offset. A Kafka consumer's position in a partition. Offsets are committed (checkpointed) to track progress. When a consumer restarts or a partition is reassigned, consumption resumes from the last committed offset. See Ch 6.
OpenTelemetry. A vendor-neutral standard for observability instrumentation. One SDK, one wire protocol (OTLP), one semantic convention. Instrument once, export to any backend. The ecosystem converged on OpenTelemetry because fragmentation was costing everyone more than collaboration. See Ch 9.
Orchestration. Coordinating multiple agents to accomplish a goal. The orchestrator decomposes tasks, assigns them to agents, tracks progress, and handles failures. Orchestration patterns include supervisor-worker, pipeline, and round-robin. See Ch 27.
Outbox pattern. A pattern for reliable event publishing that writes events to a database table (the outbox) atomically with business operations, then publishes from the outbox asynchronously. Avoids the dual-write problem where a database commit succeeds but event publishing fails. See Ch 8.
Overlap. In chunking, the number of tokens shared between adjacent chunks. Overlap preserves context across chunk boundaries, improving retrieval when relevant information spans chunks. Typical overlap is 10-20% of chunk size. See Ch 14, Ch 15.
P¶
p99 (99th percentile). The latency below which 99% of requests complete. For AI workloads, p99 can be 10-100x higher than median due to variable prompt lengths and provider queue times. p99 is the latency that matters for timeouts -- the median request completes far sooner. See Ch 1, Ch 9.
Partition. A Kafka topic is split into partitions, each an ordered, immutable sequence of messages. Partitions are the unit of parallelism: each partition is consumed by exactly one consumer in a consumer group at any time. More partitions allow more consumers to work in parallel. See Ch 6.
Partition tolerance. In CAP theorem, the ability of a system to continue operating despite network partitions. All distributed systems must be partition tolerant because networks are unreliable. The choice is between consistency and availability during partitions. See Ch 1.
PII (Personally Identifiable Information). Data that can identify an individual: names, emails, phone numbers, social security numbers. AI systems must redact or filter PII before logging, caching, or sending to model providers. PII handling should fail closed. See Ch 22.
Planning horizon. How far ahead an agent plans before acting. A short horizon (one step) is reactive. A long horizon (full task graph) is deliberate but fragile -- the plan may become invalid after the first action if the world changes. See Ch 25.
Postmortem. A document analyzing an incident after resolution. Postmortems cover timeline, root cause, impact, and action items. Blameless postmortems focus on system improvements rather than individual fault. See Ch 33.
Precision. The fraction of retrieved documents that were relevant. If 5 documents were returned and 3 were relevant, precision is 0.60. High precision means you are not returning irrelevant results. Contrast with recall. See Ch 16, Ch 17.
Prefill. The phase of inference where the model processes the input prompt. All input tokens are processed in parallel, making this phase compute-bound. The time to complete prefill is the time-to-first-token. See Ch 10, Ch 13.
Prompt injection. An attack where malicious input causes a model to ignore its instructions and follow attacker-controlled instructions instead. Injections may appear in user input, retrieved documents, or tool outputs. Defense requires input sanitization, output filtering, and structural controls. See Ch 22, Ch 24.
Provider. A third party serving model inference over an API. In this book, providers are interchangeable at the interface level and very much not interchangeable in behavior. The gateway abstracts provider differences. See Ch 18.
Q¶
QPS (Queries Per Second). A measure of request throughput. For AI systems, QPS alone is misleading because requests vary dramatically in cost. Token throughput is more meaningful for capacity planning. See Ch 23.
R¶
RAG (Retrieval-Augmented Generation). A pattern where relevant documents are retrieved from a corpus and included in the model's context to ground its responses in specific information. RAG reduces hallucination and enables models to answer questions about data they were not trained on. See Ch 14, Ch 15, Ch 16, Ch 17.
Rate limiting. Restricting request volume to prevent overload or enforce quotas. For AI workloads, rate limiting should be token-based, not request-based, because request cost varies by 100x or more. See Ch 1, Ch 23.
ReAct (Reason-Act-Observe). An agent architecture alternating between reasoning (producing a thought), acting (executing a tool), and observing (receiving results). The loop continues until a termination condition is met. ReAct improves both accuracy and interpretability over pure action sequences. See Ch 25.
Recall. The fraction of relevant documents that were retrieved. If 4 documents are relevant and 3 were returned, recall is 0.75. High recall means you are not missing relevant results. Contrast with precision. See Ch 16, Ch 17.
Reciprocal Rank Fusion (RRF). A method for combining ranked lists that uses ranks instead of scores. For each item, compute 1/(k + rank) where k is a constant (typically 60). Sum these contributions across all lists. Higher totals rank higher in the fused output. RRF is immune to score scale differences because it only uses ordinal position. See Ch 16.
Reflection. In an agent context, analysis of an observation to decide next steps. When an action fails, reflection asks: should I retry? Try something different? Give up? Reflection learns from failure rather than blindly repeating it. See Ch 25.
Replication. Maintaining copies of data across multiple nodes for durability and availability. In database systems, replication strategies include synchronous (all replicas confirm before commit) and asynchronous (commit returns before all replicas confirm). See Ch 5.
Reranking. A second-stage ranking step that applies a more expensive model to reorder retrieval results. Cross-encoder rerankers score each query-document pair independently, improving precision at the cost of O(n) inference calls. See Ch 17.
Reserve-then-reconcile. A budget tracking pattern where pessimistic estimates are reserved at request admission and reconciled to actual usage at completion. Prevents budget overshoot from concurrent in-flight requests. See Ch 1, Ch 18, Ch 23.
Retry storm. A cascade where retries generate more retries, amplifying load on an already-stressed system. Retry storms often trigger when timeouts fire during a brownout and each timeout spawns retries. Prevention requires retry budgets and latency-aware circuit breakers. See Ch 1, Ch 18.
RFC (Request for Comments). A document proposing a technical change for review before implementation. RFCs ensure significant decisions are examined by multiple perspectives and create documentation that outlasts the original engineers. See Ch 32.
Runbook. A documented procedure for responding to a specific operational situation. Runbooks reduce incident response time by providing step-by-step instructions. AI system runbooks must cover provider-specific failure modes. See Ch 33.
S¶
Saga. A pattern for managing distributed transactions through a sequence of local transactions with compensating actions. If step 3 fails, steps 1 and 2 are compensated (rolled back). Sagas provide eventual consistency without distributed locks. See Ch 8.
Schema validation. Checking that tool arguments conform to declared types, required fields, and constraints before execution. Malformed calls are rejected without reaching business logic. See Ch 24.
Semantic caching. A caching strategy that returns cached results for queries "similar" to previous queries, not just exact matches. Semantic caching is risky -- it can return answers to questions nobody asked -- and should only be enabled with eval coverage. Contrast with exact-match caching. See Ch 18.
Session timeout. The time a Kafka broker waits without a heartbeat before declaring a consumer dead. Default is 10 seconds. For AI workloads where processing takes 20-60 seconds, session timeout must be increased to 2-3x maximum processing time. See Ch 6.
Sharding. Distributing data across multiple nodes based on a partition key. Each shard holds a subset of the data. Sharding enables horizontal scaling but complicates queries that span shards. See Ch 4, Ch 5.
SLA (Service Level Agreement). A contractual commitment on service quality, typically including availability (99.9%) and latency (p99 < 1s) targets. Violating SLAs may trigger financial penalties. See Ch 28, Ch 31.
SLI (Service Level Indicator). A quantitative measure of service quality: request latency, error rate, throughput. SLIs are the data from which SLO compliance is calculated. See Ch 28, Ch 31.
SLO (Service Level Objective). An internal target for service quality, stricter than the external SLA. If your SLA is 99.9% availability, your SLO might be 99.95% to provide buffer. Alert when SLO is breached, not just SLA. See Ch 28, Ch 31.
Sliding window. A memory pattern that retains only the most recent N messages or tokens, discarding older content. Sliding window memory bounds resource usage but loses long-term context. Contrast with summary memory. See Ch 20.
Small model. A capability tier representing lightweight models optimized for cost and speed. Used instead of specific model names. Small models are suitable for classification, routing, and simple extraction but may struggle with complex reasoning. See terminology in STYLE-GUIDE.md.
Span. The fundamental unit of distributed tracing. A span has a name, a start time, an end time, a parent (or none for root spans), and attributes. Spans form a tree within a trace. See Ch 9.
Static batching. A batching strategy that groups requests together and processes them as a unit. All sequences are padded to the length of the longest. When the batch completes, all sequences complete. Simpler than continuous batching but wastes compute on padding. See Ch 10.
Streaming. Delivering model output incrementally as tokens are generated rather than waiting for the complete response. Streaming improves perceived latency (users see progress) but complicates error handling (retries are impossible after the first byte). See Ch 13, Ch 18.
Strong consistency. A consistency model guaranteeing that reads always see the most recent write. Requires coordination between replicas, which adds latency and may cause unavailability during partitions. Necessary for financial data and audit logs. See Ch 1.
Structured log. A log entry with machine-parseable fields. Instead of
"Request failed for user 123", you log {level: "error", user_id: 123,
error: "timeout"}. Structured logs are searchable and aggregatable;
unstructured logs require regex. See Ch 9.
Summary memory. A memory pattern that maintains a compressed summary of conversation history rather than raw messages. Summaries preserve semantic content while bounding token usage. Requires periodic summarization model calls. See Ch 20.
T¶
Task decomposition. Breaking a complex goal into subtasks with dependencies. "Transfer money and set up auto-pay" becomes two tasks where the second depends on the first. The planner tracks which tasks are complete, which are blocked, and which are ready to execute. See Ch 25.
Technical strategy. The high-level plan connecting business objectives to engineering choices. Technical strategy answers "what should we build and why" at a level above individual RFCs. See Ch 34.
Tenant. A customer, organization, or account in a multi-tenant system. The book uses "tenant" rather than "customer" or "org" for consistency. Tenants are the unit of isolation, billing, and quota enforcement. See Ch 31.
TF-IDF (Term Frequency-Inverse Document Frequency). A scoring method that weights terms by how often they appear in a document (TF) and how rare they are across the corpus (IDF). BM25 extends TF-IDF with saturation and length normalization. See Ch 15.
Thought. In a ReAct loop, the model's internal reasoning made explicit. A thought might be "I need to find the user's account balance before I can calculate fees." Thoughts provide interpretability and structure. See Ch 25.
Time to first token (TTFT). The latency between request submission and receiving the first token of the response. Mostly fixed overhead: authentication, queue wait, initial context processing. Highly variable under load. The component of LLM latency that timeout strategies most often underestimate. See Ch 1, Ch 10, Ch 13.
Token. The fundamental unit of text for language models. Tokens are subword pieces, roughly 4 characters on average for English. Tokens matter because models charge per token, context windows are measured in tokens, and latency scales with token count. See Ch 11.
Token budget. A per-tenant allowance denominated in tokens or currency over a window. Distinct from a rate limit, which is denominated in requests. Token budgets match how providers actually bill and rate-limit. See Ch 1, Ch 23.
Tokenization. Converting text into a sequence of token IDs. Different models use different tokenizers; the same text produces different token counts with different tokenizers. Tokenization also affects how the model "sees" the text. See Ch 11.
Tool calling. The capability for a model to output structured function invocations alongside or instead of text. The model specifies a function name and arguments; the application executes the function and returns the result to the model for incorporation into its response. See Ch 24.
Tool definition. A schema that declares a tool's name, description, and parameter types. The model uses this schema to understand when and how to invoke the tool. The application uses it to validate incoming calls. See Ch 24, Ch 26.
TPS (Tokens Per Second). A measure of generation speed. Typical values range from 30 TPS for frontier models to 100+ TPS for smaller models. TPS determines how long generation takes for a given output length. See Ch 13.
Trace. A tree of spans representing a single request's journey through the system. The root span is the entry point; child spans are operations called by the parent. All spans in a trace share a trace ID. See Ch 9.
Trace context. The data propagated across service boundaries to correlate
spans. In HTTP, this is the traceparent header. In Kafka, this is a message
header. Without context propagation, traces break at every boundary. See Ch 9.
Transformer. The neural network architecture underlying all modern LLMs. A transformer is a stack of identical layers, each containing an attention mechanism and a feed-forward network. The number of layers determines depth; the hidden size determines width. See Ch 10.
V¶
Vector database. A database optimized for storing and querying high- dimensional vectors. Vector databases use ANN algorithms (typically HNSW) for efficient similarity search. Examples include Pinecone, Weaviate, and pgvector. See Ch 16.
Vector search. Finding items similar to a query by comparing vector representations. Vector search captures semantic similarity (documents about related concepts match) but may miss lexical matches (exact keywords). See Ch 16.
W¶
W3C Trace Context. The standard format for trace context propagation.
The traceparent header contains version, trace ID, parent span ID, and
flags. Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01.
See Ch 9.
Notes on terminology¶
This book uses consistent terminology throughout. Preferred terms and their alternatives:
| Preferred | Not |
|---|---|
| LLM gateway | AI gateway, model proxy, LLM proxy |
| provider | vendor, backend, upstream model |
| retrieval | RAG pipeline (reserve "RAG" for the pattern itself) |
| agent runtime | agent framework, agent engine |
| eval | evaluation suite, evals (plural only when countable) |
| tenant | customer, org, account |
| request | call, invocation, completion |
| token budget | context budget, token limit |
Model capability tiers used throughout: - frontier: Most capable models available - mid: Balanced capability and cost - small: Optimized for speed and cost
See STYLE-GUIDE.md for the complete terminology guide.