AI Infrastructure at Scale: Lessons from Serving Billions of Requests

AI Infrastructure at Scale: Lessons from Serving Billions of Requests

Serving AI at scale is fundamentally different from traditional web services. You're not just routing HTTP requests—you're orchestrating GPU clusters, managing model artifacts that consume gigabytes of RAM, and optimizing for both latency and throughput in ways that defy conventional wisdom.

I've architected systems serving billions of AI requests monthly. This article shares the hard-won lessons—the patterns that work, the anti-patterns that fail, and the tradeoffs that define production AI infrastructure.

The Scale Problem

Let's establish baseline numbers. A typical production LLM workload:

  • Request volume: 10M+ requests/day
  • Token throughput: 100B+ tokens/day
  • Latency requirements: P50 < 200ms, P99 < 2s
  • Availability: 99.99% (four nines)
  • Cost budget: $0.001-0.01 per 1K tokens

These constraints create a brutal optimization landscape. Every millisecond and every cent matters.

GPU Cluster Architecture

The Memory Wall

GPUs are memory-constrained, not compute-constrained. An H100 has:

  • 80GB HBM3 memory
  • 989 TFLOPS FP8 compute
  • But model weights for GPT-4-class models exceed 200GB

The Solution: Model parallelism strategies:

| Strategy | How It Works | Best For | |----------|--------------|----------| | Tensor Parallelism | Split layers across GPUs | Single node, low latency | | Pipeline Parallelism | Split model stages across GPUs | Multi-node, throughput | | Sequence Parallelism | Split sequence dimension | Long context windows | | Expert Parallelism | Route to different "expert" GPUs | MoE architectures |

Production Choice: I use tensor parallelism within nodes (NVLink is fast) and pipeline parallelism across nodes (Ethernet is slower but acceptable for coarse-grained communication).

Dynamic Batching

The key throughput optimization: batch requests dynamically.

Performance Impact: Dynamic batching improves throughput by 5-10x compared to single-request inference, with only marginal latency increase (10-20ms).

Continuous Batching (vLLM-style)

Traditional batching waits for all sequences in a batch to complete. Continuous batching (inflight batching) is better:

This improves GPU utilization from ~30% to >80%.

The Request Routing Problem

Not all requests are equal. Smart routing is critical.

Model Routing

Route to appropriate model based on task complexity:

Cost Impact: Proper model routing reduces inference costs by 60-70%.

Geographic Routing

Latency to first token is critical. Deploy inference clusters geographically:

Implementation: Use Anycast IP with health checks. If Tokyo cluster is overloaded, failover to Singapore or US-West.

A/B Testing at the Edge

Test model variants without client changes:

Caching: The 1000x Optimization

Caching is make-or-break for AI infrastructure economics.

Embedding Cache

Embed the same text twice? That's waste.

Hit Rate: 40-60% in production. At $0.0001 per 1K tokens, this saves thousands daily.

Semantic Cache

Exact match is too restrictive. Cache semantically similar queries:

Performance: Additional 15-25% cache hit rate on conversational applications.

Response Cache

For deterministic tasks (code generation, data extraction), cache full responses:

TTL Strategy:

  • Code generation: 1 hour (code doesn't change)
  • Factual queries: 24 hours (facts are stable)
  • Creative writing: No cache (want variety)

Vector Database Optimization

RAG applications live and die by vector search performance.

Index Selection

| Index Type | Use Case | Recall | Query Time | |------------|----------|--------|------------| | Flat (Brute Force) | Small datasets (100M) | 85-95% | Fast with disk |

Production Default: HNSW with ef_construction=128, M=16. Tuned for 99% recall at sub-10ms query times.

Hybrid Search

Vector similarity alone is insufficient. Combine with BM25:

Improvement: 15-20% better retrieval accuracy than vector-only search.

Embedding Model Selection

Not all embeddings are equal:

| Model | Dimensions | Strength | Best For | |-------|------------|----------|----------| | text-embedding-3-large | 3072 | General purpose | Default choice | | text-embedding-3-small | 1536 | Fast, cheap | High volume, simple tasks | | Cohere Embed v3 | 1024 | Multilingual | Non-English content | | voyage-2 | 1024 | Domain-specific | Technical documents | | E5-Mistral | 4096 | Long context | Legal, academic |

Cost Tradeoff: text-embedding-3-small is 10x cheaper and 80% as good. Use it for first-pass filtering, then re-rank with larger model.

Cost Management

AI infrastructure costs can spiral. Here's how to control them.

Usage-Based Alerts

Token Optimization

Every token costs money. Optimize relentlessly:

Prompt Compression:

Response Streaming:

  • Stream tokens as they're generated
  • Allow users to cancel mid-generation
  • Reduces wasted tokens on abandoned requests by 30%

Quantization:

  • Use INT8 or FP8 quantized models where possible
  • 2x throughput, minimal quality loss (<2%)

Reserved Capacity vs. On-Demand

For steady-state workloads, reserved GPU capacity is 40-60% cheaper:

| Deployment Model | Cost per H100/hr | Best For | |------------------|------------------|----------| | On-Demand | $2.50 | Spiky workloads, experimentation | | Reserved (1 year) | $1.50 | Baseline production traffic | | Spot/Preemptible | $0.75 | Fault-tolerant batch jobs |

Strategy: 70% reserved for baseline, 30% on-demand for spikes.

Observability

You can't optimize what you can't measure.

The AI Metrics Stack

Model Metrics:

  • Token latency (TTFT, TBT—time between tokens)
  • Token throughput (tokens/sec)
  • Error rate by model and version
  • Cache hit rates

Business Metrics:

  • Cost per request
  • Cost per user
  • Revenue per token
  • Request success rate

Quality Metrics:

  • Hallucination rate (measured via evals)
  • User feedback scores
  • Task completion rates

Distributed Tracing

Trace requests through the entire pipeline:

Tooling: OpenTelemetry + Jaeger for traces, Prometheus + Grafana for metrics.

Reliability Patterns

Circuit Breakers

When a model endpoint fails, fail fast:

Graceful Degradation

When primary model is overloaded:

Request Queuing

Smooth traffic spikes:

The Future: Edge Inference

The next frontier is running models on the edge:

Current Feasible:

  • Embedding models (BGE-small, 100MB)
  • Classification models (DistilBERT, 250MB)
  • Small LLMs (Phi-2, 2GB)

Edge Deployment Stack:

  • ONNX Runtime for model execution
  • WebNN for browser inference
  • Core ML / NNAPI for mobile

Use Cases:

  • Real-time transcription
  • On-device personalization
  • Privacy-sensitive processing

Conclusion

Building AI infrastructure at scale is a systems engineering discipline. The winners won't be those with the best models—they'll be those with the best infrastructure to serve those models reliably, efficiently, and cost-effectively.

The patterns in this article—dynamic batching, smart routing, aggressive caching, and graceful degradation—are the difference between a demo and a production system.

The future belongs to infrastructure engineers who understand both GPU architectures and distributed systems. That intersection is where the next generation of AI applications will be built.