Deploying AI Models
Model Serving at Scale. Master PagedAttention (vLLM), TGI Docker deployment, Ollama for local inference, quantization formats (GGUF, GPTQ, AWQ), and mitigating Serverless Cold Starts.
Model Serving at Scale. Master PagedAttention (vLLM), TGI Docker deployment, Ollama for local inference, quantization formats (GGUF, GPTQ, AWQ), and mitigating Serverless Cold Starts. This hands-on tutorial focuses on practical implementation of deploying ai models concepts.
Deploying AI Models
Serving a model for one person is a solved problem. Serving it for 10,000 concurrent users requires deep optimization. In this chapter, we explore how vLLM, TGI, Ollama, and quantization make large-scale AI inference practical.
1. PagedAttention: The vLLM Revolution 🚀
Older serving engines wasted up to 80% of GPU memory because the Key-Value (KV) cache was stored in contiguous blocks. If a request was shorter than expected, that memory was locked and useless. PagedAttention (used in vLLM) breaks the memory into "pages," similar to how operating systems handle RAM.
- Impact: vLLM can handle 24x higher throughput than standard HuggingFace Transformers.
- Result: You can serve 2x the users on the same expensive GPU.
📚 Official Docs
2. HuggingFace TGI: Docker-Based Serving 🐳
Text Generation Inference (TGI) is HuggingFace's production serving solution — pre-packaged as Docker containers:
📚 Official Docs
3. Ollama: Local Model Deployment 💻
Ollama is the simplest way to run models locally — perfect for development, privacy-sensitive use cases, and air-gapped environments:
📚 Official Docs
- Ollama Documentation
- llama.cpp GitHub — The engine powering Ollama
4. Quantization Formats: Running Big Models on Small Hardware 📦
| Format | Bits/Param | Size (70B model) | Quality Loss | Best For |
|---|---|---|---|---|
| FP16 (Full) | 16-bit | 140 GB | 0% | Training, maximum quality |
| GPTQ (4-bit) | 4-bit | ~40 GB | ~2% | GPU inference, HuggingFace TGI |
| AWQ (4-bit) | 4-bit | ~40 GB | ~1-2% | GPU inference, better than GPTQ |
| GGUF Q4_K_M | 4-bit | ~42 GB | ~2% | CPU/Mac inference via Ollama/llama.cpp |
| GGUF Q2_K | 2-bit | ~21 GB | ~5-10% | Very limited RAM (edge devices) |
5. Serverless vs. Provisioned GPUs ☁️
| Metric | Serverless (RunPod/Modal) | Provisioned (AWS/GCP) |
|---|---|---|
| Cost | Pay-per-second. Great for low traffic. | Fixed monthly. Better for constant traffic. |
| Cold Start | 10-60 seconds to "wake up". | Instant. |
| Scaling | Auto-scales to zero. | Manual or Kubernetes-based. |
Cold Start Mitigation Strategies
- Warming: Send a "fake" lightweight request every 5 minutes to keep the container alive.
- Base Images: Pre-bake model weights into the Docker layer (cached on the server).
- GGUF Quantization: A 4-bit GGUF model loads 4x faster than FP16.
- Smaller Models: Use LLaMA 3.2 3B for edge cases where speed > quality.
Quiz
Quiz
Question 1 of 3What is the primary innovation of vLLM's PagedAttention?
Key Takeaways
✅ vLLM with PagedAttention provides 24x throughput — use it for production GPU inference.
✅ TGI offers Docker-native deployment with Kubernetes support.
✅ Ollama is the fastest path to local model inference with zero configuration.
✅ GGUF Q4_K_M is the best quantization format for CPU/Mac inference.
✅ AWQ outperforms GPTQ for GPU inference at the same bit-width.
✅ Match your VRAM budget to model size + quantization before choosing hardware.
Official Resources
What's Next?
Deployed! Now, let's track everything.
Next Chapter: Advanced Monitoring: LangSmith, Langfuse, and A/B Prompt Testing.