Systematically Improving RAG Applications
A comprehensive technical reference for building and improving Retrieval-Augmented Generation systems.
About This Book
This book teaches a data-driven approach to building RAG systems that get better over time. Unlike tutorials that show you how to build a RAG system once, this book shows you how to build systems that improve continuously based on real user behavior.
The content is designed for two audiences:
- Product Managers who need to understand RAG capabilities, make strategic decisions, and measure success
- Engineers who need to implement, optimize, and maintain RAG systems in production
Throughout the book, content is clearly marked for each audience using admonitions. You can read the full book or focus on the sections most relevant to your role.
Book vs Workshops
This book is a different draft from the workshops. While the workshops came directly from the course lectures, this book synthesizes content from workshops, transcripts, talks, and office hours into a comprehensive technical reference organized for both Product Managers and Engineers.
Book Structure
The book is organized into four parts, plus appendices and supporting materials.
Book 1: Foundations
Build the mental models and infrastructure for continuous improvement.
| Chapter | Title | Description |
|---|---|---|
| Chapter 0 | Introduction - The Product Mindset | Foundational concepts, the improvement flywheel, common failure patterns |
| Chapter 1 | Evaluation-First Development | Synthetic data, precision/recall, statistical significance |
| Chapter 2 | Training Data and Fine-Tuning | Embeddings, re-rankers, contrastive learning, loss functions |
Book 2: User-Centric Design
Understand and serve your users better.
| Chapter | Title | Description |
|---|---|---|
| Chapter 3 | Feedback Systems and UX | Feedback collection, streaming, citations, perceived latency |
| Chapter 4 | Query Understanding and Prioritization | Query clustering, topic modeling, economic value analysis |
Book 3: Architecture and Production
Build robust systems that scale.
| Chapter | Title | Description |
|---|---|---|
| Chapter 5 | Specialized Retrieval Systems | Metadata extraction, RAPTOR, multimodal retrieval |
| Chapter 6 | Query Routing and Orchestration | Router architectures, tool interfaces, latency analysis |
| Chapter 7 | Production Operations | Semantic caching, monitoring, cost optimization, scaling |
Book 4: Advanced Topics
Techniques for complex scenarios.
| Chapter | Title | Description |
|---|---|---|
| Chapter 8 | Hybrid Search | Lexical search, BM25, Reciprocal Rank Fusion |
| Chapter 9 | Context Window Management | Lost in the middle, token budgeting, dynamic context assembly |
Appendices
Technical reference materials for deeper dives.
| Appendix | Title | Description |
|---|---|---|
| Appendix A | Mathematical Foundations | Retrieval metrics, statistical testing, loss functions |
| Appendix B | Algorithms Reference | RAPTOR, clustering, router selection algorithms |
| Appendix C | Benchmarking Your RAG System | Standard datasets, methodology, running benchmarks |
| Appendix D | Debugging RAG Systems | Systematic methodology, failure modes, debugging tools |
Supporting Materials
| Resource | Description |
|---|---|
| How to Use This Book | Reading paths, prerequisites, navigation guide |
| Glossary | Key terms and definitions |
| Quick Reference | Formulas, decision trees, checklists |
Case Studies
Real-world examples that thread through the book.
| Case Study | Description |
|---|---|
| Construction Company | Blueprint search system evolution from 27% to 85% recall |
| Voice AI | Restaurant voice assistant with real-time requirements |
| WildChat | Analysis of 1M+ real conversations |
Reading Paths
For Product Managers
Focus on business value, decision frameworks, and success metrics.
Quick Start (4-6 hours):
- Chapter 0 - Understand the product mindset
- Chapter 1 - Learn why evaluation comes first
- Chapter 3 - Design feedback systems
- Chapter 4 - Prioritize improvements
Full Journey: Read all chapters, focusing on "For Product Managers" sections.
For Engineers
Focus on implementation details, code examples, and technical tradeoffs.
Quick Start (6-8 hours):
- Chapter 0 - Build foundational intuition
- Chapter 1 - Set up evaluation infrastructure
- Chapter 2 - Implement fine-tuning
- Chapter 7 - Production operations
Full Journey: Read all chapters, focusing on "For Engineers" sections.
Full Journey
Read chapters in order for the complete picture. Each chapter builds on previous concepts.
How Content Is Organized
Throughout the book, content is marked for specific audiences:
For Product Managers
Business context, decision frameworks, ROI analysis, success metrics.
For Engineers
Implementation details, code examples, algorithms, technical tradeoffs.
PM Pitfall
Strategic mistakes to avoid.
Engineering Pitfall
Technical mistakes to avoid.
Example
Concrete examples and case studies.
Info
General information and context.
Getting Started
New to RAG? Start with Chapter 0: Introduction to build foundational understanding.
Have an existing system? Start with Chapter 1: Evaluation-First Development to establish baselines.
Looking for something specific? Use the Quick Reference or Glossary.
Ready to begin? Start with Chapter 0: Introduction - The Product Mindset for RAG.