RAG grounding lessons from production email classification

Retrieval-augmented generation sounds straightforward until you ship it against a live mailbox. The demo retrieves three plausible paragraphs and the model sounds confident. Production retrieves three plausible paragraphs where two are from the wrong thread and the model still sounds confident. That's worse than no retrieval at all.

Lesson 1: Metadata beats embedding similarity

For email, thread ID, timestamp, and participant set often outperform pure cosine similarity. I filter candidates by conversation context before vector search runs. Similarity on the wrong thread is how you get fluent wrong answers.

Lesson 2: Structured extraction before open-ended generation

At Allstate I learned this on Graph + Azure OpenAI pipelines: force JSON-schema output for intent metadata (category, confidence, urgency) before any generative step. Parsing failures become visible metrics, not silent hallucinations buried in prose.

Lesson 3: Empty retrieval is a valid answer

Agents need permission to say “I don't have grounded context.” Building that stop condition — in prompts, in tool contracts, in evaluation — is as much engineering work as the embedding index itself.

Lesson 4: Evaluate retrieval separately from generation

If you only score final answers, you can't tell whether the model or the retriever failed. I maintain small golden sets per intent category and measure recall@k on the retrieval step alone. Fix retrieval first; tuning temperature second.

These lessons come from email, but they generalize to any domain where stale or adjacent context poisons trust. RAG isn't a library call — it's a data pipeline with a language model at the end.