RAG Architecture Patterns: Proxy, Caching, and Routing

A few months ago, our RAG (Retrieval-Augmented Generation) setup did what most first versions do: the Streamlit app called OpenAI directly, pulled chunks from MongoDB, and stitched an answer together. It worked. It also had three problems nobody wants to discover in production.

First, the OpenAI bill. Users ask the same question in ten different ways: "How many leave days do I get?" and "What's the annual leave entitlement?" are the same question, but if you're calling the LLM for every single one of them, you're paying for the same answer over and over.