RAG Explained: How to Connect AI to Your Business Data

A property management company called us last year with a problem. They had a ChatGPT subscription and were excited about AI. But when their leasing agents asked it questions about their specific properties — vacancy dates, pet policies for Building C, the maintenance request process for their Denver locations — it hallucinated confidently. It made up pet deposit amounts. It invented amenities that didn't exist. Their team stopped trusting it within a week.

The issue wasn't ChatGPT. It's brilliant at general knowledge. The issue was that ChatGPT doesn't know anything about their 47 properties, their lease terms, their maintenance vendors, or their internal processes. Why would it? That information doesn't exist on the internet.

RAG — Retrieval-Augmented Generation — is how you fix this. It's the bridge between a general-purpose AI that knows everything about the world and nothing about your business, and an AI assistant that can actually answer questions about your specific operations, products, and data.

We built that property management company a RAG system connected to their property database, lease documents, maintenance logs, and policy handbook. Now their AI answers tenant questions accurately, helps leasing agents with property-specific details, and generates reports from real data. Total build time: 3 weeks. Monthly cost: about $350.

How RAG Actually Works (No PhD Required)

I'm going to explain this without the academic jargon. If you want the technical deep-dive, there are great papers for that. This is the business-owner explanation.

Imagine you hired a brilliant new employee. They have incredible general knowledge — they can write, analyze, reason, and communicate beautifully. But they've never worked in your industry, don't know your products, haven't read your internal docs, and have no idea what happened in last week's team meeting.

You have two options:

• Option A (Fine-tuning): Send them through a 6-month intensive training program where they memorize everything about your business. Expensive, takes forever, and you have to retrain them every time something changes.
• Option B (RAG): Give them a perfectly organized filing cabinet and say: "Before answering any question, check the relevant files first." They combine their general intelligence with your specific information. Updating is instant — just add new files to the cabinet.

That's RAG. Technically, three things happen:

Step 1: Your Data Gets Indexed

Your documents — PDFs, spreadsheets, databases, Notion pages, Google Docs, Slack messages, whatever — get converted into a format the AI can search quickly. This format is called "embeddings," which are essentially numerical representations of meaning. The sentence "Our pet policy allows dogs under 50 pounds with a $300 deposit" gets converted into a cluster of numbers that capture the meaning of pets, weight limits, and deposits.

These embeddings get stored in a vector database — a specialized database optimized for finding semantically similar content. When someone later asks "Can I have a dog in the apartment?", the system matches the meaning of that question to the meaning of your pet policy, even though the words are completely different.

Step 2: Relevant Data Gets Retrieved

When a question comes in, the system searches your indexed data for the most relevant chunks. Not keyword matching — semantic matching. It understands that "Can I bring my golden retriever?" is about pet policy, not about golden anything.

A good RAG system retrieves 3–10 relevant chunks from your data. Too few and the AI might miss important context. Too many and you waste tokens (money) and potentially confuse the model with conflicting information.

Step 3: AI Generates an Answer Using Your Data

The retrieved chunks get inserted into the AI's prompt alongside the user's question. The AI then generates an answer grounded in your actual data. Instead of making up a pet policy, it references the specific document that says "Dogs under 50 pounds, $300 deposit, breed restrictions apply per Addendum B."

The key insight: the AI's general intelligence handles the reasoning and language. Your data handles the facts. You get the best of both worlds.

When You Actually Need RAG (and When You Don't)

You Need RAG When:

• AI needs to reference proprietary information. Product specs, internal policies, customer data, pricing that isn't public, historical records. Anything the AI couldn't find on the open internet.
• Accuracy is non-negotiable. Customer-facing support bots, medical or legal information, financial data, compliance-related answers. Hallucination isn't just annoying — it's liability.
• Information changes frequently. Inventory levels, pricing, staff schedules, policy updates. RAG picks up changes as soon as you update the source documents. No retraining needed.
• You have 50+ pages of reference material. Once your knowledge base exceeds what fits in a single AI prompt (~50 pages for most models), you need RAG to find and retrieve the relevant sections.
• Multiple departments need AI access to shared knowledge. Sales needs product info. Support needs troubleshooting guides. HR needs policy documents. RAG lets one system serve all of them from the same indexed knowledge base.

You Don't Need RAG When:

• Your knowledge base fits in a prompt. If your FAQ is 20 questions and your product has one pricing tier, just paste the info into the system prompt. RAG adds complexity and cost for no benefit.
• You need real-time transactional data. "What's the current stock price?" or "How many items are in my cart right now?" — these need direct API calls, not document retrieval. RAG is for reference material, not live transactional queries.
• The AI's general knowledge is sufficient. If you're building a general writing assistant, a coding helper, or a brainstorming tool, the model already knows what it needs to know. Don't add RAG because it sounds impressive.
• Your data is too messy. RAG amplifies data quality. Clean, well-organized documents produce accurate answers. A chaotic Dropbox folder full of outdated PDFs with conflicting information produces confident-sounding wrong answers. Fix the data first.

How We Build RAG Systems for Businesses

We've built RAG systems for about 40 businesses at this point. They range from a 5-person law firm with 200 contract templates to a 300-person logistics company with 50,000 pages of operational documents. Here's the process that works.

Phase 1: Data Audit (Week 1)

Before touching any technology, we map out every data source the AI will need:

• What documents exist? Where do they live? How current are they?
• Which documents have conflicting information? (This happens more often than you think — the employee handbook and the company wiki say different things about PTO policy.)
• What format is the data in? PDFs, Word docs, spreadsheets, databases, Notion, Confluence, Google Drive?
• How frequently does the data change? Daily price updates need different infrastructure than quarterly policy revisions.
• Who owns the data? Are there access restrictions? Does some information require authentication to view?

This phase is boring but critical. The biggest RAG failures we've seen weren't technology failures — they were data failures. The AI confidently cited a policy document from 2019 because nobody told the system which version was current.

Phase 2: Chunking Strategy (Week 1–2)

"Chunking" is how you break your documents into pieces for the AI to retrieve. This sounds trivial. It's not. Bad chunking is the #1 technical cause of poor RAG performance.

• Chunks too small (50–100 words): The AI retrieves fragments without enough context. It finds "$300 deposit required" but not "for pets over 25 pounds in buildings A and C only."
• Chunks too large (2,000+ words): The AI retrieves entire chapters when it only needs one paragraph. Wastes tokens, dilutes relevance, costs more.
• Sweet spot (200–500 words): Large enough for context, small enough for precision. With overlap between chunks so no information falls through the cracks.

The real art is semantic chunking — breaking documents at natural boundaries (section headers, topic changes, paragraph breaks) rather than at arbitrary character counts. A chunk should contain one complete idea, not half of two ideas.

Phase 3: Embedding and Storage (Week 2)

We convert your chunks into embeddings using models like OpenAI's text-embedding-3-small ($0.02 per 1M tokens — pennies for most businesses) and store them in a vector database.

For most business use cases, we use one of three vector databases:

• Supabase pgvector: Our default choice. If you're already on Supabase (or PostgreSQL), add vector search without another service. No additional cost beyond your existing database. Handles up to ~1M vectors before you need to think about optimization.
• Pinecone: Purpose-built vector database. Better performance at scale (1M+ vectors). Starts at $70/month. We use this for larger deployments where search speed is critical.
• Weaviate (self-hosted): Open source, full control, no vendor lock-in. Requires DevOps knowledge to manage. Good for companies with strict data residency requirements.

For 90% of small-to-mid businesses, Supabase pgvector is the right call. You're not operating at the scale where specialized vector databases matter, and keeping everything in one database dramatically simplifies your architecture.

Phase 4: Retrieval Tuning (Week 2–3)

This is where most DIY RAG projects fall apart. The retrieval step — finding the right chunks for a given question — requires more than just "find the most similar embedding."

• Hybrid search: Combine vector similarity (semantic meaning) with keyword matching (exact terms). Someone searching for "Form W-9" needs keyword matching — the semantic meaning of "W-9" won't help the vector search much.
• Metadata filtering: Don't just search all documents. Filter by category, date, department, or document type first. If someone asks about Denver pet policy, search only Denver property documents — not your entire knowledge base.
• Reranking: The initial vector search returns the top 20 candidates. A reranker (we use Cohere's rerank model, $1/1,000 queries) reorders them by actual relevance to the specific question. This step alone improves answer accuracy by 15–25% in our experience.
• Source diversity: If 3 of the top 5 chunks come from the same document, the system actively pulls from other sources to provide a more complete answer.

Phase 5: Generation and Guardrails (Week 3)

The final step: take the retrieved chunks and generate an answer. But not without guardrails:

• Source citation: Every answer includes references to which documents it drew from. Users can click through to verify. This builds trust and catches errors.
• Confidence thresholds: If the retrieved chunks have low relevance scores, the AI says "I don't have enough information to answer this accurately" instead of guessing. Knowing what you don't know is more valuable than confident nonsense.
• Scope boundaries: The AI only answers questions within its domain. A property management bot should not give legal advice even if its knowledge base includes lease agreements with legal language.
• Hallucination detection: We compare the generated answer against the retrieved sources. If the answer contains claims not present in any source, it gets flagged for review.

Real RAG Deployments: What They Cost and Delivered

Law Firm: Contract Search

Problem: 5 attorneys spending 3–4 hours each per week searching through 4,500 past contracts to find relevant clauses, precedents, and templates.

Solution: RAG system indexing all contracts with metadata (contract type, date, client industry, clause categories). Attorneys type natural language queries like "non-compete clause for a SaaS employee in California" and get the 5 most relevant precedents with highlighted sections.

Build time: 3 weeks. Monthly cost: $280 (Supabase + OpenAI API + hosting). Result: Contract research dropped from 3–4 hours/week per attorney to 20–30 minutes. At $350/hour billing rates, the firm recovered about $4,500/month in billable time. The system paid for itself in 3 days.

E-Commerce: Customer Support

Problem: 12,000 support tickets/month. 40% were answerable from existing product documentation and return policies. 6 support agents spending half their time on repetitive questions.

Solution: RAG-powered chatbot connected to product catalog (2,300 SKUs), return policy, shipping info, and order tracking API. The bot handles tier-1 questions and escalates complex issues to human agents with full context.

Build time: 4 weeks (the product catalog integration was complex). Monthly cost: $520 (Pinecone + Claude API + monitoring). Result: Bot resolved 47% of tickets without human intervention. Average resolution time dropped from 4.2 hours to 8 minutes for bot-handled tickets. They reassigned 2 agents to proactive customer success work instead of reactive support.

Manufacturing: Internal Knowledge Base

Problem: Technical knowledge trapped in the heads of 3 senior engineers who'd been with the company for 20+ years. New hires took 6–9 months to become productive because so much operational knowledge was undocumented.

Solution: We recorded interviews with the senior engineers (about 40 hours of audio), transcribed and organized them, combined with existing technical manuals and SOPs, and built a RAG system new employees could query in natural language. "What's the troubleshooting procedure for the CNC lathe when it throws error code E-47?" — and the system pulls from both the official manual and the tribal knowledge the senior engineer shared about that specific error.

Build time: 6 weeks (including the interview process). Monthly cost: $180. Result: New hire onboarding time dropped from 7 months to 3 months average. The senior engineers reported getting 60% fewer interruptions for basic questions. One of them told me: "It's like I cloned myself without the attitude."

5 RAG Pitfalls We See Constantly

1. Garbage In, Confident Garbage Out

RAG doesn't fix bad data — it amplifies it with articulate confidence. If your knowledge base has outdated pricing, contradictory policies, or poorly written documentation, the AI will surface those problems fluently. Clean your data before indexing it. A 200-page knowledge base that's accurate and well-organized will outperform a 5,000-page dump of every document your company has ever created.

2. Ignoring the Update Pipeline

Your RAG system is only as current as its last index. If pricing changes weekly but you re-index monthly, the AI will confidently quote last month's prices. Build the update pipeline before you build the retrieval system. For most businesses, this means: source document updates automatically trigger re-indexing of affected chunks. Not a manual process someone remembers to run.

3. One-Size-Fits-All Chunking

Different document types need different chunking strategies. FAQ pages chunk well by question-answer pair. Legal contracts need larger chunks that preserve clause context. Product catalogs need row-level chunking that preserves column relationships. Applying the same 500-word window to all document types is a recipe for mediocre retrieval across the board.

4. No Evaluation Framework

How do you know if your RAG system is working well? Most businesses just... try it and see if answers "seem right." That's not enough. Build a test set of 50–100 questions with known correct answers. Run them against your system regularly. Track retrieval accuracy (did the right chunks come back?) and answer accuracy (was the generated answer correct?). Without this, you can't objectively measure improvements or catch regressions.

5. Over-Engineering the Architecture

I've seen companies spend $50,000 on a RAG system with a dedicated vector database cluster, custom embedding models, multi-stage retrieval pipelines, and a team of three engineers to maintain it — when their knowledge base was 300 pages and got 50 queries a day. For that scale, Supabase pgvector with off-the-shelf embeddings handles it perfectly for $100/month. Match the architecture to the actual scale, not the aspirational scale.

RAG vs. Fine-Tuning vs. Long Context: When to Use What

RAG isn't the only way to make AI work with custom data. Here's how the three main approaches compare:

RAG (Retrieval-Augmented Generation)

• Best for: Reference knowledge, frequently updated content, large document collections
• Cost: $100–500/month for most businesses
• Update speed: Minutes (re-index changed documents)
• Limitations: Retrieval can miss relevant content. Doesn't change how the model reasons or writes.

Fine-Tuning

• Best for: Changing the model's behavior, tone, or reasoning patterns. Teaching it a specialized domain vocabulary or specific output formats.
• Cost: $500–5,000 per training run, plus higher inference costs
• Update speed: Hours to days (requires retraining)
• Limitations: Expensive to iterate. Can degrade general capabilities. Not good for frequently changing information.

Long Context Windows

• Best for: Small knowledge bases (<50 pages) that change infrequently
• Cost: Higher per-query cost (you're sending more tokens each time)
• Update speed: Instant (just change the prompt)
• Limitations: Gets expensive at scale. Models can "lose" information in the middle of very long contexts. Doesn't scale beyond ~200 pages of text.

Our recommendation for most businesses: start with long context (paste key docs into the system prompt) if your knowledge base is under 30 pages. Move to RAG when you outgrow that. Consider fine-tuning only if you need the AI to behave fundamentally differently — like writing in a specific clinical tone or following a strict output format that prompt engineering can't achieve.

In practice, we build about 10 RAG systems for every 1 fine-tuning project. RAG solves the more common problem (AI needs to know your stuff) while fine-tuning solves the rarer problem (AI needs to think differently).

Starting Your First RAG Project

If you've read this far and think RAG might help your business, here's the honest starting point:

• Step 1: Identify the use case. Don't build "a RAG system." Build a support bot that answers product questions. Or an internal tool that searches your SOPs. Or a sales assistant that pulls relevant case studies. Specific use case = measurable success.
• Step 2: Audit your data. Gather every document the AI would need. Remove duplicates, outdated versions, and irrelevant content. Organize what's left by topic or category. Be honest about quality — if your documentation is a mess, fix it first.
• Step 3: Start with the simplest architecture. Supabase pgvector + OpenAI embeddings + Claude or GPT-4 for generation. You can always add complexity later. The first version should take 2–3 weeks to build, not 2–3 months.
• Step 4: Build the test set before you build the system. Write 50 questions with expected answers. You'll use these to measure how well the system actually performs and to catch regressions when you make changes.
• Step 5: Deploy to a small group first. Let 5–10 users test it for a week. Collect every bad answer. Fix the retrieval, update the chunks, adjust the prompt. Then gradually expand access.

The businesses that succeed with RAG treat it as a living system, not a one-time project. The initial build gets you 80% accuracy. The ongoing tuning — based on real user questions and feedback — gets you to 95%+. Budget for at least 2–3 months of active optimization after launch.

If this sounds like something your business needs — whether it's a customer-facing support bot, an internal knowledge search, or an AI assistant connected to your data — we build these systems and can walk you through what makes sense for your specific situation and data.