Retrieval-augmented generation (RAG) is the most important enterprise AI pattern of this cycle. Instead of hoping a model “knows” your discount policy, you fetch the current policy and ask the model to answer only from that text. Done well, RAG turns scattered SaaS knowledge into a citable assistant. Done poorly, it becomes a slower search box that still hallucinates.
What RAG actually is
A user question is converted into a search against your approved corpus. The top chunks—paragraphs, table rows, ticket macros—are stuffed into the prompt. The model writes an answer and, if you design the UI correctly, shows the sources. No mystical memory of your ERP is required.
Choose sources like an archivist, not a hoarder
- Include: current SOPs, certified metric dictionaries, closed-won playbooks, product docs with owners.
- Exclude: draft Confluence pages, personal Drive folders, expired contracts, and “final_v7_REAL” spreadsheets.
- Split: keep HR, legal, and customer PII in isolated indexes with stricter roles.
Starter corpus for a mid-market operator
| Source | Chunk style | Refresh | Permission mirror |
|---|---|---|---|
| HR / IT knowledge base | Heading-aware sections | Hourly | KB space ACLs |
| ERP item & customer masters | Row + key attributes | Near real time | Company / site role |
| SaaS product docs | Page sections | On publish | Public vs internal |
| Ticket macros | Whole macro | Daily | Queue membership |
Chunking, metadata, and the queries you forget
Naive 500-token splits destroy tables and numbered procedures. Keep headings with their body. Attach metadata: entity, region, effective date, and document owner. Then support filters such as “Germany + 2026 price list only.” Most bad answers are retrieval misses, not model stupidity.
Evaluate like a product manager
- Build a set of 50 real questions from Slack and tickets—not invented happy-path prompts.
- Score citation correctness separately from writing quality.
- Track “I don’t know” as a success when the corpus has no answer.
- Re-run the set whenever you change embedding models or chunk sizes.
Security is part of retrieval
If your search index is built from a service account that can see every SharePoint site, the chatbot will too. Filter chunks after retrieval using the user’s groups, or maintain per-tenant indexes. Log every retrieved document ID. This is non-optional for ERP financials and employee data.
Conclusion
RAG is how AI becomes useful on business data without waiting for a perfect data warehouse. Start with a small, owned corpus, mirror permissions, cite everything, and measure retrieval quality weekly. That discipline beats another proof-of-concept that dies after the demo.