RAG and Memory: Making AI Answer From Your Company Data
By Darus S Alfatah · 2026-10-19 · Teknis · 12 menit
Author note: Uncontrolled memory grows unbounded. We regularly summarise and drop what is irrelevant.
Hallucination and the Retrieval Fix
Language models answer from patterns in training data. Asked about your product prices, internal SOPs, or customer contracts, they lack that information — but often answer confidently anyway. That is why AI feels untrustworthy in business settings.
RAG (Retrieval-Augmented Generation) fixes this with a simple idea: before answering, the system retrieves relevant document chunks and includes them as context. Answers become anchored to real sources, and users can check references.
The Standard RAG Pipeline
- Ingest — pull documents from drives, wikis, databases, or internal sites.
- Chunking — split into small pieces (300–800 tokens) with overlap.
- Embedding — convert each chunk into a vector and store it.
- Retrieve — for a question, find the most similar chunks (top-k 5–10).
- Generate — send question plus context, requiring citations.
Common Pitfalls
First, careless chunking. Splitting by 1000 characters breaks tables and procedures mid-way, so retrieved context is incomplete. Split by structure — headings, paragraphs, tables.
Second, vector-only search. Semantic search is good for conceptual questions but poor for product codes, invoice numbers, or names. Combine with keyword search (hybrid) and metadata filters.
Third, no evaluation. Many teams assume RAG “works” because answers look plausible. Without a test set and rubric, quality regressions go undetected.
Memory: Beyond One Conversation
RAG answers “what does the system know”; memory answers “what already happened”. Conversation memory stores preferences, past decisions, and open tasks. Implement it as a rolling summary, a key-fact store, or per-user vectors.
The risk: memory can store wrong or outdated information. Let users view and delete memory — which also helps with data protection compliance.
Measuring RAG Quality
- Retrieval recall — what share of test questions retrieve the right document.
- Faithfulness — whether answers are supported by context rather than invented.
- Answer relevance — whether the answer addresses the question.
- Citation accuracy — whether references point to the right source.
Start with 50 questions representing real cases; that is enough to catch most problems.
When RAG Is the Wrong Tool
If a question needs aggregation across the entire database (“total sales last month by city”), RAG is not appropriate — use SQL or an analytics tool. RAG also struggles with data that changes every second without a sync mechanism.
Maintaining the Index
An index is not a one-off project. Schedule synchronisation: add new documents, update changed ones, delete expired ones. Store date and source metadata so results can be ranked by recency. Without maintenance, the chatbot will confidently quote old policies — the hardest error for users to notice.
Also track documents that are retrieved often but rarely help. That usually means the chunks are too broad and should be split further.
Conclusion
RAG is the cheapest way to make AI “know” your business without expensive fine-tuning. The key is not the best model but data quality, clean chunking, hybrid search, and consistent evaluation.
Sources & References
Practical Note
Show sources next to each answer. If users often open a source and cannot find the claim, the model is adding information beyond the context. Tighten the instructions and re-run your test set.
An example of memory upkeep: Every five turns, summarise the conversation into a few points and drop old, irrelevant detail.
Frequently Asked Questions
Does BankAI support streaming responses?
Yes, BankAI’s endpoint supports streaming (Server-Sent Events) and non-streaming per the OpenAI standard.
Are there RPM/TPM limits?
As long as your token quota lasts, usage is not tightly capped on TPM/TPD/RPM, but safeguards keep the service stable for everyone.
How do I choose the right context window?
Match it to the combined prompt + history + answer you need; larger-context models help with long documents.