Large language models are strong on general knowledge, but they do not know your procedures, product recipes or customer contracts. Retrieval-Augmented Generation (RAG) first finds the document passages relevant to a question in the organisation's own archive, then asks the model to answer based only on those passages. Set up properly, answers are current, citable and in line with the user's permissions.
This guide follows on from our enterprise AI readiness checklist; we recommend completing data and governance readiness first.
How RAG works
- 01Document ingestionDMS, shared folders
- 02Chunking and indexText + vector
- 03Permission-aware retrievalRelevant passages
- 04Answer generationFrom context only
- 05CitationsDocument and revision
The quality of the system is usually decided by the retrieval step, not the model. If the wrong or incomplete passages are retrieved, even the strongest model produces a wrong or generic answer. So most of the effort should go into document preparation, chunking and retrieval.
Scope and use case
Starting with “an assistant that can answer anything about the company” leads to a result that is hard to evaluate and hard to trust. Instead, start with a clear user group, a limited document set and a measurable goal.
| Scenario | Document set | Success measure |
|---|---|---|
| Production operator support | Work instructions, setup cards, maintenance manuals | Time to reach the right instruction |
| Quality and audit preparation | Procedures, standards, past audit findings | Share of audit questions answered with sources |
| Sales and quotation support | Product catalogue, datasheets, approved answer library | Quotation preparation time |
| HR and employee questions | Policies, regulations, FAQs | Number of repeat questions reaching HR |
Document preparation
RAG makes the mess in your archive visible. If there are three versions of the same procedure, the assistant cannot know which one to cite. Document control discipline turns directly into answer quality here.
Permission model and security
The assistant must never answer with the content of a document the user is not allowed to see. That is why permission checks happen at retrieval: every passage carries the access list of its source document, and only passages the user is authorised for are sent to the model. Filtering after the answer is generated is not enough.
- Authentication is integrated with the corporate directory (Active Directory, Entra ID, LDAP).
- Document permissions are synchronised from the source system (DMS, file server); no separate permission list is kept by hand.
- For confidential documents and personal data, running the model and index on-premise or on in-country infrastructure is considered; cross-border transfers are subject to KVKK Article 9.
- Questions and answers are logged; the retention period and access for logs are defined.
- To defend against attacks such as prompt injection, instruction-like text inside documents is treated as data.
Chunking, indexing and retrieval
- 01Chunk by structure
Split documents by headings, clauses and table boundaries, not a fixed number of characters. Add the document title and section path to each chunk; a chunk should make sense when read on its own.
- 02Use hybrid search
Combine semantic (vector) search with keyword search. For queries that need an exact match, such as part numbers, standard clauses or error codes, keyword search is decisive.
- 03Rerank
Narrow the broad candidate set from the first search with a reranking step; send only the few most relevant chunks to the model.
- 04Filter with metadata
Filter by department, product family, site or document type; add user context (for example, the site they work at) automatically.
For Turkish documents, always test the embedding model's Turkish performance on your own data. Adding a synonym and abbreviation dictionary so that technical terms, abbreviations and product codes match correctly noticeably improves retrieval quality.
Answer generation and citations
- The model is instructed to answer only from the passages provided and to say “not found in the documents” when the information is missing.
- Every answer cites the document name, number, revision and relevant section, and the user can open the source in one click.
- For critical content such as procedure steps, the answer quotes the source text instead of summarising it.
- For questions that need a safety, quality or legal judgement, the assistant refers the user to the responsible person.
- Users can mark each answer as “helpful / wrong”; feedback is added to the evaluation set.
Common mistakes: indexing old and current revisions together; checking permissions after answer generation; allowing answers without citations; going live without an evaluation set because “we tried it and it looks fine”; not updating the index when a document changes.
Evaluation and acceptance
Before going live, build an evaluation set of real user questions and the documents where the correct answers are found. Rerun the set after every model, prompt or chunking change, so you can see whether an improvement breaks something else.
| Measure | Question | Example acceptance threshold |
|---|---|---|
| Retrieval hit rate | Is the right source passage among the top results? | In the top 5 for 90% of questions |
| Faithfulness | Is every claim in the answer present in the cited source? | 95% or higher |
| Correctness | Does the answer match the expert's answer? | 85% or higher in expert review |
| Correct refusal | For questions not answered in the documents, does it say “not found”? | 90% or higher |
| Permission violations | Was content outside the user's permissions returned? | Zero |
| Response time | How long does the user wait? | A few seconds for most questions |
Conclusion
A successful document assistant is built on a clean, current document archive, a permission model enforced at retrieval, answers that cite their sources and an evaluation set that is run regularly. OREL DMS manages document revisions and permissions in a single source; with OREL AI and OREL DM, that archive becomes an assistant that runs inside your organisation and cites its sources.