Is your business knowledge ready for an AI assistant?

Choose the questions before the documents

Start with one team and a defined job, such as helping service staff find product setup instructions. Write down the questions they actually ask, including the terminology customers use. Identify which approved documents should answer each question.

Keep unsupported topics outside the first release. If a task depends on a live stock balance or an individual account, decide how that current information will be retrieved. An old document export cannot reliably stand in for a live business system.

Give every document an owner and a useful date

Record the owner, approved version, effective date, review date and authoritative location for each document. A recent upload date does not prove the content is current. Resolve competing versions and exclude abandoned drafts from retrieval.

Agree how edits, withdrawals and access changes reach the search index, and how quickly. Test a document replacement and removal before rollout. Depending on the connector, deleting a source file may leave a searchable copy behind. Assign someone to investigate failed updates rather than assuming synchronisation worked.

Check what the assistant can actually retrieve

Inspect extracted text from scans, tables and PDFs. Keep headings, units and exceptions connected to the relevant passage when documents are split into smaller chunks. A price without its conditions or a procedure without its warning can change the meaning of an answer.

Apply access controls to retrieval using the authenticated user's permissions, before passages reach the model. Carry those controls through to document chunks and refresh them when access changes. A prompt saying “keep this confidential” is insufficient protection. Test with accounts that should and should not see a document, including after access is revoked.

Evaluate complete tasks with expected answers

Build a small test set from real work, using synthetic or appropriately approved examples. For each question, record the user role, expected supporting passage and acceptable response. Include common questions, ambiguous wording, outdated documents and questions the collection cannot answer.

Review retrieval and the final answer separately:

• Retrieval: did it find the correct, current passage within that user's access?
• Answer: is the response supported, complete and useful for the task?
• Uncertainty: does it ask for clarification or say the answer is unavailable?
• Traceability: can the user open the supporting document and check the claim?

Agree pass criteria before the pilot. Treat unauthorised disclosure as a release blocker, and have a subject expert review consequential errors. Re-run the tests after document, permission, retrieval or model changes.

Start with a collection you can maintain

A useful first RAG implementation has a clear boundary: a named team, maintained content, tested access rules and recorded failure cases. Compare it with the team's current search process, including time spent checking and correcting answers.

Before expanding, confirm who owns content quality, monitors retrieval failures and handles unanswered questions. Those responsibilities belong in the project scope alongside the technology.

Planning an AI knowledge base in Australia? Explore Tej Studio's AI and automation consulting, or start a conversation with the task and document collection you want to assess.