An AI knowledge base is not a folder uploaded to a chatbot. It is a maintained collection of approved sources with clear ownership, access rules, and a way to show where an answer came from.
/ 01
Inventory sources before ingestion
List policies, service descriptions, procedures, product facts, templates, contracts, training materials, and frequently answered questions. For every source record the owner, intended audience, sensitivity, authority, effective date, last review date, and next review date.
Start with the questions people actually ask and the decisions they need to make. A smaller collection that answers 50 important questions with traceable sources is more useful than thousands of unreviewed files.
/ 02
Resolve contradictions
Older documents often disagree with current pricing or policy. Do not ask retrieval software to decide which source is correct. Establish a source-of-truth hierarchy, add effective dates, and archive superseded material outside the active index.
Create a contradiction queue during ingestion. When two authoritative-looking sources disagree, retrieval should not blend them; the content owner must choose or document the condition under which each applies.
/ 03
Design answers around evidence
A useful system cites the approved source and relevant section, distinguishes sourced fact from inference, acknowledges missing context, and routes sensitive or uncertain questions to a person. Test whether citations actually support the answer rather than merely discussing the same topic.
Permissions must apply before retrieval. Filtering the final answer is weaker than preventing unauthorized documents from entering the model context.
/ 04
Make freshness operational
Assign review dates and owners. Monitor questions that return no answer, low confidence, unsupported citations, or conflicting evidence. Update the source rather than repeatedly patching a prompt around incorrect knowledge.
Version the index and retain a reproducible evaluation set. Before publishing an update, rerun the same questions and compare answer support, refusal behavior, and access isolation.
/ 05
Build a 30-question evaluation set
Use ten straightforward questions, ten questions requiring more than one source, five questions the system should refuse because of permissions, and five questions with no approved answer. For each, write the expected sources, acceptable answer boundaries, and reviewer.
Track grounded answer rate, correct refusal rate, citation support, unauthorized retrieval attempts, and unanswered-question trends. A single overall 'accuracy' number hides the failures that matter.
- Answer supported by an approved source
- Citation supports the exact claim
- Correct source version used
- Restricted source not retrieved
- Unanswered question is acknowledged
- Human escalation includes useful context
/ Practical takeaway
A trustworthy knowledge system combines approved sources, field-level access, contradiction handling, citations, freshness ownership, and a repeatable evaluation set.
Primary references
These sources support the factual and technical guidance in this article. Product decisions still require review against your own systems, policies, and risk.
NIST Generative AI Profile OWASP: prompt injection and RAG poisoning guidance