An AI knowledge base your team can actually trust

The problem it solves

Knowledge in a growing company spreads across a wiki nobody updates, a ticket tracker, a chat tool, Google Drive, a CRM and the codebase. New hires ask senior engineers the same questions. Support escalates things the docs already answer. The person who knew why a system works the way it does left last year, and their reasoning is somewhere in a closed pull request.

An internal AI assistant that can search all of that and answer with links back to the source is genuinely useful – if people trust it. Trust is the hard part.

Buy first, build if you need to

Several enterprise search AI products, plus the AI features inside tools you may already pay for, connect to common sources and handle permissions out of the box. If your sources are mainstream SaaS tools and your questions are general, start there. Check their connectors, their permission model and their data terms against your needs.

A custom build earns its cost when you have proprietary systems or databases no vendor connects to, permission rules more specific than “same as the source tool”, a need to answer over code and operational data together, strict data-residency requirements, or an assistant that has to take actions in your systems. I’ll help you make that call honestly – see build vs buy software.

Permission-aware retrieval

The fastest way to kill an internal assistant is for it to quote a salary spreadsheet or a board memo to someone who shouldn’t see it. So access control is designed in from the start:

• Every indexed chunk carries the access-control information of its source document – users, groups, workspaces.
• Retrieval filters by the asking user’s identity before ranking, not after generation. The model never sees what the user can’t.
• Permission changes in the source propagate to the index promptly, including revocations.
• Identity comes from your SSO, so access maps to the groups you already manage.
• Every query and the documents retrieved for it are logged for audit.

Citations and honest answers

Every answer links to the passages it came from, so people can check and the assistant earns trust over time. When the retrieved sources don’t contain an answer, it says so instead of guessing. When sources conflict – an old wiki page against a newer ticket – it shows both with their dates. These behaviors are tested, not hoped for.

Docs, tickets and code need different handling

• Docs and wikis: split by headings, keep tables intact, carry the page title and last-edited date on every chunk.
• Tickets and chat: thread-level context matters more than single messages; resolved state and dates matter a lot; noise needs filtering.
• Code: chunk by function, class or file rather than by token count, attach the file path and repository, and combine with keyword search so exact identifiers match. Commit messages and pull request descriptions often hold the “why”.

Underneath, this is a retrieval-augmented generation system with hybrid keyword and vector search plus reranking. The general techniques are covered in RAG development and RAG architecture; this service is about the extra problems that come with internal, mixed-source, permissioned content.

Keeping it current and measuring it

Indexes are updated incrementally from source webhooks or change feeds, with deletions handled as carefully as additions. Stale documents are down-ranked by age where that makes sense.

Quality is measured against a labeled set of real questions collected from the team, covering retrieval recall and answer faithfulness, and rerun on every change. In production I track thumbs up and down, unanswered questions and source coverage, which tell you which docs to write next. More on the method in LLM evaluation.

How we start

A two-week sprint: we pick two or three sources and a group of users, collect their real questions, and ship a working assistant with permissions and citations to that group. By the end you have usage data and accuracy numbers, and a clear view of whether to expand, buy instead, or stop.

Related

• RAG development services, measured on retrieval quality
• RAG architecture for production systems
• An AI readiness assessment that ends in working code
• LLM evaluation for product features
• Build vs buy software: a framework for making the call