How to Build an AI Assistant Over Your Company's Documents

To build an AI assistant for company documents that people trust, retrieve a small number of relevant passages for each question, hand only those to the model, and make it show where every claim came from. The demo takes a day. What decides whether the thing survives contact with real staff is how you handle messy files, old versions and access rights, and that is where this guide spends its time.

Check whether you need retrieval at all

Retrieval (looking up relevant passages and giving them to the model along with the question) is the standard pattern, often called RAG. It is not always necessary. Anthropic's own engineering guidance says that for a knowledge base under about 200,000 tokens, roughly 500 pages, you can include the whole thing in the prompt, and that prompt caching makes this faster and cheaper (Anthropic, Contextual Retrieval).

If your handbook, policy set or product manual fits in that window, start there. You skip chunking, embeddings and a vector store, and you can add them later when the library outgrows the prompt. Our comparison of RAG, fine-tuning and agents covers the wider strategy question; the rest of this guide assumes you do need retrieval.

Index the documents people ask about

Most failed projects index everything on the shared drive and then wonder why answers are vague. Start with the 200 to 500 documents that generate real questions, and take the list from where people already ask: support tickets, the internal chat channel, the questions new starters keep repeating.

Leave out drafts, duplicates, meeting slides and anything marked superseded. Three libraries show how the choice changes:

• A policy and contract library: index the current signed version of each document and nothing else. The assistant should never quote a clause from a template that was replaced last year.
• A support knowledge base: index resolved articles and macros, and exclude ticket threads, which contain half-right answers from the past.
• A set of engineering test reports exported as PDFs, with long tables of results: index them, but treat the tables as a separate content type (next section).

Make tables, scans and revisions survive ingestion

Plain text extraction flattens a table into a stream of numbers with no row or column labels, and an embedding of that stream retrieves badly. Use a layout-aware parser that returns tables as structured rows, then store each table as its own chunk with its caption and the heading above it. If a table is too large for one chunk, repeat the header row in every piece.

Scanned PDFs need optical character recognition (OCR, turning page images into text) before anything else. Skipping it rules out citations: Claude's citations feature, for example, cannot cite scanned PDFs without extractable text (Anthropic citations documentation). Check a sample of OCR output by eye, because a misread 0 or 8 in a limit or a price will be quoted with full confidence.

Chunks also lose their context. A passage reading "revenue grew 3% over the previous quarter" does not say which company or period. Anthropic's tests of prepending a short generated context to each chunk cut top-20 retrieval failures by 35% with contextual embeddings, 49% when combined with contextual keyword (BM25) search, and 67% with reranking added, from a 5.7% baseline. Its one-time cost was $1.02 per million document tokens (Anthropic). Those are results on Anthropic's own datasets, so test the technique on yours before relying on them.

For revisions, store a document ID, version number, effective date and status on every chunk. When a new version arrives, mark the old one superseded in the same job that ingests the new one, and filter to current versions by default.

Put citations and permissions in the data layer

A citation is useful only if a person can open the source at the right place. Keep the file name, page number and section heading with each chunk, and require the model to answer only from supplied passages. Where your model provider offers native citations, use them: Anthropic documents that its citations are parsed into the response format and are guaranteed to contain valid pointers to the provided documents, which a prompt asking the model to quote sources cannot promise.

Permissions are a stricter problem. Telling the model "do not reveal HR documents to non-HR staff" is not a control, and OWASP warns that prompt-based restrictions may not be honoured and can be bypassed through prompt injection (OWASP, Sensitive Information Disclosure). Store the access groups on each chunk, look up the signed-in user's groups, and filter the search before anything reaches the model. If you use Postgres with the pgvector extension, note that its approximate indexes apply filters after the index scan, so a restrictive filter can return too few results unless you use iterative scans (version 0.8.0 onward) or an index on the filter column.

Permissions enforced in the search query hold up; permissions written into the prompt are only a request.

What it costs to run

Here is a worked example with stated assumptions: 2,000 documents of 15 pages each at about 500 tokens a page, which is 15 million tokens, and 2,000 questions a month. Each question sends six retrieved chunks of about 800 tokens plus 700 tokens of instructions and history (about 5,500 input tokens) and receives a 400-token answer. Prices checked October 2026 from the vendors' own pages.

At these volumes the model bill is small. At 2,000 questions a month, answering with Sonnet 5.5 costs about 1.5 cents a question, and the same question on Haiku 5.5 costs under a tenth of a cent. The expensive parts are the build, the parsing of difficult files and the person who owns the content, so budget for those before you argue over model price. Our breakdown of AI running costs at scale shows how these numbers move as usage grows.

A two-week plan to build an AI assistant for company documents

1. Days 1 to 2: write 50 real questions with the correct answer and source page for each, using staff and support history. Include 10 that the documents cannot answer, where the right reply is "I could not find that".
2. Days 3 to 4: pick the first 200 to 500 documents, record owner, version and access group for each, and drop the rest.
3. Days 5 to 6: build ingestion with a layout-aware parser, OCR for scans, table chunks and the metadata fields above. Open a random 30 chunks and read them.
4. Days 7 to 8: add search (embeddings plus keyword search combined), the permission filter and a reranker if the question set shows missed passages.
5. Days 9 to 10: add the answer step with citations, a refusal rule when nothing relevant is retrieved, and a link from each citation to the source page.
6. Days 11 to 12: run the 50 questions, read every failure, and fix the layer that caused it, whether parsing, chunking, search or prompt.
7. Days 13 to 14: release to ten users, log every question with its retrieved passages, and review the log daily.

Our default stack for a first version is Postgres with pgvector, a hosted parser, one mid-priced model and a thin web or chat front end. That is enough for a few thousand documents and keeps the access data next to the vectors.

Measure whether people trust it

Accuracy on your question set is the first number: aim for correct and cited on at least 90% of answerable questions, and a correct refusal on nearly all of the unanswerable ones, before widening access. After launch, watch the share of answers where users open a citation, the share marked unhelpful, and the questions that returned no answer, which show you which documents are missing. Turn the 50 questions into an automated regression suite so a change to the parser or prompt cannot quietly break old answers; the approach is described in our guide to evals and guardrails.

If you want a second opinion on scope, corpus and architecture before you commit, book a free 30-minute LLM integration workshop with us.

Sources: Anthropic, Contextual Retrieval; Anthropic, Citations; Anthropic, Pricing; OpenAI, text-embedding-3-small; OWASP, Sensitive Information Disclosure; pgvector.