AI Chatbot With Knowledge Base: How to Build One

Three things get called "chatbot" and they behave very differently:

• Scripted bots follow a decision tree. If the user's question isn't on the tree, they fail.
• Plain LLM chat (ChatGPT in an iframe) answers anything, but from training data. It doesn't know your refund policy, your API, or last week's release notes, and it will guess.
• A knowledge-base chatbot sits in between: an LLM writes the answer, but only after the system retrieves the relevant passages from your content and puts them in front of the model. When the passages don't exist, a well-built bot says so instead of inventing one.

The technique is called retrieval-augmented generation (RAG). It is the standard architecture behind Intercom Fin, Zendesk AI agents, Chatbase, and most "train a bot on your docs" products. None of them fine-tune a model on your data; they retrieve and prompt.

There are two pipelines: one that runs when your documents change, one that runs on every question.

Ingestion (runs when docs change)

1. Collect sources. Help-center articles, PDFs, product docs, a crawled marketing site, Notion pages, policy documents.
2. Chunk. Each document is split into passages of roughly 300–800 tokens with some overlap, so a single answer's context fits in one chunk. Chunking on headings beats chunking on fixed character counts.
3. Embed. Each chunk is turned into a vector (a list of numbers) by an embedding model. Semantically similar text ends up with similar vectors.
4. Store. Vectors plus the original text and metadata (source URL, title, last-updated) go into a vector store: pgvector inside Postgres, or a hosted index like Pinecone.

Answering (runs per question)

1. Embed the question with the same model.
2. Retrieve the top 3–8 chunks whose vectors are closest. Many systems also run a keyword search and merge the results (hybrid retrieval), which helps with product names and error codes.
3. Build the prompt: system instructions ("answer only from the context; if the context doesn't cover it, say you don't know"), the retrieved chunks, the conversation so far, the question.
4. Generate the answer, streaming tokens to the UI, and attach citations back to the chunks used.
5. Log the question, the retrieved chunks, and the answer. This log is how you find gaps in your docs later.

1. Pick the sources and decide the boundary. Start with the content that already answers most support questions, usually a help center and product docs. Leave out anything you wouldn't want quoted verbatim to a customer.
2. Choose the three models. A chat model (Claude, GPT, or an open-weight model via Ollama), an embedding model (OpenAI, Cohere, or a local one), and a vector store (pgvector is enough for well under a million chunks). Keep them swappable; prices and quality shift every few months.
3. Build the ingestion pipeline. An admin page to upload files and paste URLs, a worker that parses, chunks, embeds, and writes to the store, and a re-ingest path that only re-embeds documents whose content hash changed.
4. Build the chat UI. Streaming responses, markdown rendering, a "Sources" block under every answer, per-user conversation history, and an embed snippet so the same bot can live on your site or inside your app.
5. Add guardrails and measurement. A refusal path when retrieval returns nothing relevant, an escalation button that creates a ticket with the transcript attached, token-cost tracking per conversation, and a fixed test set of 30–50 real questions you re-run whenever you change chunking, prompts, or models.

Steps 3–5 are where the time goes. The prompt at the top of this page generates that shape as a Next.js app, with the ingestion routes, pgvector schema, streaming chat, and citations wired, so the work left is your content and your prompt wording.

There are three realistic routes. Which one is right depends on how you pay and what you need to own.

Buy if your support team already lives in Intercom or Zendesk, volume is low enough that per-resolution pricing doesn't hurt, and you don't need the bot anywhere except the help widget.

Build or generate if per-resolution pricing scales badly for you, the bot needs to sit inside your own product with your auth and your UI, your data has to stay in your database, or you want to change the model when a better or cheaper one ships.

The recurring cost is model usage, and it's small compared to per-resolution pricing.

• Embeddings are a one-time cost per document, on the order of a hundredth of a cent per chunk. A 100-page documentation set costs cents to index, and incremental re-ingestion keeps it that way.
• Chat completions run roughly $0.01–$0.05 per conversation on a small model such as Claude Haiku or GPT-4o-mini for typical support-length exchanges. A thousand conversations a month is $10–$50 in API spend.
• Hosting is a Postgres instance with pgvector and a Node app. On Supabase and Vercel that starts free and stays under typical SaaS seat pricing for a long time.

Compare that with $0.99 per resolved conversation on Intercom Fin: 1,000 resolutions is $990 a month before the platform subscription. Cost is rarely the reason to buy; speed and not having to own anything is.

• No refusal path. If the prompt doesn't tell the model to decline when the retrieved context is empty or irrelevant, it will answer anyway, and it will be confidently wrong.
• No citations. Users trust an answer that links to the doc it came from and distrust one that doesn't. Citations also make bad answers debuggable.
• Chunks that are too big or too small. Too big and the relevant sentence is buried in noise; too small and the answer is split across chunks that don't all get retrieved. Chunk on document structure, and test.
• A stale index. Docs change; the vector store doesn't know unless something re-ingests. Trigger re-ingestion from the doc source, not from a calendar.
• Mixing tenants. If the bot serves multiple customers or workspaces, every retrieval query has to filter by tenant. Forgetting this leaks one customer's documents into another's answers.
• No evaluation set. Without 30–50 fixed real questions and expected answers, every change to chunking or prompts is a guess. Run the set on every change.

The bot is only as good as what it can retrieve, and some sources retrieve much better than others.

• Works well: help-center articles, product documentation, policy pages, FAQs, release notes, anything already written to answer a question.
• Needs preparation: long PDFs (split by section and keep headings), spreadsheets (add a row-per-record description), and marketing pages (strip navigation and boilerplate before chunking).
• Usually not worth it: scanned PDFs without OCR, Slack or email threads where the answer is spread across twenty messages, and internal wikis nobody has updated in a year. The bot will cite them, and the citation will be wrong.

A useful starting rule: if a human support agent would paste a link to it, ingest it. If they'd say "let me check with someone," it isn't ready to be a source.