How I Build Production-Ready AI Chatbots with RAG

Most AI chatbots fail for one simple reason: they answer from general knowledge instead of your data. Retrieval-Augmented Generation (RAG) fixes that by fetching the most relevant pieces of your own content and handing them to the model before it answers.

The architecture in one picture

Ingest documents (PDFs, help articles, Notion pages).
Chunk them into 300–800 token pieces with a little overlap.
Embed each chunk and store the vectors in a database such as Supabase (pgvector) or Pinecone.
At question time, embed the query, run a similarity search, and pass the top results to the model as context.
Stream the answer back to the user with citations.

Chunking matters more than you think

Bad chunking is the number one cause of wrong answers. A few rules that work well for me:

Split on headings first, then on paragraphs.
Keep tables and code blocks intact.
Store the source title and URL with every chunk so you can cite it.

Keeping answers grounded

Tell the model explicitly to answer only from the provided context and to say "I don't know" otherwise. Then log every "I don't know" so you can improve the knowledge base over time.

const system = `You are a support assistant. Answer only from the CONTEXT.
If the answer is not in the context, say you don't know and offer to connect a human.`;

Hand-off to humans

The best chatbots know their limits. When confidence is low or the user asks for a person, create a ticket with the full conversation attached. This one feature turns a "nice demo" into something a business will pay for.

If you're planning an AI feature for your product and want a second opinion on the architecture, get in touch.