In short
The most damaging thing a knowledge-base chatbot can do is say "I don't have enough details" when the answer is sitting in the knowledge base. Customers trust it less than a bot that's wrong, because they can see the answer on the website. This year we chased that failure through Tec-AI, the AI platform we build and run in Reading, and found three separate causes. None of them was the language model. If you want the basics first, our guide to RAG chatbots covers how retrieval-augmented generation works.
How retrieval works in Tec-AI
Knowledge-base entries are split into chunks of up to 400 characters with a 50 character overlap, splitting on headings first, then numbered lists, then paragraphs. Products aren't chunked; they're embedded by title. Each chunk is embedded and stored in PostgreSQL with pgvector, alongside the id of the customer who owns it, because every customer's knowledge base must stay separate.
At question time we embed the question, search that customer's vectors, keep the matches above a similarity threshold and put the best few into the prompt. Every failure below sat somewhere in that one sentence.
Failure 1: an index that returned nothing
In May a product we could see in the knowledge base simply never came back from search. The query returned zero rows.
The vectors were indexed with IVFFlat using 100 lists. IVFFlat groups vectors into clusters and, by default, searches only the single nearest cluster (probes = 1). Add a WHERE user_id = ... filter and that one cluster often contains no rows belonging to the customer asking. The index does exactly what it was told and the answer is empty.
IVFFlat, probes = 1 -> 0 rows IVFFlat, probes = 100 -> 5 rows, similarity ~67-70% HNSW (no IVFFlat) -> the same 5 rows
We dropped IVFFlat and moved to HNSW (m = 16, ef_construction = 64, ef_search = 100). For a multi-tenant table where every query is filtered by owner, it's the safer default. IVFFlat can work, but only if you tune probes against your filters, and it fails silently when you don't.
Failure 2: an embedding model that couldn't tell clippers from dryers
Tec-AI serves a lot of Greek-language content, including product catalogues. Our first embeddings were 1536-dimensional, sized for a hosted model. We later switched to a small multilingual model with 384 dimensions. It was fast and cheap, and on product questions it was poor: it could not reliably separate hair clippers from hair dryers, and the top match for a clear question often scored around 50%.
We replaced it with multilingual-e5-large (1024 dimensions), using the query and passage prefixes the model expects. Changing dimensions has a cost people underestimate: every stored vector becomes unusable. Until a regeneration script had re-embedded the whole knowledge base, retrieval returned nothing at all. Plan that migration like a data migration, because it is one.
We also changed what gets embedded. Each entry is now embedded from its title, its content and a handful of questions a small local model generates for it (plus the SKU for products). Customers ask questions, so matching against likely questions pulls the right entry up the list.
Failure 3: cutting the list before checking it
The last one was found on our own website. The Tec-AI widget on tecdynamics.co.uk kept telling visitors it didn't have details about things answered in our FAQ.
The search asked the database for exactly as many rows as fit in the prompt, often four, and applied the similarity threshold afterwards. When the right answer ranked fifth, or fifteenth, it was never even considered. The fix is dull and effective: fetch a larger pool (five times the limit, minimum 25), apply the threshold, then trim to the limit. A full-text search runs as a fallback for exact terms that vector search can miss.
The same fix (released on 13 August) corrected a statistics count that made half of the knowledge base look unsearchable. That false count had us chasing a data corruption bug that didn't exist for longer than we'd like to admit.
What we check first now
- Run the failing question directly against the database and look at the raw rows and scores before touching the prompt. In all three cases the model never saw the answer.
- If the vector table is multi-tenant and filtered, prefer HNSW, or test IVFFlat with realistic filters and probes.
- Test embeddings on your hardest near-duplicates (two similar products, two similar policies) in every language you serve, not on easy questions.
- Over-fetch candidates, threshold, then trim. Never trim first.
- Treat an embedding model change as a full re-index with a cut-over plan.
We build private, UK-hosted AI assistants on this platform for businesses that want answers grounded in their own documents. See AI development or ask us about your use case.