Retrieval-augmented generation works when retrieval is good. Most RAG failures are not model failures: the chunk is wrong, the embedding misses the question, the filter is ignored or the answer is not grounded in the retrieved text.
Key takeaways
- Chunk by meaning, not a fixed character count; keep headings and source metadata.
- Store embeddings with tenant, document and permission metadata.
- Retrieve, rerank and then generate; do not send every chunk to the model.
- Measure quality on a fixed evaluation set before you optimise the prompt.
What matters in practice
- Require citations and a refusal path when the context does not contain the answer.
- Evaluate retrieval separately from answer quality.
How to implement it
- Ingest and clean. Extract text, remove navigation, keep headings and source URLs.
- Chunk with metadata. Store document ID, title, section, tenant and access level.
- Embed and index. Use pgvector with an index appropriate to the dataset size.
- Retrieve with filters. Filter by tenant and permissions before similarity search.
- Rerank and generate. Take the top candidates, rerank and pass only the best passages.
- Evaluate. Measure recall@k, citation accuracy and answer correctness on a fixed set.
A working example
const embedding = await embed(question);
const { data } = await supabase.rpc("match_documents", {
query_embedding: embedding,
match_threshold: 0.7,
match_count: 8,
tenant_id: tenantId,
});
const answer = await generateAnswer({ question, context: data });Cost, latency and reliability
| Concern | What drives it | What to do |
|---|---|---|
| Wrong answer | Retrieval missed the right chunk | Improve chunking, embeddings and filters; inspect recall |
| Answer ignores sources | Prompt does not require citations | Require passage IDs and refuse when absent |
| Cross-tenant leak | Filter applied after retrieval | Filter by tenant and permission in the query |
| Slow | Too many chunks or a large model | Rerank fewer chunks and use a smaller model |
| Stale | Index not updated | Re-index on publish and track document versions |
Security and privacy
- Keep provider keys on the server and scope them to the minimum permissions.
- Treat retrieved documents and user input as untrusted; separate instructions from data.
- Redact sensitive fields before sending them to a third-party model.
- Log prompts and outputs only as long as you need them, with access controls.
- Provide a human escalation path for high-impact or low-confidence answers.
Common mistakes
- Chunking by character count and splitting a table or procedure in half.
- Putting tenant filtering only in the prompt.
- Sending 20 chunks to the model and hoping it finds the answer.
- Evaluating only the final answer and not retrieval recall.
How to evaluate it
- Build a fixed set. Real questions, ambiguous cases, no-answer cases and adversarial prompts.
- Score the output. Correctness, citation quality, refusal behaviour, latency and cost.
- Review failures. Group them by retrieval, prompt, model or product gap.
- Version changes. Record the prompt, model and retrieval settings with each release.
Evaluation rubric
| Dimension | Pass condition | How to test |
|---|---|---|
| Correctness | Matches the source or is clearly marked as uncertain | Human review against a reference answer |
| Grounding | Every factual claim has a source | Citation coverage and spot checks |
| Refusal | Declines when the context is insufficient | No-answer and adversarial cases |
| Safety | Does not follow injected instructions | Prompt-injection test set |
| Latency | Within the product's interaction budget | p95 on a production-like request |
| Cost | Within the per-task budget | Tokens, retrieval calls and retries |
Rollout plan
- Internal only. Test with the team and a fixed evaluation set; log every failure.
- Small pilot. Expose it to a narrow user group with a visible feedback control.
- Human in the loop. Require review for high-impact answers or actions.
- Measure. Compare task completion, correction rate, latency and cost with the baseline.
- Expand or stop. Expand only if the rubric passes; keep a kill switch and a fallback.
Data governance
| Question | What to decide | Why |
|---|---|---|
| What data is sent? | Minimum fields, redaction and retention | Third-party processing and privacy |
| Where is it stored? | Logs, embeddings and evaluation sets | Access and deletion |
| Who can see it? | Roles, support and vendor access | Least privilege |
| How long is it kept? | Prompts, outputs and feedback | Retention policy |
| How is it deleted? | User, tenant and backup paths | Privacy requests |
| What is the fallback? | Search, form or human handoff | The model will fail sometimes |
Frequently asked questions
No. LangChain helps with loaders, splitters, retrievers and tracing, but a small custom pipeline can be easier to debug. Use it when the integrations save real time.
Start with 300–800 tokens with overlap, then test retrieval on real questions. Structure matters more than the exact number.
Store tenant and permission metadata with every chunk and apply the filter inside the vector query. Test with two tenants and adversarial questions.
Build a set of questions with known source passages. Measure recall@k, citation accuracy, answer correctness and refusal behaviour.
Sources and further reading
- LangChain.js documentation (js.langchain.com)
- Supabase pgvector guide (supabase.com)
- pgvector (github.com)
- OWASP Top 10 for LLM Applications (genai.owasp.org)
Related reading
Planning an AI feature?
Describe the user, the task and the data. I will tell you whether a prompt, RAG or a deterministic workflow is the right starting point.
How to cite this guide: Nur Hossain Repon, “RAG LangChain Supabase Next.js 2026 build”, Nhrepon Portfolio, reviewed October 2026, https://nhrepon.com/posts/rag-langchain-supabase-next-js-2026-build.
Comments
No comments yet.