79% of my RAG benchmark didn’t need RAG

That was an uncomfortable result. 😅

I was trying to measure retrieval quality. Instead, I discovered that most questions could already be answered from the model’s internal knowledge. So I changed the benchmark.

For the past few weeks, I’ve been treating an LLM product running in production in Brazil — chat + document RAG, paying customers — like any backend that moves money:

• 📏 Measure first
• 💸 Treat cost as correctness
• 🛡️ Assume adversarial conditions
• 🧪 Make invariants executable

A few results:

→ On a 50-question Brazilian Constitution benchmark, increasing retrieval from 8 to 24 chunks improved full-evidence retrieval from 86% to 94%.

But 79% of the questions were answerable closed-book. So I moved to Brazilian public tender notices, where memorization is far less useful. Hybrid retrieval (BM25 + dense + RRF) scored:

• Tuned set: 88.2%
• Unseen holdout: 87.3%

No meaningful overfitting. We also measured the retrieval ceiling: even perfect retrieval would add only ~2.6 points. At that point, more tuning becomes engineering theater.

Vectors stay local in LanceDB; S3 stores what is needed to rebuild them. And I’m now mapping privacy commitments, including LGPD-related ones, back to deployment configuration and executable checks. Stack: Next.js, TypeScript, MongoDB, AnythingLLM, LanceDB, AWS S3, Caddy.

The lesson:

A good experiment doesn’t only tell you what to build.

Sometimes it tells you what to stop building.