Why embedding-based RAG can’t answer the questions enterprises actually ask

Retrieval demos well

Point a retrieval-augmented model at a folder of documents and ask it something qualitative, and it will do well. It finds the passages that resemble your question, and a capable model writes a fluent answer. This is a genuinely useful capability, and for finding a document it is the right tool.

Then somebody from finance asks how many active clients the firm has in a region, or a researcher asks how many participants across every study meet an inclusion criterion. The system retrieves some text that mentions clients or participants, and the model produces a number. The number is wrong, and, more importantly, there is no way to tell that it is wrong without doing the work by hand.

Where retrieval runs out

The failure is structural. Aggregation, joins, deduplication and reconciliation are operations over structured records. An embedding index does not contain records; it contains chunks of text and their positions in a vector space. There is no operation you can perform on that index that constitutes counting.

Three things follow from this:

• No aggregation. The model cannot sum what it has not been given as data.
• No entity resolution. The same client in three systems remains three clients, because nothing has decided otherwise.
• No provenance. The answer can cite the chunk it summarized, which is a different thing from citing the system of record.

Access control gets flattened

There is a second problem that surfaces later and hurts more. Building an index over everything tends to flatten the permissions that governed each source. A document that only three people could read becomes a retrievable chunk. Organizations usually discover this after the pilot has been opened to a wider group.

Querying a governed model instead

Unify the structured data into a governed model first, then answer questions by querying it. This is less exciting than it sounds and it is where the difficulty genuinely lives: deciding that cust_id here and ClientRef there are the same entity, that these two date formats mean the same thing, that this orphaned reference is a data quality problem rather than a fact.

That design work is what SetMeld generates and what your team approves. The payoff is that answers are computed rather than recalled, carry attribution back to the systems that produced them, and respect the same access controls as those systems. Which is to say: they can be used for something.