Unified knowledge graph vs. siloed data sets

I wanted to see what it takes to get enterprise data to a point where it is usable by AI, and this article represents the output of an experiment to do just that.

“Getting your data ready for AI” has been a hot topic in the data world recently, and there is no wonder why. When hundreds of companies allocated a hefty budget to employ a “God in a box” in their infrastructure hoping that it would solve all their problems (or at least deliver a bit of ROI), hopes were high. Unfortunately, not even God can properly contend with four separate COBOL mainframes from the 60s and a Frankenstein SAP instance built from desecrated body parts of five acquisitions and one messy merger.

So, after a year or two of disappointing enterprise AI explorations, companies have done one of three things:

1. They have abandoned their AI initiatives, deciding to instead just use it for the simple things like email copy, slop image generation, and replacing their entire software engineering department with an army of agents and one very overworked human. These companies are content with the status quo for AI and do not care about an AI strategy that gives them an advantage over every other organization.
2. They have turned to AI shops that focus on delivering AI for a specific vertical (like Ramp for HR or Harvey for legal). These organizations have an AI strategy focused on delivering cost savings and other goals to “keep up with the Joneses” in their industry. They hand their data over to an external provider so it can do the same kind of work everyone else is doing.
3. They have decided to clean up their data because they consider their data a core asset to their business and a competitive advantage they can unlock with AI.

It is for the third kind of enterprise this article is written.

The Experiment

The experiment tests 20 questions ranging from simple retrieval of a single entity to complicated calculations across the data set. All 20 questions are run against data systems with varying levels of quality:

• silo_messy represents the current status quo in many organizations. Data is split between four different silos, and within them, there are opaque naming schemes and common data problems.
• silo_clean improves on silo messy. Data is still siloed between four systems, but this time, common data problems are cleaned up and field names are discernible.
• silo_prompt is a duplicate of the silo_clean data system, but this time, the AI is given context for previously unknown institutional knowledge.
• unified_sql takes the silo_prompt ecosystem and unifies it into a single SQL database.
• graph ingests the tabular data and turns it into a knowledge graph.

If you want to rerun or modify this experiment yourself, then I encourage you to check out this experiment’s GitHub repository.

Before we begin, a disclaimer. These results are only indicative of some general trends around these kinds of database systems. There is always more work that can be done with any of these that could improve or destroy their performance, and some may perform differently if asked different questions. This is not a peer reviewed paper, and any one section could be expanded into an entire research project. But I think I have set up an experiment that tests a pretty wide breadth of enterprise challenges, if a bit imperfectly.

So, with a general understanding of the experiments, let us look at some results.

Overall Results

messier datacleaner data

Show the underlying numbers

System	What it is	Solve rate	Passed
silo_messy	the bank as you would actually find it	48.3%	58 / 120
silo_clean	every defect repaired	50.8%	61 / 120
silo_prompt	clean, plus about 1,100 tokens of institutional knowledge	82.4%	98 / 119
unified_sql	entities resolved, one database	89.2%	107 / 120
graph	entities resolved, one knowledge graph	92.4%	110 / 119

Right away, we can see that each successive improvement yielded, on average, better outcomes in AI Q&A accuracy, but by far, the biggest jump in improvements is that between silo_clean and silo_prompt, suggesting that the single biggest improvement you can make for your AI query is providing institutional knowledge to the model.

We have seen this with the companies we work with as well. Gathering institutional knowledge is less of an engineering problem and more of an organizational problem. That is why we built an agent to help customers gather their internal knowledge, and if you are looking to get more ROI on your AI data systems, it may be a good idea to follow suit.

The threat of confident wrong answers

Certain questions are given the option to answer a question or to abstain from answering if the model gets stumped. We would all rather have a correct answer, but especially when you are dealing with important information like finances, an abstention is better than a confidently wrong answer.

PassedConfidently wrongAbstained

Show the underlying numbers

System	Passed	Confidently wrong	Abstained
silo_messy	48%	52%	0%
silo_clean	51%	46%	3%
silo_prompt	82%	18%	0%
unified_sql	89%	11%	0%
graph	92%	7%	1%

We will dig into the actual dynamics of the specific abstentions a bit later. But the general trend is that messier data causes more confident wrong answers. Once data was cleaned enough between silo_messy and silo_clean, the AI agent was able to realize that it did not have the institutional knowledge to answer a question. Providing that institutional knowledge helped the model answer most questions for silo_prompt and beyond, but it was not until we structured things as a knowledge graph that the model was able to once again realize data problems that would prevent it from answering correctly.

The cost of data silos

Then there is the number that justifies unification. Take every run in the three siloed arms and bucket it by how many of the four databases the agent actually had to open.

Show the underlying numbers

Databases opened	Solve rate	Runs solved
1	91%	50 / 55
2	70%	73 / 104
3	59%	16 / 27
4	45%	78 / 173

Every additional database takes a bite out of accuracy, and by the time the agent has all four open the odds have roughly halved.

Token cost

Accuracy is all well and good, but another thing a lot of organizations are considering is token cost. After all, you cannot say you have achieved a positive ROI on AI if your I (cost in tokens) is greater than your R (benefit from using AI).

$ per question asked$ per correct answer

Show the underlying numbers

System	Tool calls / run	Tokens / run	$ per question	$ per correct answer
silo_messy	18.2	100,888	$0.22	$0.452
silo_clean	17.1	100,082	$0.21	$0.423
silo_prompt	11.1	92,017	$0.18	$0.222
unified_sql	12.2	95,040	$0.27	$0.243
graph	9.6	149,321	$0.32	$0.299

I see two interesting insights from these findings. First, as set up in this experiment, the graph uses the fewest tool calls of any system and the most tokens. It knows where to go because it paid about 92,700 tokens up front to read the vocabulary. We can call this a “cover charge,” and there is certainly more research beyond the scope of this experiment to be done to lower it.

We can measure the cover charge. Regress the graph’s token spend on the messy silo’s, question by question (fit a straight line through the twenty points), and you get:

graph_tokens = 92,700 + 0.78 × silo_tokens        (r = 0.72, n = 20)

Performance based on type of question

The cleanest way I have to say what each intervention buys is to sort the twenty questions into three classes.

0%100%

Class of question	messy	clean	prompt	SQL	graph
Control (2 questions)	67%8/12	67%8/12	75%9/12	75%9/12	73%8/11
Data engineering (4 questions)	83%20/24	100%24/24	96%23/24	100%24/24	96%23/24
Semantic (14 questions)	36%30/84	35%29/84	80%66/83	88%74/84	94%79/84

Control questions

Control questions involve finding (or not finding) an answer in one table in one database. Even for the siloed systems, the answer is always in one database.

0 of 6 runs6 of 6 runs

Question	messy	clean	prompt	SQL	graph	messy → graph
Q1 · balance on one loan	6	6	6	6	6	6 → 6
Q2 · confirm nothing is wrong	2	2	3	3	2	2 → 2
All control	8/12 · 67%	8/12 · 67%	9/12 · 75%	9/12 · 75%	8/11 · 73%

Every system aced a question about finding an existing record (Q1), but across the board struggled to report that data did not exist when asked to find data that was not there (Q2). Adding more institutional knowledge helps the model, but it still only accurately reported that data did not exist about half the time.

Data engineering questions

0 of 6 runs6 of 6 runs

Question	messy	clean	prompt	SQL	graph	messy → graph
Q3 · which table is live	6	6	6	6	6	6 → 6
Q5 · the payment posted four times	6	6	6	6	6	6 → 6
Q6 · the double-counted reversal	6	6	6	6	6	6 → 6
Q4 · loans with no origination record	2	6	5	6	5	2 → 5
All data engineering	20/24 · 83%	24/24 · 100%	23/24 · 96%	24/24 · 100%	23/24 · 96%

I expected the agents to have more of a problem with data engineering. I purposely included data cleanliness problems: lack of unit definitions (numbers recorded in cents rather than dollars), duplicate data, a completely stale table, and so on. But for the most part, cleaning data was barely a problem for them. It turns out agents are very good at data cleaning on the fly and have a built-in internal dialogue to diagnose how to handle these problems.

The only place where messy data hurt overall performance was Q4, where the data engineering problem was hidden in a haystack of many rows that needed aggregation. But when an agent is able to look at a problem directly, it does not crack a sweat.

Semantic questions

0 of 6 runs6 of 6 runs

Question	messy	clean	prompt	SQL	graph	messy → graph
Q9 · money not yet applied	6	6	6	6	6	6 → 6
Q8 · what “commitment” means	6	6	6	6	5	6 → 5
Q11 · the break with no cause	5	6	6	6	6	5 → 6
Q16 · brief the new RM	5	3	6	6	6	5 → 6
Q7 · the same interest, twice	4	3	6	6	6	4 → 6
Q13 · what does this customer owe	1	1	6	6	6	1 → 6
Q10 · how much can they draw	2	2	2	5	6	2 → 6
Q17 · will this payment post today	0	0	6	5	6	0 → 6
Q19 · two loans closed, one still owes	0	0	6	5	6	0 → 6
Q12 · are two records one company	0	0	6	3	6	0 → 6
Q15 · two systems, same meaning?	1	2	1	6	5	1 → 5
Q14 · exposure — which number	0	0	3	5	5	0 → 5
Q20 · the five largest exposures	0	0	3	4	6	0 → 6
Q18 · reconcile the portfolio	0	0	3	5	4	0 → 4
All semantic	30/84 · 36%	29/84 · 35%	66/83 · 80%	74/84 · 88%	79/84 · 94%

By far, most questions assess semantic reasoning: facts that are not in the database at all. They require interpretation of data or unexpected data digging.

Imagine you designed a database specifically to answer questions on a pre-designed dashboard. Q1 through Q6 represent that scenario. The rest of the questions are ones the database was not designed and pre-indexed to answer.

The quantity of these questions is also the main reason why performance jumped once institutional knowledge was provided in the silo_prompt data set. Having insider knowledge is invaluable for semantic reasoning.

How much a bigger model buys on each version of the bank

For this fun little study, I performed the tests with both Sonnet and Opus. This by no means is an exhaustive list of the models you could use for extraction, but it does give you a little hint as to how more powerful models would improve accuracy.

silo_messysilo_cleansilo_promptunified_sqlgraph

Show the underlying numbers

System	Sonnet 4.6	Opus 4.8	Lift
silo_messy	48.3%	48.3%	+0.0 pts
silo_clean	48.3%	53.3%	+5.0 pts
silo_prompt	78.3%	86.4%	+8.1 pts
unified_sql	80.0%	98.3%	+18.3 pts
graph	90.0%	94.9%	+4.9 pts

On the messy bank, upgrading the model bought literally nothing: 29 out of 60 both times. The facts the agent needed were not in any of the four databases, so Opus had nothing more to query than Sonnet did, and it got the same 29. If your plan is “wait for the next model,” that flat line is what happened when I tried it on this bank.

unified_sql is the best substrate in the study if you can afford Opus, since it missed one question in sixty. On Sonnet it is ten points behind the graph. So SQL has the higher ceiling, and the graph loses less when you drop to the cheaper model. I speculate that this is due to SQL queries being more common in training data than graph-based queries (SPARQL).

And Sonnet on the graph outscores Opus on the best silo: 90.0% against 86.4%, a 3.6-point edge, at less cost per correct answer. So if you have a fixed budget, spend it on the data before the model.

Conclusion (Lessons Learned)

After running this rough experiment, a few things stand out:

• Collecting institutional knowledge is of paramount importance for semantic questions. Data alone will not be enough to inform your AI how to answer questions.
• A full ontology for a knowledge graph is less important for simple questions (adding institutional knowledge to a prompt is all that’s needed). It’s needed for more complicated questions, but on simpler questions loading it will only drive up token costs.
• Modern agents are surprisingly capable of cleaning dirty data on the fly, but problems arise when querying dirty data in aggregate.
• Finally, unifying data into a single datastore increases accuracy specifically for questions that would span multiple databases. The more databases an agent needs to query, the more likely they are to miss some important bit of information.

Notes on the numbers

• Two of the 600 runs failed to complete and are excluded rather than counted as failures, which is why a few denominators read 119, 83 and 11 instead of 120, 84 and 12.
• Dollar figures use Sonnet 4.6 pricing at the time of the run. The model comparison is the only place Opus 4.8 appears.
• Every chart above has a table twin underneath it, and the raw run logs are in the repository if you would rather cut the data a different way.