Your AI agents are going to be wrong until you build a knowledge graph

Your AI agents are going to give wrong answers, regularly, until you build a high quality knowledge graph.

If you work in data at a large company, you’re probably thinking you already have a plan for this. You bought a data catalog (or you’re migrating to a nicer one), you have a modernization program with a name like “the mesh,” somebody on your team owns the semantic layer, and there’s a pilot agent running in claims. That’s a lot of activity, and I’m not going to tell you any of it is wasted. But I had a call last week with someone who runs data projects at a large insurance company, and by the end of it we had both agreed that the piece in the middle, the piece that turns all of that activity into agents that answer correctly, is the piece almost nobody in leadership can name.

I’m writing this because that call put words to something I’ve been circling for months across a lot of similar conversations. Everyone can describe the destination (agents that answer business questions the same way every time) and everyone can list vendors. The stretch of road between the two is what I want to lay out here, in plain terms, because it’s the stretch where I keep watching smart companies stall. (I say this as someone who once spent a full weekend hand-writing SQL joins across four systems to answer a question my boss had asked in passing, so I have personal experience with the stall.)

What an agent does when it doesn’t have a knowledge graph

Some definitions first, because I’m going to use a few terms that mean slightly different things to everyone. A knowledge graph is a database where every record is a real thing (a client, a policy, a claim) and every relationship between things is stored explicitly, so “this policy belongs to this person who was underwritten by that office” is a one-line query and not a two-week project. A taxonomy is the agreed vocabulary for what those things are called and how they nest (think of the folder structure of a very well organized filing cabinet). An ontology is the taxonomy plus the rules about how things relate to each other. When people say business context, they mostly mean the taxonomy plus a paragraph of human explanation for every field.

Now, imagine you point an agent at a company that has none of that. You ask it, “What’s the total net worth of clients who bought a term policy in the last five years?” Here is roughly what happens. The agent looks at the list of data sources it can reach. It reasons about which of them might contain policies and which might contain net worth. It writes a little script to pull from each one. It writes another little script to join them, guessing at the join key. It returns a number.

Some of the time that number is right. But large language models are non-deterministic (ask the same question twice and you may get two different plans of attack), and every step above is a place for a guess to go sideways. The agent that says, “I found a field called NW_AMT in the marketing warehouse, that’s probably net worth, let’s go,” is doing exactly what a new analyst would do in their first week, and we would never let the new analyst’s number go into a board deck without someone checking it.

Compare that to an agent sitting on top of one high quality knowledge graph. Its job collapses to writing one query against a database where “client,” “policy” and “net worth” each mean one specific thing. Same question, same workflow, same answer, every time. That’s the whole argument for the knowledge graph, and I don’t think it’s controversial. My friend on the call said as much: most people figure out the why on their own.

The part nobody has a name for

So if the why is easy, what’s the problem? Her framing was three questions: why, what and how. The why is “we need agents that answer correctly.” The how is vendor selection, and every enterprise is currently drowning in vendors happy to help with that. The what is the actual list of transformations the data has to go through, whether a person does them or a system does, before any platform on top can work. She said the what is nearly absent from the leadership vocabulary, including among her own peers, and that she spends a good chunk of her week trying to convince people that capturing business context is worth the time.

I’ve had this conversation with enough large companies over the past few months that I believe her, and the missing piece is the same one every time. Here is what it looks like on the ground, using her examples (anonymized, since this was a chat between peers and not a sales engagement).

Her company stores the same underlying facts in a dozen formats, and I mean the same client’s same address, entered under different attribute names, different types and different validation rules depending on which system it landed in first. The mainframe systems were built product-first, so the unit of record is the insurance policy and the human being is an attribute hanging off it. Years ago they ran a golden record project (a golden record is one master row per client, produced by matching and merging every duplicate the company has ever created) and it got so complex to maintain that they’re now building a canonical client model instead. Some of the older databases have field names like C1332 and no documentation at all, so the schema reads like a ransom note cut out of license plates, and the only person who could translate it retired to a lake in 2011.

The best example she gave was net worth. There is a client net worth value in marketing, a different one in sales, and a third one in underwriting. All three are called net worth. All three are correct within their own department, and each department will tell you with a straight face that theirs is the one that matters. So there is no canonical definition, and the semantic layer (the layer whose entire purpose is to hold canonical definitions) can’t be finished until someone gets three departments in a room and makes them agree. A catalog will happily list all three net worths side by side, and an agent will happily average them, and no amount of vendor evaluation changes either fact.

The Mars Climate Orbiter problem

In 1999 NASA lost a $125 million spacecraft because one team’s software reported thrust in pound-force seconds and the other team’s software read the same number as newton-seconds. Both teams’ code worked. Both teams’ numbers were right. What was missing was a shared definition of the unit, written down somewhere both sides had to look. Enterprises have this problem in nearly every table: a field holds a number, and nobody recorded whether it’s Fahrenheit or Celsius, dollars or thousands of dollars, calendar days or business days. An agent reading that field will return a wrong number with total confidence, and unlike a crashed orbiter, a wrong number in a dashboard doesn’t get a congressional investigation. It gets forwarded.

What the road actually looks like

Since the what is the missing piece, here is the what, at the level of a process and not an architecture diagram. This is the sequence we walk through with clients, and it works whether a system runs it or a very patient person does.

1. Pick one data set, ideally one business domain (policies, claims, client). Connect to it read-only. Nothing about the existing system changes, and nothing has to be migrated first.
2. Scan it. Pull whatever meaning can be extracted from the schema, sample values, and any documentation that happens to exist.
3. Generate a list of everything the scan could not figure out. What entity does this table represent? What does C1332 mean? What unit is this column in? Which of the three net worths is this one?
4. Get subject matter experts to answer that list on their own time, as a checklist they can chip away at whenever they happen to be awake (if that’s 3 a.m., great). Management can see who has answered and who hasn’t. The alternative most companies use today is scheduling meetings with each SME, which takes weeks, burns hours of review time per table, and produces notes nobody can find later.
5. Use those answers to map the data set into the shared taxonomy and merge it into the knowledge graph.
6. Repeat for the next data set. When enough of them are in, point your catalog, your BI tool and your agents at the graph and let them query one thing.

Step 3 is the one I care about most. The reason digital transformation projects run over time and over budget is that nobody knows how much unknown stuff is left, so nobody can estimate. Once you have the list of what the data doesn’t explain about itself, you can count it, assign it and predict when it will be done, and the count will probably be uncomfortably large and will still be the first estimate anyone at the company has been able to defend.

Full disclosure, since I’d be a bit ridiculous pretending otherwise: this is what SetMeld does. It scans, builds the checklist, collects context from the SMEs, and produces the knowledge graph. What you do with the graph is your business. Hook it to your catalog, feed it to your agents, export it somewhere else. The automated part is the part I like showing off, and it’s also worthless if a company hasn’t done the work of deciding what its data means, which is why the checklist matters more than the code.

What to do before your next digital transformation

Pull three tables from your oldest system and try to write, in plain English, what every field means and what unit it’s in. Then go find your version of net worth: pick one term the whole company uses, get every department to write down their definition, and put the definitions side by side on one page. If those two exercises are easy, you’re ready for agents. If they aren’t, that page is your project plan, and you should start on it before you sign anything.