Unify siloed data into a high quality, queryable knowledge graph

Introduction

Organizations accumulate data in dozens of disconnected systems. Research databases, operational stores, vendor platforms and departmental applications each hold a fragment of the picture, and each uses its own schema, identifiers and conventions. Answering a question that spans these silos typically requires weeks of manual integration work by data engineers, and the result is brittle, one-off plumbing that must be rebuilt whenever a source changes.

SetMeld eliminates this integration burden. The platform connects to existing structured datasets, uses AI to design the transformations required to unify them, and continuously loads the result into a single high quality knowledge graph. Analysts, applications and AI agents can then query all organizational data through one interface, with full provenance back to the original sources.

SetMeld consists of two cooperating components. SetMeld Composer is the design environment in which data sources are connected, scanned and mapped to a unified schema. SetMeld Pipeline is the execution engine that carries out those mappings, extracting data from the sources and loading it into the knowledge graph on a one-time or continuous basis.

Architecture

SetMeld supports two deployment models: a fully self-hosted deployment inside the customer’s own network, and a managed deployment on SetMeld Cloud. Both models share the same core components and workflow. The choice between them is driven by the customer’s security, compliance and operational requirements.

Self-hosted deployment

Every SetMeld component runs inside the customer’s on-premises environment or private cloud network. No data ever leaves the customer’s infrastructure. SetMeld Composer scans the existing siloed datasets and reasons about them using an LLM of the customer’s choosing, hosted within the same network. SetMeld Composer then builds configurations for SetMeld Pipeline, which extracts data from the sources and loads it into a unified knowledge graph that also resides inside the customer’s network. Agents, software and analysts query the graph directly.

This model is designed for regulated buyers and institutions with strict data residency requirements. Because the reasoning model is supplied by the customer, sensitive schema information and data samples are never transmitted to a third party.

SetMeld Cloud deployment

In this model both components run in SetMeld Cloud, using an LLM that SetMeld operates. They connect to the customer’s siloed datasets over secure channels, scan them and extract data. The resulting knowledge graph is loaded back into the customer’s own network, so the authoritative unified store remains under the customer’s control. Queries from agents, software and analysts are served from that customer-resident graph.

The cloud model minimizes operational overhead. Customers do not need to provision compute for SetMeld Composer, SetMeld Pipeline or the reasoning model, and they receive platform updates automatically.

Optional data lake export

In either deployment model, the knowledge graph can serve as an upstream source of record for an existing data lake. After data has been unified and validated in the knowledge graph, SetMeld can push it onward to the customer’s lake. This preserves existing analytics workflows built on the lake while ensuring that the data flowing into it has already been cleaned, unified and enriched with provenance.

Using SetMeld Composer

SetMeld Composer guides users through four stages: connecting data sources, generating a unified schema, syncing data, and querying the result. An embedded AI assistant is available at every stage. The assistant performs the heavy lifting, narrates its actions, and asks for clarification whenever it encounters ambiguity, so that a system administrator can complete a full integration without writing transformation code.

Connect any structured dataset

Integration begins by connecting the source systems. System administrators provide credentials for any structured dataset through a guided connection flow. If a connection fails, the assistant diagnoses the problem and coaches the user through resolving it, whether the cause is a network restriction, a permissions gap or a misconfigured credential.

Users can also upload documentation alongside each source. Data dictionaries, schema notes and internal wikis give the AI additional context about what the data means, which improves the quality of the mappings designed in the next stage. If any point of confusion remains, the model asks the user for clarification rather than guessing.

Once connected, SetMeld Composer scans each database and extracts an AI context: a structured description of every entity, field and datatype, enriched with inferred meaning. This scan serves a second purpose. Because it exercises the same access paths that SetMeld Pipeline will later use, a successful scan is a proper end-to-end test that data extraction will work once the configuration is pushed to the SetMeld Pipeline.

Automatically design transformations

The AI context gathered during scanning is fed into SetMeld’s AI pipeline, which designs the integration. The pipeline determines the exact ontology and schema needed to unify the connected sources into a single coherent model. Entities that appear in multiple systems under different names and structures are identified and aligned, and relationships across sources are made explicit.

During this stage, the pipeline also inspects the data for common quality problems such as inconsistent formats, duplicate records, orphaned references and conflicting values. Problems that can be fixed automatically are corrected as part of the transformation design. Problems that cannot be fixed are reported clearly to the user, along with potential workarounds, so that no data issue is silently ignored.

By default, every generated schema extends the GIST upper ontology, a minimalist business ontology maintained by Semantic Arts. Anchoring all mappings to a common upper ontology enforces clean ontology hygiene: concepts are defined once, categories do not proliferate, and schemas produced for different deployments remain mutually consistent. Organizations that have already invested in their own ontology can bring it instead, and SetMeld Composer will normalize the connected sources to that model.

As part of transformation design, the pipeline also determines entity resolution strategies for each entity type, defining how records from different sources are recognized as referring to the same real-world thing. The output of this stage is a complete set of configurations for SetMeld Pipeline: extraction plans, transformation logic, entity resolution rules and load specifications.

Run SetMeld Pipeline

When the design is approved, the configurations are pushed to SetMeld Pipeline for execution. The SetMeld Pipeline extracts data from each source, applies the designed transformations, resolves entities across systems, and loads the unified result into the knowledge graph.

The SetMeld Pipeline can run in two modes. It can execute once, producing a point-in-time unified dataset, or it can run continuously, allowing data to flow from the source systems into the knowledge graph as it changes. Continuous operation keeps the graph current without manual intervention. Every run produces detailed logs of each query, transformation and load operation, giving operators full visibility into exactly what the SetMeld Pipeline did and how long each step took.

End result: a high quality, queryable knowledge graph

The outcome of the workflow is a single knowledge graph that unifies all connected data under one clean schema. Agents, software and analysts can query all organizational data in one database query, without knowing which source system originally held each record. Results carry datasource attribution, so every answer can be traced back to the systems that produced it.

Because the graph is high quality by construction, it is a dependable foundation for downstream use. Business intelligence tools connect to it as a single source. Applications integrate against a stable, well-documented schema. AI agents query it with provenance and access controls intact, making the graph a trustworthy substrate for the agentic workflows organizations are actively building.

Summary

• SetMeld Composer connects to any structured dataset through a guided, assistant-supported flow and verifies extraction end to end at scan time.
• The AI pipeline designs the unifying ontology, transformations and entity resolution strategies automatically, anchored to the GIST upper ontology or the customer’s own ontology.
• SetMeld Pipeline executes those designs once or continuously, loading clean, unified data into a knowledge graph with full logging.
• Deployments run entirely on-premises with a customer-chosen LLM, or in SetMeld Cloud with the graph remaining in the customer’s network.
• Agents, software and analysts query all data through a single interface, with datasource attribution on every answer and optional export to an existing data lake.

The result is that integration work which previously took an engineering team weeks is reduced to a guided session in SetMeld Composer, and the organization gains a durable, queryable asset rather than another one-off pipeline.