The execution engine behind the knowledge graph
When a design is approved, its configurations are pushed here for execution. It extracts data from each source, applies the designed transformations, resolves entities across systems, and loads the unified result into the knowledge graph, once or continuously.
Two run modes
Run once for a point-in-time unified dataset, or run continuously so data flows from source systems into the knowledge graph as it changes. Continuous operation keeps the graph current without manual intervention.
- Every run is logged. Each extraction query, transformation and load operation, with duration, so operators have full visibility into exactly what the SetMeld Pipeline did.
- Entity resolution at load time. Records are merged into the graph using the strategies designed for each entity type.
- Scales horizontally. SetMeld Pipeline instances run in parallel across sources.
Extract, transform, resolve, load
Extract
Reads from each connected source using the access paths already proven by the SetMeld Composer scan.
Transform
Applies the transformation logic designed in SetMeld Composer, including the automatic quality corrections.
Resolve
Applies entity resolution strategies so a customer, a patient or a part in three systems becomes one node.
Load
Writes the unified result into the knowledge graph, preserving datasource attribution on every record.
Operating the Pipeline
Timings on every step
Each query and load operation reports how long it took, so a slow source is obvious rather than mysterious.
Re-runnable
Rescan a changed source, review the diff to the design, and push again. Schema drift becomes routine maintenance.
Lake export
Push the validated graph onward as the upstream source of record for an existing data lake, preserving the analytics you already built.
Put a continuous sync on your most difficult pair of systems
We will scope it on a call and tell you honestly how long it will take.