Egeria's Nanny Connectors Aim to Keep AI Context Current and Trusted

Context is an operations problem

In a post on the LF AI & Data Foundation blog, the fifth in a series on recent Egeria work, Chessell argues that a knowledge graph is a sensible way to ground a model, but it is a snapshot and often a single perspective. For one team experimenting with one assistant, that may be enough. For widespread AI use, she writes, the graph has to be something you can trust.

She lists what that demands of context. It has to stay current as data lands in new places, pipelines are rewired and systems retire. It has to be consistent, so that the same customer, product or supplier is not seen as half a dozen different things. It has to say where things came from, be delivered to teams and applications with an audit trail, be understandable beyond raw table and column names, and be governed.

None of this, in her view, is a modelling problem. It is an operations problem: somebody, or something, has to keep looking after the catalog. In a small deployment that is a steward with a spreadsheet. At enterprise scale, she says, it has to be automated.

What the Nanny Connectors are

That is the job of Egeria's Nanny Connectors. The name comes from the idea that a nanny looks after the household so the family can get on with life. The connectors watch an open metadata deployment, analyse it, tidy it up, connect it to the rest of the estate and hand its contents to the people and applications that need them.

They are integration connectors and governance services that run in the same integration daemon and governance engines as other Egeria connectors. They use the same open metadata APIs and the same audit log.

Chessell says this needs to be a family of connectors rather than a feature of one product because a typical enterprise runs many platforms: databases, a lake, a warehouse, Kafka topics, catalogs from different eras, a scheduler, a notebook platform and SaaS applications. No single platform sees the whole picture, she writes, so Egeria acts as a neutral, open metadata layer between them.

Each connector is named after a person whose work contributed to modern IT, including Charles Babbage, Ada Lovelace, Gregor Mendel, Charles Darwin and Barbara Liskov.

Babbage and Lovelace: interpreting what has been collected

The Babbage Analytical Engine is an integration connector that orchestrates a set of analysis services called the Lovelace Services. Each is a governance service that performs one focused analysis on the knowledge graph and records the result as a classification on the element it analysed. The current set includes:

• Award Karma Points: notices and rewards people who contribute to the catalog.
• Build Zone Membership Profile: summarises the make-up of each governance zone.
• Profile Pipeline Runs: analyses the runs of each job in an Open Lineage log store and records frequency, regularity, duration, failure rate and data volume against the job.
• Refine Data Scope: works out from the reads and writes of each dataset the pattern in which it is written and the window of data it covers, based on Open Lineage events.
• Summarise Data Quality: turns data quality assertions and tests in the Open Lineage log store into a survey report with the pass rate of each quality dimension.

Chessell notes that the raw Open Lineage events are already produced by pipelines. What is missing is interpretation: whether a job is reliable, how much history a dataset holds, and whether its quality is good enough. The services are selected independently, so a deployment runs only the analyses it wants.

Mendel and Darwin: duplicates and lineage

The Mendel Automated Duplicate Manager deals with near-duplicates that build up when many systems describe the same real-world thing. Its survivorship rules decide which properties the combined element inherits. When a potential duplicate is found, Mendel either validates it if the match is close enough, or raises a to do for a steward. It also revisits its own earlier decisions and withdraws a match that has stopped being one, but it never overturns decisions taken by a human steward.

Once enough validated duplicates cluster together, they are merged into a single consolidated element. Where the merge must choose between conflicting values, the losing value is written to the audit log, so a steward can see what was left behind.

The Darwin Product Dependency Manager traces the origin of each digital product's data. Lineage is captured at column level, but people usually ask coarser questions, such as which systems feed a server or which products a product depends on. Darwin works up through three levels: schema elements to data assets, data assets to the software servers that host them, and data assets to the digital products that package them. It maintains the resulting DataFlow and DigitalProductDependency relationships automatically. Dependencies asserted by a person but not proven by lineage are not deleted. They are recorded as exceptions for review.

What it means for AI deployments

The post's central point is that trustworthy AI context depends on continuous maintenance, not only on building a graph. For an application asking about a customer, Chessell writes, the difference between one well-formed answer and six competing ones is the difference between being useful and being misleading.

For organizations planning to run many agents and applications, the Nanny Connectors are presented as an open, vendor-neutral way to automate that upkeep using the same APIs and audit log as the rest of Egeria. The source text provided covers the Babbage, Lovelace, Mendel and Darwin connectors; it does not detail the others named in the series.