Why AI Agents are Replacing Traditional Data Engineering

‍

TL;DR: Matt Glickman, co-founder and CEO of Genesis Computing, joined Joe Reis, co-author of Fundamentals of Data Engineering, for a conversation that starts with an uncomfortable premise: for organizations that care whether things are calculated consistently, AI has made the situation worse than it was five years ago, not better. The barrier to writing code was never only a constraint. It was also a governor on sprawl, and it has been removed. This post covers what a context graph is and why Matt argues it has to be primary infrastructure rather than a side project, why he inverts the “talk to your data” premise, who inside an enterprise actually has the mandate to sponsor this, and why he believes the largest opportunity in enterprise AI is capturing institutional knowledge as an owned asset.

Listen: The Joe Reis Show on Spotify · Watch on YouTube

‍

‍

What This Means if You Own the Data Function

• AI has lowered the barrier to creating pipelines, and that is not purely good news. The scarcity of people who could write code was also the thing preventing uncontrolled sprawl.
• You cannot safely change a data estate you have not mapped. Impact analysis requires knowing the dependencies, and in most enterprises those dependencies were never written down.
• Every previous attempt at cataloging failed for the same reason: it depended on humans doing metadata entry as a side project.
• The buyer for this is the CDO or CIO, not the engineer underwater in pipeline failures. Incremental automation sold to the person drowning does not change the operating model.
• Most organizations already have the mandate and are not acting on it. Matt’s read from CIO and CDO conversations is that confidence, not authority, is the blocker.
• The durable asset is not the model. It is the captured knowledge of how your organization actually works.

‍

What Goldman Learned in 2008 About Platforms

Matt spent more than two decades running data and analytics at Goldman Sachs, where a single unifying platform called SecDB defined how the entire front office represented a trade, a security, and risk. One language, one database, one development environment, one shared set of concepts, used across commodities, FX, equities, and bond trading. The system is still running: Goldman’s engineering team reports that SecDb has been in continuous use for more than thirty years, maintained by over 3,000 developers who have written more than 200 million lines of code in its proprietary language.

His view of why that mattered is direct, and he does not hedge it:

“I’m convinced that the reason Goldman survived and Lehman did not is that we had a platform we could change on a global scale, at a pace you couldn’t possibly match if everyone was doing their own thing, in their own way, with their own nomenclature.”

The specifics are worth understanding, because they are the argument for centralized context in an agentic environment. Things that were irrelevant in 2007 became existential in 2008. Collateral mattered. Margin mattered. The possibility that a counterparty could simply cease to exist stopped being hypothetical. Because the platform was already deployed firm-wide, Goldman could add a data warehouse on top of it and give every desk a macro view alongside their micro one, quickly.

Matt adds a condition that most platform arguments leave out: none of it works without a mandate from the top. Absent organizational support for a single way of doing things, you get what he calls normal enterprise chaos. This becomes the through-line of the entire conversation.

‍

What Is a “Strat,” and Why Does It Matter for AI?

A strat was Goldman’s term for a person who held business knowledge, technical knowledge, and data knowledge in a single head. Roughly analogous to a quant, though not identical. Goldman’s own history of the platform describes strats as multi-disciplinary specialists who blend quantitative finance, engineering, and technology, and credits them with building SecDB in the first place. Matt describes roughly a thousand of them at Goldman’s peak, embedded inside business lines rather than pooled in a central IT function.

The reason the role produced outsized results is mundane and important: problems got solved without meetings. No handoff between the business person who understood the question, the data person who understood the model, and the engineer who understood the system.

“Most people who understand finance don’t care about architecting databases. Most great data modelers don’t care about how to price a bond. Most deep computer scientists want to go theoretical.”

Matt is explicit that this is his North Star for building with AI. The strat profile works, and it does not scale, because the combination of skills and interests is genuinely rare and the people who have it are expensive and mobile. His bet is that the combination can be assembled in a system rather than hunted for in a hiring market.

‍

Is AI Making Enterprise Data Chaos Better or Worse?

Worse, in the near term, and Matt says so plainly. This is the most contrarian claim in the episode and the one most likely to match what data leaders are actually seeing in their environments right now.

The logic runs against the standard narrative. For years, the number of people who could write code and had access to production systems was treated purely as a bottleneck. Matt’s point is that it was also functioning as a control:

“It used to be that the barrier to entry, having a certain number of people who knew how to code and had access to systems, was a limiting factor, but in some ways it was also a governor that prevented sprawl and chaos.”

That governor is gone. Anyone can now generate a pipeline or an application on demand. For an organization whose actual question is whether metrics are calculated consistently and whether anyone can explain how a number was produced, Matt argues the situation is arguably worse than it was five years ago.

The 2026 data supports him. Gartner projects that the average Global Fortune 500 enterprise will run more than 150,000 AI agents by 2028, up from fewer than 15 in 2025, while only 13% of organizations believe they have the right governance in place today. Gartner’s own framing for what CIOs are now managing is ungoverned agent sprawl. CloudBees, surveying more than 200 enterprise technology leaders, found that 81% report an increase in production issues tied to AI-generated code.

He also names the intermediate failure state that has replaced the demo problem. Teams are no longer stuck at “we built a cool demo.” They are stuck at “it works on my laptop, so I am just going to use it.” That is a harder problem, because the output is now in circulation.

For anyone running data governance in a regulated environment, this section is the one to sit with. The volume of net new, undocumented, individually authored data logic entering your estate went up sharply, and the mechanism that used to limit it was never a policy you could reinstate. Matt develops the companion argument in a separate conversation on IBM’s Making Data Simple podcast, covered here in why code review stopped working as a control: review scaled only because human typing speed capped output, and that cap is gone.

‍

What Is a Context Graph?

A context graph is a continuously maintained map of an enterprise’s data systems, the dependencies between them, and the human knowledge that explains why they are arranged that way. Matt describes it as a second brain for the enterprise, acquired the way a search engine crawls the web rather than the way a catalog collects submissions.

The construction is directional and starts where the data lands:

“It’s acquired Google-crawl style: start in one place, a warehouse tends to be a nice place, because it’s at the end of these distributed chains. Start there and work backwards: where’s the data coming from, who’s touching it, which APIs are being used to pull data in, where do those APIs live, where do those source systems live.”

The premise is that any non-trivial enterprise runs on interconnected systems, source databases, code repositories, communication tools, document repositories, and ticketing systems, and that at any given moment some percentage of the connections between them are failing. The organization’s real job is minimizing that percentage. None of it is written down.

Matt is pointed about why previous attempts failed. Cataloging vendors promised to inventory systems, semantics, and data dictionaries, and they failed for a structural reason rather than a technical one: every one of them depended on humans performing metadata entry as a secondary task. Nobody does their secondary task.

Two properties make the Genesis Context Graph operationally different from a catalog. Genesis builds it during onboarding by connecting to repositories, databases, documents, and catalogs, a process described in more detail in the agentic data engineering overview.

It is a digital twin, borrowed deliberately from manufacturing. The point is not documentation. It is simulation. If I want to make this change, what is the blast radius, what is the impact, and what do I need to test?

It tracks change over time, which Matt argues no enterprise does today. His example is the one every infrastructure leader recognizes: the cloud bill that nobody questions until it is enormous, when in reality it degraded gradually and no one was watching the trend.

“Typically CIOs only hear about something when it breaks. Some human observes something, and then there’s a root-cause analysis and a fire drill.”

His conclusion is a positioning statement as much as an architectural one: this has to be primary infrastructure, not a secondary system bolted on after the agents are already running.

‍

Why Start With Data Engineering Specifically?

Because the data plane is the ground floor, and it is far easier to work up a stack than down it.

Matt’s reasoning is practical. If you understand where data originates, where it lives, and how it moves, you can follow it upward to see how it is consumed. Starting in the middle, or at the BI layer, means inheriting abstractions without the ability to change what sits underneath them. Working from the data plane means you can actually mutate the state you need to change.

Joe Reis pushes on this productively, noting that most agentic data engineering pitches he receives amount to replicating the current task list of a data engineer, which he finds uninteresting. It is a fair challenge, and the distinction Genesis draws is between suggesting code and completing work: a full dbt project across nine phases, executed end to end with human input required once, is a different artifact than an autocomplete. That framing sets up the strongest section of the conversation.

‍

What Does Agentic Data Engineering Look Like From First Principles?

Data engineering disappears as a visible function, and the direction of the relationship between people and data reverses.

Matt’s answer has two parts, and neither is about doing today’s tasks faster.

Humans move to idea generation and idea analysis. The machinery fades into the background. A person has an idea for a product, and an agent determines what data exists, what does not, acquires what is missing, shapes it correctly, and operates it efficiently. His argument is that people were never suited to this work in the first place. The systems were too messy and changed too often to automate, so humans did it, and the workaround calcified into a career. He cites the Ralph Wiggum meme about being the data engineer nobody knows the name of, which lands precisely because the role is only visible when something breaks.

And then the inversion, which is the most quotable line in the episode:

“People shouldn’t talk to their data; their data should talk to them.”

This is a direct shot at the premise underneath most of the current market. The goal is not a better interface for querying. It is a system that monitors the environment and surfaces things unprompted. New data becomes available on a market, and it is pulled in, combined with existing data, and brought to a person as a proposition rather than waiting to be asked about.

The organizing concept is what Matt calls a meta mission. Humans set the objective and the reward function: find new ways to trade more efficiently, find better supply chain monitoring, find and evaluate new suppliers. Agents gather what they need and act. Humans stay involved on the questions that actually require judgment, which are whether the mission is still valid and whether the system is optimizing for the right thing.

“We have to elevate humans away from all the mechanism that keeps systems running today.”

In the Genesis platform this maps to a specific structure. A Blueprint defines the reusable workflow, a Mission carries the objective and success criteria, and Tasks are the individually verifiable steps inside it. The synthetic data generation blueprint for Databricks is a working example of that pattern running inside a customer’s own environment.

‍

Who Inside the Enterprise Actually Sponsors This?

The CDO or the CIO. Not the data engineer who is underwater keeping pipelines alive.

This is the most directly useful section for anyone building a business case, and Matt is blunt about the failure mode. Sell incremental automation to the person drowning in pipeline failures and you will get a sale and no change to the operating model. She wants to type less. That is not the same as changing how the function works.

“You sell to the CIO. You sell to the CDO, who now actually has something to do. The CDO role has traditionally suffered, looking for a problem to own, an ROI problem. Now the CDO has teeth.”

The teeth are measurable. Deloitte’s 2026 Chief Data and Analytics Officer Survey, covering C-suite data leaders at companies with $1B or more in revenue, found that 78% say AI has given them more power as decision makers, 63% now describe themselves as the primary drivers of data and analytics decisions, and 94% expect their influence to grow over the next year.

His observation from the field is worth taking seriously if you are the one who has to make this argument internally, and it echoes what he has said elsewhere about how data engineers are multiplying their own output with agents. Nearly every CIO, CDO, and CEO he speaks with already has the mandate. What they lack is confidence that they can act on it non-incrementally. The constraint is not authority. It is conviction.

He pairs this with a design lesson from Tesla that has nothing to do with autonomy and everything to do with adoption: the decision to show the driver a live view of what the car is seeing. Most drivers have had the experience of watching the system identify a cyclist or a pedestrian before they did. That visibility is what earns the willingness to let go, and it was a deliberate product choice rather than a technical necessity.

‍

How Do You Sequence This Without Breaking the Organization?

Incrementally, with agents observing and collaborating before they replace anything. Matt is direct that the failure mode being circulated as evidence against agents is usually a sequencing failure rather than a capability failure.

“You can’t, on a dime, say, ‘Okay, we’re just going to take all the people out and replace them with agents.’ That doesn’t work.”

The self-driving parallel holds up better here than it usually does. Tesla did not ship autonomy. It shipped sensors that watched how humans drove, built up knowledge, automated the easiest scenario first, which was highway driving, and expanded outward as confidence accumulated on both sides. The system got better and the humans got more willing.

Joe adds a structural point that Matt agrees with, and it is one worth raising with any executive sponsor. He describes a large retailer where one division has been told to increase a metric by 30% with 20% fewer people, and the person closest to it expects failure, because the org chart is not built for it. Agent workflows are lateral. Most enterprises are hierarchical and top down. Shoehorning one into the other produces exactly the failure stories that then get cited as proof the technology does not work.

Matt’s own framing for where the value sits is an iceberg. Above the waterline are today’s data professionals. Below it is the much larger population who consume data and are currently blocked, waiting on the people above the line. His open question, and he thinks it resolves faster than most expect, is whether anything should remain above the waterline other than an automated system serving everyone below it.

‍

What Is the Biggest Opportunity in Enterprise AI?

Capturing how an organization actually works and owning it as an asset, independent of any model provider.

This is where the conversation opens up beyond data engineering, and it is the most strategically interesting argument Matt makes.

His framing of frontier models is that they arrive like a talented new graduate. Broad competence about how the world works, zero knowledge of how your institution operates. Every organization is currently solving that by handing context to the model over and over, per task, per session, and retaining nothing.

“Getting that out of human brains and into a singular AI brain, a context graph, that can be reasoned about, optimized, questioned, and even traded on.”

The word doing the work there is traded. Matt extends the argument into mergers and acquisitions, and it is a genuinely novel application:

‍

The acquisition case follows directly. Diligence today cannot answer whether the value of a target is real IP and durable process, or one person’s undocumented judgment that walks out after the earnout. If a buyer could inspect the target’s captured operating knowledge, that becomes an answerable question, and integration becomes a merge rather than an archaeology project.

Matt’s example of a company starting from this premise is Tower Peak Partners, a London-based private markets investment firm running Genesis inside Snowflake Cortex and building as an agentic firm from the outset. Their team has described the productivity change as structural rather than incremental three months in. Their operating principle mirrors what he saw at Goldman: if it is not in the system, it does not exist. A bad decision that was modeled with full data is acceptable. A bad decision nobody can explain is not.

“If you start today with that operating principle, and you only hire for the things that can’t be done as this AI wave accelerates, you have a major advantage over competitors who are going to have to rewrite their systems and processes to ride this wave.”

Matt is deliberately less interested in the version of this argument that gets the most attention. Sam Altman’s prediction of a one-person billion-dollar company gets most of the airtime. Matt calls it interesting but says the more consequential story is non-technology companies in their own industries riding the wave, which is where the vast majority of enterprise value actually sits.

‍

Does Genesis Run on Genesis?

Yes, and Matt gets specific rather than gesturing at it.

Agents run the analysis work in finance that would traditionally have been done by people, including building a calculator to simulate the business without anyone on the team writing code. Marketing operates the same way. The stated rule is that anything with a digital input and a digital output should be done by agents.

The exception he names is instructive. Enterprise sales stays human for a long time, because it runs on trust between people. His aside is that it will outlast plumbing, since plumbing will eventually go to robots and relationships will not.

‍

The Takeaway

The argument worth carrying out of this conversation is that the hard part of enterprise AI was never the model.

Frontier models arrive capable and contextless. Every organization is currently paying, repeatedly, to explain itself to a system that forgets. Meanwhile the knowledge that actually makes the organization work sits distributed across people who can leave, and it degrades quietly, and nobody is tracking whether it is getting better or worse.

Matt’s position is that the enterprises that win capture that knowledge into something they own, then let agents operate against it. The data estate is where he argues you have to start, not because it is the most valuable layer, but because it is the ground floor and everything else is built on top of it. The pattern shows up in deployment: GXS Bank compressed pipeline development from months to hours once agents had the context to work from.

“Turning the liability of how much knowledge is in which people’s heads, and how much of it is actually defensible, into a true asset, is the real opportunity of enterprise AI.”

Listen to the full conversation with Joe Reis and Matt Glickman on YouTube or Spotify.

‍

Frequently Asked Questions

What is a context graph in enterprise data? A context graph is a continuously maintained map of an enterprise’s data systems, the dependencies between them, and the human knowledge explaining why they are arranged as they are. It is built by crawling the environment rather than by asking people to fill in metadata, starting at the warehouse and working backward through pipelines, APIs, and source systems. It functions as a digital twin of the data estate, enabling impact analysis before a change is made.

How is a context graph different from a data catalog? Data catalogs depend on humans entering metadata as a secondary task, which is why most catalog initiatives stall. A context graph is acquired automatically by crawling systems and inferring dependencies, and it tracks how the environment changes over time rather than describing a static snapshot. Its purpose is simulation and impact analysis, not inventory.

Is AI making enterprise data governance harder? In the near term, yes. The number of people who could write code and access production systems was a bottleneck, but it also acted as a governor limiting uncontrolled sprawl. With code generation available to everyone, organizations are producing more undocumented, individually authored data logic than before, which makes consistency and explainability harder rather than easier.

Who should sponsor an agentic data engineering initiative? The CDO or CIO. Selling incremental automation to engineers currently overwhelmed by pipeline maintenance produces marginal efficiency without changing the operating model. Restructuring how data work happens requires an executive mandate, and Matt Glickman’s observation is that most data executives already have that mandate but lack confidence to act on it non-incrementally.

What is a “meta mission” in agentic data engineering? A meta mission is a standing objective set by a human that agents pursue continuously, such as finding more efficient trading opportunities or identifying and evaluating new suppliers. Rather than assigning discrete tasks, humans define the goal and the reward function, and remain responsible for judging whether the mission is still valid and correctly specified.

Will agentic AI eliminate data engineering jobs? Matt Glickman’s position is that the function itself fades into the background rather than the people disappearing. He points to a much larger population of business users who consume data and are currently blocked waiting on data teams. The shift moves people from maintaining mechanism toward defining objectives and analyzing results.

What is the difference between a copilot and an autonomous data engineering agent? A copilot waits for human instruction on every action and suggests code. An autonomous data engineering agent monitors the environment, decomposes an objective into verifiable steps, and executes them, escalating to a human when confidence is low. Matt Glickman discusses the validation problem this creates in a companion conversation on why code review stopped working as a control.

How does knowledge capture change M&A diligence? Acquisition diligence currently cannot determine whether a target’s value is durable institutional process or the undocumented judgment of a few individuals who may leave. If a target’s operating knowledge is captured in an inspectable system, a buyer can evaluate whether unique IP and process actually exist, and integration becomes a matter of merging captured knowledge rather than reconstructing it through interviews.

‍

About the Joe Reis Show

Joe Reis is co-author, with Matt Housley, of the O’Reilly book Fundamentals of Data Engineering, co-founder of Ternary Data, and an instructor at the University of Utah. He writes regularly at joereis.substack.com. He describes himself as a recovering data scientist and has worked in data for two decades. The Joe Reis Show features unscripted long-form conversations on data engineering, architecture, AI, and careers, and is available on Spotify and wherever podcasts are found.

‍

About the Author

Matt Glickman is co-founder and Chief Executive Officer of Genesis Computing. He spent more than two decades at Goldman Sachs running data and analytics, then joined Snowflake in 2014 to lead product, where he helped make financial services the company’s largest industry and built the Snowflake Marketplace into a platform for sharing data and applications. He co-founded Genesis Computing in 2024 with Justin Langseth. LinkedIn

‍

About Genesis Computing

Genesis Computing is the agentic data engineering platform. Genesis deploys pretrained autonomous AI data agents securely inside enterprise environments across Snowflake, Databricks, AWS, and Azure, using blueprints for repeatable multi-step workflows and the Genesis Context Graph for enterprise-wide context. Genesis was recognized in Gartner’s “Data Engineering 2.0” research (G00852814, April 2026).