How Do You Know the Data Is Right?

‍

How Genesis Prover turns business assumptions into a specification it can verify

I’ve spent most of my career building data and analytics systems, and some of the hardest problems to find have been in systems that appeared to be working normally. Data was arriving on time and dashboards were updating, but the numbers didn’t quite represent what people thought they did.

Often the explanation was something ordinary: missing values had been treated as zero, a group of customers had been excluded from a calculation, or a join had changed which records contributed to a total. The pipeline kept processing data because there was nothing in those decisions that would prevent it from running.

As AI agents take on more of this work, those assumptions deserve more attention. We need to be clear about what the results are supposed to mean, and turn that understanding into something software can verify.

The assumptions behind the numbers

A request for “average customer spend” leaves quite a bit unsaid. Does the average include customers who made no purchases? What does a missing amount mean? Are we averaging over customers, orders, or accounts? Different answers produce different numbers, even when everyone believes they’re following the same requirement.

These choices show up in null handling, filters, joins, and grouping logic. An inner join can remove customers with no matching orders. Replacing a null with zero can change an average. Joining orders to multiple category records can count the same order more than once. Each operation is valid; whether it belongs in the calculation depends on the business definition.

A migration can carry those choices forward or change them inadvertently. A new pipeline has to make them in the first place. In either case, knowing that data arrived and was processed tells us little about whether the intended groups, values, and relationships survived.

‍

Where an agent’s assumptions become requirements

When a requirement leaves something open, whoever implements it has to fill in the gaps. An engineer might ask the person who owns the report, inspect an existing query, or make a judgment based on experience. An agent faces the same ambiguity, and can turn a reasonable guess into code without anyone noticing that a decision was made.

That decision can be difficult to see in a finished pipeline. A filter may look sensible in isolation while excluding an entire group the business expected to include. Asking the agent whether it implemented the request correctly doesn’t resolve that ambiguity if it still holds the same assumption.

Human input is essential, but reviewing every detail of every change becomes harder as agents produce more work. As our CEO Matt Glickman has said about work in regulated industries:

depending upon human engineers to keep up with and review all agentic work is a non-starter.
Matt Glickman, CEO

We need a way to make those assumptions visible and settle them as part of the specification, so they can be checked consistently as the implementation changes.

Turning assumptions into a mechanized spec

Genesis Prover brings together the stated requirements and the unstated assumptions embedded in existing queries, transformations, and system behavior, and turns them into a mechanized specification. By that I mean a spec expressed as rules that software can evaluate against the data and the implementation.

That requires making the meaning of each rule explicit: which records belong in a group, how missing values are handled, what counts as a unique entity, and which relationships must be preserved. For example, a team might agree on these rules for a customer report:

Illustrative rules, agreed with the business:

• Null handling: An average discount includes orders with known discounts, including zero. Missing discounts are excluded from both the sum and the count; if none are known, the average remains null.
• Group inclusion: Every active customer appears in the monthly report, including customers with no orders. Those customers have an order count of zero.
• Counting: Each order contributes once to a customer’s order count, even if it has several matching category records.

The existing system helps reveal these assumptions, but its behavior alone doesn’t establish that they’re right. An inferred rule may reflect an old workaround or a mistake. Where the intended behavior is unclear, the business owner needs to resolve it before it becomes an agreed requirement.

Once those decisions are captured in the mechanized spec, they provide a common basis for proving the work and for generating synthetic challenge data. Both depend on the same explicit definition of what should happen.

‍

Proving the work against the spec

The spec gives Prover something to verify that is independent of the implementing agent’s interpretation. Prover checks whether the resulting system satisfies the agreed rules, with evidence that the team can inspect.

For a migration, Prover also reads the original and resulting systems directly, comparing data, logic, permissions, and downstream views and queries. Those comparisons help identify what changed. The spec establishes which behavior is required, including where the team has deliberately chosen to correct the old system.

‍

1. The agent builds the migration, pipeline, or fix using the agreed spec.
2. Prover verifies the result against the spec and identifies deviations or gaps in the evidence.
3. The agent makes corrections and refers unresolved business decisions to their owners.
4. Prover repeats verification after each revision. Any change to the spec requires an explicit decision.
5. A person reviews the evidence and decides whether to approve the work.

This makes it possible to distinguish an implementation error from an intentional change in behavior. It also gives reviewers a specific question to resolve when a requirement is ambiguous. The agent doing the work doesn’t get to redefine success just because its output differs from the spec.
‍

Knowing what the evidence covers

A mechanized spec is useful only if we’re clear about which parts have actually been verified. If an account can read only part of a table, a comparison may leave out the very group we need to examine. A rule that couldn’t be evaluated still needs attention, even if the other results look right.

Prover records which account it used, what that account could access, and whether its inventory of the system was complete. Limitations remain visible so the person approving the work can see what the evidence supports and what is still unresolved.

The spec needs to say what we mean, and the evidence needs to show which parts we’ve verified.
Justin Langseth, Co-founder and CTO

Using the spec to generate challenge data

The same spec gives us a way to exercise cases that may be missing from the data available for verification. If every customer in a sample has an order, that sample won’t reveal whether customers without orders are being dropped. If every discount is populated, it won’t tell us how the system handles nulls.

Prover uses the mechanized spec to generate synthetic challenge data that exercises those rules. For the example above, we can include a customer with no orders, an order with multiple category matches, and a group whose discounts are all null. The spec supplies the expected behavior for each case.

It also lets us check combinations with a clear expected answer. Under the illustrative discount rule, three orders with discounts of 0%, 10%, and null should produce an average of 5%. Treating the null as zero would produce about 3.33%. That small dataset exposes the assumption and makes the discrepancy easy to explain.

The value of challenge data comes from choosing cases that can reveal a violation of the spec. We’ll go further into how to do that in the next post.

‍

Coming in the next post: from the spec to the cases that challenge it

We’ll look at how to generate synthetic challenge datasets from a mechanized spec, combine conditions that ordinary samples miss, and check the results against the business rules those datasets are meant to exercise.

Want to see how Prover verifies your data work? Talk to us.

Matt Glickman quote from Genesis Computing’s press release, PR Newswire, August 25, 2026.