In June, at Microsoft Build, Raki Rahman showed agents running production transformations on OneLake and I wrote that most data teams don't have a Raki. This month, on a Microsoft Reactor livestream, he came back for a full hour and showed how it's done. Every data engineer using Fabric should watch it. Every Head of Data should watch it twice, the second time with their favourite AI assistant taking notes.
First, the credentials, because they matter. Raki is a principal engineer on the SQL Server telemetry team, which runs real-time telemetry from more than three million SQL Server instances into Fabric. Around six petabytes in bronze, five in silver, a couple of hundred terabytes arriving daily, a five-minute freshness SLA. The team doing this is four engineers. His problem statement is the entire industry's problem statement, said plainly: "the only way we can scale without scaling our headcount is to be able to use AI."
So he built the machine that makes that safe. The hour walks through it piece by piece. A VS Code dev container with Spark, Livy, Scala and Python, built once and stored in a registry. Production OneLake mounted read-only into that container with BlobFuse, so the agent can, in his words, "trick it into thinking OneLake is a local hard drive." A SQL Server container impersonating the Fabric warehouse so dbt runs locally against the same T-SQL. dbt for everything silver-to-gold, with tests as the contract. A hand-authored skill file with ten provisions, every one of them paid for in advance, from a ban on the agent committing or pushing to a catalogue of every encoding his vendors have ever sent him. Five tasks, five git worktrees, five agents in five dev containers on five machines he reaches over RDP, because a Fabric workspace binds to exactly one git branch. His words again: "It's physically impossible. There's a one-to-one relationship." Then CI re-runs the whole suite against production data before any human merges, and fabric-cicd ships it.
It is a genuinely superb talk. It is also a bill of materials with an hour of assembly instructions.
This is not a rebuttal. I agree with nearly every design decision in that hour, and I can prove it, because we made the same ones.
He says the job has no tolerance for slop, that a data engineer has one job and it is presenting trustworthy data. Agreed, which is why an agent's claim about data has to survive independent execution against real data, never the agent's say-so.
He says the central move is to "box them into a framework that they cannot escape out from." Agreed. A model with a smaller action space and a hard contract is a model that ships.
He says pick industry standards over bespoke frameworks, because "any LLM in 2026 knows exactly what dbt is," while your in-house framework must first be taught, and the teaching eats the context window you needed for the work. Agreed, emphatically. This is the same reason we landed on dbt and dlt.
He says the agent needs a "fearless playground where it's able to write code, read data, and is able to do things without causing any damage." Agreed. That sentence is the whole isolation problem in twenty words.
And he says the data platform is not where the agentic tooling will come from, that he wants "the best tool that is in the industry" for the engineering itself. A principal engineer, on a Microsoft channel, saying the agentic layer sits above the platform. We are building an entire company on agreeing with that sentence.
Here is the part worth the second viewing (or asking your note-taking agent to summarize). Two components of that machine came off the shelf, dbt's discipline and dbt's agent skills, and Raki is rightly enthusiastic about both. Everything else is artisanal. The skill file and its ten provisions. The interface convention across half a million lines of Scala that lets an agent find the right code by string-matching instead of drowning its context window. The exemplar drivers. The test fixtures and the test-structure spec the agent must conform to. The container image. The BlobFuse wiring. The warehouse mock and the parity work to trust it. The CI pipeline that reuses the container against production data. The job registry, the vacuum registry, the permission policy, the encoding catalogue, the five-machine rig. Fourteen hand-built components, by my count, before an agent writes a single trustworthy line.
Raki can build that. Years on the platform, a codebase already shaped for it, and the kind of engineer who mocks a warehouse engine before breakfast. The talk cheerfully assumes you can too. I can't — which is why I have a co-founder, Umesh, with a cost-based optimizer running where the rest of us keep opinions. The talk's assumption is the industry's current operating model. We have collectively decided that the way to adopt agentic data engineering is for every data team on the planet to hand-assemble the same fourteen components, separately, forever, each maintained by whoever built them, each one person's resignation letter away from being archaeology.
His own best argument says otherwise. A bespoke framework is something the model has to be taught, and the teaching is a tax. His harness is bespoke by necessity — he built it before anything better existed, for a Scala estate no product covers. Yours would be bespoke by choice.
That gap is what Vibedata is. Vibedata is a coding agent specialized for data engineering, and the harness around it: the isolation, guardrails, context, and cross-platform reach a general coding agent lacks the moment the work is data.
When an engineer states an intent, the harness provisions what Raki assembles by hand, before the agent writes a line. A git worktree for the code. A sandboxed container for the compute. An ephemeral workspace on the data platform itself for the data, on Microsoft Fabric (and also DuckDB or MotherDuck today), through what we call Adaptors. The workspace exists for the life of that one intent and is torn down on merge. Ten intents means ten isolated worlds, running concurrently, on the same platform whose workspace binds to a single git branch. Nobody RDPs into anything.
The write is the point. A coding agent can read carefully, but a wrong transform still meets the real rows somewhere, and the only question that matters is where it fails — in a branch built to be thrown away, or in the table your boss's boss reads. Raki's mount gives his agents safe reads and local dummy writes; the quarantine of the real write path is what the workspace-per-intent model makes structural.
The rest of his design shows up as product decisions too. His skill file pauses to ask the user for the raw table; our agent is gated the whole way — no generated code until a design exists, a reviewer has been through it, and a human has approved it. His no-commit rule lives in a markdown file the next maintainer can soften; our equivalents live in hooks the agent cannot call and gates the harness refuses to skip. And the stack the agent is boxed into is the one his own argument selects, dlt for ingestion and dbt for transformation with Elementary watching the results, chosen precisely because the models arrive already knowing them.
Credit where it is due, twice over. His bronze-to-silver Spark estate, streaming petabytes with a four-person team, is his own engineering and remains comfortably his. And he keeps his gold-standard tests where the agent cannot touch them, which is the right instinct. We enforce the same separation on our own product, where the tests that grade our agents live in a repository the build agent cannot reach.
Last week I asked what your harness is, and whether you have one at all. The strongest evidence then was that four independent voices had converged on the same conclusion in one week. The strongest evidence now is better. The platform vendor's own principal engineer has moved from demonstrating a harness to teaching one, component by component, as a professional survival skill. When the how-to exists, the category exists. The only question left is whether every team builds the fourteen components by hand or whether it arrives as a product with opinions.
Raki's answer to "who maintains your harness" is that he does. He is a better answer than most companies will ever be able to hire. If yours is less certain, we are provisioning sandboxes — bring us your ugliest source and see what the harness does with it.
Nate Jones, Databricks, Snowflake and Microsoft all described the same gap in one week: your data platform needs a harness, not just a coding agent.
A software bug throws an error; a data bug lies silently for weeks. Why agentic data engineering needs a harness for Day-2 operations, not just build...
New Proof, Provocation, and Practice posts - straight from the team building the harness.