How Snowflake Built an Agent to Fix Broken Data Pipelines

How Snowflake Built an Agent to Fix Broken Data Pipelines

Skip to content

Blog / Data Engineering / The Path to Autonomous Data Engineering: Fixing Data Pipeline Failures with AI

Oct 19, 2026 / 5 min read Data Engineering Copy post link Open in Claude Open in ChatGPT

The Path to Autonomous Data Engineering: Fixing Data Pipeline Failures with AI

Zachary Blackwood

For years, a pipeline failure at Snowflake followed the same script: Someone got paged. They dug through logs, traced the root cause, wrote a fix, tested it and opened a pull request (PR). Then they did it all again the next night.

Today, an agent runs that script for us. It investigates every monitored pipeline failure we have and goes from incident detection to a ready-to-review PR in under 16 minutes on average. Here's how we built it and what it changed for our team.

The problem that kept getting worse

I'm a data scientist on Snowflake's internal data team, and I've watched our pipeline count grow from something one person could reason about to something genuinely overwhelming.

More pipelines sounds like a good problem: more data, more value. But every new pipeline is a new place to fail, and every new dependency is one more thing that can break at 2 a.m. Our failure volume wasn't rising because our pipelines got worse. It was rising because we had more of them, and our team wasn't growing at the same rate.

What we actually wanted to build

Most tooling stops at the alert: Something broke. Go look. We wanted an agent that could explain the problem and help solve it. That meant it had to own the full cycle:

Scan pipelines continuously for code smells and performance issues.

Catch failures proactively, before they escalate.

Dig into logs and the codebase to identify the likely actual root cause.

Test proposed fixes before anything touches production.

Open a clean PR with full context so the human can quickly review and merge.

That last requirement matters most. The agent proposes the repair. A person still decides what ships.

How we built it

Figure 1: First option was “one server to rule them all.”

We looked at a few architectures before landing on the right one. Our first instinct was a Snowpark Container Services server running a REST API: self-contained, with everything in one place. But it meant we would be limited by how many runs a single server could handle.

Figure 2: Second option was to run each agent in a separate job.

This allowed us to scale up to many different agents running in parallel without being bottlenecked by a single server.

Eventually, we plan to go beyond this and use the new EXECUTE AGENT TASK from CoCo automations (now in public preview) to simplify and scale our runs even further.

No server to manage. No shared compute pool. One SQL call triggers an agent run. Snowflake handles the rest.

Figure 3: Eventual option: simply run an agent directly.

We needed a flexible way to define a workflow with templated instructions. So workflows are defined in JSON. You specify the instructions file, which repos the agent can access, a turn budget, which model to run and where to send notifications when the work is done.

{ "name": "fix-broken-pipeline", "owner": "your.name@snowflake.com", "agent": { "turns": [{"file": "org/repo:path/to/instructions.md"}], "model": "opus", "max_turns": 50 }, "repos": [{"repo": "org/repo", "branch": "main"}], "timeout_minutes": 30, "notifications": { "slack_channel": "#data-eng-alerts", "emails": ["your.name@snowflake.com"] } }

Running it is just SQL:

CALL RUN_WORKFLOW('fix-broken-pipeline', '{"pipeline_id": "12345"}')

CREATE TASK SCHEDULE = '12 hours' AS CALL RUN_WORKFLOW('code-smells')

The task handles the clock. The agent handles the work. These workflows can be quite simple, or we can combine multiple simple workflows and human feedback to create more complex workflows.

Figure 4: An example of one of our more complicated workflows

The impact: Comprehensive investigations of failures

We've been running this in production, and, over the last five months across the managed pipelines our team owns, have seen:

Near-complete triage coverage for monitored pipeline failures. Designed to investigate every failure, not just the loud ones.

Under 16 minutes from incident detection to PR creation.

100s of PRs merged.

500+ pipelines improved.

Hundreds of daily wasted credits saved by pipeline improvements.

That last number is an important one. Compute waste from underperforming pipelines accumulates invisibly. Most teams never surface it because it requires someone to proactively look, but our agents look constantly.

Autonomous data engineering

This work has become a direct inspiration for what our product team will bring to customers. Read more about our vision for autonomous data engineering here .

Forward-looking statements

This article contains forward-looking statements, including about our future product offerings, and are not commitments to deliver any product offerings. Actual results and offerings may differ and are subject to known and unknown risks and uncertainties. See our latest 10-Q for more information.

Learn more about the author

Zachary Blackwood Staff Data Scientist

Share this post

Subscribe to our blog newsletter Get the best, coolest and latest delivered to your inbox each week

By submitting this form, I understand Snowflake will process my personal information in accordance with their Privacy Notice.

Where Data Does More

Start for free

Watch a demo

Subscribe to our monthly newsletter

Stay up to date on Snowflake’s latest products, expert insights and resources—right in your inbox!

Product

Platform

Snowflake CoWork

Data Engineering

Analytics

AI

Applications & Collaboration

Pricing

Support

Support

Priority Support

Status

Industries

Advertising, Media & Entertainment

Financial Services

Healthcare & Life Sciences

Manufacturing

Public Sector

Retail & Consumer Goods

Telecom

Technology

Company

About Snowflake

Leadership & Board

Careers

Investor Relations

Trust Center

Brand Guidelines

Contact

Newsroom

Environmental, Social & Governance

Snowflake Ventures

End Data Disparity

Snowflake Summit 26

Learn

Resource Library

Live Demos

Fundamentals

Training

Certifications

Snowflake University

Developer Guides

Documentation

Data Governance

© 2026 Snowflake Inc. All Rights Reserved

Site Terms

Communication Preferences

Cookie Settings

Do Not...