We Taught An Ai Agent To Qa Our Cloud Console In Plain English

How an AI Agent Automates QA for the Cerebras Cloud Console Skip to main content

Introducing CS-4: The Fastest AI Accelerator in the Industry Learn more >>

Sep 08 2026 We Taught an AI Agent to QA Our Cloud Console — in Plain English Kartik Malunjkar Urmi Banerjee Hagay Lupesko

Cerebras is known for speed. Our platform serves the fastest tokens in the industry — but speed isn't only silicon; it's also how fast we can ship the console, playground, and billing flows developers touch every day. And the faster we ship, the louder one question gets: how do we know we didn't break anything? The traditional answer is brittle machinery. A human writes a script full of CSS selectors, the frontend team renames a div, and overnight half the suite goes red — not because the product broke, but because the test did. That tax scales with how fast you move. So we asked a different question: what if a test read like a QA engineer's notebook, ran itself like a QA engineer, and fixed itself when the UI drifts? That's console-prompt-tests — an internal framework where every test is a paragraph of plain English, run by an LLM agent driving a real browser. Here's what we learned building it. A few months in, the payoff is concrete — each stat below reads before → after :

Real run. Frame one of an actual test run: the agent opens a real browser on the Cerebras Cloud sign-in page and takes it from here — no fixtures, no mocks.

A test is a paragraph, not a script Here is a complete, real test from our suite. Not pseudocode — this is the actual file the system runs: A .pt file — a prompt test. No data-testid, no getByRole(), no network-idle waits. Just intent. There isn't a selector in sight — no data-testid , no getByRole('button', { name: /github/i }) , no waiting on a network-idle event that may never fire. Just intent, written the way you'd describe the flow to a teammate. An agent reads that intent, looks at the actual page in front of it , and finds the GitHub button the way a human would. Rename a class, reshuffle the DOM — the test shrugs. It was never looking at the markup; it was looking at the page. That single property — tests that describe behavior, not markup — is what makes the rest of the system possible.

The same UI refactor that reddens a selector suite leaves the prompt test untouched — the single largest source of E2E flakiness, gone.

How it works: an agent, a browser, and a protocol Under the hood, three pieces do the work. 1. The prompt runner. A Python harness ( run_prompt_with_agent.py ) loads a .pt file, hands it to an LLM coding agent, and runs the show: logging, timeouts, a fresh browser session, and a per-test screen recording — so every run leaves a .mov you can watch later. 2. Playwright MCP. The agent doesn't get raw access to the browser. It drives a Playwright instance through the Model Context Protocol (MCP) — a tool interface exposing exactly what a tester needs: navigate , click , type , wait for text , screenshot . The agent plans, MCP executes, the browser responds, the agent looks again. A perception–action loop, not a fixed script. 3. A real, signed-in browser. These tests run against a live deployment — a staging environment or a local Docker stack — with real auth. We handled the annoying parts (GitHub's first-login device-verification email, NextAuth session minting for headless CI), so the agent lands in an authenticated console and gets straight to the flow under test.

A hardcoded script fails the instant a button moves. An agent that can see the page adapts, retries, and only fails when a human would fail too — when the thing it was told to do genuinely can't be done. Every loop leaves behind a recording, logs, and screenshots.

The payoff of that loop is resilience : the suite bends with the UI instead of shattering against it.

Real run · TC-017The page the agent lands on and drives. TC-017 changes the selected model (top-right of the Usage panel) and verifies the code snippet updates to match — described in the .pt as one plain-English step.

From test plan to test suite, automatically Writing dozens of end-to-end tests by hand is exactly the toil we were trying to escape. So we mostly don't. From a PDF test plan. generate_tests_from_pdf.py ingests a QA test plan and emits ready-to-run .pt files, one per scenario. The test plan becomes the test suite. From the live UI. generate_tests_from_baseline.py c aptures a DOM baseline of each page and generates sanity tests from what's actually there — so coverage tracks the product as it evolves, and DOM diffs flag when a page changes out from under us.

The suite describes itself. Because every CUJ carries a plain-English summary, the same corpus that runs the tests also feeds the coverage agent that decides whether a given PR is tested.

Today the suite covers dozens of critical user journeys — the CUJ catalog — across the whole console: get-started, playground, API keys, analytics, logs, members, billing, org settings, and the chat app. Each is a row in an auto-generated catalog describing exactly what it checks, tagged with markers ( Smoke, Billing, release-gate …) so a 90-second smoke pass or the full regression sweep is a single command. Knowing what to test on every PR A test suite is only as good as your confidence that it covers the change in front of you. So we pointed the same intelligence at our own pull requests. A PR coverage agent cross-references a diff against the CUJ catalog and asks the question a release manager asks: does an existing test already exercise this change — and if not, what would a new test need to add? Because every CUJ carries a plain-English description of what it verifies, the agent can reason about coverage semantically instead of grepping for file paths. Gaps become visible before a release, not after an incident. This closes a loop that most test suites leave open. It's not enough to run the tests you have; you need to know when the tests you have aren't enough. The part we're most proud of: tests that heal themselves Here's the reality of any large E2E suite: overnight, some tests go red. The expensive part isn't running the suite — it's the morning triage. Real bug? Flaky network blip? Did the UI just move a button? Or did the infrastructure fall over? So we built the triage into the system. Every night, a batch runs and a router skill — heal-batch — reads the...