AI-assisted bug triage without the snake oil

Half the QA tooling pitches arriving in your inbox right now have AI on the box. Most of them, on inspection, are doing very little — a thin LLM wrapper around bug descriptions, a smell of "we wrote a system prompt and called it a feature."

We're cautious about adding AI to QA workflows for a simple reason: the cost of a hallucinated bug report is much higher than the cost of writing one yourself. A wrong reproduction step wastes 40 minutes of a developer's time. A wrong root cause sends a fix in the wrong direction. AI gets QA reporting wrong with confidence, which is the worst failure mode.

That said — there are three places where the math works in your favor.

1. Duplicate detection

This is the one that pulls the most weight per dollar of compute. When a tester files a new bug, embedding-based similarity against the last N open bugs is boring and correct — it doesn't generate new content, it just surfaces existing ones.

We've seen teams with 2,000+ open bugs cut duplicate noise by 60–70% with a similarity score plus a "is this the same as bug #X?" prompt at file-time.

2. Repro-step generation from session recordings

If you have a recording of a tester clicking through and the bug being filed, a model can convert that timeline into clean reproduction steps. This is the right shape of LLM use because:

• The input is structured (DOM events, network calls).
• The output is reviewed by a human before it ships to the developer.
• The model is summarizing, not inventing.

Steps to reproduce (AI draft, please review):
1. Navigate to /checkout
2. Add 2 items to cart
3. Apply coupon "SUMMER20"
4. Click "Place order" — observe 500 error in Network tab

The first time you see steps you actually want to ship written from scratch by a model that watched the bug happen — that's when AI in QA stops being a gimmick.

3. Regression risk prediction

Given a list of files changed in a PR and the team's historical bug data, a model can suggest which areas of the app are at elevated regression risk. This is suggestion, not gating. It seeds your manual exploratory testing — "we changed the cart code; here are the four places we've historically broken the cart."

What we don't do

We deliberately don't:

• Auto-close bugs that the AI thinks are duplicates. The model nominates; a human closes.
• Auto-write tests from natural language. We let the model suggest which test you need; we don't let it write the assertions.
• Generate fake test data that looks plausible but doesn't represent your real users' behavior.

The right rule of thumb: AI can help you decide what to do next. It shouldn't decide what shipped.

Where this is going

Honestly? The win in AI-assisted QA over the next 18 months won't come from giant models reasoning about bugs. It'll come from small, boring, focused models doing one task very well — embedding, summarization, classification — embedded into the places where humans are doing rote work today.

Less magic. More leverage. That's the bet.