Every CI vendor now has an "AI-powered" testing story. Some of it is real. A lot of it is a demo that looks great on a clean sample repo and falls apart on a real, messy, five-year-old codebase. Here's the checklist we'd actually use to tell the difference — and we built GreenCI, our own AI-powered testing product, to hold up against every item on it.
Does it run tests before committing them, or just generate and hope?
Generating a test is trivial. Generating a test that actually passes against your real, running application is the hard part. If a tool writes tests without executing them first, you're getting a first draft you still have to debug — which defeats a lot of the point.
Does it tell "flaky" apart from "your app actually broke"?
This is the difference between a tool that's useful and one that quietly erodes trust in your test suite. A tool that "heals" every failing test by rewriting it until it passes will happily paper over a real regression. The right behavior is the opposite: fix the test when the test is wrong, and loudly flag it when the app is wrong.
Does it lock you into a proprietary format?
Check what actually gets committed to your repo. If it's plain, standard framework code (plain Playwright, for example) you keep every test even if you cancel the subscription tomorrow. If it's a proprietary DSL or format that only runs inside the vendor's platform, you've bought a dependency, not a test suite.
Where does your code actually go?
Ask directly: is your source code sent to a third-party model, and is it used for anything beyond generating your tests — including training future models? Look for tools that support bringing your own LLM provider, so your code goes straight from your CI runner to a provider you chose, not through the vendor's servers.
Can it start from zero, or does it need an existing suite?
A lot of teams evaluating these tools have little or no E2E coverage today — that's usually the actual problem they're trying to solve. A tool that only "enhances" an existing suite doesn't help a team starting from nothing. Ask specifically how it handles a repo with zero tests today.
Try it, don't just read about it
The fastest way to evaluate any of this is to run a tool against your own repository and watch what it actually does. GreenCI is free to try — sign in, point it at a repo, and check it against every item on this list yourself rather than taking our word for any of it.