Do automated tests actually make AI-built software better?
AI writes code several times faster than a person types it. It also writes buggier code. Those two facts are not in tension - they are the whole reason to care about tests. The question was never "should I write tests." It is: do the tests earn the time they cost? So instead of arguing from taste, let me look at what the research actually found, and then at what it means when the author of the code is a fast, confident machine.
AI ships faster, and it ships more bugs
The productivity side is real - teams using AI assistants generate working code far faster than before. The quality side is also real, and less advertised.
Veracode's 2025 analysis found that roughly 45% of AI-generated code contained a security flaw. CodeRabbit's "State of AI vs Human Code Generation" report put AI-authored changes at about 1.7x more issues than human-written ones, with critical and major defects skewing higher. And it gets worse when you let the model iterate on its own output: a 2025 study of iterative generation found critical vulnerabilities rising the more times the AI revised the code.
None of this means "don't use AI." I use it every day. It means the speed comes with a tax, and the tax is paid in defects you did not see because the code looked right. Which is exactly the kind of defect a test catches and a glance does not.
Yes, tests reduce defects - the numbers are strong
The best evidence here is old and boring, which is what makes it trustworthy. In a four-team industrial study across Microsoft and IBM, teams that adopted test-driven development shipped with 40% to 90% lower pre-release defect density than comparable teams working on the same products without it. The cost was honest too: roughly 15% to 35% more time up front.
That is the trade in one sentence: pay a bit more now, get dramatically fewer bugs later. For anything you intend to keep running, that is one of the best deals in software - and it predates AI by twenty years. AI just widened the gap, because it raised the bug rate on one side of the ledger.
But coverage is not the number to chase
Here is the nuance the honest studies insist on. A high coverage percentage is not the same as good tests. A case study on the Apache projects found that code coverage, on its own, only marginally predicted post-release defects. You can hit 90% coverage with tests that assert almost nothing - they run every line and check none of the behaviour. That is theatre, and it rots like any other unmaintained code.
So do not optimise the percentage. Optimise the question "would this test fail if the feature broke?" A smaller suite of tests that assert real behaviour - the actual edge cases, the money math, the thing that went wrong last time - beats a big suite that exists to make a dashboard green.
What this means for AI-built software specifically
Two kinds of test pull their weight against AI's particular failure modes:
- Fast unit tests on the logic. The AI's confident-but-wrong assumptions live here - the off-by-one, the wrong default, the edge case it did not consider. A unit test that asserts the real behaviour surfaces them in seconds.
- Browser / end-to-end tests (Puppeteer, Playwright). AI's signature failure is code that compiles and looks right and is subtly broken in the actual flow. The only cure is to drive the real thing - click the button, submit the form, read the result on the page - and assert what a user would see.
The two together let you close the loop: the AI writes the code and the tests, then runs them, so "done" means demonstrated, not claimed. That is what makes it safe to let a fast, confident junior work without you re-reading every single line - the tests are the babysitter you do not have to be.
The honest cost
Tests are not free. There is the upfront time, the maintenance, and the discipline to write ones that assert something real instead of something convenient. For a throwaway spike where correctness does not matter, skip them - they are wasted effort. For anything that touches money, data, or a customer, or anything you plan to change again, they are the cheapest insurance you will buy.
And AI shifts this trade further toward "yes," because writing the tests is now fast too. The old excuse - "I do not have time to write tests" - is mostly gone. The agent will write them in the same session it writes the feature, if you make it.
So: do automated tests make AI-built software better? The evidence says yes, clearly, if the tests assert real behaviour rather than chase a coverage number. Given that AI ships faster and buggier, that is the entire game - keep the speed, catch the mess. It is why every tool I hand a client comes with a test suite: so the thing still works the week after I give them the keys, not just the day I did.