I've been waiting for someone to actually study this properly, and now they have.
CodeScene has published peer-reviewed research that quantifies something many engineers have suspected but struggled to articulate: AI coding assistants don't perform uniformly. They perform dramatically worse in the kind of codebases where you most want help.
The study analyzed AI-generated refactorings across 5,000 real programs using six different large language models. The researchers measured whether AI-generated changes preserved behavior while keeping tests passing. The headline finding: in code with lower health scores, defect risk increased by at least 30% compared to healthier codebases.
Here's what makes this finding significant: the 30% increase was observed in code that was still relatively maintainable - systems scoring 7.0 or above on CodeScene's health metric. The study explicitly notes that truly low-quality legacy modules (scores of 4, 3, or 1) weren't even included in the analysis. The researchers suggest, based on prior research, that breakage rates in deeply unhealthy legacy systems are likely non-linear and could increase steeply.
Translation for anyone who doesn't live in codebases: the code where you most desperately want AI help - the scary, undocumented, spaghetti legacy system that nobody wants to touch - is precisely where AI is most likely to make things worse.
As a former engineer who has spent quality time with some genuinely horrifying fintech codebases, this result is completely unsurprising and also somewhat devastating. The promise of AI coding tools has always been "let AI handle the grunt work so humans can focus on interesting problems." If the grunt work in your messy codebase is exactly where AI fails most spectacularly, that promise collapses.
The research frames this with an elegant insight about what it means to write code in the AI era. The traditional maxim says code should be written for humans to read. But as the paper argues, if AI is increasingly modifying code, it may also need to be structured in ways machines can reliably interpret.
The practical implications are real:
