Vibe Coding and QA: Why Testers Matter More When AI Writes the Code
Software Testing Mentor and RCV Academy 
Andrej Karpathy's tweet describing his own workflow got specific about the tradeoff. With AI tools good enough, he wrote, you can "forget that the code even exists." He was describing a weekend project. Within months, teams were shipping production features the same way, and the code did not stop needing to work correctly just because nobody was reading it closely anymore.
From weekend hack to production habit
Karpathy coined the term vibe coding in a post on X in February 2025, describing a style of programming where you describe what you want, accept what the AI produces, and move on without reviewing every line. It was a casual observation about a personal project, not a methodology for shipping software other people depend on.
The gap between those two things is the whole problem. A throwaway prototype can tolerate code nobody fully understands. A production application handling real user data cannot, and the line between the two gets blurry fast once a team is used to accepting AI output without friction.
What actually breaks when nobody reviews the diff
Security researchers have already measured what happens when review gets skipped. Veracode's 2025 GenAI Code Security Report tested output from more than 100 large language models across 80 coding tasks and found that AI generated code introduced security vulnerabilities in 45 percent of cases, including flaws from the OWASP Top 10 list of common weaknesses. Cross site scripting was one of the worst categories tested.
That number held up across newer and larger models, not just older ones, which suggests the issue is not something that gets automatically solved as the underlying models improve. Nobody explicitly asked the AI to write insecure code. It made the choice on its own, the way it makes every other choice, based on patterns in its training data rather than a security review.
The productivity story is not as clean as it sounds
Vibe coding is usually sold on speed. A randomized controlled trial from METR, a nonprofit AI evaluation organization, complicates that pitch. The study had sixteen experienced open source developers complete real tasks in codebases they knew well, randomly assigning each task to allow or disallow AI tool use. Developers predicted AI would cut their completion time by 24 percent. It actually increased their completion time by 19 percent, and even after finishing the tasks, developers still believed AI had made them faster.
METR is upfront that this was a specific snapshot: experienced developers, mature codebases, tools available in early 2025. It is not a verdict on every use of AI coding tools forever. But it is a real data point against the assumption that faster generated code automatically means faster delivered software, and it lines up with something testers already know from experience: the time saved writing something is not the same as the time it takes to verify it actually works.
Where testers fit into this workflow
None of this means AI generated code is unusable. It means it needs the same scrutiny any other untested contribution gets, arguably more, since a human author under time pressure at least knows what they were trying to do. A generated pull request often does not carry that context with it.
Practically, that means treating AI written code as an unverified submission rather than a finished feature. Run it through the same security and functional test coverage you would apply to any other change, since the Veracode numbers say you cannot assume secure defaults. Prioritize exploratory testing around the areas most likely to hide subtle bugs: authentication, input handling, and anywhere user data crosses a trust boundary. These are exactly the areas where pattern matching from training data tends to produce plausible looking code that fails on edge cases.
Testing AI generated output well also increasingly means understanding how the AI itself can fail, not just how traditional code fails. ISTQB's Certified Tester Testing with Generative AI (CT-GenAI) syllabus covers exactly that: how to evaluate and test around the specific risks generative tools introduce. RCV Academy's Generative AI and AI Agents for QA course goes further, covering how testers can work directly with Copilot, Claude Code, and MCP based agent workflows rather than just testing their output after the fact.
Frequently asked questions
Does vibe coding mean testers become obsolete?
The evidence points the other way. If AI generated code carries a higher security defect rate than human written code, someone verifying that code becomes more necessary, not less.
Is AI generated code safe to use in production?
It can be, but not by default. Treat it the same way you would treat a contribution from a new, unreviewed developer: valuable, but requiring the same test coverage and code review as anything else before it ships.
How should QA teams respond to AI generated pull requests?
Apply your existing test and review process rather than a lighter one. Given that models chose an insecure implementation nearly half the time in controlled testing, AI generated code deserves at least the scrutiny of human written code, not less.
What is the actual difference between vibe coding and normal AI assisted coding?
Vibe coding, in Karpathy's original description, means accepting AI output with minimal review and prioritizing speed over understanding the code. AI assisted coding can use the same tools while keeping a human reviewing, testing, and understanding what gets shipped. The tools can be identical. The discipline around them is what differs.
AI will keep writing more of the first draft of the code teams ship. Whether that draft is safe depends entirely on whether someone tests it like they mean it. RCV Academy's Generative AI and AI Agents for QA: Complete SDET Masterclass is built for testers who want to be that someone, covering the tools writing the code and the verification habits that code actually needs.
Categories: : AI SDET, AI Tools, Automation, qa, SDET
