Malicious Code Generation: Why You Shouldn't Auto-Execute AI-Generated Test Scripts

Picture this: your CI/CD pipeline is set up to pull AI generated test scripts and run them automatically. Nobody reads them first. They just run. This felt like a productivity win six months ago. Today it is a liability you haven't fully accounted for.

AI models generate code at impressive speed. But speed and safety are not the same thing. When test automation teams skip the review step between generation and execution, they open a door that attackers are already aware of.

The Code Quality Problem Isn't Going Away

The security concern with AI generated code starts long before anyone introduces malicious intent. The baseline quality is the first problem.

A systematic study by researchers at NYU Tandon School of Engineering tested GitHub Copilot across 89 distinct code generation scenarios, producing 1,689 programs in total. About 40 percent of those programs contained vulnerabilities, including bugs and design flaws that could be exploited by an attacker. The research, published as "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions", found these weaknesses across a range of Common Weakness Enumeration categories aligned with high risk security issues.

Copilot improved since that original study. But the underlying problem hasn't been solved by any model. Research from Veracode in 2025 testing over 100 large language models across 80 coding tasks found security vulnerabilities in AI generated code across SQL injection, XSS (cross site scripting), log injection, and insecure cryptographic algorithms. One finding from that dataset: Copilot introduced security vulnerabilities in 32.8 percent of Python code and 24.5 percent of JavaScript code.

For test automation specifically, the consequences of a vulnerable script are not confined to a failed assertion. Test scripts often run against staging environments with real credentials, internal APIs, and database connections. A script that contains an SQL injection vector is not just a bug. It is a live attack path pointed at your test infrastructure.

The Slopsquatting Problem You Probably Haven't Heard Of

Beyond the vulnerabilities baked into generated code, there is a second threat specific to how AI models construct dependencies.

Large language models do not look up packages in real time. They predict tokens that look statistically correct based on training data. When the right package name is uncertain, the model invents one that sounds plausible.

Researchers at USENIX Security 2025 analyzed 576,000 code samples generated across 16 LLMs in Python and JavaScript. They found that roughly 20 percent of recommended packages were hallucinated, meaning names that did not exist on npm or PyPI. Among open source models, the hallucination rate for package names reached 21.7 percent.

This matters because attackers have noticed. The attack, now called slopsquatting, works by pre registering those hallucinated package names on public registries with malicious payloads inside. When a developer or an automated agent runs the AI generated install command, the malicious package is what gets installed.

The proof of concept is concrete. Security researcher Bar Lanyado registered huggingface-cli on PyPI in late 2023 as a deliberately benign package after observing that multiple AI models repeatedly recommended that name instead of the real huggingface-hub. Within three months, the empty package received more than 30,000 downloads. One team pulled it into a project README as a real dependency. That was intentionally harmless. An attacker doing the same thing with a payload inside would have had 30,000 install attempts to deliver it.

For test automation pipelines that auto execute generated scripts, slopsquatting is a direct threat. The script runs, installs its dependencies, and the malicious package executes before any test logic even starts.

What OWASP Says About Executing AI Output

The OWASP LLM05:2025 Improper Output Handling guidance directly addresses this risk. It covers what happens when LLM generated content is passed to downstream systems such as databases, web browsers, operating system shells, or code interpreters without proper validation, sanitization, or escaping.

One of the examples OWASP LLM05:2025 lists is explicit: an LLM is used to generate code from natural language inputs to streamline development tasks. The guidance notes that this approach risks exposing sensitive information, creating insecure data handling methods, or introducing vulnerabilities like SQL injection. It also warns that the AI may hallucinate nonexistent software packages, potentially leading developers to download malware infected resources.

The mitigation OWASP recommends is just as direct: never execute LLM generated code through eval() or similar dynamic execution mechanisms in production environments. That guidance applies equally to automated test pipelines. If your pipeline treats AI output as executable without review, you are running exactly the pattern OWASP is warning against.

A Safe Workflow for AI Assisted Test Automation

The solution is not to stop using AI for test generation. The tooling is genuinely useful for generating test outlines, edge case ideas, and boilerplate Playwright or Selenium code. The solution is to add one non negotiable step before any AI generated script touches an environment.

Human review before execution. Every script generated by an AI model should be read by a human before it runs. Not skimmed. Read. This means checking what the script actually does, what packages it imports, what commands it calls, and whether any of that logic was requested in the prompt.

Verify every dependency. Before any install command runs, check that the package actually exists on the registry and that it matches what you expect. A package that exists but has a suspicious publish date, no history, and no contributors is worth investigating before installing.

Run in isolation first. New AI generated scripts should execute in a sandboxed environment before they run against any infrastructure with credentials or real data. A local container with no outbound network access and no mounted secrets is the right place for a first run.

Use static analysis on generated code. Tools that scan for common weakness patterns can catch SQL injection risks, insecure function calls, and path traversal problems in generated code before it runs. Many of these tools are fast enough to sit in a pre commit or pre execution hook.

Pin your dependencies. Lockfiles prevent a package from being swapped out after you have reviewed and approved a script. If your test pipeline resolves dependencies at runtime with no lockfile, an attacker who registers a hallucinated package name after your review step can still slip in.

The Automation Boundary

Auto execution feels like the natural end state of AI assisted testing. Generate the test, run the test, get the result. The appeal is obvious. But the step in the middle, where a human reads what the model produced, is the step that keeps your test infrastructure from becoming an attack surface.

AI models generate code by predicting tokens. They are good at producing code that looks correct. They are not good at guaranteeing it is safe. That distinction matters most in automated pipelines, where the speed advantage only compounds risk if review is removed from the equation.

The teams getting the most value from AI test generation tend to be the ones who use the AI for speed and apply human judgment for safety. Both matter. Removing either one creates problems the other cannot compensate for.

Want to go deeper on building AI assisted test workflows that are safe for production? The test automation and SDET courses at RCV Academy cover secure automation practices as part of learning how to build testing pipelines that hold up under real conditions.