To verify AI-generated code before production, run every AI-written change through three enforced gates: automated tests that prove it works, security and dependency checks that prove it is safe, and a senior engineer's review that proves it is right. Then store the evidence from each gate with the release, so you can show exactly what shipped and why.
Why does AI-generated code need a different verification process?#
Because AI assistants now produce code that compiles far more reliably than code that is secure, so "it runs" no longer tells you much.
Three numbers set the scale of the problem:
- Correct syntax, weak security. Veracode's Spring 2026 GenAI Code Security Update found syntax pass rates above 95%, while only 55% of generation tasks produced secure code. The tests covered 80 tasks in Java, JavaScript, C# and Python, checking for SQL injection, cross-site scripting, log injection and insecure cryptography.
- High use, low trust. In the Stack Overflow 2025 Developer Survey, 84% of developers said they use or plan to use AI tools, yet 46% said they don't trust the accuracy of the output, up from 31% a year earlier.
- Less stable delivery. The 2024 DORA research estimated that a 25% increase in AI adoption was associated with a 1.5% drop in delivery throughput and a 7.2% drop in delivery stability (RedMonk's summary).
What changes in practice: the author may not understand every line, change volume goes up, and review becomes the bottleneck. Verification has to be automated, enforced on every change and impossible to skip, not left to spot checks.
What can go wrong when AI-written code skips review?#
Known vulnerability classes, made-up dependencies and subtly wrong logic reach production looking like ordinary code.
- Injection and XSS. Veracode's 2025 report found models failed to defend against cross-site scripting in 86% of relevant cases.
- Hallucinated packages. A USENIX Security 2025 study generated 576,000 code samples with 16 LLMs. On average at least 5.2% of the packages suggested by commercial models and 21.7% of those from open-source models did not exist, giving 205,474 unique fake package names. An attacker who registers one of those names gets their code installed by anyone who trusts the suggestion.
- Code and tests that share the same mistake. When one model writes both the implementation and its tests, both can encode the same misunderstanding. The suite goes green and the bug ships.
- Changes beyond the task. Agents may edit files nobody asked them to touch: configuration, permissions, error handling. Those edits are easy to miss in a large diff.
Which three gates should every AI-written change pass before release?#
Tests, security checks and senior engineer review. All three are enforced by the pipeline, whether a person or an AI wrote the change.
| Gate | What it proves | What runs | Blocks the merge when |
|---|---|---|---|
| Tested | It works | Unit, integration, contract and regression tests; coverage on changed lines | Any test fails, or changed logic has no test |
| Secured | It's safe | Static analysis, secrets scanning, dependency and licence checks, infrastructure and container scans | A high or critical finding has no approved exception, or a package isn't vetted |
| Reviewed | It's right | A senior engineer reviews intent, design, edge cases and risk | No approval from a code owner |
Make each gate a required status check under branch protection, so no person and no agent can merge around it. This is the standard our Quality Engineering service is built on: tests prove it works, security checks prove it's safe, and a senior engineer's review proves it's right. Each release also produces evidence.
How do you use AI to generate tests without trusting them blindly?#
Let AI draft the tests, then review them like any other code. A person confirms that each test checks the requirement, not just whatever the code currently does.
- Write acceptance criteria first. Put them in the ticket before anything is generated, so tests have a source of truth other than the implementation.
- Generate tests separately. Generate them from the criteria in a separate session or prompt from the one that wrote the code.
- Make every new test fail once. Run it against the old code or deliberately break the logic. A test that has never failed proves nothing.
- Read the assertions. Reject tests that only check that no exception was thrown, or that snapshot the current output.
- Measure test strength on critical paths. Mutation testing shows whether the suite catches injected bugs, which tells you more than coverage does.
- Quarantine flaky tests quickly. A gate that fails at random gets ignored, and then it stops being a gate.
Our test automation work aims for suites fast enough to run on every change and stable enough to trust. AI-generated tests are welcome there, but they get reviewed before they count.
How do you add security scanning and dependency checks to the CI/CD pipeline?#
Run fast scans on every pull request and deeper scans on merge and on a schedule. Fail the build on high-severity findings and on any dependency that hasn't been vetted.
- On every pull request: secrets scanning, static analysis on the diff, and rules that target the classes AI gets wrong most often (injection, XSS, log injection, weak cryptography).
- Dependency gate: scan for known vulnerabilities, and also check that every new package exists, has a real history and is on your approved list. This is your main defence against hallucinated packages. Pin versions, commit lockfiles and pull through an internal registry proxy.
- At build: generate a software bill of materials (SBOM), sign the artifacts and record provenance. Under SLSA, Level 1 requires automatically generated provenance showing who built the artifact, which build process was used and what the top-level input was. Level 2 adds signed provenance from a hosted build platform.
- Before deploy: scan infrastructure-as-code and container images, and run dynamic tests against staging for web-facing services.
- Exceptions: every accepted finding gets a named owner and an expiry date.
To check coverage against a recognised framework, map these controls to NIST's Secure Software Development Framework (SP 800-218). If you don't know where your gaps are, our Security & DevSecOps practice finds them, ranks them by risk and builds the checks into your pipeline.
What should a human reviewer actually check in an AI-written pull request?#
The things automation can't judge: whether the change does what was asked, fits the system and fails safely.
- Intent: Does the diff match the ticket, and nothing more? Send oversized or off-scope changes back to be split.
- Design fit: Does it reuse existing modules and follow your conventions, or duplicate logic that already exists?
- Trust boundaries: Is input validated? Is authorisation checked at every entry point? Is there nothing secret in code, and no personal data in logs?
- Failure paths: What happens on timeouts, retries, nulls, partial failures and concurrent requests?
- Dependencies: Is every new import justified and vetted?
- Tests: Do they assert the requirement, as described above?
- Explainability: Can the author explain every line? If not, it doesn't merge.
Label AI-assisted pull requests so reviewers know what they're looking at and you can compare defect rates over time.
What release evidence should you keep for auditors and incident reviews?#
For each release, keep a record that links the change to its requirement, its gate results, its reviewer and the exact artifact you deployed.
- Change record: ticket, pull request, commits, and an AI-assisted flag naming the tool or agent
- Test results: run IDs, pass/fail and coverage on changed lines
- Security results: scan reports, the SBOM, and accepted exceptions with owners and expiry dates
- Review: approver identity and timestamp
- Build provenance: artifact digest and signed provenance
- Deployment: who deployed, when and where, plus the rollback plan
The pipeline should capture all of this automatically. Nobody should be assembling it by hand the week before an audit. During an incident, the same record answers "what changed?" in minutes. For teams selling to the US government, the SSDF also underpins federal self-attestation, which expects evidence such as provenance and vetted components.
How do you start: piloting quality gates on one team or repo#
Pick one repository that sees real AI-assisted work, measure a baseline, run the gates in report-only mode, then switch them to blocking.
- Choose the repo. It should be active, use AI assistants or agents heavily, matter to the business, and have an owner who wants the change.
- Record a baseline. Track change failure rate, rework, review time, escaped defects and vulnerabilities found after merge.
- Turn the gates on in report-only mode. Tune the rules until findings are real and noise is low.
- Switch to blocking. Make the three gates required checks.
- Add evidence capture so every release produces its record.
- Compare and decide. If the numbers improve, roll the gates out repo by repo.
This matches how we start engagements: a paid pilot with one pod, one outcome and success criteria agreed up front, with results in days and no long-term contract.
Conclusion#
AI has made code cheap to write. It hasn't made code safe to ship. Treat every AI-written change the same way: tested, secured and reviewed by a senior engineer, with evidence kept for each release. Start with one repo, measure the result and scale what works.
Want to pilot quality gates on one of your repositories? Talk to an engineering lead →
Frequently asked questions
Is AI-generated code less secure than code written by people?
Research shows AI code often has known security flaws when the prompt gives no security guidance. Veracode's 2026 testing found that only 55% of generation tasks produced secure code. The fix is not to ban AI. Put every change through the same security gate, whoever or whatever wrote it.
Can AI review AI-generated pull requests instead of a human?
AI review tools are useful for catching routine problems before a person looks at the code. They can't confirm that a change matches business intent, fits your architecture or handles risk the way your organisation needs. Keep a senior engineer's approval as the final required check.
How do we stop AI assistants from adding packages that don't exist?
Add a dependency gate that checks every new package against an approved list or internal registry before the build can pass. Pin versions and commit lockfiles. Reviewers should also question any import they don't recognise, because attackers can register package names that models invent.
Won't three gates on every change slow our delivery down?
Scans that run on every pull request should be fast, with deeper scans on merge or on a schedule. Keeping changes small also keeps review quick. Start the gates in report-only mode so you can tune out noise before they block anything.
Do we need to label pull requests as AI-assisted?
It isn't required, but it helps. Reviewers know where to look harder, and you can compare defect and rework rates between AI-assisted and other changes. Saving the label with your release evidence also helps when you investigate an incident.
Why should AI-generated code be reviewed before production?
AI assistants write code that looks right but can contain bugs, security holes, made-up packages or wrong assumptions about your system. Tests catch some of this; a senior engineer's review catches the rest, such as logic that passes tests but breaks a business rule. Reviewing before production is far cheaper than fixing an incident after it.
Who can check my AI-developed application before release?
A senior engineer who did not write the code, or an outside engineering team that runs the three gates for you: automated tests, security and dependency checks, and a code review with written findings. Calidad Infotech's engineering pods do this for teams that build with AI tools: they review the code, fix what they find and hand over the evidence with the release.
- AI-generated code
- code review
- DevSecOps
- test automation
- software supply chain
- quality engineering
Newsletter
Get new articles by email
A short monthly round-up of what we've written on engineering pods, quality and AI. No spam.
