Human Expert Review Requirements in AI-Augmented Security Assessments

"AI-powered pentesting" gets used as a label for almost anything these days. A scanner with an LLM-generated summary on top wears the label. So does a multi-agent platform that chains exploits across a live network. These are different products, built for different purposes, with different standing in front of an auditor. Lumping them together is the first mistake a technical buyer makes.

Which decisions get made by AI and which get made by a human, and whether that division matches what the engagement actually needs, is what matters. The useful diagnostic sounds almost identical to the bad one, but it isn't: does the AI find the vulnerability, or does the AI help a human find it? Those two answers describe different capabilities and different audit outcomes. A buyer who skips this distinction ends up with a mismatch: paying for a scoped human-led assessment and getting an automated scan, or paying for a one-time scan when the environment changes weekly and needs continuous coverage. Getting the taxonomy right up front is what the rest of this piece is built on.

What AI genuinely does well in a pen test

Automation has earned its place in modern testing. It raises the floor on discovery and lets human testers cover more ground in less time. Naming what AI does well is the honest starting point, because the case for human review only holds up if it stands next to a fair account of what AI contributes.

Reconnaissance is the clearest win. Mapping an attack surface across a large environment means correlating DNS records, certificate transparency logs, cloud metadata, and public code repositories, often across thousands of assets. No human team can match the speed or scale AI brings to that correlation work. Pattern-based vulnerability detection is the second clear win. SQL injection variants, cross-site scripting patterns, missing security headers, known CVEs: these are vulnerability classes with well-understood signatures, and AI tooling now finds them faster, more consistently, and with fewer false positives than the generation of scanners before it.

The frontier of this work is visible in PenForge, presented at ICSE-NIER 2026. Its Meta-Planner runs reconnaissance, ranks candidate attack types, and builds specialized exploit agents on the fly. On CVE-Bench, in the zero-day setting, PenForge reached a 30.0% exploit success rate (12 of 40), a threefold improvement over the prior state of the art. That's a serious result, and research published in 2026 more broadly documents substantial gains in LLM-driven penetration-testing success rates across recent benchmarks.

The ceiling sits exactly where those benchmarks stop measuring. CVE-Bench and similar frameworks test known vulnerability classes in defined conditions. They don't test business logic. They don't test the kind of multi-tenant attack chains that unfold across a real production environment with its own history, its own shortcuts, and its own unwritten assumptions. A benchmark score is a real achievement inside the boundary it measures, and that boundary excludes most of what makes a breach expensive.

The specific judgment calls AI cannot make

AI is weakest at business logic flaws and multi-step chainable exploits, but those happen to be the classes most likely to produce an actual breach. The mismatch is between where automated coverage is strong and where business risk concentrates, and it means the gap in AI coverage lands unevenly, concentrated precisely where the stakes are highest.

Take a payment flow where an attacker can generate fraudulent refunds just by skipping or reordering steps in the transaction. There's no signature to detect here. Finding it requires understanding what the application is supposed to do, then reasoning about what happens when a user does something the designers didn't anticipate. Take a multi-step authorization chain, where a standard user reaches admin access through three misconfigurations that look unrelated on their own. Spotting the escalation path means holding an integrated attack narrative across the whole system, not scoring each finding on its own. Or take a cloud IAM policy that technically validates but hands a compromised Lambda function access to every S3 bucket in the account. Catching that requires understanding the business intent behind the policy, not just confirming its syntax is correct.

PenForge's own authors point to the structural reason this happens. AI agents perform poorly in zero-day scenarios because contextual information about the target, especially the attack entry point and the attack type, is often unavailable and has to be inferred. The paper's own analysis names richer tool-usage knowledge and extended benchmarks as open problems that still need to be solved, not shortcomings already fixed.

There's an accountability dimension layered on top of the capability gap. A TU Wien ethics analysis of agentic pen-testing tools finds that LLM-driven agents show indeterminacy in what they do, what impact it has, and who it affects. You can't explain their outputs before or after the fact, so incident attribution gets harder, and safety review before deployment becomes a structurally hard problem, not a solvable one. An attack narrative, the chain from initial access through privilege escalation to a specific business impact, is a judgment call about which findings combine, in which sequence, to reach an objective that actually matters to a specific business. That judgment doesn't fall out of finding more vulnerabilities faster.

Why compliance frameworks require a qualified human

An AI-only report, however technically sophisticated, runs into a wall at audit time. Auditors expect documented evidence that a qualified human evaluated the systems in scope, and the frameworks that govern this expectation say so, either directly or through what they functionally require.

SOC 2's Common Criteria 4.1 names penetration testing as one type of separate evaluation in a non-binding point of focus, and CC7.1 centers on vulnerability detection. Neither criterion spells out a mandatory methodology, testing frequency, or tester credential requirement in so many words. In practice, auditors look for severity-rated findings with CVSS scores and for tracked remediation with evidence of closure, and both of those expectations imply a human who is accountable for the methodology and who signs off on the result. The written standard and the lived audit experience aren't identical, and a buyer who only reads the control language without accounting for what auditors actually ask for will misjudge what's required.

HIPAA's Security Rule, at §164.308(a)(8), already requires a periodic technical and nontechnical evaluation, repeated in response to environmental or operational changes that affect the security of electronic protected health information. HHS proposed in January 2025 to make annual penetration testing an explicit requirement for covered entities and business associates. Those provisions remain a proposal, not current law, and organizations should prepare for the possibility without treating the mandate as already in force.

There's an efficiency case that follows from all this. One well-structured pen test can satisfy SOC 2 CC4.1, relevant ISO 27001 Annex A controls, and other framework requirements at the same time, but only if the report carries evidence of qualified human review and attestation behind it. Without that evidence, the same technical findings satisfy nothing. A report that needs a named, credentialed security researcher to sign it as third-party attestation cannot be produced by automated output alone, because the signature is the accountability, and accountability needs a person behind it.

The "agent-augmented human testing" model in practice

The working model splits labor along the line the earlier sections already drew. AI handles the scanning, the first-pass reconnaissance, and the known-pattern detection, and it drafts findings into readable reports. Humans own the judgment calls: constructing chained exploits, assessing business logic, building the attack narrative, and signing off on the final result.

In the pre-engagement phase, a human defines scope around business objectives rather than a list of IP ranges, working out what a real attacker would actually want and where they'd start looking. During exploitation, a human takes the findings AI surfaced and reasons about which combinations produce a real path to a meaningful objective, then builds custom exploits for anything existing tooling can't prove on its own. Post-exploitation work, testing privilege escalation, lateral movement, and persistence, the actual blast radius of a compromise, calls for adaptive decisions that no current automated framework performs reliably.

Reporting is where the distinction becomes visible to the client. A human-written report tells an attack narrative that starts at a specific entry point, moves through specific findings in a specific order, and ends at a specific objective that matters to that business. That's a different document from a list of CVSS scores. Attestation closes the loop: a named, credentialed expert reviews every finding before it reaches the final report, which is what makes the document hold up in front of auditors and enterprise customers.

Access level shapes how much of this a human can actually do. Whitebox access, to source code, cloud configurations, and product documentation, lets a human expert build the full attack surface map that AI reconnaissance alone can't produce. Greybox or blackbox scoping leaves the human working with the same visibility gaps the AI has, which limits how much judgment can be exercised regardless of skill. Continuous scanning of code changes as they ship closes a different gap: new code, shifting dependencies, and changing attack paths move faster than an annual assessment can track, so testing tied to actual changes catches what a fixed calendar misses.

Distinguishing genuine hybrid testing from AI-washed scanning

The signs of a scanner dressed up as a pen test are recognizable before a contract gets signed, and checking for them at the RFP stage is the main defense against paying for security theater. If you ask the right questions during evaluation, you find out what you're actually buying well before the engagement starts.

The percentage of engagement time spent on manual testing versus automated scanning shows where the effort actually goes. Which specific testers will work the engagement, and what their certifications are, is the question that separates real accountability from marketing: a vendor offering only company-level credentials without naming individuals is a flag. Asking for a sample report is one of the most useful tests available, because a report that reads like scanner output, a cover page, pages of TLS observations, missing security header findings, and the same remediation paragraph repeated across multiple findings, gives itself away immediately. Asking a vendor to describe a business logic vulnerability their team found recently works as a direct test of whether that kind of testing happens at all; a vendor who can't answer concretely isn't doing it. Asking directly whether the vendor uses AI or automated agents, and which findings come from them versus a human, separates vendors with something to hide from vendors with nothing to hide: a strong answer discloses exactly where AI is used in the process and confirms that a human validates every high-severity finding before it reaches the client. Asking whether retesting is included, for how long, and how quickly critical findings get communicated during testing rather than buried until the final report, rounds out the picture of whether the engagement is built around ongoing risk reduction or a one-time deliverable.

Certain answers should end the conversation. A vendor who can't specify a manual testing percentage, who offers only company-level certifications with no named individuals, who promises comprehensive results in under a week for a complex scope, who holds critical findings until the final report instead of flagging them as they're found, or who never mentions researcher sign-off or third-party attestation, is describing a scan, not an assessment. A price that sits well below market for the stated scope is often the clearest tell of all: a web application pen test quoted at a rate that barely covers tool licensing is very likely a vulnerability scan wearing a pen test's name. Scope matters as much as price. A provider who narrows the engagement to a subset of systems without justifying why that subset reflects the real attack surface can leave a client exposed in exactly the place they didn't look: organizations with well-secured applications have been compromised through their SSO systems because the provider tested the wrong thing.

Why the accountability gap in fully autonomous testing persists

AI capability in offensive security is improving fast, and the natural assumption is that human review is a stopgap that better models will eventually make unnecessary. That assumption misreads what the requirement actually is. Accountability and judgment are structural properties of what makes a finding credible to an auditor, an enterprise customer, or a board, not capability problems waiting on faster inference or bigger training runs.

The TU Wien ethics analysis names three dimensions of indeterminacy in agentic pen-testing tools, and none of them go away just because the underlying model gets better. Actions are drawn from a non-deterministic policy whose outputs resist explanation. Impact is open-ended because the LLM supply chain behind the agent is opaque. The population of users affected is itself indeterminate. Together these spread moral attribution across users, tool-makers, and third parties in a way that doesn't collapse back down to a single accountable party just because the agent's success rate climbs.

Even an AI agent that reliably found every vulnerability a human expert finds would not answer who is responsible for that finding, who signs the attestation letter, or who stands behind the methodology when an auditor asks. A success rate on a benchmark doesn't produce a signature.

The researchers building the most capable automated tools say so themselves. PenForge's own authors, publishing at ICSE-NIER 2026 on a system that beat the prior state of the art by a factor of three, call for incorporating explainable mechanisms and human review as a necessary condition for developer trust.