Penetration Test Report

What is a penetration test report?

A penetration test, or pen test, is an authorized attack on a system by security testers. NIST, the US standards agency, describes it as testing in which assessors mimic real-world attacks to find ways around the security features of an application, system or network. Most tests look for combinations of weaknesses that together give more access than any one of them would.

The report is what the test produces. NIST's testing guide says the report generally describes the weaknesses found, gives a risk rating and explains how to fix them. When you assess a vendor, you ask for this report, or a summary of it, as evidence that the vendor tests its own product.

A pen test is not a vulnerability scan. A scan uses automated tools to flag possible weaknesses, in seconds or minutes per system. A pen test is mostly manual: testers try to exploit what they find to prove it is real, and an engagement can last days or weeks. PCI DSS, the payment card security standard, requires companies in its scope to test at least annually and after significant changes (requirement 11.4 since version 4.0).

What a full report contains

The PCI Security Standards Council's pen testing guidance suggests an outline with these main parts. It stresses that the outline is only a suggestion and that formats differ between testers:

• Executive summary: the scope and the major findings in brief.
• Scope: exactly which networks, systems and applications were tested.
• Methodology: how the testers worked and which tools they used.
• Limitations: anything that restricted the test, such as set testing hours.
• Findings: each weakness with its severity, the systems affected and a description.
• Retest report: a follow-up with the original findings, the retest date and whether each one was fixed.

What it looks like in a vendor's terms

Pen testing promises appear in documents such as a vendor's security addendum or in the technical and organizational measures attached to its DPA. A typical clause reads like this:

Provider will engage an independent third party to perform penetration testing of the Services at least annually. Upon Customer's written request, Provider will provide a summary of the results. Provider will remediate identified vulnerabilities in accordance with their risk.

• "Independent third party": outside testers, not only the vendor's own team.
• "At least annually": the minimum rhythm. NIST notes that, because of its cost and impact, a yearly test may be sufficient.
• "A summary": you get the executive summary, not the full findings.
• "In accordance with their risk": no fixed deadline for fixes.

Real terms make similar promises. Atlassian's Security Measures, effective October 7, 2025, commit to "annual penetration testing on Cloud and Software Products."

Why it matters to you

A security questionnaire tells you what a vendor says it does. A pen test shows what skilled testers could actually do to it. NIST also notes that a test shows how well defenders detect an attack and respond.

For a critical vendor that holds your customers' data, a recent independent test is some of the strongest evidence you can collect. The promise to test can also change, because it sits in documents vendors edit. Venpo flagged, for example, that Splunk took background checks out of two security exhibits.

What to check

• Date: when testing ended, ideally within the last 12 months.
• Scope: the product, API and infrastructure you actually use. A test of the marketing website says little about the app.
• Tester: an outside firm, or testers organizationally separate from the people who run the systems. PCI's guidance requires that separation for its tests.
• Knowledge: whether the testers had no details (black-box), some (gray-box) or full details (white-box).
• Severity: how findings were ranked, for example with CVSS (Common Vulnerability Scoring System) scores, and how many were critical or high.
• Fixes: whether critical and high findings were fixed and retested, with a retest date.
• Limitations: excluded systems, testing windows or a short engagement. A clean report from a narrow test proves little.

How Venpo helps

Venpo does not see a vendor's private pen test report. It watches the public documents where the testing promise lives, such as DPAs, security addenda and other public legal pages, and emails you a plain-English alert when one changes. If a vendor drops "annual" or "independent" from its testing commitment, the alert says what changed and why it matters.

Each vendor gets a change history with a redline of each change, so you can show what the vendor promised when you signed.