A good penetration testing company tests by hand, not just with a scanner. It staffs the work with named, certified testers, follows a recognised methodology, and hands you a report with proof, CVSS ratings, clear fixes, and a retest. Everything below is a checklist you can grade any provider against, including us.
You are not buying a commodity. You are authorising a stranger to attack the systems your business runs on, usually because a regulator, an auditor, or an enterprise customer's security review put a deadline on your desk. A good penetration test, or pentest, simulates real-world attacks the way a hacker would, to show whether your sensitive data is actually reachable. Pick the wrong provider and you get a scanner report with a new logo on it, a failed audit, or a false sense of safety. Pick the right one and you get evidence you can show a customer and a roadmap of fixes that genuinely improve your security posture.
This guide gives you the eight criteria that separate a real cybersecurity test from a checkbox, a scorecard to evaluate each provider on your shortlist, and the questions to ask before you choose the right penetration testing partner.
First, get clear on what the test is for
Before you compare vendors, be honest about the goal. A test run purely to satisfy a compliance line item is scoped differently from one meant to find how an attacker would actually reach your customer data. Both are valid. The pentesting market ranges from boutique pentesting companies to large consultancies, so knowing which outcome you need keeps you from overpaying for the wrong thing or under-scoping the thing that matters.
Most buyers start with six questions: How manual is the testing? Who actually does the work? How good is the report? Does the provider understand our environment? Will we talk to the tester directly? And does this create security value beyond passing the audit? The criteria below turn those questions into something you can score.
1. Who actually does the testing
Company size tells you little. The person doing the work tells you everything. Ask, in writing, who will run your test and what they hold. Respected credentials include OSCP, CREST individual certifications (CRT or CCT), eCPPT, CRTP, and CEH. They prove a baseline of skill.
The stronger signal sits one level deeper. Testers who have found and responsibly disclosed real vulnerabilities, contributed to open tools, or published research will find things a certificate-only tester misses. Ask for that background. A provider who is proud of their team will share it; one who hides the testers behind a sales contact is telling you something.
For our part, Onyx penetration testing services are delivered by a testing team who hold eCPPT, CRTP, and CEH, with hands-on offensive security experience. Where an engagement calls for CREST-accredited testing, we deliver it through our CREST-member partner. We name who is doing the work before you sign.
2. Manual testing, not just an automated scan
An automated scan, run with automated tools, is a useful first pass, but it is not a penetration test. Vulnerability scanning flags known signatures. It does not chain a low-risk misconfiguration into a full account takeover, reason about your business logic, or tell the difference between a finding that matters and noise. Skilled security testing by a person does that.
Ask the provider what is automated and what is done by hand. If the answer is vague, or if the sample report reads like exported scanner output, you are buying a scan at pen-test prices. The whole point of a test is the manual exploitation a tool cannot perform.
3. A recognised methodology and a written scope
A credible provider follows a published testing methodology and can name it: OWASP for web application penetration testing and API work, PTES, or NIST SP 800-115. The right approach also depends on the types of penetration testing you need, from network penetration to web app and cloud. Methodology is what makes a test repeatable and defensible to an auditor.
Just as important is a written scope agreed before anyone touches a system. It should state the targets, what is in and out of scope, the testing windows, the rules of engagement, and how a critical finding gets escalated mid-test. It should also state the approach, whether black box, grey box, or white box, matched to the assurance you actually need. No written scope, no test.
Not sure where your security stands?
Run the free 15-question self-assessment and get an instant readiness score.
4. A report you can act on
The penetration test report is the deliverable. Everything else is the work behind it. A strong report has two layers: an executive summary a non-technical stakeholder can read, and technical detail an engineer can act on the same day.
Each finding should carry a CVSS-aligned risk rating, reproduction evidence, and the attack-path context that shows what an attacker could actually reach. Vague severities and no proof make a report useless for prioritisation and unconvincing to a customer. Always ask to see a sanitised sample report before you sign. A provider who will not show you one is the clearest signal on this list.
5. Remediation guidance and a retest
Finding the problems is half the job. Telling you how to fix them, in order, is the other half. Look for specific, developer-ready remediation, not a generic "apply patches" line.
Then ask the question most buyers forget: is a retest included after we fix the findings, and within what window? A retest verifies the issues are actually closed and produces an updated report your auditor or customer can rely on. Some firms include it; others charge again. Onyx includes a remediation re-test within an agreed window and issues an updated report confirming what is resolved.
6. The right compliance fit
If compliance requirements are driving the test, the provider needs to know that framework cold. SOC 2, PCI DSS, and HIPAA each scope a penetration test and the security controls behind it differently, and a test formatted for one will not cleanly satisfy another. PCI DSS, for example, requires penetration testing at least annually and after any significant change to the cardholder data environment, and has specific requirements covering segmentation testing.
A provider who has done the work for your framework will scope the test so it produces evidence your auditor or underwriter expects, the first time. One who has not will hand you a technically fine report that fails to map to the control you needed to satisfy, and you pay for a second round.
Need a penetration test scoped properly?
We map your environment to the right test on a 30-minute scope call. No obligation.
7. Direct communication and critical-finding alerts
You should be able to talk to the person testing your systems, not only a project manager relaying messages. Direct access shortens the loop when a finding needs context and makes the debrief far more useful.
Ask about the critical-finding policy too. If a tester finds something that puts you at immediate risk halfway through the engagement, you want to hear about it that day, not in a PDF three weeks later. A clear notification policy, and a real debrief at the end rather than a report thrown over the wall, are marks of a provider that treats your risk as theirs.
8. Independence, insurance, and accreditation honesty
Three quieter signals matter more than they look. First, independence: a tester who is also selling you the products under test has a conflict. Second, professional liability insurance, which a serious firm carries and will confirm.
Third, and most telling, accreditation honesty. Some assurances are issued only by accredited parties, and a trustworthy provider is plain about who signs what. A PCI Report on Compliance is signed by a QSA. A SOC 2 opinion is issued by a licensed CPA firm. CREST-accredited testing is delivered by a CREST-member firm. A provider who blurs these lines, or implies they personally issue a certificate they do not, is a provider who will blur other lines too. We would rather tell you plainly: Onyx does the testing, the readiness, and the documentation, and where a formal accreditation is required we name the partner who delivers it.
Use this scorecard on any provider
Use this scorecard to assess the security capability of each penetration testing provider on your shortlist and evaluate them on the same eight checks. If most boxes go unchecked, keep looking, no matter how polished the sales deck is. Unchecked boxes are security gaps you will pay for later.
| Criterion | What good looks like | Your provider? |
|---|---|---|
| Who tests | Named testers, certs disclosed (OSCP/CRT/CCT/eCPPT), real research | Y / N |
| Manual depth | Manual exploitation, not just a scan | Y / N |
| Methodology + scope | Named method (OWASP/PTES/NIST), written rules of engagement | Y / N |
| Report | Exec summary, CVSS ratings, proof, sample shared on request | Y / N |
| Remediation + retest | Developer-ready fixes and a retest in an agreed window | Y / N |
| Compliance fit | Knows your framework (SOC 2 / PCI DSS / HIPAA) and scopes to it | Y / N |
| Communication | Direct tester access, critical-finding notification, a debrief | Y / N |
| Independence + honesty | Independent, insured, plain about who signs accreditations | Y / N |
Red flags to walk away from
A few signals should end the conversation early:
- No scoping call before a quote. A real test cannot be priced off an asset count alone.
- Scanner-only results, or a refusal to share a sample report.
- Won't name the testers or the certifications they hold.
- No professional liability insurance.
- Vague timelines, or promises of total coverage in an unrealistically short window.
- A pitch that talks only about asset count and price, never about methodology or value.
How Onyx measures up
We built Onyx to pass its own checklist, so here is the honest mapping against the eight criteria above.
Our tests are manual-led, run by named testers who hold eCPPT, CRTP, and CEH, with CREST-accredited testing delivered through our CREST-member partner. We work to OWASP, PTES, and NIST SP 800-115, agree the scope and rules of engagement in writing, and deliver a report with an executive summary, CVSS-aligned findings, and reproduction evidence. A remediation re-test is included within an agreed window. We scope to your framework, whether that is SOC 2, PCI DSS, or HIPAA, you talk directly to the tester, and we are an independent testing firm. Where the goal is broader than individual pentests, we also run red team assessments that probe your detection and response, not just your security measures. Where a formal sign-off is required, we name the accredited partner who provides it.
We do not publish fixed prices, because a number that is not scoped to your environment is meaningless. Every engagement is scoped on a short call: no pressure, no obligation, just a clear plan and a quote.
Talk to us about a penetration test
Every engagement is scoped on a short call. No pressure, no obligation, just a clear plan and a quote.
Prefer to see where you stand first? Run the free security self-assessment, or read more about our penetration testing service.
The checklist at a glance
Save it or share it with your team when you are comparing providers.
FAQ
How much does a penetration test cost?
It depends on scope: the number and type of targets, the depth of testing, and whether a retest is included. Cost varies widely by scope, test type, and provider seniority; a credible provider scopes your environment before quoting rather than publishing a fixed list price.
What certifications should penetration testers have?
Look for OSCP, CREST individual certifications (CRT or CCT), eCPPT, CRTP, or CEH as a baseline of skill. Certifications are a floor, not a ceiling. The stronger signal is real exploitation experience: disclosed vulnerabilities, published research, or contributed tools. Where formal CREST-accredited testing is required, confirm it is delivered by a CREST-member firm. Always ask which specific testers will run your engagement and what they hold.
How long does a penetration test take?
Most engagements run from a few days to a couple of weeks of active testing, plus reporting time, depending on scope and the number of targets. A provider should give you a clear timeline in the scoping document. Be cautious of anyone promising broad coverage in an unrealistically short window.
What is the difference between a penetration test and a vulnerability scan?
A vulnerability scan is automated and lists potential weaknesses. A penetration test is performed by a person who attempts to exploit those weaknesses, proves which are real, chains them where possible, and shows the business impact. A scan is a useful input; it is not a substitute for a test.
Do I need a penetration test for SOC 2 or PCI DSS?
Often, yes. PCI DSS requires penetration testing at least annually and after any significant change to the cardholder data environment. SOC 2 does not mandate a pen test by name or frequency, but auditors and enterprise customers frequently expect one as evidence. The test should be scoped to produce the evidence your specific framework and auditor expect.
Should I worry if a provider won't share a sample report?
Yes. A sanitised sample report is a reasonable request and a standard one. A provider who refuses is either hiding thin, scanner-style output or has no standard report worth showing. The report is what you are paying for, so see the quality before you commit.