Urgent penetration testing › Testing production safely
Every scoping conversation reaches the same question. Do we test the live system, or the copy? The answer is usually production, with named exceptions, and the reasoning is worth setting out because the instinct to protect production often produces a worse outcome.
Why staging misleads
A staging environment is only useful as a test target if it genuinely mirrors production. In practice it rarely does, and the differences are exactly the things that cause findings.
- Configuration drift. Debug modes left on, verbose error pages, different TLS settings, relaxed security headers. You find issues that do not exist in production and miss ones that do.
- Different data. Anonymised or synthetic data changes application behaviour. Logic that fails on real-world edge cases never triggers.
- Missing integrations. Payment providers, identity providers, third-party APIs are stubbed or absent, and those boundaries are where a lot of real findings live.
- Different infrastructure. A single staging box behind a different firewall tells you nothing about the production perimeter.
A test of a staging environment that differs materially from production produces a report about staging. That is worth saying out loud, because the report will not say it.
What is genuinely safe on production
The great majority of what a penetration test does. Reconnaissance, service enumeration, configuration review, TLS and header analysis, authentication and session testing, access control testing between roles, injection testing with non-destructive payloads, business logic testing, and reading things you should not be able to read.
None of that is more disruptive than a moderately busy day of legitimate traffic, and a competent tester paces it deliberately.
What is not, and should be handled differently
| Activity | Risk on production | How to handle it |
|---|---|---|
| Denial of service testing | By definition, an outage | Out of scope by default. If genuinely needed, a separate exercise in a window, agreed in writing. |
| Password brute forcing | Account lockouts affecting real users | Rate-limited and targeted at test accounts only, or the lockout policy reviewed rather than exercised. |
| Destructive payloads | Data loss | Never on production. Prove the vulnerability with a benign proof, not by deleting something. |
| Exploiting a confirmed remote code execution | Instability, and you are now running code on a live server | Confirm minimally, report immediately, agree next steps before going further. |
| Automated scanning at full throttle | Resource exhaustion on smaller estates | Throttled, and scheduled outside peak where the estate is delicate. |
| Testing a fragile legacy system | It falls over if you look at it | Named exclusion, or a copy specifically for this, with the limitation stated in the report. |
The controls that make production testing safe
Four things, all of which belong in the rules of engagement.
- A named contact reachable during the window, so anything unexpected stops within minutes rather than hours.
- Source IP disclosure, so your monitoring team can distinguish testing from a real attack, unless you are deliberately testing detection.
- Agreed stop conditions: instability, evidence of prior compromise, unexpected personal data.
- A known-good backup taken before the window opens. Not because testing should need it, but because the cost of having it is nil and the cost of not having it is not.
The hybrid that usually works
Test production for everything that is safe, which is most of it. Use a copy for the small number of activities that are not, and state clearly in the report which findings came from which environment. That gets you an accurate picture of the live system without taking the risks that genuinely warrant caution.
What does not work is testing staging because it feels safer and treating the result as though it described production. That is the choice that produces a clean report and an unchanged risk.
A note on cloud
Testing your own workloads on a major cloud platform is generally permitted without a separate request, within the provider's published policy. What is not permitted is testing the provider's own infrastructure, and the line between the two is not always where people assume. Check the policy at scoping rather than mid-test.
Tell your monitoring team, or do not, but decide
A penetration test generates traffic that looks exactly like an attack, because it is one. What your security monitoring does with that is a decision worth making deliberately rather than discovering.
Telling them is the default and the right choice most of the time. Share the tester's source addresses and the testing window so alerts can be correlated rather than escalated. This avoids an incident response process kicking off over your own test, which wastes a day and erodes trust in the alerting.
Not telling them turns the engagement into a test of detection as well as of vulnerabilities, which is genuinely valuable information. But do it knowingly: agree it with whoever owns the monitoring at a level above the analysts, make sure somebody senior knows the test is happening so a real incident can still be distinguished, and accept that you may burn a day of your own team's time.
What does not work is failing to decide. An analyst escalating at eleven on a Tuesday, a manager ringing the tester's number, and nobody able to confirm quickly whether the traffic is authorised is the worst of both options.
Practical steps before the window opens
- Take a backup, and verify it restores. Testing should not need it. The cost of having one is an hour and the cost of not having one, on the rare occasion it matters, is considerably more.
- Check your alerting thresholds. Scanning traffic can generate enough alerts to bury a genuine one. Knowing that in advance is better than finding out.
- Note the current state of anything fragile. If a system is already unstable, record that before testing, so a failure during the window can be attributed accurately rather than argued about.
- Confirm who can authorise a stop. One named person, reachable, with the authority to say halt without convening anyone.
None of this takes more than an hour and it converts most of the residual risk of production testing into something managed rather than hoped about.
Will a penetration test take our site down?
It should not, and in normal testing it does not. The activities that could are denial of service, destructive payloads and unthrottled scanning, and all three are either out of scope by default or carefully controlled. The residual risk is a fragile system that falls over under ordinary load, which is itself worth knowing about.
Should we test out of hours to be safe?
Usually unnecessary and it often costs more. Business hours testing with a contactable engineer is safer than overnight testing with nobody around, because if something does go wrong somebody is awake to stop it.
Can you test a system that is already unstable?
We can, but we will tell you that it will probably fall over and that the finding will be "this system is unstable", which you already know. It is usually better to exclude it and note the exclusion, or fix the instability first.
Do we need to tell our hosting provider?
If they own the infrastructure, yes, and you need their written authorisation. For a major cloud platform, your own workloads are normally covered by their published policy. For a smaller host, expect a ticket and a few days.
Not sure what is safe to test?
Send us the environment and we will tell you what we would test live, what we would exclude and why, before you commit to anything.