The concept of a penetration test

A penetration test – also known as a pentest – is a security and risk assessment of an application(s) and/or a system(s) performed by an IT security professional using manual, and sometimes, automated tests. While automated tests are meant to find low hanging fruits and maximize the coverage of the assessment, manual assessments are still required as many use cases cannot be tested or accurately identified with automated tests.

A penetration test assesses two aspects of a system, the security and the risk.

Security in the context of Information Security means the practice of preventing unauthorized access, use, disclosure, disruption, modification, inspection, recording or destruction of information [1]. This definition is usually synthesized with the CIA concept, where CIA stand for ConfidentialityIntegrity and Availability. The penetration test will attempt to evaluate the overall security level of the system and identify all potential issues that might compromise any aspect of the security of that system.

The pentest will also evaluate the risk related to each identified issue. The risk can be defined as the potential that a given threat will exploit vulnerabilities of an asset or group of assets and thereby cause harm to the organization. It is measured in terms of a combination of the probability of occurrence of an event and its consequence [2]. For each issue listed, the associated risk will be evaluated based on the likelihood and the impact. We, at KeyIdentity, decided to use the standard CVSSv3 [3] as scoring system. This system makes it easy to calculate the risk based on empiric values. The standard being widely used; we have a better chance to easily integrate with the client’s internal risk framework.

Audit vs Penetration test

Penetration test and audit are not the same. The scope and the goal are different. A penetration test is what we described earlier while an audit is systematic test of a system against a list of standards and/or baseline, usually in order to obtain a certification (e.g. PCI DSS). Typically, an audit is rather a checklist test where multiple items are ascertained and where the result is usually binary: PASSED or FAILED. The list of items to test is predefined, therefor an audit cannot identify unexpected issues. The tests are not deep and the report is not as granular as a penetration test.[AA1]  Nonetheless, an audit will be able to identify fundamental security issue in your IT organisation, such as missing patch management, which cannot be identify in a penetration test. Audit and penetration test are two different methods used to identify two different kinds of issues.

Scan limitations

The automated tools are a blessing when it’s about incremental tests or deterministic procedural tests. These allow the assessor to execute repeated tasks quickly and thus saving some precious time. However, scans should be used with caution for the four following reasons.

Overflow of data

Scan can sometime be very noisy. When misconfigured, it can generate lots of informational data, which could be interesting for the assessor, but sometime, it just adds noise around the valuable data, which require the pentester more time for analysis. Beside information date, the automated tool can report false positive vulnerabilities, which means each findings should be verified by the assessor. In some case, we might end up with scans that output too many false positive, and the assessor is spending more time filtering the output rather than actually testing the system.

Feeling of a job done

We talked about the false positive, now let’s talk about the false negative. Scanner can misinterpret result from the system and thus missing a potential vulnerability. This due to the parser that often require complex logic in order to identify specific data with usually regular expression. Sometime, a simple custom error message may fool the scanner. Scanner may give this satisfying feeling of job done so the assessor doesn’t need to manually re-evaluate the scanned target. In such case the assessor is entirely relying on the parser that could fail and miss potentially important issues. This is why it is important to understand scanners and their capability and reliability. Like everything in security, nothing is 100% bullet proof.

Destructive Scan

Scanners can sometimes be destructive. For instance, the Christmas tree port scan method that caused in some cases a Denial-of-Service, or the web app scanner that inject multiple payloads into a form used to send SMS resulting in an expensive bill at the end oft he month. Pro-active scanner usually have an impact on the targeted system. We once again stress that point, it is important to know what scanner to use, how they work and what could be their impact.

Blind spots

A scanner cannot cover as much as a manual penetration test. Many use cases require human understanding of the application or specific (complex) execution sequence in order to trigger the vulnerability. This is why automated tools are not enough when execution a penetration test.

Logic not defined

It is not yet possible to configure the scanner so it understand the logic of an application. For instance, a donation page where people can transfer any amount to the author. What if the payment system allow negative values. An attacker could transfer money the other way round. Another typical example is the permission verification. Since it is not defined in the scanner, the automated tool has no clue whether a set of data is meant to be disclosed or not.

The four elements of a good pen-test

Four elements highly impact the quality of a penetration test: the time allocated for the assessment, the assessor’s skills, the system stability and the documentation available.

Time allocation

It is a known fact in the security community, no one can offer a 100% secure solution. But this also means a penetration test cannot identify 100% of the security issues. This is mainly due to the limited time allocated for the penetration test[AA4] . In general, the methodology of an assessment consists of identifying all user inputs – value that can be manipulated by the user and later be process by the system – and for each inputs, find a way to deceive the function that will process that value in a malicious way. This often involve the injection of specially crafter payload into the user inputs. Sometime, this means testing thousands of different payloads. This is where automated tool can come handy. However, scanners have their limitations, which manual tests to be done. Depending on the time allocated for the assessment, the penetration tester usually doesn’t have enough time to test the entire set of payloads, in all possible orders in all possible user inputs. Therefore, some tests will be missing, which might have potentially disclosed more vulnerabilities. That is why the allocated time for the penetration test should be properly evaluated and recommended by the penetration testing team to the client based on the scope, the documentation provided and, when possible, the thread model designed in an earlier phase of the SSDLC (in case we are talking about an application).

Sometime, whenever a launch date is based on an event that cannot be re-schedule (e.g. portal for political election), additional resources may be allocated in order to overcome the limited time available. However, nine women cannot make a baby in a month, same goes for penetration test: two penetration testers will not be able to cover in one week as much as one penetration in two weeks. This is why a mitigation plan should be established in parallel to tackle as much potential attack as possible. A mitigation plan typically includes the deployment of third party solution to filter any user input (firewall, web app firewall, etc).

Assessor’s skills

Over an unlimited period of time, a senior and junior penetration tester will eventually find the same findings when using the same methodology. However, an experienced penetration tester will develop through time some skills to understand and guess what is happening behind the scene in the server. The assessor will also naturally identify a set of potential exploits based on the server’s response. This helps the assessor to priorities the test (set of payload to use, specific order to execute function, etc) depending on estimated likelihood to find a vulnerability. This optimize the ratio finding per time spent, which is very valuable in an assessment limited in time.

System stability

Several deployment environments, also known as tier, are usually used in large companies in order to test future services and configurations without impacting the running environment (production). In such structure, you have at least two tiers, testing and production. Usually the test environment has less capacity then the production. Therefore, it may happen that some latency can be experienced on the testing environment. While small latency (2 seconds) can already have a big impact on automated test, more than 2 seconds will impact both the automated and the manual test by slowing down the assessor’s work resulting in a waste of time and resource, which ultimately will reduce the amount of finding.

Furthermore, when performing a pentest, the environment should not be experiencing other tests (such as QA or stress test) as it may alter the behaviours of the system and thus potentially altering the assessor’s results.

If the assessment takes place in the testing environment, this one should be as close as possible from production. Otherwise, the penetration testing might miss some valuable finding such as outdated version or misconfiguration.

A penetration test should be executed on a system that reached at least the “release candidate” state as each changes on the system may potentially open or close a vulnerability, making the work of the assessor inconsistent. This slows down the penetration test and may also result in a more vulnerable system than before the assessment.

In general, we recommend to avoid performing a penetration test on a production environment. If for any reason the penetration test must take place in a production environment, usually the assessor must take extra precaution in order to prevent damaging or impacting the environment. [AA6] This extra care will limit the set of tests executed by the pentester and usually prevent any use of automated tool, which will slow down the assessor’s work. Furthermore, it is almost impossible for a penetration test to guarantee 100% fail free assessment. There is still a risk that a test executed by the pentester may trigger a function that will badly impact the environment.

This is why penetration test should be iterative. For each new release that may open new vulnerabilities, a new penetration test should be scheduled.

Documentation available

A penetration test always starts with a discovery phase. In this discovery phase, the assessor will get acquainted with the system to be assessed, understand its purpose and the way it should usually work. This phase includes lots of guessing, trial and error and sometimes luck. All those unreliable methods can be skipped, and thus fasten the assessor’s work, if proper and adequate documentation is given ahead of the assessment to the penetration tester. Reading the documentation is part of the penetration test and should be taken into consideration when planning the time frame. Too much information might drown the assessor with valuable but also useless information. This is why well-structured and good quality documentation is a key element to a good penetration test.

Further to “simply” explain the project, the documentation should also contain permissions and roles table. Let’s take as an example a Human Resource application that manage internal employees. If assessor find a way to bypass the user interface limitation and access the salary of another employee. As a HR employee from the same department, this might be authorized, however, the access should be restricted to HR from different department and also to any other employee. For a given observation, the outcome could either be a critical vulnerability or a normal behaviour. This could be reported properly only with sufficient documentation from the customer.

Lastly, it is important to accurately communicate the scope of the assessment. First of all, for legal and authorization matter, but also to prevent the assessor from spending time on systems that are not relevant to the customer. Sometime, a subnet or a domain is not accurate enough.

There is no such things as 100% secured

A penetration test cannot identify 100% of the security issues, the main reason being that a penetration test is limited in time, while an attacker has unlimited time to penetrate the system. Apart from that, there are also reasons inherent to the system, for instance, the “human” component. Human are usually the weakest link in a system. Human can make error, human can be bribed, can be malicious, lazy, clumsy, etc. All this can put a system at risk. So even if you have the most secure solution on the market, you are still at risk because of human interaction.

The second reason is the complexity of a system. A system usually relies on different layers. For instance, a web application is based on a framework (WordPress for instance) which is hosted on a web server (IIS for instance) which rely on an operating system (Windows 10 for instance). So even if a penetration could identify all potential security issues at that time, there is still a risk that there is a vulnerability in the underlying layers. One finding in a different layer could potentially compromise the entire server, including the web application.

Another reason is the endless conflict between usability and security. Let’s say you want to authenticate on a platform you don’t use regularly. If you enter the wrong password, should it tell you “Your password is incorrect” or “Your username and/or password is incorrect”. The first one is better from a usability perspective, while the second is better from a security perspective as it prevents username listing. What is more important? Slowing down malicious users, or increase the user satisfaction. The goal is to have enough measures to deceive all malicious user from compromising your platform while offering the best user experience.

There is no evidence that someone or a company has ever produced a 100% secure system. Eventually, every system has been compromise after some time.

What to expect from a penetration test?

The goal of a penetration test is sometime misunderstood. A pentest is not meant to fully secure your application. A pentest is meant to understand the current security posture of a project, evaluate the risk related to it and provide clear solution to fix the identified security issues.

The entire value of a penetration tester resides in the final report. The report should list all the security issues identified during penetration test. Each reported finding should explain in detail the following:

  • What is the issue
  • What is the impact in case the issue is exploited
  • What is the likelihood that this security issue is exploited
  • Where is the issue located
  • How to replicate the exploitation steps for internal test
  • How to fix this issue

Some IT security companies tend to exaggerate the calculated risk and/or find some very unlikely scenarios, which sometimes are not even related to the initial scope. This only to legitimate their work, building fake evidence of their skills and showing how they are essential to their customers by blowing up the critical findings. By doing that, they do not help their customers, but rather selfishly blind their them in hope to get further contracts. This is a example of what a penetration test should not be.

The final report should help the project manager to understand the risk. It is important to explain the remediation with enough detail so the project manager and technical support can properly evaluate the budget and time needed to fix the issues. The final goal being to provide all necessary information for the project manager to take the best educated choice for the future of its application/system, meaning:

  • Shutting down the project
  • Accepting the risk
  • Mitigating the risk
  • Remediating the security issues

It is worth noting that the risk reported during an assessment is purely technical and doesn’t take into consideration business risks. Let say for instance we found a security issue that allows a user to escalate privileges and reboot a server. Typically, this vulnerability will be rated 6.8 (medium). However, if the same server also hosts the financial transaction platform, then the risk should be rated higher. The assessor might not be aware of all the components linked to the assessed system due to lack of documentation and/or confidentiality. This is why our assessments only use the “Base Score” part of the CVSS framework, which calculate the technical risk only. With this “Base Score”, the project manager can update the CVSS score based on their internal knowledge of the project and infrastructure.


A penetration test is an assessment that provides valuable and critical information about a system when done properly that couldn’t be brought otherwise. However, it has its limit. A pentest is meant to highlight risks and suggest ways to reduce it, nothing more. Before you start a penetration test, you should know exactly what it is and what to expect from it and I hope this blog post help you to answer those questions.


Published by

Antonin Beaujeant

Information Security Consultant at Foxmole