Is penetration testing useful?

4 Comments

Comment by Sandro Gauci on July 20, 2008 at 7:48pm

I think you should elaborate more with reasons for not going for Penetration Testing. For example - your section on "fixing holes can introduce new security breaches". Give real examples of when this can happen and explain the logic behind it. I'm not saying that this is not true - I've seen cases when this happened. But simply stating something like that is neither clear not constructive.

I do think that Penetration Testing is many times advertised as the way to secure your network / software / whatever, while in fact it is just the start. However neither that, nor the reasons you give make Penetration Testing useless.

A good penetration tester will recommend solutions that last. He/she will not just explain how he hacked the system, but will also expand on what other attacks can happen. A good report will include information on how to prevent intrusions from becoming major security incidents.

Apart from that - a Penetration Test many times serves to remind the management that security attacks can indeed be successful on their systems. It can remind them that what they read in the news is not just something that can only happen in the stories. Penetration Testing also has a tendency of being step 1 of a real assessment of the security posture of a system, leading to jobs which can help improve the situation for real. It is much easier to convince upper management that there is need for certain things after a Penetration Test. If the upper management is already educated, then a security assessment might be a better solution.

Is it good or not?
Definitely! But its no panacea.

Comment by blackjw on July 20, 2008 at 11:37pm

Sandro thanks for your comment.

Your are right, I did not really elaborate on the subject. In fact it was just some thought for further explanation in a future post. This one was published just after having read Secrets and lies by Bruce Schneier. He makes really good points on how security works and doesn't work.

The main idea in this post is that organization often hire outsource penetration teams because they want to know if they are vulnerable or not so why an organization should spend money on something that we already know the result. Maybe I am wrong, but I feel that penetration testing is misunderstood by most organizations.

"fixing holes can introduce new security breaches". Let say that a pentester found out that he can brute-force passwords because employees used dictionary-based password. He/She can recommend to use stronger password. What can prevent an employee to write down is password next to his monitor because it is too complicated to remember.



Yes, pentest can be useful if it is part of a security process and should not be seen as a product. Organization should not think that they are secure because the team says. A good penetration tester is as good as is own knowledge. Penetration testing can create a climate of fear and uncertainty that can help to improve security in an organization but is it what we really want?

Last thing, How fast should fixes be applied and how long should a solution last?

Comment by Sandro Gauci on July 21, 2008 at 8:57am

Secrets and Lies is a good book :)

I think the reasons that you think that pentesting is useless are:
* the lack of an successful intrusion from the pentest team does not mean that your organization is secure
* it tries to prove something that is already known - that the organization is vulnerable to attack

However there are various reasons why one could ask for a Penetration Test. I don't know how familiar you are with corporate politics and organizations, but there is a tendency that things do not get done unless its absolutely necessary. It is much easier to get something done if you have a report saying that it should be done because there is indeed a problem.

Just because you or the sysadmin already knows the "result", doesn't mean that the right people (the people who sign the bills) do.

Penetration Test is not the ultimate approach but it will also serve as part of a process (to use Schneier's words) for certain organizations. Something like "we've done everything right, now lets test it out".

Regarding the "fixing holes can introduce new security breaches". Good example. Dictionary based passwords are definitely a problem, and they should be included in a pentest. Part of the solution is to not use a dictionary based passwords ofcourse. However as you mention, making it too complex can introduce other problems. My solution is that once dictionary passwords are out of the way, create password lockout or slow down policies. The policies have to account for people who forget which password they used. So I don't agree with lockout policies that limit to 3 attempts. Something like 10 attempts would be more appropriate. The logic is simple - a brute force attack relies on the fact that you can try a large amount of passwords in a short time. By introducing ways to dramatically slow down a brute force attack, it no longer remains an effective attack.

FUD is another thing altogether.

How fast should fixes be applied / how long should they last? As soon and much as possible ? ;-)
Fixes that rely on only software patches are very short sighted. A more suitable fix is one that takes the bigger scope, looks at the attacks that apply to the system and find out how to make a solution that affects the economics of these attacks. Think Dan Kaminsky's solution to the DNS issue. It (should) makes DNS poisoning in general less successful as an attack, rather than tackles the specific technical issue that he should be disclosing in Blackhat soon.

Comment by blackjw on July 21, 2008 at 1:44pm

Sandro,

You made an excellent comment here. It is appreciated. You got the point of what I meant. Actually, it's not easy for me to articulate clear arguments (I'm working on it) because English is not my first language.

I'm currently reading Beyond Fear by Bruce Schneier and I like how he explains things like the trade-offs we have to do and the agenda of players involve in a security system.
***
I believe that pentest has its place but does it worth the cost for most of organization? The team will eventually come up with solutions in order to fix holes that were found but they have a price. Is the organization willing to pay.

Penetration Test is not the ultimate approach but it will also serve as part of a process (to use Schneier's words) for certain organizations. Something like "we've done everything right, now lets test it out".

I could secure my house with door locks, burglar system and even a 24/7 guard but I'm leaving in an area where breaks in are rare. So, door locks should be enough. There're still a risk but it doesn't worth to pay neither for a burglar system nor a 24/7 guard. Let's called a specialist (Penetration tester) if my house is secure.

Penetration testing will tell you (I hope) if the trade-offs we made are the right one, if involve parties are satisfied with the actual risk.

Add a Comment