Phishing simulations are worse than useless

We've all seen "security theater" - things that look or feel like they're making a difference, but empirically do not make us safer. Every time we take our shoes off in the airport, or rotate our passwords every 90 days, we take part in a security theater ritual.

"Okay, maybe those things don't make us safer, but they also don't hurt" is the most common objection I hear to getting rid of security theater practices. But is that actually true? Taking off my shoes in the airport wastes time, not to mention the sanitary issues of walking around with a bunch of other shoeless humans on a dirty floor. And research has shown that password rotation policies cause end users to pick more predictable passwords - exactly the opposite of the outcome that we want!

That bring us to phishing simulations. I'll admit that I used to be a passionate believer in doing phishing simulations, but over the years, my view has changed. These days, I think it's fairly common even for PhishSim advocates to admit that, much like sugary breakfast cereal, simulations are only a part of a balanced security diet. But I would go one step further - I would argue that phishing simulation the way it's done today is actively harmful to organizational security posture, and to the end users that are subjected to it.

"But phishing simulations are a risk reduction measure that provide defense in depth!"

In one of the most comprehensive real-world studies of phishing simulations, researchers evaluated the outcomes of ten phishing simulation campaigns sent to over 19,500 employees of a large healthcare organization. Their conclusions were definitive (emphasis added):

First, we find no significant relationship between whether users have recently completed cybersecurity awareness training and their likelihood of failing a phishing simulation. Second, when evaluating recipients of embedded phishing training, we find that the absolute difference in failure rates between trained and untrained users is extremely low across a variety of training content. Third, we observe that most users spend minimal time interacting with embedded phishing training material in-the-wild; and that for specific types of training content, users who receive and complete more instances of the training can have an increased likelihood of failing subsequent phishing simulations. Taken together, our results suggest that anti-phishing training programs, in their current and commonly deployed forms, are unlikely to offer significant practical value in reducing phishing risks.

Of course, if you read the statistics from any vendor that provides PhishSim services, you'll come away with the exact opposite impression, with claims like "the average phishing test program results in a 37-fold ROI" and "regular training [is] shown to reduce risk from 60% to 10% within the first 12 months."

The Ponemon Institute is one of the most commonly-cited research organizations in this space. They published studies in 2015 (sponsored by Wombat Security, a security awareness vendor acquired by Proofpoint) and in 2021 (sponsored directly by Proofpoint), and most of the stats cited by other vendors originate from these papers.

To be clear, vendors sponsoring research doesn't automatically mean the research is bad. But when you look at the underlying methodology in the paper, this was actually a survey asking respondents to estimate the "percentage decrease in the cost of phishing as a result of employee training interventions."

I have no reason to believe that the respondents were being untruthful. However, humans are notoriously bad at self-assessment, and a subjectively-worded survey of a few hundred people is hardly the same as a controlled research study. I'll take hard data over polled opinions every day of the week.

"But if phishing simulations prevent even one successful attack, that's a good thing!"

By this logic, disabling all communication tools at work would make the company even safer, because then nobody could even receive a phishing email, let alone click on it. We all know that isn't going to happen though, because the purpose of a cybersecurity function is to enable the business, not shut it down entirely.

Even if you assume that all of the marketing stats around phishing simulation are true (and again, there's very little reason to believe that they are), the best possible outcome we can hope for is a partially-effective security control that degrades over time.

Our investments in security controls need to (1) be proportionate to the value that they provide, (2) minimize the amount of unneeded friction that they cause, and (3) maximize the utilization of our time and effort. Rather than trying to train our employees how to recognize phishing emails, why not invest in training them how to use passkeys?

The best security controls require end users to think about security less, not more.

"But we need to educate our users about security and common attacks!"

I agree! Education is a good thing. That's why all of our college professors made us take tests first, and then shamed all the people that failed the test, and then taught us the subject matter, right?

Of course not. When done well, security education can help turn your employee population into a crowdsourced threat intelligence feed. But this doesn't have to mean sitting through an annual, multi-hour training, nor does it mean punishing people for clicking on a link (and yes, remedial security training counts as punishment.)

What I've found works best here are two things: first, making it dead simple to contact the security team and forward suspicious communications for review. Second, providing the organization with a constant trickle of relevant articles and news about recent attacks, with commentary about how to recognize that attack if it happened to our company.

This approach positions the security team as a trusted resource, rather than a group of remote authority figures looking to punish any mistake they can find.

"But my regulators require me to do phishing simulations!"

Are you sure that they require you to do a phishing simulation specifically? Or do they require you to perform regular security awareness training, where PhishSim is simply one way to achieve that goal?

I've worked in some very regulated environments, and every time, we've been able to convince our auditors and regulators that our approach to security education met their underlying requirements.

There are so many ways for poorly-constructed phishing simulations to irrevocably destroy the security team's trust and goodwill with an organization, from announcing fake bonuses to using an imaginary Ebola outbreak.

Getting creative in your approach to security education, and investing in security controls that remove the burden of risk decisions from end users, is definitely harder than just pressing "send" on your next phishing simulation. But it's more than worth it in the end.