March Email Security Risk Assessment Report: A Focus on Office 365

With this blog I am happy to bring our 8th consecutive quarterly release of the Mimecast Email Security Risk Assessment (ESRA) aggregate report to your attention (also with an associated infographic). We now have been running ESRA tests, collecting and analyzing the data and publishing reports for two years!

In last December’s blog I focused on the overall false negative rate that we had seen across all of incumbent email security systems we had tested against to that point. It is quite a diverse list of incumbent systems.

In this report I am going to focus on the data we have collected over these two years specifically with Microsoft Office 365™ (Exchange Online Protection or Advanced Threat Protection) as the incumbent email security system. But before I do that, for those readers who are new to ESRA testing, I first need to explain what it is all about.

How Does the ESRA Work?

In an ESRA test the Mimecast Secure Email Gateway service reinspects a participating organization’s emails that were deemed to be safe by their incumbent email security system. This is based on actual inbound email traffic, not on test emails. We run this test over a period of time, usually between a week and a month at each organization. An ESRA test passively inspects and records the security results of real emails that have been delivered to their employees.

In security terms an ESRA test is a false negative hunting test, where the Mimecast email security service inspects delivered emails for missed spam, phishing, malicious files and URLs and impersonation emails.

Before I get into the Office 365-specific results, it’s worth noting that we recently added the detection of malicious URLs within delivered email to our ESRA testing capability. In aggregate the Mimecast ESRA testing has detected 463,546 malicious URLs that were contained in 28,407,664 delivered emails. This comes out to an average of one malicious URL getting through an organization’s email defenses for every 61 delivered emails. Given how many emails a typical organization gets in a day, that is a lot of malicious URLs waiting to be clicked in employees’ inboxes!

Office 365 Misses a Variety of “Bad” Emails

Now to the Office 365 specific results:

• Of the 232 million emails we have inspected in aggregate, 105 million—or almost half—of those had passed through Office 365 as the incumbent email security system.
• Of the 75 organizations for whom we have conducted ESRA testing, more than half, or 47 of them used Office 365 as their incumbent security system.
• We have found that Office 365’s false negative rate for spam to be 16% (as in 16% of delivered email was actually false negative type spam) versus 11% for ESRA testing across all incumbent security systems. That may not seem like a big difference, but to an organization receiving a lot of spam, it can be quite burdensome.
• We also found that Office 365 let in more than its fair share of impersonation attacks, more than 33,000 of them, as well as unwanted, potentially dangerous or malicious file attachments—also more than 33,000 of them.

We promise to keep testing and to keep reporting on what we find. While perfect security is not possible, better security most certainly is. And we see our ESRA testing as a great way to keep the focus on false negatives and how best to minimize them. So, stay tuned for our 9th quarterly report, due out this summer.