- When Microsoft 365 went down on 12 November, Mimecast kept accepting email for its clients and held close to 100 million messages safely until their mailboxes came back, so no mail was lost.
- Isolated regions, interchangeable copies of every service and mail stored across many locations kept mail flowing into our platform for 18 hours, with no manual intervention.
- Clients using continuity kept reading and sending email through Mimecast. Demand on the user portal rose 35-fold, which strained it and shows us where to improve next.
Designing for the day the mailbox goes away
When Microsoft 365 went down on 12 November, mail kept arriving. This is an engineering account of what our platform did with it
By Paul Sowden
Key Points
The first 15 minutes
At 17:50 UTC (12:50 ET) on Thursday 12 November, our alerts for failed delivery began firing and the queues of email waiting for delivery started to grow. Inbound mail slowed, but kept arriving. In other words, we could no longer pass mail on to Microsoft, but the rest of the world was still sending it to us.
The gap between mail coming in and mail going out is the whole story. When the far end of an email pipeline disappears, senders don't stop sending. The internet’s address book (DNS) keeps pointing them at us, and our gateways (the servers that receive email on our clients' behalf) keep accepting it. The question becomes what you do with mail you've taken responsibility for but can't hand over.
Our first question was the one every on-call engineer asks: is this us? Delivery to non-Microsoft destinations held up, which told us the problem was downstream, not with us. Microsoft’s first acknowledgement on X at 17:54 UTC (12:54 ET) confirmed it. What follows is the view from the sending side: our platform, holding the mail and trying to deliver it.
Queues began climbing within minutes of onset in all of our nine regions, indicating this was a global event.
What SMTP and email does when the destination disappears
Email was designed for unreliable networks, so failing to deliver is a normal state, not an exception. Four properties of SMTP (Simple Mail Transfer Protocol, the shared rules every email server follows) decide what happens next.
- MX decides who the sender talks to. A sending server looks up the recipient domain's MX (mail exchange) record, which says where that domain's email should go. For our clients, that points at us, not at the mailbox provider, such as Microsoft. The sender's conversation is with our gateway.
- A 250 transfers responsibility. When our gateway replies 250 OK (the standard code for ”message received”), the message is ours. The sender considers it delivered and forgets it.
- 4xx means "try later"; 5xx means "give up". When we pass mail on and get a temporary 4xx error, a timeout or a refused connection, we keep the message and retry. The email standard (RFC 5321) expects senders to keep retrying for days, not minutes.
- Store-and-forward is the design. Every hop (each server along the way) may hold a message until the next hop is ready.
Now consider the alternative. If MX pointed straight at the mailbox provider, every sender on the internet would queue its own copy of each message. That means thousands of independent queues, each with its own retry schedule, some giving up early. The recipient would see nothing and could read nothing until the mailbox came back.
A gateway in the MX path turns that into one queue, with one retry policy, that we can observe and that users can still read through continuity (our service that lets people keep reading and sending email while their main mailbox is down). That is the architectural bet this post is about.
Mimecast Apex: Scalability and resilience
I joined Mimecast in the early days of the company, and the core design philosophy has remained a constant throughout the years: Everything fails. That applies to our own services and infrastructure, and equally to the services and infrastructure we connect to, like Microsoft 365.
A well-known principle in computer science, the CAP theorem, says that when a system keeps its data on many computers and the network between them breaks, it has to choose: always give the right answer, or always give an answer. Networks always break eventually, so the real question is what you give up while they're broken: wait until you're sure, or answer straight away and accept the data might briefly be out of date.
In the early years of Mimecast, our approach favoured consistency (the right answer) over availability (always an answer). In 2011, after a network incident affecting our UK region, we heard loud and clear from our clients: email must always flow. Our approach shifted to prioritising availability when forced to make that choice. That decision is now realised through three core tenets:
Regional Isolation
We operate nine distinct regions. That gives clients data residency (their data stays in the part of the world they choose), and it gives us natural isolation: the platform is designed so that a significant failure in one region does not affect clients in another.
Stateless services run as a fleet
Multi-tenant, stateless services such as the Mimecast Apex™ Gateway run as many interchangeable copies, so a problem with any one is absorbed by the rest.
Stateful workloads are heavily sharded
Where a workload must hold data, we split both the data and the work into many small pieces, called shards, so an interruption to any one shard touches only a small fraction of mail.
This philosophy is what enabled the Mimecast Apex platform to support our clients during a global event.
Mimecast Apex: Architecture under sustained load
For a full day, the platform ran in a state it is built for but rarely sees: still accepting mail, though not at normal rates, while delivering almost nothing to a large share of destinations.
Delivery success fell in every region within an hour of onset and stayed near the floor until about midday UTC the next day. We saw it fall in two ways: a cliff in regions such as the US and Australia, a steady 18-hour decline in others such as the UK.
Where the mail went
Accepted messages for unreachable destinations moved into Termite, our purpose-built email queuing platform (think of it as a secure holding area for mail waiting to be delivered). As a stateful workload, Termite is heavily sharded and distributed. It is designed both to move the billions of emails we process every day and to hold them for an extended period if the mailbox disappears. Messages are copied across multiple physical locations before we return the 250 ”received” code. Nothing is acknowledged to a sender until it is safely stored in more than one place.
The name is deliberate. Like a termite colony, Termite runs as a set of independent units, each working for the whole.
On a typical working day, Termite will be actively managing approximately 8 million emails globally at peak moments. On 13 November we saw nearly 12x that volume with regional peaks totaling nearly 100 million.
*US & USB are both United States of America.
Continuity: the queue is the mailbox
The architecture behind continuity is beautifully simple, because it reuses what’s already there.
Reading. The mail users can’t reach is the mail sitting in Termite, waiting for their mailbox to return. Rather than copying it somewhere else, we show that same queue to users as an alternative mailbox, through Mimecast’s end-user tools such as the Personal Portal and Mimecast for Outlook. There’s no second store to build, keep in sync or switch over to.
Sending. Outbound mail already flows through the Mimecast Apex Gateway, so users can keep sending through it, whether or not their main mailbox is working. Replies leave on the same path as every other message.
Recovery. When the mailbox returns, Termite delivers the held mail as normal, and the inbox ends up complete, with nothing to reconcile by hand.
The human element
From the first alert we ran a round-the-clock incident room, with every affected service under constant watch. As the incident unfolded, AI created additional monitors, so our coverage grew alongside the problem. For example, a Cursor agent gathered queue statistics from every region, including how quickly each queue was emptying, and posted them to the incident Slack channel every 15 minutes, giving the room one consistent view of the drain. One of its updates, covering 14:46–14:58 UTC
What strained, and what we'd change
I’d love to say everything ran perfectly, but it didn’t (and you wouldn’t believe me even if I said it did). Our mail flow systems ran with no intervention: every email that could be delivered during the incident was delivered, and the queues drained once Microsoft came back online. We did, however, see intermittent problems with our end-user portal for a period. With their Microsoft 365 mailboxes unavailable, users turned to Mimecast to reach their mail. Mailbox views rose 35x, putting real strain on the end-user portal. Engineers adjusted service runtimes and stopped non-critical services to free the capacity it needed.
The layer in front
Everything fails. We wrote that down as a design principle years ago, and on 12 November it stopped being a principle and became a Thursday afternoon.
Most of the time, this layer is invisible. Mail arrives, gets checked and is passed on, and nobody thinks about the queue in between. Its value only shows on the day the destination mailbox disappears. The job is simple to describe and hard to build: take responsibility for every message, turn an outage into a delay rather than a loss, give people a way to keep working, and hand mail back at a pace the mailbox can handle when it returns.
In 2011, our clients told us email must always flow. Holding close to 100 million messages in a single day tested whether we'd listened. Mail flow held without intervention. The portal strained under 35 times its usual demand, and that's showing us where our next piece of work is.
None of this was software alone. Thank you to the people in our incident room, who watched the graphs around the clock and had the discipline to let the system do its job. And thank you to the engineers at Microsoft, who spent those same hours restoring a service that millions of people depend on.
Every dependency fails eventually. Design for the day it does.
Related Tags
- Email & Collaboration Security