How Thanawalla Digital Defends Clients’ Salesforce Enterprise orgs Against Catastrophic Data Events

On May 11, 2021, without warning, Salesforce instances around the world began denying users access to their systems. The shutdown was rapid and showed no early signs. Users already logged in could continue working to a point, but new logins were rejected. The splash page simply informed users that their instance was down for maintenance.

What admins saw in the first hours.

Any experienced Salesforce Admin knows the first move: go to the Trust site at status.salesforce.com, built and maintained by the Trust team at Salesforce, a team that reflects one of the company’s core values. On that Wednesday, the Trust site itself offered an equally ominous page with little indication of what was actually happening. That level of opacity is highly unusual, even during a major outage.

The timing made everything feel more alarming. The outage came during the same week that Colonial Pipeline had been hit by a ransomware attack, and it immediately prompted speculation that this could be a similar incident. It was not, but in the first hours, no one could say that with confidence.

As a Salesforce consulting partner, our support lines lit up globally within minutes. In the moment, we were just as disadvantaged as our clients in identifying what was happening. The questions came in fast:

What actually caused the outage.

As the afternoon progressed, Salesforce shared that the culprit was a faulty internal DNS table update. In plain terms, the internal routing tables that direct HTTPS requests had lost their ability to resolve to Salesforce’s internal IP addresses. No requests were being sent to aberrant locations. No client data had been accessed by unauthorized parties. The security platform that authenticates who should see which data remained fully active throughout the entire event.

This was genuinely comforting news. A misdirected routing table is a serious operational failure, but it is categorically different from a breach. The data was never in danger of exposure.

The Rapid Response team goes to work.

The Salesforce Rapid Response team is a remarkable thing to watch in action, and fortunately we do not see them perform at full capacity very often. The team identified, isolated, and began remediating the problem with the kind of precision that reflects years of preparation. They re-threaded the DNS tables manually across the entire Salesforce ecosystem and completed the arduous work in hours. Trust restored.

How TDigital responded for our clients.

While Salesforce worked to restore access, our team executed our own client communication protocol immediately, confirming data backup status against each client’s SLA. When orgs began coming back online, our real work began.

At its core, Salesforce’s number one commitment is Trust. On May 11, 2021, they demonstrated that preparation is the foundation of that commitment. Partners across the world were able to rely on Salesforce’s remediation strategy and their own protocols to ensure that trust was never compromised, even during one of the most disruptive outages in the platform’s history.

One lesson we took forward from that day: we have since modified our backup recommendations for all clients to include near-real-time offsite backups, not just scheduled intervals. A few hours of potential data gap is too much exposure when your business depends on that data.