972-383-9791 | ContactUs@t.digital

On May 11, 2021, without warning, Salesforce instances around the world started to deny users access to their systems.  The shutdown was both rapid and without any early signs.  Users who were logged in were able to continue to access services up to a point, but new logins were rejected.  The splash page informed users that their instance was down for maintenance.

If you have been a Salesforce Admin for any length of time, you know that your immediate next step is to go to the company’s Trust site (status.salesforce.com), which is built and maintained by the Trust team at Salesforce.  The company places Trust at the forefront of all that it does; a core value.

Upon visiting the site on Wednesday, you were greeted with an equally ominous page that gave little indication of what was happening. This is very unusual, even during a system-wide outage.

The outage came during a week when Colonial Pipeline had been attacked by a Ransomware attack and prompted many to immediately wonder if this was a similar incident.

As a Salesforce consulting partner, our support lines, across the globe, lit up.  In the moment, we were just as disadvantaged in identifying for our clients what was happening.  The immediate questions ranged from (obviously) “what is happening…”, “when will service return” and, most importantly, “is my data safe”. The first two questions were difficult to answer as we had no direction from Salesforce (yet).  The third one was relatively easy to answer.  Our Support Clients participate in a regular backup-service, and we were able to indicate exact date and time of the last offsite back (lessons learned, we are modifying our recommendations to include almost-real-time backups).

Teamwork

As the afternoon progressed, Salesforce shared with us that the culprit was a DNS (internal) table update gone bad.  Deciphering the geek-speak, this was very comforting news, because it meant that none of our client’s data had been compromised.

The team started re-threading the tables manually across the entire Salesforce ecosystem and completed the arduous task in hours.  Trust restored.

How Salesforce Partners Stepped up their Game

Most Salesforce registered partners have a game plan for these types of events. At Thanawalla Digital, we commenced a communication protocol to our clients with the assurance that their data was backed-up as of the date/time their SLA dictated.

With the help of the Salesforce Rapid Response team, we were able to deliver a robust messaging to our clients during the outage, and we were able to independently verify that data integrity had been maintained.

At its core, Salesforce believes that Trust is their number 1 goal.  They demonstrated that, indeed, preparation is key to that commitment.  Partners across the world were able to rely on Salesforce remediation strategies to ensure that trust was not compromised during the outage.

Reach out to us directly to better understand how we harden our clients’ Enterprise orgs against catastrophic data events.