Real-time monitoring for Liquibase database changes is here!

Salesforce down: What you can learn from this outage

May 20, 2019
Salesforce down: What you can learn from this outage

Everyone has bad days. Every company has been through some kind of outage due to a buggy database deployment. Even the best of the best, with highly trained staff, world-class best practices and well thought out processes, make mistakes. On Friday, May 17th, 2019 Salesforce.com had a bad day.

The company deployed a faulty database change script that broke permission settings in production. This meant that users suddenly obtained read and write access to restricted data. This opened the door for an unauthorized employee to steal or tamper with the company’s data. As a result, Salesforce needed to take large parts of their infrastructure down to find and properly fix the issue. The outage lasted 15 hours and 8 minutes. According to Gartner’s formula ($5600/minute), this outage cost approximately $5 million. Plus, since so many companies rely on Salesforce, it was a very visible and embarrassing outage. (It’s earned several hashtags, including  and .)

I couldn’t help but think about all of those Salesforce employees who needed to work like mad to take the whole database down, find the offending database script, and restore everything. All because of one change script. That’s not a fun way to spend a Friday night (and Saturday).

Before Friday, Salesforce customers have had very little disruption in service. Many loyal customers were tweeting about how rock solid the service has been and that’s impressive. They clearly had their stuff together over at Salesforce.

But everyone has blind spots.

Look for Blind Spots in Your Database Process

The bottom line is that this outage was preventable. There are tools out there, like Datical, that can auto-generate permissions on objects to ensure this doesn’t happen. And there are other ways to prevent this, like expanding test efforts and mirroring production data with solutions like Delphix to ensure scripts aren’t pushed to production without understanding what effect they might have.

If you haven’t looked into more robust database automation for your company yet, it’s well worth your time. We’d love to talk to you about how to prevent database production errors. We’re experts in this area. We’d love to connect with you about how we help our customers prevent database production errors every day. Contact us to learn more.

Article author
Erika Kalar