It’s widely known, and endlessly repeated, that the last, best line of defence against the potentially devastating effects of a ransomware attack is your backups.
So why do we keep hearing things like this:
We’re also feeling relatively confident, we have a very good backup system … and then we find out at about four or five hours after the [ransomware] attack that our backup system is completely gone.
Ski Kacoroski, System administrator, Northshore School District
The quote above comes from a recent Malwarebytes podcast, racing against a real life ransomware attack, in which host David Ruiz interviewed sysadmin Ski Kacoroski about a ransomware attack on the Northshore School District in Washington State.
Kacoroski’s alarming discovery—that the backups he was relying on to restore the school district’s damaged systems were unusable—is not unusual in the aftermath of a ransomware attack. The glib and depressingly common response from some in the IT community is to assume that those involved were idiots, and to blame them for their misfortune, observing with hindsight that they should have known they needed to spend more on this, run that, patch this, check that, etc.
A more realistic, more useful, perspective assumes that system administrators and security folk like Kacoroski are competent, intelligent people who are doing their best to meet multiple requirements in complex environments with limited resources. Starting there, the obvious conclusion from experiences like Kacoroski’s is that backups are hard to get right.
Why do backups fail?
Following the interview with Kacoroski, we set out to find out why getting backups right is so difficult. To help us we approached backup expert Matt Crape, a technical account manager at VMWare, and put exactly that question to him in a follow-up podcast episode, Why backups aren’t a “silver bullet” against ransomware.
This is what we learned from Crape:
Backups are difficult
Crape observed that people often imagine backups are easy, because their only experience of performing backups is doing them at home, where it is easy: You just plug a USB hard drive into your laptop every night and press a button.
But add a few hundred computers and you’re living in a different world.
Step one, says Crape, is figuring out what you’re trying to achieve. To do that you have to work though a series of important but difficult questions, including:
Are you backing up just your data, or your data and your applications? Are you archiving medical information or personally identifiable information that comes with regulatory requirements that dictate where, how, and for how long you can store it? How many copies of the data and applications will you make and where will you keep them? How long will you store each type of data? Do you need versioning? How often are you going to back everything up? Are you going to run the same schedule for all your data, no matter how important it is or how often it changes, or are you going to run different schedules for different things? And how will the scheduling, and the amount of data travlling over the network at different times, affect performance?
A backup archived to tape or the Cloud is only half the story too. It can only be considered a success if you can restore a working system from it, and there are a few things that can derail that.
SQL databases typically have to be stopped before you can take a back up that will usefully restore, for example. Many applications also depend on the existence of other services too (such as DNS, email or authentication) and you’ll need to understand and record those relationships, and have a plan for restoring systems in the right order if you want it all to come back to life.
You also need a process for reviewing those decisions regularly. Businesses evolve and change, and your backups have to keep up.
And finally, having done all that, you’ll need to do something far more difficult—convince someone it’s all worth paying for.
Backups are expensive
According to Crape “That money conversation was always the hardest part”. The problem with backups, he says, is that 99% of the time you don’t need them, so they can seem like money down the drain.
Ransomware changes the calculation considerably. Aside from their day-to-day uses, organisations have historically seen backups as a way to cope with natural disasters and other severe but infrequent events. It is easy to understand why they might put off dealing with that problem until tomorrow in favour of more immediate concerns.
But a ransomware attack isn’t a lightening strike or a once-in-one-hundred-year flood. According to IDC, “more than one third of organizations worldwide have experienced a ransomware attack or breach that blocked access to systems or data in the previous 12 months”. Other organisations might give you slightly different figures, but there’s no doubt that ransomware attacks are frighteningly common.
Crape suggests that the best way to make the argument for properly staffed and funded backups is to make the conversation about the cost of losing key systems: “How much downtime can we afford for this specific server?What’s the cost of that vs the cost of storing backups for three years?”
Backups are targets
“Had the Empire had better physical security for their backup archives, the Star Wars franchise would be markedly different”.
Matt Crape, Technical Account Manager, VMWare
Backups contain all the information that makes a company tick, which makes them targets for both theft and sabotage. For a modern fable on the menace of insider threats and the importance of physical security for backups, just watch Rogue One: A Star Wars Story, says Crape. “The Death Star blew up because of a backup.”
Ransomware gangs understand that your backups could deprive them of a multimillion dollar payday and will seek them out and delete them if they can. It’s also not unusual for criminal hackers to spend days, weeks, or even months inside the networks of organisations they’ve breached. They use that time to perform reconnaissance and elevate their privileges, so they can reach all parts of the network, including its backups (even Cloud backups). If they can find them, they will destroy them before running their ransomware.
When it is finally run, many kinds of ransomware will also look for and disable or delete shadow copies—a form of local backups—on the machines they infect, cutting off the possibility of restoring those machines with a quick rollback.
If your ransomware recovery plan relies on backups, you will need copies of your data that are offline and off-site, where they are permanently beyond the reach of an attacker who may be resident in your network for months.
Everyone assumes they’re working
According to Crape, another reason that backups let us down when we need them most is that people simply assume they are running correctly. “It’s not uncommon to hear about folks who just don’t check the status, ever”, he told Ruiz. “They’ll check it the first couple of days and then it gets old so they stop paying attention to it, or they turn off notifications because it’s just been running fine. You go to do a restore and you find out, oh, this thing hasn’t run in six months.”
It’s not enough to monitor that the application ran without failing, says Crape. A backup job can run without failing, but that doesn’t mean it did anything; and just because the job ran properly, that doesn’t mean the tape isn’t blank; and having something on tape doesn’t mean you have something that will usefully restore.
If you want to know if your backups are working, you have to test them. And that means doing a full restore into another environment.
Listen to the podcast
To learn more about why backups fail and how you can use them to effectively combat ransomware, listen to the full podcast below, or in your favourite podcast player from Apple, Spotify, or Google.
