Kirstin Burke: We always go back and forth about the topics that we think are relevant for us to speak to, or with, or about for you. And, one of the things that we went back and forth on this month was all around ransomware. Don’t have to go too far, look too far. Maybe you’ve been affected by it, but it remains one of the top topics of the year, no matter who you are, no matter what business you’re in. And it goes beyond security tools, right? So you can have the best security tools in the world, but if and when something happens, right, a lot of the other things that support your infrastructure start to kick in, and one of them is data protection.
And one of the things you hear a lot about nowadays, in the marketplace, is that data protection is ransomware protection. And Shahin and I started talking about this, and, you know, hey, is this something we ought to talk about? And I started getting a lot of attitude and pushback on that. Is it or is it not, you know, is that sufficient? And so we thought we’d tackle that today because it really is something that you hear a lot about in the market. And if you’re not paying close enough attention or asking the right questions, it might affect the decision you make and the quality, or your ability to actually recover from a ransomware attack. So tell me, what’s about that attitude I got?
Shahin Pirooz: So, going way back, going way, way back in IT, it used to be that you really didn’t pay attention to or care about backups until you lost data. And then all of a sudden, backups became the most important thing on the planet. And so the rest of your career, you made sure backups were there, and you did tests, and you did validation and all that.
So, I think we’ve come to a place where we realize backups are pretty critical, and it’s become, you know, there’s a lot of regulatory concerns that require data protection of some kind. Disaster recovery, which implies there is data protection that is being either moved off site or replicated someplace else.
Data protection has become kind of like the table stakes for IT. And, it is really one of those proactive steps you want to take for data loss, no matter how it happens, whether it’s a fire, weather, you know you have to move out of your office in the pandemic and you have to restore someplace else, or ransomware.
The problem is, we have a lot of data protection vendors and storage vendors today coming out and talking about the fact that all you need is a good data protection solution and you have ransomware protection. And there’s some truth to that. There is fact associated with that.
Anecdotally, I’ll tell you, we’ve had incident responses we’ve done where the system couldn’t be recovered from rollback or anything like that, from the tools that are in play. And the only way to be able to get it back was either to buy the encryption key to pay the ransom and get the encryption key or, if you had good backups, you can restore it.
So that’s a great use case where you can say, oh, maybe they’re right, that is ransomware protection. However, what kind of backup are you using? How is it set up? How is it configured? And so a lot of the players today are now talking about immutable backups are the important thing.
So it has to be something that the bad actors can’t change. Okay, that’s great. How is that done? So what’s going on today is that bad actors are not attacking systems without access. So, when we think about any part of the security stack, we need to take into account, when you study for your CISSP, one of the things you get taught is the CIA triad.
CIA triad is about confidentiality, integrity, and accessibility. And sometimes they swap the A to be access or accessibility. The reason that’s important, as we’re talking about backup and data protection, is everybody wants to focus on the immutability of backup, which is really the integrity part of it. So integrity means it doesn’t change, you know that whatever you had, you’re going to get back the same way.
Confidentiality means that data is protected, so it can’t go anywhere. So usually air gapping is how that’s solved. It’s someplace else, it can’t be modified, it can’t be touched. The final piece of it is access, and that’s really the missing component in when backup providers come up with an answer that is all you need is immutability, and you’ve got ransomware protection.
The challenge is that almost every IT admin sets up their backups using their domain admin credentials. So, what’s a bad actor do when they get into a network? They compromise the domain credentials. And if I have a domain admin credential, I can log into the backup software and I can delete those immutable backups.
And there are systems that you can put in place, there are protections that you can put in place that say that it needs two domain admins to approve a deletion. But not everybody sets those things up. So, when you brought up the question of, is this accurate? My reaction was, well, it can be, but by default it’s not.
It’s very similar to, you know, other conversations we’ve had where we say is one tool enough to protect the environment? Immutable backups alone aren’t enough. You have to have all three sides of that CIA triad. You have to have air gap backup solution. It can’t be in the same data center because then they can encrypt it. It can’t be on the same VMware environment because if they encrypt all your VMs, they encrypt your backups. It has to be immutable, so it’s not easy to modify and change, and it has to have access controls in place. So, confidentiality, integrity, and access, those three attributes have to be there for it to truly be a ransomware defense, or recovery mechanism more accurately.
So if you’ve got a backup system that is backing up to cloud, let’s say, and it is an immutable backup system, and you have access controls in place that say that you need more than one person to approve a deletion or to make changes or whatever the case may be, then yes, absolutely, a backup solution can address the issue of recovery. And there’s been multiple recoveries where even in environments where we’ve been fully securing the environment and the ransomware hits and we’ve got our EDR or MDR solution and we’re able to recover all of the systems and be able to roll back using volume shadow copy. Every once in a while, there’s a system that has VSS corrupted, and when your volume shadow copy is corrupted, you’re not going to be able to recover that system unless you have a backup. So, even the tools that say we can do rollbacks, they rely on core system services. So, for anybody to believe that IT and security don’t work hand in hand, the left and right side of the room, is the mistake anybody has going into this space. So, you have to have backups, but you also have to have the protective controls, like an EDR solution, that can do rollbacks.
Kirstin Burke: It’s so interesting you just said that, because as I’m listening to you talk, and I think we see this and it’s part of why we have the five practices that we do, right? Everything is so interrelated. And so my next question to you is going to be, what is the level of intersection, interaction, codependency between IT and security to really make sure that you have real ransomware protection? Because it’s not just backup, it’s not just security tools. And it seems sometimes, even in the industry, that we treat these things as silos. How do these teams or functions need to come together to really ensure that you are effectively, that you are effectively protected from ransomware?
Shahin Pirooz: I think there’s a lot of use cases out there where you see those two teams are in conflict and it creates a lot of challenge. It becomes more of a competition between those teams as opposed to a collaboration. And it has to be a collaboration. I’ve seen implementations of security in such a way that it is an overwhelming regulatory body over IT that limits them from doing anything productive. That’s not good for the company.
Security should not be there to inherently slow everything down. Security should be based on governance and risk. Understanding your risks and understanding how you’re going to address those risks and then governing that you are in fact doing those risks, protections, against several things, either internal regulations or external regulations or industry. So whether it’s government, industry, or personally implied regulations, you need to govern your environment in such a way that you’re mitigating risks.
So mitigating risk doesn’t mean preventing people from being productive. And there’s a lot of security people that think we have to completely control and block access to everything so that we’re secure. And it’s a little bit too far off the beaten path. It becomes a hindrance to doing business if you have that kind of mindset coming to it, going into it. So how do they need to play together? They need to be collaborative and say, you know, if we get hit with ransomware, we need to be able to recover from backup. The security team doesn’t own the backup. The IT team owns the backup. The security team owns the EDR. So if the EDR tool can’t do the rollback and we have to go to IT, we better have a good connection between us that says, do you have a backup system that’s been tested and can recover and can do a restore in whatever your mean time to recovery is, whatever RTLs and RPOs you’ve set for your data protection.
What’s interesting about all this is we now call it ransomware protection. This is just traditional disaster recovery. The disaster happened to be ransomware. But it’s really nothing new. It’s something we’ve been doing for decades and decades in IT and technology. And we still have to decide risk-based approach to how much risk can the company take if this system is down for X hours. Therefore, what is our recovery time objective, and what is our recovery point objective? How long does it take us to restart, and how far back can we go? How much data can we lose? So those are the only two factors that apply here. And if the answer is we have transactions that happen every day that we can’t lose, then maybe your recovery point objective is 4 hours. I can’t lose more than 4 hours of data. Our team can reload those last 4 hours. And your recovery time objective is you need to be back up and running in 4 hours. So now you have to have, that’s a very short RTO and RPO that you have to be able to recover. And you better have a backup system that has immutability so that the bad actors can’t encrypt it, is air gapped, but it better be air gapped in such a way that you can restore in 4 hours, and it doesn’t take you days to restore, and, lastly, has access controls so the bad actors can’t delete those backups.
Kirstin Burke: Well, and I think you just mentioned kind of the things that were in my mind. I’m an organization, I have my backup strategy. I feel very confident about it. Bad actor X comes in, does their damage, and I feel good about it, but bad actor doesn’t stop at, hey, I’ve got your data. They delete it, they encrypt it, or you get your data back, but then there’s been something left in it, done to it, so that two weeks, three weeks, four months, whatever, later, something happens again. So what’s going on there where that backup strategy either didn’t go far enough or had holes? When you think you did everything right and find out that the compromise has come from a different angle.
Shahin Pirooz: Yeah. The best way to think through that is, this goes back to you have to have both sides of the house talking to each other. So if you don’t have a good endpoint security solution in place that can detect malware, ransomware, or whatever it is that’s being dropped onto your system, then the backup’s just going to back up that malware, ransomware, and when it fires and triggers and you restore to the point before it fired, you might be restoring to a point after it had landed on your systems. And so it’s in your backup. And so, a good answer, a good solution would be checking to see if there’s new files getting added, identifying what those files are, looking for those files and the backups, and that all is interactions between the security team and the backup team. So good backup solutions will let you say, when did this file show up on this system? So as soon as the security team identifies here’s the file we found, you can go back and say this showed up on, let’s just say September 10. Let’s make sure we restore before September 10. Now, you’ve lost your four hour RTO RPO, but you’ve gotten rid of the ransomware.
Kirstin Burke: So a lot of these things come down to human factors, right? I mean, on the security side, you’ve got the folks that are out there looking for bad guys, containing them, all of that on the IT side, right? You’ve got maybe someone who’s doing the testing, the planning, all of that. Each of these two teams have different objectives, and so, I see point in time when they have to come together. I’ve got this issue we have to work. How do you see these two teams working consistently together? Because it really seems like this intersection is not just situational, but there’s just got to be this constant back and forth.
And I know you’ve led, you’ve been the leader over both teams. You’ve also been the leader over one or the other. How should organizations think about kind of these teams running independently but together? And how does that work? I mean, even down to like, do you have joint scorecards? Do you have joint teams? Like, how do you make sure you’re connecting all the dots?
Shahin Pirooz: So the best way to address that is to, we’ve always taken this approach of supplier and customer. If you take that supplier-customer model in everything you do, the supplier of security is the security team to the IT team. The supplier of data protection is the IT team to the security team. So who’s the customer, who’s the supplier? What are the requirements? What are the expectations? And if you’re creating scorecard items that come from that context, then at least you can say, how often are we backing up? When’s the last time we tested? But the missing piece, I think that most people don’t get, we used to focus on tabletop exercises that were across the organization. Today, tabletop exercises mean red team blue team, we shifted to a security-only context of tabletop exercise. I think it’s really important to make sure that you’re including your IT partners in those tabletop exercises to say, okay, the EDR didn’t work. That’s the scenario we walk through. What do we do next? And be able to have those steps and processes and escalation matrices all in place to say, we got to get a restore, who do we call for the restore? It’s Saturday at midnight. So those are all attributes of being able to work through a roundtable conversation with all the players to say, who’s responsible for this? Who’s responsible for that? Here’s a scenario. What happens if this doesn’t work? What do we do next? So best laid plans don’t work. You know, have a meta plan that once you engage with the enemy, that it works. They do stuff to mess with you.
Examples might be, you go to do a restore and you find that your backup server, the system that controls the ability to restore is one of the systems that got encrypted. So now you can’t get to your backups. So now you have to rebuild the backup server to talk to the backups that are off site to be able to restore them and hope to God you can set up the configuration properly. So you always have to have these backup plans in place and tabletop exercises to detect. But you can’t simply think about it as what is our incident response for we got hit with ransomware. What are the options? You need to go and include all the players.
Kirstin Burke: So, you know, in all the research and when you read about this topic, they talk about the importance of doing a threat assessment. So before you get to the teams and before you get to the tabletop exercises or whatever, kind of assessing your threat, right? Part of that is the risk mitigation, or risk management and identifying what that is. How do you recommend, or how does DataEndure for its clients, we’ve got a situation like this where someone is clearly and understandably worried about ransomware as one of those disaster scenarios that can happen.
How do we go about helping them do a threat assessment? And is it usually just focused on backup and data protection, or how has that changed over time? And how do we really recommend people look at that threat assessment to determine what they need to do?
Shahin Pirooz: So in my opinion, this is where the backup companies actually have their act together, because at the end of the day, disaster recovery heavily relies on the ability to recover from backup. So what we at DataEndure, to answer your question, we have a resiliency workshop that we do that’s a complimentary engagement. It takes about 8 hours of time, but it requires involvement from all the key stakeholders in all the different departments to sit down and talk about what their top applications are, what their processes are.
If you have to, for example, do order processing on paper instead of in systems because everything’s down, how do you do that and what does it take? And does everybody know what to do? If you have to ship orders manually instead of just in time because your network’s down or whatever the case, those are the factors that you want to nail down. The resiliency workshop gives you high level visibility into to what are the key applications in your environment and how to recover those and how to prioritize those. And it gives you a really good, cool spider map of how am I looking in context of being able to recover these applications.
We have a full resiliency assessment, which goes to the guts of it and builds the disaster recovery plan and disaster recovery methodology, including technologies, including backup solutions, all that. So it could go to a full engagement to help you implement or assess where you’re at with your solution. Or it could be as simple as, let’s just do a quick assessment to see what you look like, let’s do a workshop to see what you look like. And the outcome of that might be we need to do something deeper, or it might be we’re in pretty good shape and we’re doing the right things and we’re going to keep doing those right things.
But, you know, for every organization, it’s like we can do security assessments and risk assessments that talk about what’s your attack surface look like, but that’s only telling you how to put controls in place to protect from the bad actors causing damage. But they’re not the only way your business is at risk. Security is not the only way your business is at risk. Weather could be a factor, pandemic could be a factor. Any number of things, a power outage –
Kirstin Burke: Disgruntled employee.
Shahin Pirooz: Disgruntled employee that deletes everything. There’s a lot of, everything from insider threats to geographic threats to environmental threats. Those are all real things that the answer to all of those is, you better be able to restore your data so you can get back up and functional. And a disaster recovery plan, not a ransomware plan, is the answer to all of those things.
Kirstin Burke: Right. What is the one thing, as we wrap up, what is the one thing you think is most misunderstood about this area? So if you were to just leave folks with something, you know, that they might make an assumption about, or they might think that you might just want to set the record straight.
Shahin Pirooz: Yeah. I would say that there’s two things, not just one. The first thing for me is that people really don’t take the time to understand and define what their RTOs and RPOs should be from a risk perspective. So what’s the impact to the company if this thing is down for X amount of hours? That’s the question. And on an application by application perspective, we need to do that. And then build a backup plan for that. Too many times we just back up everything.
Kirstin Burke: Right.
Shahin Pirooz: Right? And you don’t need to. Back ups –
Kirstin Burke: Then you’re frustrated either with the cost, the time –
Shahin Pirooz: And you can’t, you can’t back up everything all the time and do it. You know, if you have a big environment, it’s very expensive to back up. So that’s the first thing.
The second thing is, there’s one of two factors. Either we don’t test restores, which is the confidentiality side, confidentiality means when I get it back, I know that it’s there, it’s functional and it’s we have access to it. So, the last piece of it is, it’s either not tested or we make mistakes. Like, we don’t put it off site. We do the backups on site. We may have a separate backup server, or it might be on the VMware environment like next door servers, or we may be backing up our storage system has backups. We’re going to keep it on storage. And that air gapping and immutability are critical to backups because the first thing a bad actor is going to do is wipe out the backups before they encrypt. The worst thing that’s going to happen to you is the facility gets impacted and you have to restore someplace else. If your backup servers and your backup data is in that same facility, you have the same challenge that it got deleted. So I think those are probably the two big things. I would say it’s not misunderstood, but it’s misused. Not properly figuring out what your RTO RPOs should be and then keeping your backups right next to your servers.
Kirstin Burke: Got it. Well, thank you once again for all the insight. As you can see or hear, DataEndure has different ways to help the organizations that either we work with or that need help. We have different ways to help you assess kind of where you are, where maybe you need to go. So if you’re in a situation where maybe you’re now hearing this and thinking, okay, maybe I need to think about data protection a little different, please reach out to us. We do have several complimentary tools and assessments that are able to help you at least get a handle on where you are and then lay the groundwork for where it would be best for you to go. So thank you. Thank you for joining us. We’ll see you next time.