Join Paul Ducklin and SolCyber CTO Davic Emerson as they talk about the human element in cybersecurity in our new podcast TALES FROM THE SOC.
In this episode, our insightful duo investigate the issue of how far to trust automated cybersecurity advice.
When you come across a threat that seems mundane at first glance, because it’s just “the uninteresting malware that comes before the real malware,” is it OK to let AI try to mop it up on its own?
Don’t miss this wisdom-filled podcast from the experts at SolCyber.
If the media player above doesn’t work in your browser,
try clicking here to listen in a new browser tab.
Find Tales from the SOC on Apple Podcasts, Audible, Spotify, Podbean, or via our RSS feed if you run your own podcatcher app.
Or download this episode as an MP3 file and listen offline in any audio or video player.
[FX: PHONE DIALS]
[FX: PHONE RINGS, PICKS UP]
ETHEREAL VOICE. Hello, caller.
Get ready for “Tales from the SOC”.
[FX: DRAMATIC CHORD]
DUCK. Hello, everybody.
Welcome back to “Tales From the SOC (dramatic music).”
I’m Paul Ducklin, and I’m joined once again by David Emerson, CTO and Head of Operations at SolCyber.
David, welcome back.
DAVID. Thank you.
DUCK. What have you been up to lately?
DAVID. I guess I would like to say something heroic, but it’s actually been quite mundane, which is the best you can hope for in this industry.
DUCK. But sometimes that can be much harder than dealing with the super-dramatic, can’t it?
I know you have a tale of that sort for us…
DAVID. Yes.
The mundane is the hardest part.
Doing something regularly and doing it well is what we espouse, and that’s what will keep you safe ultimately.
We did have an incident.
We had a customer, in the healthcare sector…
DUCK. Oh, dear.
They’ve been really under the pump lately, haven’t they?
DAVID. They have, yes.
They’ve got some issues…
DUCK. I’m sure you got that story from the London hospitals over in the US.
They had to cancel blood tests and they had to cancel surgery, not because the hospital did anything wrong, but the company that they outsourced their blood tests to got a ransomware attack.
So they couldn’t actually process data, and the cooks got away with a whole lot of private stuff as well.
DAVID. They’re not alone.
And the whole industry really is just an entangled mess in that regard: outsourcing, and liability transfer, and whatnot.
It’s really a mess.
Our incident essentially was a ConnectWise vulnerability.
You know, a really common piece of software essentially permitted the malicious propagation of command-and-control software or some kind of a downloader.
It appeared to have no initial action, since it was being deployed through a ConnectWise orchestration tool using a privileged administrative account.
It wasn’t something that was especially drastic in the tools as an alarm.
DUCK. Right.
And that’s the tool you’re supposed to trust to get the updates out to make you more secure.
DAVID. Yes, this was leveraging orchestration software that is intended to improve your security posture.
A well-managed system is more secure intrinsically.
DUCK. It’s certainly the part of the system that you have to trust an awful lot.
DAVID. It is!
It is capable of entirely destroying your environment through malice or through poor configuration. [LAUGHTER]
Yes, absolutely.
DUCK. Well, fortunately in this case, you guys headed off the ultimate disaster, which is great.
So the cooks got so far, but no further.
DAVID. Yes.
I think it’s important to note, as well, part of the detection in this case…
Engaging SolCyber and engaging any any kind of team like a SOC that actually uses humans to solve portions of the problem, rather than pure tools and pure process, involves the ability to have feedback in an incident.
So the customer in this case noticed that things were wrong.
We noticed that things were wrong.
It was not necessarily a pure tool detection, and that’s important because tools are not everything.
Ultimately, humans have far more flexibility.
They have far more ability to synthesize and engage on something like an incident, or something undefined, or an insider threat.
This was not an insider threat in this case…
…but insider threats are actually very often hard to detect, hard to defend against.
DUCK. That’s where someone thinks, “You know what, I’ve gone rogue, or I’ve taken a bribe, or I’ve got an ax to grind with my employer.”
“I’m going to set the official tool to do an unauthorized task, and then I’m going to sit back and cackle.”
DAVID. Yes.
Almost no tools are going to be able to defend against that.
If you were authorized to take that activity or do that action, that’s something that a tool is not going to identify as anomalous.
So,f yes, insider threats…
This almost appeared as an insider threat, and it’s classically difficult to defend against.
DUCK. They were using a vulnerability in this tool that let them basically pretend to be an insider threat, even though they were actually triggering it from outside.
Is that right?
DAVID. Yes.
On over 1000 machines, they had deployed what appeared to be essentially an installer, so it was a prelude to actually deploying a malicious payload.
I would consider the installer a malicious payload, but it didn’t do something intrinsically malicious, right?
It essentially waited for further commands.
DUCK. So, this could be crooks who want to deploy ransomware as their final goal.
Or it could be IABs, initial access brokers: people just getting in there, seeing how far they can go, and then jumping on the dark web and saying, “Hey, 250 healthcare computers! Install what you will! For sale or rent!”
DAVID. Yes, it could have been either.
Our response should be the same, obviously.
DUCK. Absolutely.
DAVID. And so the coordinated response identified a few issues.
Firstly, it was very clear that it came from this ConnectWise vulnerability, a known remote exploit.
A really, really unfortunate thing to have out in the wild, especially in an orchestration tool, but that’s no different than SolarWinds – we’ve seen this before.
So it was important that we that we shut that down.
It was important that we identify the infected machines, shut them down, or remove them from the network.
In some cases, they weren’t running the appropriate endpoint agents.
So that would be anti-malware, or something that would give us the ability to further address some of the remediation steps that were necessary.
DUCK. Oh, dear.
So that’ll be computers that are “running the software” if you look at the license schedule, except that it never actually got installed?
DAVID. Yes, very common, very common.
And this is not a knock – obviously, we’re not identifying the customer or anything – but this is true across the board.
We see, all the time, inventory that is not appropriately configured, even at customers that have SolCyber.
They have access to the tools necessary to protect their environment, but they may not realize that the accounting computer is sitting under someone’s desk, and isn’t registered in their orchestration tool.
They may not realize that they’ve just done a merger or acquisition of another company, and it came with 50 machines and those machines actually aren’t running anti-virus or anti-malware.
DUCK. It’s really hard to get to 100-point-zero percent coverage, isn’t it?
But it’s kind of where you need to be aiming.
DAVID. Right.
DUCK. 98% is really good, but it can still leave you with a giant gaping hole.
DAVID. This issue also crops up notably in some of the most critical systems in a network.
So it’s also important to note that some critical systems, either intentionally or just because of the nature of their segregation as ‘critical’, are unprotected, or protected minimally.
And I sometimes refer to that as “Too Important to Do Anything About,” which is really an unfortunate paradigm, because “too important to do anything about” means that some of the most critical systems are often what ends up being compromised.
As an object lesson in what it looks like if you don’t, look at some of the OT [operational technology] out there, such as manufacturing firms that have CNC machines that run Windows XP.
There is nothing I can do to protect that system at this point.
It hasn’t had security updates in ages; it often doesn’t run modern anti-malware; it often doesn’t run modern detection software in any form.
DUCK. And even if it did, it lacks all those mitigations that are now standard in Windows, where Microsoft finally caught up with things like data execution prevention, ASLR, and all of that.
That stuff can’t be retrofitted to Windows XP, because it was put into Vista instead.
DAVID. Yes.
DUCK. You may not like that, but that’s how Microsoft chose to do it.
DAVID. As a mental exercise, that’s the extreme example.
That’s what happens when you do not patch.
You eventually end up with a system that is intrinsically vulnerable, in ways that you can’t fix anymore.
You can’t protect it simply by having additional software on top of it: that will be deactivated as soon as someone exploits one of the numerous documented remote exploits for that system.
The only real answer at that point is to take it offline.
So, if it’s going to be obsolescent or knowingly unpatched, I hope it’s not available in any form to an attacker that might be in the network otherwise!
DUCK. So, in this particular Tale From the SOC…
…you got there in time, you detected whatever it was – the downloader, the installer, the process hollower.
Was that the end of it?
DAVID. Well, no.
More than 100 machines were affected.
Simple remediations, of course, were possible initially.
Install anti-malware; get the systems under some form of network protection; control their access to the network so that this can’t spread further; get the ConnectWise server offline.
Outside of that, ultimately, this had a large enough impact that we weren’t really going to be able to resolve it without boots on the ground.
And so the next step, really, was getting our customer hooked up with an incident response [IR] firm of some kind.
We maintain IR firms as our partners; we have ones that we recommended.
In this case, we recommended one – they’re wonderful; we love our IR firm partners.
But at the end of the day, you know, IR is much more expensive when it’s done ad hoc, when it’s not under a retainer of some kind.
So we got them a quote for ad hoc IR, for someone that could actually respond to this incident and get them back up and running again, and that quote was expensive.
Even the friends and family discount was expensive, because it would involve tearing resources off of other projects and whatnot.
DUCK. [LAUGHS]
DAVID. But we relayed it to them, and with that quote, because we’re humans and not just part of a giant system that nobody can run off the rails, we asked them if they had any other alternatives.
Because we recognized that this was going to be a lot of money for what we saw as a fairly routine kind of breach.
And we ended up talking to their CFO, and walking through whether they had some kind of cyberinsurance, and if that cyberinsurance had a rider or an IR retainer included.
And it turned out they did, at about a third of the price of what we were able to provide for ad hoc IR.
They actually had a retainer that they could activate.
And we recommended they go with that… use their existing retainer, that they probably didn’t really think about ever since signing that cyberinsurance contract some time ago.
So they did that.
And honestly, that response was sufficient; it was what they needed.
DUCK. That was somebody actually visiting computers and going around and viewing things to make sure that they were correct?
DAVID. It was, yes.
In concert with their staff, because their staff matter, right?
You can’t just have an IR firm and nobody to support them from the actual company.
DUCK. Learning is really important, isn’t it?
We talked about that last time.
When you do something for the 12th time, or the 24th time, it’s a lot easier than the very first time you do it.
This is a great example, isn’t it, of a threat where if you tried to do the “full AI, full automation/closed loop, work everything out and fix it”…
…that would nowhere near have solved the problem, would it?
Because (A) you wouldn’t have patched the hole, so the next crooks or the same crooks would have just wandered in a bit later, and (B) you might not have absolutely found everything, so the crooks might still have an inside foothold.
And (C) you wouldn’t have got an idea of how many moving parts there are that you need to look after in the future.
DAVID. There are so many reasons that you couldn’t have automated this.
Humans are way more flexible; companies are run by humans; these processes are made by humans.
All of that alone would be enough to preclude the comprehensive resolution of this situation with some kind of automation, especially an automation that was not somehow knowledgeable or trained in this particular organization.
And that’s true of all of our customers.
They’re all unique.
When you have over 100 infected machines at a healthcare facility, discussions include things such as, “Which ones can we shut off and not cause a problem?”
A problem that may threaten life, or may threaten the records of someone’s healthcare, or whatnot.
And that’s not a discussion you’re going to have with ChatGPT.
It’s not a discussion you’re going to have even with a more sophisticated actual AI, you know, like the ones that exist in many security, information, and event monitoring tools.
It just isn’t how AI works.
It’s not trained in that sort of synthetic analysis.
But our customer can have that discussion with our SOC.
They can say, “OK, well, what would you do?”
“This is my EHR system; this is the database from my EHR system. Should we take this off the network?”
There’s even discussion around breach notification.
What AI is going to help you decide whether you’ve just experienced a breach or not?
DUCK. [UPBEAT AI-TYPE VOICE] “Have you tried mixing bleach with vinegar?”
DAVID. [LAUGHING] Yes, exactly!
DUCK. “Some people find that very helpful.” [LAUGHS]
DAVID. OK, if you look at the large language models, there are some that could probably give a well-crafted mimicry of a good answer.
DUCK. But would it be an answer that actually fitted your needs?
DAVID. No.
Except in a generic sense, no way!
DUCK. By the way, folks, don’t mix bleach and vinegar.
That was a very poor example of a joke…
I was assuming that everybody realizes that causes a reaction that releases chlorine gas, which is highly reactive and rather toxic.
So don’t do that!
It will clean your sink brilliantly, but you may wake up on the floor with a burned chest.
I thought I’d better say that in case, David. [LAUGHS]
DAVID. [LAUGHS] Too much mustard gas in the kitchen!
That’s never good.
DUCK. It was meant to be facetious, but it was probably not really the kind of thing to joke about.
In the same way that there’s not really any malware that it’s okay to joke about or go, “Oh, it didn’t matter. It wasn’t ransomware. It wasn’t a data stealer.”
But it was the thing that was designed to lead to the very next thing.
So, jumping in quickly in a way that’s sensitive enough that you don’t break what you’ve got, but you head the crooks off at the pass…
…nothing could be really more important than that.
DAVID. It’s true.
Only in hindsight is something ‘fairly harmless’.
In hindsight, much of what was deployed here was fairly harmless, but you don’t know that initially, and you really don’t know that until you’ve done enough analysis to determine that there wasn’t an exfiltration of data, for example.
This was probably a prelude to an exfiltration of data, in this particular example.
DUCK. David, I think that’s a fantastic place on which to end, because it emphasizes the fundamental aspect of SolCyber that I really love, namely that it is a cybersecurity service designed by humans, and run by humans, for humans.
If our listeners are interested in learning more, how do they get hold of you?
How do they get hold of the company?
How do they find out what it costs, and how do they sign up for a demo?
DAVID. Our website is at https://solcyber.com.
If human interaction is your style, just write us at amos@solcyber.com, and you will get an actual human response – I encourage you to try that out.
DUCK. And for those of you wondering why A-M-O-S… that is Amos the Armadillo, the corporate mascot.
DAVID. It’s a banded Armadillo of some kind.
DUCK. It’s a state animal of Texas, which is where SolCyber is headquartered, and it is a great metaphor.
Because it’s an animal that can just move around normally, but it has a shield on its back that helps it to escape from its prey.
As David said, if you do want to get in touch: https://solcyber.com.
If you’d like to read business and technical articles, including community content that isn’t all about sales schpiel, head to https://solcyber.com/blog.
Thanks for listening everybody, and until next time, stay secure!
Catch up now, or subscribe to find out about new episodes as soon as they come out.
Find us on Apple Podcasts, Audible, Spotify, Podbean, or via our RSS feed if you run your own podcatcher app.
By subscribing you agree to our Privacy Policy and provide consent to receive updates from our company.
