Metrics and Measurement
Co-hosts Duck and David ask some tricky cybersecurity questions, including, “How you you know you’re doing the right things to start with?”
If the media player above doesn’t work in your browser,
try clicking here to listen in a new browser tab.
Find TALES FROM THE SOC on Apple Podcasts, Audible, Spotify, Podbean, or via our RSS feed if you use your own audio app. Or download this episode as an MP3 file and listen offline in any audio or video player.
[FX: PHONE DIALS]
[FX: PHONE RINGS, PICKS UP]
ETHEREAL VOICE. Hello, caller.
Get ready for TALES FROM THE SOC.
[FX: DRAMATIC CHORD]
DUCK. Welcome back, everybody, to TALES FROM THE SOC.
I am Paul Ducklin, joined by David Emerson, who is CTO and Head of Operations at SolCyber.
Hello, David.
DAVID. How’s it going?
DUCK. Pretty well.
Spring is starting in England, which is quite pleasant.
Let’s crack on with an intriguing question that is surprisingly important and easy to get all wrong: Metrics and measurements – Do they really matter?
It’s all very well to have a lot of lists of things that might be bad, but how do you decide which things you should be looking for in the first place, whether they really have happened, and what really is the most appropriate thing to do about them when you see them so that you don’t end up running around in a panic?
DAVID. It’s a big topic.
I think we’re probably going to end up going back to a lot of intent, which we’ve discussed many times in the past.
I do definitely believe that reporting and metrics are key to any successful program, particularly in cybersecurity.
DUCK. So how do you deal with what you might call breach fatigue or notification overload?
How do you avoid getting into that situation where it’s just something that you’re so used to hearing that it kind of doesn’t mean anything anymore?
DAVID. That’s kind of starting in the middle of the problem, in a way.
It’s important; it’s something that happens – if you’re collecting enough logs, if you’re collecting enough events, you are going to end up with alert fatigue.
But I think that solving that is really a mid-level problem.
It’s not one of the foremost problems when you’re just starting out building a program.
The reason I say that is this: It probably isn’t your primary concern.
Your primary concern, when you’re trying to build a program, really ought to be splitting the alerts and the events, the things that you will respond to, from the forensic data that you’re collecting.
You should be collecting every log possible in your entire environment.
You should be chucking them all into some kind of retention mechanism for an appropriate length of time to do forensic work after something happens.
You should not be in the mindset that you’re going to be alerting on every single one of those logs.
DUCK. Right.
DAVID. And that kind of second-order concern of, “OK, what events, what alerts do I actually want to respond to?” – that’s where you get into something like alert fatigue.
But when you’re planning a program, in the beginning, you should separate the notion of comprehensive logging from the notion of comprehensive alerting.
And then I think you don’t get to alert fatigue until you’re actually ready to begin maturing the program beyond that initial collection phase.
DUCK. If you raise an alert for every log event, you’re basically just having two logs.
DAVID. Yes.
DUCK. You’ve got one where the data goes, and another that says, “Hey, here’s a link to the data.”
It’s a little bit like the old days, when people who weren’t quite used to email yet would send you an email and they’d then phone you up and say, “Hey, I just sent you an email. Are you going to read it?”
Maybe if you only phone me up when it really, really, really is important, that would be more helpful to all of us?
It’s the same sort of thinking, isn’t it?
DAVID. Yes.
And it’s also not valuable.
Part of the point of these solutions is some kind of intelligence, or some kind of value-add.
And that added value doesn’t exist if there’s just a tremendous amount of noise.
If it’s 1:1 with input and output, then there is no value to that system.
That system is a pass-through.
DUCK. You know what, David, because I know you love literary illusions․․․
․․․this reminds me of a Borges short story, the name of which I’ve forgotten, about the country that became so obsessed with mapping that the Emperor decided they should create a 1:1 scale map of the entire empire.
DAVID. Yes.
Yes!
DUCK. And they are under this sheet of vellum the size of the nation.
DAVID. That’s a wonderful allusion.
And it’s also a really good short story.
DUCK. Can you remember the name of it for our listeners?
DAVID. “On exactitude in science.”
It’s about a paragraph long.
DUCK. [LAUGHING]
Yes, the difference between ‘accuracy’ and ‘precision’ is often very important.
DAVID. It’s very important!
We have to describe it to a lot of our customers.
Sales is going to hate me for doing this on the podcast․․․
DUCK. [LAUGHING]
DAVID. One of the things that our customers love to ask about, prospective customers as well as current customers․․․
․․․they love to talk about “impossible travel”; rules around “impossible travel.”
And for those who are not aware, rules around impossible travel are pretty simple.
If a log entry is seen to geolocate the subject in one place, and then another log entry, a number of hours later which it would be impossible to account for travel via normal conveyance from one location to another, is seen to locate that subject in another place.
DUCK. So that’s, say, “15 o’clock, Paul Ducklin logs in from Oxford.”
DAVID. Right.
DUCK. And then, “At 15:03 o’clock, Paul Ducklin logs in from Pyongyang.”
DAVID. Yes, exactly.
That’s “impossible travel” – it’s very easy to understand.
But it’s actually almost entirely useless in practice in the modern era.
DUCK. Yes.
DAVID. But because it’s easy to understand, customers ask about it when there’s a breach that would have been caught by impossible travel alerts, potentially.
Customers love to say, “Oh, but what about the impossible travel alerts that you should have had?”
And there’s not a lot of understanding until it’s explained that if we actually took action on every impossible travel alert that had been raised in the modern era, I don’t think anybody would get any work done.
You would just have this constant stream of impossible travel alerts, because Paul Ducklin’s traffic is routing all over around the world, like traffic does in today’s era.
DUCK. The average location alert is often worth very little indeed.
DAVID. Yes.
DUCK. I regularly get them via my ISP, or via Facebook or something, which likes not to give you the IP number you connected from, which you can actually do something with, but the place.
It almost never places me in the city where I live.
Very frequently, it will put me in Wales, which is not far away by American standards, but we’re talking hundreds of kilometers, and hours of driving.
And equally frequently, in Scotland – in places that are nearly 1000 kilometers away, and 10 hours of driving if you’re lucky.
How much worse when you’re using content delivery networks, or a VPN, or load balancers, and things like that that are designed to spread your load around the world for efficiency and reliability.
DAVID. Oh, it’s utterly useless, yes.
In my case, the connection I’m coming to you from is located in New York City.
I’m physically in Washington, DC – that’s an artifact of the co-working space that I’m in.
If I were to log in to my Office 365 from this computer here, it would locate me somewhere in New York.
And then if I switch to my phone on a cell tower, it would locate me somewhere in Washington, DC.
DUCK. Yes.
So “Pyongyang” might be a good indicator, and that might be worth looking at, but panicking every time there’s something that simply looks slightly different may or may not be meaningful.
It’s your job to build measuring tools, and processes, and procedures that mean that you are very likely to react to the ones that matter, but much less likely to waste time looking at the things that don’t.
Risk assessment on an ongoing basis.
Is that fair to say?
DAVID. Yes, it is.
You have to build a system that realizes value from the tools that you have in your environment; from the measurements that you are capable of making.
And it can’t be a 1:1 representation of those tools and measurements, or very rapidly it descends into the chaos of unfiltered, low-value alerts.
DUCK. So what are some other examples other than just “impossible travel,” which I’m sure we’ve all experienced at some time?
DAVID. Another example, I would say, is more strategic.
At the project management and strategy level of a cybersecurity program, it has to have metrics that it uses as indicators of success; as key indicators that the program is working.
DUCK. Right.
DAVID. And there are some of these that are common that you probably have heard of: “mean time to response,” or “mean time to resolution.”
Many of those I’m not super-big on, but there are some really practical ones, such as inventory counts and what coverage in your inventory do you have?
DUCK. So that’s trying to measure something like, “What is the chance that if I go to an auditor and say, ‘I don’t have any Windows XP anywhere in my environment,’ that you are, say, 99.9% right”?
DAVID. That would be a very convenient metric to have as an indicator if you felt that was appropriate for your environment.
DUCK. So you don’t have to be perfect, but you have to have some idea of whether you’re just ticking a box, or you’re actually coming up with a good mixture of accuracy and precision.
DAVID. My guiding principle, if you’re implementing a program here, is to use tools, not tool-shaped objects.
DUCK. [LAUGHS]
DAVID. Especially in cybersecurity, there is a tendency to have the operation of the tool be the output, and that is not the case.
DUCK. And to have the tool be a hammer. [LAUGHTER]
DAVID. Yes.
It’s got to be something other than the operation of the tool that is the valuable activity, or else that tool is just a tool-shaped object.
Even useful things can be tool-shaped objects in the wrong hands, and those hands often are hands that have not found a way to bind the operation of that tool to something which matters in real life, or in real business, to the enterprise.
And those are your metrics.
Your metrics should in a way reflect the tool’s link to the operational success of the business.
Metrics are a wonderful way to bridge that gap.
That’s a guiding principle for me.
When you’re thinking about your KPIs; when you’re thinking about the metrics of your program – they are the things that derive value from the tools you have, and the things that avoid those tools being merely tools.
That avoids them being just tool-shaped objects that you’re operating for your own sake, but not necessarily getting some kind of an outcome out of.
DUCK. How do you avoid accidentally ending up back with that 1:1 map?
DAVID. Discretion is one way.
Being able to differentiate between “We didn’t catch this alert or this breach because we don’t have enough defense in depth,” and “We didn’t catch this alert or this breach because we didn’t have impossible travel enabled.”
It’s too easy; it’s too facile to look at the alerts that you didn’t get and assume that if you’d gotten one of those alerts, you would automatically have stopped this breach that has occurred, or this incident that has occurred.
Maybe it’s not always a breach; it might be an outage.
That leads you inevitably down this path of 1:1 representation, and it is inappropriate.
DUCK. That’s the “more tools, more tools” problem, isn’t it?
DAVID. Yes.
More tools, more alerts.
DUCK. And that’s no good if all that’s going to happen is it leaves you struggling even to tread water.
DAVID. Yes.
Sometimes this takes higher-order thought – something like risk management, true risk management, where you’ve defined the things that are important to you, you’ve defined what constitutes valuable things or valuable data to your company, or valuable processes that must operate.
DUCK. So that’s almost a sort of metric for the metrics.
Deciding which ones you can simply do without, which ones that you will only look at if certain other things happen first, and so on.
DAVID. It’s at least a hierarchical pyramid.
DUCK. [INTERRUPTS BY LAUGHING] Sorry for laughing, David.
I’m just trying to imagine a non-hierarchical pyramid․․․․ that would be a cool shape.
DAVID. [LAUGHTER] Well, you know, it’s not a Ponzi scheme․․․ I guess I’m sort of differentiating it from a Ponzi, or something.
It’s a pyramid of the activities that feed up into other activities, that feed up into other activities, that feed up into success.
And so what I’m advocating here is not linking the successful operation of a critical process to something as entirely irrelevant and common as impossible travel.
DUCK. Right.
DAVID. If you find that you’ve been breached, and that the client list for some critical system has been exfiltrated, and that’s a big deal to your company, it probably isn’t because you missed an impossible travel alert.
It’s probably because you missed numerous other alerts, or maybe you had an improperly-defined security protocol or security posture around that data.
It’s not because of any one minor alert, and getting into that mindset where you’re not trying to solve this by some kind of sensible link with critical data.
I think that you can have success defining what it looks like to protect a piece of critical data.
But you just have to have that higher-order thought.
You can’t be like, “Oh, we missed it because this person logged in from Pyongyang.”
That isn’t the reason you missed it.
You probably missed it because of 19 other things that didn’t nest together properly into some kind of defense in depth.
There wasn’t a single tiny metric that caused this.
If anyone listening has ever worked in a SCIF, a highly secure environment, right?
DUCK. S-C-I-F?
DAVID. Yes, S-C-I-F.
DUCK. Oh, I thought you meant S-K-I-F-F, as in a rowing boat. [LAUGHS]
DAVID. [LAUGHING] Oh, no, that would probably be more connected than the average skiff.
DUCK. [MORE LAUGHTER] I thought you were going to tell us about the importance of not catching crabs, and listening to your coxswain and stuff.
An S-C-I-F. [Sensitive Compartmented Information Facility]
DAVID. It’s a very inconvenient place to work, at the end of the day.
It’s a place disconnected from wireless, from probably even some wired connections.
These are conference rooms essentially that are just highly secure, and intended for the operation of a particular system scope, or intended for meetings.
DUCK. So this is one of those places where there’s a little safe outside that you put your mobile phone in?
DAVID. Yes.
DUCK. And a place where you store your rucksack, and then you get searched and then you’re allowed in?
DAVID. Exactly, exactly.
And they are wildly inconvenient to exist in.
DUCK. Yes.
DAVID. Your internet is not available, or changes nature drastically when you enter them.
And that’s a really good mental exercise for what life would look like if you actually had control over a bunch of these really tiny threat vectors.
The reality is that most problems don’t require that sort of comprehensive solution.
For the most part, it’s far more important, in today’s modern society, that we have activity; that we have facility; that we have the ability to work through a problem collaboratively, using the internet and so on.
And so to your point, no, it really isn’t helpful to say, “Every single access to this application is going to happen from this particular chair.”
That is unlikely to lead to a flexible, operational system.
But in concert with other metrics, it’s entirely possible that it is an important data point – it just isn’t going to be a 1:1 alert.
“Oh, someone’s not in that chair and is doing work”․․․ not useful information for most systems.
DUCK. It sounds like really what you’re talking about is a system where you are continually reviewing, continually revisiting.
And you’re adapting your processes and procedures to give you good flexibility and a little bit of risk, but a small enough amount that you feel that you can manage it, and stay within the bounds of all the compliances that you need to have.
How do you go about doing that?
Where do you start if you haven’t done so already?
DAVID. Defining the thing that you’re trying to protect.
Because if what you’re trying to protect is a state secret, is of national security scope, the appropriate activities around protecting that data are very different than if what you’re trying to protect is a patent, or if what you’re trying to protect is communications between you and me.
All of those things are probably deserving of *some* level of protection.
DUCK. Thank you, David. [LAUGHS]
DAVID. But to varying degrees.
We are recording this podcast; it’s going to be public.
Clearly, it isn’t something that we’re going to be that broken up about if someone else finds their way into the unedited version of this.
There’s not a huge difference between the unedited version of this, and the edited public podcast.
DUCK. Adapting your behavior to suit the level of security that you think you can afford and is reasonable is also a perfectly acceptable thing to do.
An example I know you’ve given before is if you suddenly think, “Oh, golly, it’s far too expensive for me to become HIPAA compliant,” then if you’re not deep in the middle of the health care industry, maybe one way out of that is there’s some data that you no longer need to collect․․․ so stop doing it!
Then you don’t have to worry about it at all.
DAVID. Right.
That is a strategic activity; that is knowing yourself, knowing your data, understanding the risk that you’re assuming, and also understanding the places where you should simply opt out.
Understanding the data that you shouldn’t take on, or the activities you should not engage in, because they’re going to cause too much operational overhead for your particular situation.
I think that’s the most important part.
To begin not in the middle with, “How do we not have alert fatigue,” but to begin at one of the ends.
If you’re in need of tools and metrics, and you’re just starting out, begin at the beginning.
If you’re a program that already exists, I would advocate to begin at a much higher level, which is to say, begin with the crown jewels.
What are the things you’re trying to protect?
What are the operation you’re trying to ensure continues to operate?
Depending on how critical that is, and depending on the ways in which you need to protect that particular aspect of your enterprise, the alerts around that, and the level of acceptable noise, and the funding that you need in order to triage those alerts and events will be more apparent to you upon observation.
DUCK. Sometimes the best way to protect the things that you think are the most difficult might be not to do those things anymore at all, because they don’t bring enough benefit to match the risk that they introduce, or the cost they introduce for your organization.
Then you’ve got a lot more time to do all the other security things that can help you protect the stuff that you actually can’t live without in your business.
DAVID. Some of what you’re indicating there is also a cultural problem.
DUCK. Yes.
DAVID. A lot of security programs don’t feel empowered to push back on the activities of the business in a way that may be appropriate and beneficial.
I’m not talking about pushing back on, “Oh, you should secure this or you should secure that.”
That is table stakes.
But pushing back on the actual strategic operations of the business is something that should be more commonplace.
That is to say, if I’m a security practitioner, and I notice that we aren’t encrypting the database, maybe the answer is, “Let’s encrypt that database and take the performance impact and whatever,” but also maybe, “I’m failing to see a much broader problem, which is why do we have that data in the first place?”
DAVID. Yes.
If nobody thought it was worth encrypting, why was it even worth collecting?
DAVID. Right.
DUCK. Because, surely, if it was worth that much to you, you would have crossed that bridge already, as a natural part of deciding to retain the data?
DAVID. Exactly.
And that is something I see too infrequently.
It goes back to “using tools, not tool-shaped projects.”
People have this ways of engaging with the world that oftentimes involve more, more, more, and additional apparatus around existing business processes.
It works, to a certain extent, but it’s expensive.
It’s operationally cumbersome.
Sometimes, the better question is, “Why are we doing this activity at all? Why is this a thing that our business needs to do?”
Maybe it does need to be done, but if nobody has ever asked that question, you won’t really have a sane mapping of your events, your alerts, your procedures, your policies, everything․․․
․․․all up to revenue, or all up to value, or all up to assets that need protection, whatever it is for your particular enterprise.
DUCK. And even if you decide, “Well, I’m going to get somebody else to do this for me, or to help me do it,” or if you decide, “This is too difficult for me,” or, “This is not core to my business, so I’m going to sign a deal with SolCyber and I’ll let them look after it”․․․
․․․that doesn’t absolve you from the need to describe what it is that you’re trying to do, but it does free you up to take advice in perhaps changing your business operations.
You have to be willing to hear that, perhaps, some of the things you’re doing, you shouldn’t be doing at all.
And to take that, if you like, as an opportunity to do things that you weren’t doing before that could actually help you grow your business!
DAVID. Yes.
At the very least, even in the case of signing up a service provider that will now provide managed security services to you, you still need to know what it is your business does; what systems are critical; what people are critical; what processes must not be interrupted.
If you don’t know those things, applying a generic skim of rules across every single business aspect is going to result in very low-fidelity alerting, either in the direction of excessively noisy, or in the direction of entirely ineffectual and quiet, because “we don’t really know.”
That’s table stakes, to know what your business does; to know what’s important to your revenue; to your security; to your operational reliability.
DUCK. As we’ve․․․ well, *I’ve* certainly said before, because you know I love my cliches, David: “If you can’t measure it, you can’t manage it.”
But perhaps even more importantly in this case, “Less can very definitely be more.”
You don’t have to keep on doing all the things you’ve been doing, with extra security layered on top, if some of those things aren’t necessary at all.
DAVID. Yes, that’s absolutely the case.
And I see all the time that people have done that.
DUCK. David, thank you so much for your very thoughtful and intellectually-focused words.
Some of the things you’ve said are not necessarily wjat everyone wants to hear, because they just like to turn on all those tools and have it all happen automatically.
But as you said, it really is about building a culture where security works for you in your business, and for your customers, and for the regulators, and for the greater good of all.
So, David, thanks to you, and thanks to everybody who tuned in and listened.
If you like this podcast, please subscribe in your favorite podcast feed.
Please like and share us on social media – that really helps us a lot.
Please pay a visit to solcyber.com/blog, where you’ll find loads of community-focused advice, not sales spiel, helping you deal with some of the things that David’s been talking about today.
And remember everybody․․․
Until next time, stay secure.
[FX: CALL ENDS]
Catch up now, or subscribe to find out about new episodes as soon as they come out. Find us on Apple Podcasts, Audible, Spotify, Podbean, or via our RSS feed if you use your own audio app.
Why not ask how SolCyber can help you do cybersecurity in the most human-friendly way? Don’t get stuck behind an ever-expanding convoy of security tools that leave you at the whim of policies and procedures that are dictated by the tools, even though they don’t suit your IT team, your colleagues, or your customers!
By subscribing you agree to our Privacy Policy and provide consent to receive updates from our company.
