Breaking Good: Where does goodware end, and malware start?
Many legitimate tools, from programming libraries to adversary simulators, can directly be used either for good or for bad. What to do?
AI – Enterprise best practices
Embrace the good bits, avoid the downsides.
If the media player above doesn’t work in your browser,
try clicking here to listen in a new browser tab.
Find TALES FROM THE SOC on Apple Podcasts, Audible, Spotify, Podbean, or via our RSS feed if you use your own audio app. Or download this episode as an MP3 file and listen offline in any audio or video player.
[FX: PHONE DIALS]
[FX: PHONE RINGS, PICKS UP]
ETHEREAL VOICE. Hello, caller.
Get ready for TALES FROM THE SOC.
[FX: DRAMATIC CHORD]
DUCK. Welcome back, everybody, to TALES FROM THE SOC.
I am Paul Ducklin, joined as usual by David Emerson, CTO and Head of Operations at SolCyber.
Hello, David.
DAVID. Hey there.
DUCK. David, this episode is a topic that several customers have said, “Hey, why don’t you record something about this?”
AI – Enterprise best practices.
Any business, organization, charity, whatever – you don’t have to be a multinational, a great big global company.
It’s just where companies are being told, “Thou shalt start using AI more and more, because it’s great.”
How do you do so without ending up in the swamp?
DAVID. In the swamp!
It has been a popular topic, for sure, lately.
DUCK. Let’s start with the topic that seems to dominate at least the social network and the media headlines, namely:
“We’re getting more and more CVEs, more and more vulnerabilities, more and more exploits. And how are we ever going to catch up or keep up? What on earth can we do? “
DAVID. [PENSIVE] What on earth can we do?
Well, I think the first thing to keep in mind is that this technology effectively accelerates all things that were occurring anyway, which is spurious bug reports; which is correct bug reports; findings of esoteric vulnerabilities; as well as findings of the ordinary.
All of these things are merely accelerated.
And to the extent that they are accelerated, I think what we have to start becoming accustomed to is not so much the question of, “Can we do this?” but, “Should we do this? Is this the best use of our time?”
And so, if you are in a position where you’re confronted with a million bugs that always were there, or some of them are hallucinations, or whatever, the actual fundamental truth of triage has not changed, which is you now have a million bugs.
Which of these do you want to solve?
Which of these are actually important?
Some of them may be bugs that are outside the domain of a normal user, meaning they are so esoteric as to be inaccessible to an attacker under normal circumstances.
Those probably aren’t high priority.
Others of those might be hallucinations.
Those are not high priority – triage them out.
And then some of them are legitimate, and some of them are really interesting bugs that nobody else has found, and they are exploitable, and there’s something that you should fix.
Again, that act is triage; that act is not one of despair, just triage of what is being turned out.
And I think that that’s what people are missing – there’s nothing new under the sun, including the need to decide what is the most important thing to work on now.
DUCK. I guess if you look at the last few decades of cybersecurity, every time a potential what you might call “cybercrime accelerator” has come along – think polymorphic viruses; automated malware generation tools; botnets; fuzzing, when that came out and just started churning out bugs․․․
They did provoke a lot of fear, didn’t they?
People go, “Well, maybe we should take these same tools, and put the tools to fight the tools, in this infinite loop.”
But, as you say, that’s not always necessary, is it?
Sometimes, less is more, and maybe just removing the code that has lots of vulnerabilities – that are only there because nobody’s looked at it for 20 years, because hardly anybody’s using it – would be a simpler, faster, and much more effective solution.
Would you agree with that?
DAVID. Yes.
I think that fear, while also being cheap, sells well – fear is going to be marketed.
Also, it might just be human psychology․․․
DUCK. Yes. [LAUGHS]
DAVID. But it doesn’t change the fact that there are constructive ways to view this problem – as a problem of triage, as aforementioned, or in some cases as a problem of contribution.
What luck do we have now, given that if there is a need to optimize something across a codebase that maybe is not well understood by any one person, we have technology which can help us do that?
We have technology which can help us test the ramifications of those sorts of optimizations that we might not previously have been able to do.
So, yes, there’s a lot of AI slop right now in the open-source contribution domain, but there’s also a lot of contributions happening – real actual contributions.
So I don’t necessarily think that it’s all doom.
I think it’s more just a reframing of a problem, because the level of productivity that now is possible has shifted significantly.
DUCK. And for a company that’s not itself involved, say, in IT in general, or in cybersecurity in particular, for whom looking after IT is just a necessary issue like making sure that the plumbing in the office building works․․․ you might as well focus on issues that could affect you and your business directly if you do something wrong.
And the one of those that seems to create a lot of understandable concern at the moment is:
“What if I wake up one day and I’ve gone from one tier of usage to another, and I’ve got exploding AI token bills that I just can’t afford?”
DUCK. It’s the cloud ops problem.
We had this in 2011 with AWS.
DUCK. Yes.
DAVID. People would drive up five-figure bills and not even understand why.
DUCK. Or they’d say, “Oh, I’ll take the free tier,” and then when their service just stopped working, they’d go, “I wonder why?”
DAVID. Yes, yes, the free tier is a real seductress!
The exploding bills problem is a very dull fix – there are spending cap tools that you can use.
If you’re granting API tokens to your developers, break them out by function so that you can say, “This program is costing me this much.”
For example, I have an API token that just runs code review.
I know how much code review cost me – it cost me about a dollar a day; it’s totally worth it.
If that suddenly became $700 a day, well, it couldn’t because there’s a circuit breaker at $20 a day.
And so, a $20 a day code review day, which is expensive but not ruinous, would result in code review stopping, and me or one of the devs getting an email that says, “Hey, you don’t have code review tokens anymore. You allowed me to spend 20 bucks a day on that.”
Those kinds of functions exist.
They are not exciting, they are not technical to set up, but you must set those up – you absolutely must.
AWS has the same things.
DUCK. Yes.
DAVID. They have usage limits.
And actually, in some of their services that are known to be especially problematic, they put fairly sensible usage limits in that might even be a little lower than you would want, but you can always request them to raise the cap.
And that’s the kind of circuit breaker that will protect you.
There are some technical vulnerabilities in AI to additional cost – things like agentic loops, where an agent is just kind-of stuck ruminating on something, and you end up getting a five-figure bill because the agent actually legitimately burned those tokens.
But, again, even that․․․ you have to have a cap of some kind; you have to have an expectation of spend.
DUCK. Well, that’s been in operating systems like Unix since the beginning.
Things like CPU limits, ulimit() settings.
Just so that one person who writes a well-meaning program that figures, “I can find an odd perfect number after all, if only I try harder and harder․․․
DAVID. [LAUGHS]
DUCK. ․․․gets itself backed off a little bit, until someone with a bit of wisdom can come in and intervene.
DAVID. Yes, yes – that’s a good example.
DUCK. That’s really just a business management thing.
It’s kind-of technical, and I guess people are worried, “Well, what if that limit kicks in and it means that I was so close to solving a problem automatically, but now I’ve left this problem open, and attackers are going to get in,” and whatnot.
Well, that’s always going to be a problem if you’re at risk of what’s essentially a denial-of-service attack.
DAVID. It will always be a problem.
Those ulimit() settings came from shared systems – a system that was a shared resource.
DUCK. Yes.
DAVID. This is a similar sort of situation.
This is a system where all your charges roll up into someone’s bill.
Of course, the resources here are significantly less bounded because these AI providers of the major models would love to send you a six-figure bill.
It’s a boring thing that you have to do, because you are trying to all be on the same bill.
I think that’s a pretty straightforward one.
DUCK. Yes.
DAVID. But it’s something you have to think about.
Think, in advance, what you want to spend on AI.
What’s more surprising to me is that some people that I’ve spoken to clearly don’t understand that AI is actually fairly expensive.
DUCK. Yes!
The idea is: it’s kind of free/cheap; sack all the staff; and then employ this stuff that’s created by a company that exists itself to make profit.
DAVID. Yes.
DUCK. They don’t realize that the reason they can solve much more complicated problems with AI now than they could a few years ago is that we have lots more CPU, lots more RAM, to burn through AI tokens.
DAVID. Yes.
DUCK. Like you say, this sort of thing happens because we can do it, not because we should.
DAVID. Exactly.
There was an exploit that made the news in OpenBSD.
It was the OpenBSD/IP stack – I think it was actually just an IPv4 exploit, but whatever.
DUCK. Yes – it had been there for 27 years, or something.
You just send it a packet and you get a denial of service.
DAVID. Exactly.
The part of the headline that didn’t make as much news is that it cost Anthropic, or the researcher that did it – it cost them something like $25,000 or $50,000 in tokens to find that.
That is not free, right? [LAUGHS]
Yes, it’s really impressive․․․
DUCK. And they’re getting mate’s rates, I’m sure․․․
DAVID. Yes. [LAUGHS]
DUCK. ․․․f they’re inside the company.
[LAUGHS] Their $25,000 goes a lot further than yours or mine.
DAVID. That would have been our $100,000, yes.
I’m not saying that AI is not a good deal if you need to do something.
But it isn’t free; it’s not even cheap.
And no, you shouldn’t just do everything in the world with AI because you can.
It’s a resource at the end of the day – there’s a physical reality.
DUCK. So, David, another concern that hopefully at least some people, if not everybody, has learned to have about AI is:
“What happens when you feed your data into the model and it kind-of absorbs it, and could regurgitate it later because it’s become part of collective wisdom?”
How do you manage the risk that somebody with the best will in the world inside your company might feed privileged or protected data into an AI engine and it might pop out later where it shouldn’t be?
This unknown, unknowable data breach risk that could come back to bite you later?
DAVID. This one is the most common question that I’ve received so far.
The number one step is proactive.
You need to get an enterprise plan with a major provider.
If you’re going to be using AI, which I honestly think most companies are going to be, get an enterprise plan with a major provider.
At the very least, get something that, on paper, legally, contractually, doesn’t train on your data, and offers no retention or governs that retention to your tenant.
Proactively get an enterprise subscription, because your company, your employees, are going to use AI at this point.
And if you don’t have something to drive them toward; if you don’t have something that is more compelling than free chat GPT or whatever․․․
․․․they’re just going to use free ChatGPT, which absolutely will train on your data, will ingest your secrets, and maybe divulge them.
That might be more of a hypothetical problem, but it’s definitely not impossible.
And a very large percentage of employees now, according to the data that we have, are in fact using personal AI accounts, personal chatbot accounts, to do work.
That is ridiculous in my mind – that’s your leak.
DUCK. Right.
DAVID. Your leak is not this hypothetical training exploit that, “Maybe there’s guardrails; maybe there’s not.”
Worry about that a little bit – if you’re handling classified information, [LAUGHS] do not put it into a standard Claude!
For the most part, the real vulnerability is that 72% of your employees – if you don’t have an enterprise account that you’re driving them toward that is more compelling to use than free ChatGPT – are just going to use some kind of free chatbot.
And, yes, now you are exposed, for sure.
So your leak is stealth AI or stealth IT, like it always was.
It’s your biggest leak.
It’s dumber than the sophisticated guardrail concern, and it’s solvable by proactive measures, which involve giving them something that they want to use that is also contractually obligated not to be vulnerable to those problems.
DUCK. So what about a sort-of flip side of that?
“Not where you’re training the AI on data you shouldn’t have put in there, but where the AI is, if you like, training you or providing you with information that is inaccurate?”
DAVID. The same way that you manage a human.
People are expecting AI to be deterministic, because, well, that’s how they’re used to thinking about computers.
DUCK. Yes, I see what you mean.
DAVID. If you give a computer 2+2, you’re expecting it to come up with 4.
If you give an AI 2+2, maybe it’ll come up with 4.
Or maybe it will decide that the question you’re asking is part of Gottlob Frege’s theory of numbers, and 2+2 actually is an imaginary․․․ who knows?
DUCK. Maybe you were asking about Python strings instead of numbers. [LAUGHS]
DAVID. Yes, exactly. [LAUGHS]
DUCK. And actually the answer is 22. [LAUGHS]
DAVID. So, it just isn’t deterministic, or at least not at a scale that we can appreciate.
And that’s what causes these sorts of hallucinations or misunderstandings.
And you know what?
They’re a lot like how people work, which is when you ask a person to do something, a well-meaning person will mostly do that thing.
But a well-meaning person might also misunderstand your request.
A well-meaning person might also not do that thing because they forgot, or because they didn’t want to do that thing, or because they ran out of time.
The point is that it doesn’t take malice to do things in a way that is inconsistent.
And that is how AI works․․․ it is associative; it is a mimic.
So, it does things like, “Well, I don’t have a citation for that. I’m going to make a citation because what it seems like they want is a citation.”
And so, here is a citation! [LAUGHS]
DUCK. “If there really were a citation of this sort, the title might quite reasonably be written like this.”
DAVID. Yes. [LAUGHS]
DUCK. It’s not the same as, “A search engine found this exact text on this exact page, and you can click and go and verify that.”
DAVID. There have been a couple cases – I think the legal domain is where you’ve heard a lot about this already, where lawyers will․․․
․․․honestly, I think lazily – not blaming them for using AI, but I am blaming them for not checking the work of the AI.
Because even if you’re going to use AI to prepare your court documents – and by the way, AI is excellent at analyzing contracts; AI is excellent at finding court cases even, because it has a tremendous body of knowledge from which to draw.
But it is not perfect.
And if you aren’t checking your own citations, if you’re not checking the things that it’s telling you, I mean – that’s no different than having an intern or a paralegal prepare all your docs for you and just go into court with whatever they gave you.
It’s going to have a similar outcome.
You got to check the work, just like you check the output of a human.
DUCK. So remembering that non-determinism is very important.
There’s a case working through the system in the UK at the moment where evidence was, if you like, hallucinated or confected by an AI.
I think they just invited the AI to come up with something that would help their case and then went, “Oh, that looks so great! I’ll just repeat it.”
Which is, of course deeply unfair to a defendant in a criminal trial.
DAVID. Yes, that’s no good.
Air Canada was dinged for having a chatbot that was giving wrong information․․․ ultimately, we’re not blaming the chatbot.
That was Air Canada’s chatbot – and if it had been their representative giving wrong information, which absolutely happens, they’re going to be responsible for it.
I think that this is maybe also a boring problem, but it’s one to be aware of, which is that these are not deterministic systems.
They are essentially like asking a human, or an associative engine, to come up with something that you request, generatively.
That doesn’t mean it’s inaccurate all the time, but it also doesn’t mean that it’s provably accurate.
You definitely have to do a little bit of additional research.
DUCK. You have to make sure that it really fits the bill and means what it says.
DAVID. Yes.
DUCK. You probably shouldn’t be surprised if you didn’t put it through the taste test; the viability test; the ethics test; the “is this what the company wants to be known for?” test before you released it.
DAVID. You have to define for yourself how important something is to you.
You know, there are things that you would trust an intern about, and you just run with whatever they come up with.
What you don’t “just trust” the output on is whether or not someone’s going to prison.
DUCK. Yes.
DAVID. You don’t just trust the output on millions of dollars of other people’s money, right? [LAUGHS]
DUCK. [LAUGHS LOUDLY]
DAVID. I think that’s the perspective that’s missing sometimes.
DUCK. Oh dear. [LAUGHS]
We’d need a whole podcast for Web 3.0 blunders, wouldn’t we?
DAVID. Yes. [LAUGHS]
DUCK. “Oh, I thought that that smart contract was inviolable.”
DAVID. Right. [LAUGHS]
DUCK. I wonder where my $674 million, well, other people’s $674 million have gone?
DAVID. Yes.
DUCK. Don’t worry about it or wonder about it afterwards.
Check your facts first.
DAVID. Yes, absolutely.
DUCK. Perhaps we could finish up by me asking you a question of:
“How do we retain, in businesses that are under pressure to use more AI, burn more tokens, be more productive without adding new staff – how do we do all of that, but yet keep the human side of both our businesses and our business culture alive and well?”
So that we’re not just facing this “war of the machines” throwing made up stuff at one another?
DAVID. I can’t pretend to know what it looks like 10 years from now in the workplace.
But, at the moment, there is an absolutely vast surface area in which a human can contribute.
And I think that that is something that is being lost in the messaging of, “Oh, we can replace labor with automation.”
Yes, you can, to a greater degree than you could five years ago.
But, at the end of the day, on the attack side, the most common attacks remain phishing, and ransomware, and really boring things that are not sophisticated or any different than they were 15 years ago.
Likewise, in work day-to-day, we’re dealing with things that are largely administrative in nature, like quality assurance of AI output, or like dealing with the triage of AI tickets.
This, so far, has been an accelerative tool, not a total replacement of humans.
I think, used well, it will remain that way for at least some time.
So, I think that’s what companies need to keep in mind.
Humans need to have strategic but perhaps different duties than they used to have.
There is absolutely still a window here for human contribution in the enterprise, and for value creation by companies.
And I think that window is is something that we haven’t seen closed significantly in the last few years of rapid innovation.
The numbers on AI profitability and on AI productivity remain underwhelming, meaning nobody’s really making money off of this yet.
Nobody’s really proven that it has created a massive bump in productivity yet.
DUCK. Yes, at the moment, there is a sense of quantity over quality.
DAVID. Yes.
DUCK. You certainly see that in vibe coding where you go, “Oh, wow, we used $20,000 of tokens to take an AI engine that had trained itself on the source code of any number of existing browsers to build a whole new browser.”
What if you just used one of the browsers that’s had 20 years of bug fixing and careful design, instead of going, “Let’s just build another one because we can.”
DAVID. Augmented by AI, these applications that already were good can be even better.
And I absolutely agree with that particular observation, which is, you know․․․
․․․a lot of this is just waste heat.
DUCK. [LAUGHS LOUDLY] I know what you mean, yes.
DAVID. You’ve got people that are inventing things that did not need to be invented.
They existed; they needed to be improved, perhaps.
DUCK. We’ve just had the most CVEs fixed in one month on Patch Tuesday, and everyone’s fearful about that/excitable about it.
But, actually, the percentage of ransomware attacks in particular that use exploits is actually going down this year compared to last year.
The percentage that begin with an old-school phishing attack is going up, not down.
So there’s definitely room for that human side, and, I guess, for attention to quality along with churning out quantity.
DAVID. One of the things that we’d considered talking about on this podcast was:
“The RAM shortage․․․”
DUCK. [GUFFAWS] I didn’t want to mention that, David, because I thought it was a bit too far fetched. [LAUGHS]
DAVID. No, I want to mention it.
I think it’s important – it’s in the same intention vein.
Guess what?
There is enough hardware in this world to run us right into the ground, to run us all out of power.
We can’t generate enough power to power all these chips that we have.
What you should be doing instead of lamenting that you can’t buy a 32GB laptop that you’re going to browse the web on, with your web browser that’s way too inefficient, is get into permacomputing or something.
I don’t know if you’ve heard of these groups, but there are permacomputing groups all over the world.
They have some fantastic ideas on how to reuse a ThinkPad T61; they have fantastic ideas on maybe how to compress your images so that they don’t take up 30 megabytes each․․․
DUCK. And maybe, when you’re taking a picture of your buddies in the pub, don’t use burst mode that takes 1000 video frames so that you can get the best one.
Just take a photo, and if it’s no good, delete it.
DAVID. Yes.
DUCK. Life’s too short for saving a million of everything! [LAUGHS]
DAVID. Yes, absolutely.
These are the decisions we’ve never had to make, the intentions we’ve never had to have in the domain of hardware, in the last 40 years.
DUCK. Yes.
DAVID. It has always been an upward climb of ever-greater specification.
That might not always be the case.
We might simply be at the point where we’re not going to be able to consume more power for a bit.
And that’s OK – you can rewrite your software to use less power; you can take pictures that are smaller; you can power your devices differently, using solar, using your own energy, whatever.
There are so many different ways that you can affect this entire ecosystem.
I am not worried about people being unable to afford a 32GB laptop – that is just something you didn’t need to begin with.
You’re not going to get me to listen to low quality audio – I like high quality audio – but I’m willing to pay in the interface, right?
Maybe my interface can be more modest, so that I can spend my cycles on that audio stream instead?
DUCK. Ohhhhhh, I can feel a new podcast episode coming up, David. [LAUGHS]
Slightly off topic, but, “Double-blind testing of David’s audiophile abilities.” [GUFFAWS]
DAVID. [LAUGHS] No, I think a better one is, “What is David willing to put up with in the name of saving resources?”
DUCK. [LAUGHS LOUDLY] I get to choose the music, though!
DAVID. “Does he actually need graphics?”
No, not really.
“Does he need a UI?”
No, not really.
DUCK. Yes, well, that’s a way to save it, right?
You want your super-cool handmade amplifiers; maybe then you don’t need the video track. [LAUGHS]
DAVID. Yes, right.
DUCK. David, thank you so much for your time, and your good-humored look at a thorny problem.
Thanks to everybody who tuned in and listened.
I hope that you have a good idea of some of the things that you can do as an enterprise, as a business, to make sure that you get the best out of AI, and avoid the worst of the problems, such as suddenly burning through lots of money or suddenly trusting something you shouldn’t.
If you like this podcast, please like and share us on social media.
If you listen on a podcast feed, please give us a rating; maybe leave a comment?
Tell your friends, your family, your colleagues, and very definitely your boss about us.
Do pay a visit to the excellent website solcyber.com/blog, where you’ll find lots of community-centered cybersecurity advice and articles.
And․․․
Until next time, stay secure.
DAVID. Bye, everyone.
[FX: CALL ENDS]
Catch up now, or subscribe to find out about new episodes as soon as they come out. Find us on Apple Podcasts, Audible, Spotify, Podbean, or via our RSS feed if you use your own audio app.
Why not ask how SolCyber can help you do cybersecurity in the most human-friendly way? Don’t get stuck behind an ever-expanding convoy of security tools that leave you at the whim of policies and procedures that are dictated by the tools, even though they don’t suit your IT team, your colleagues, or your customers!
Many legitimate tools, from programming libraries to adversary simulators, can directly be used either for good or for bad. What to do?
When governments demand to regulate crypto, is that a practical step, or just intellectual laziness?
Hackers are no longer just external threats—they’re increasingly acting as insider threats by exploiting identities rather than simply targeting endpoint devices.
By subscribing you agree to our Privacy Policy and provide consent to receive updates from our company.
