What are the Bad Guys actually using it for, versus the Good Guys?
A brief scroll on LinkedIn these days will reveal a commonly held belief that AI-accelerated cyber-attacks are everywhere and that we’re all doomed.
Now, a couple of years ago, this may have been a huge over-exaggeration, but these days there’s a bit more truth to the statement (not the part about being doomed, fortunately!).
So, what should we be worried about? Where does the truth lie?
What we’re seeing in the here and now
From our SOC, we’ve observed several incidents in which LLM-powered coding and content generation were heavily used – and this has become increasingly frequent.
First, we saw high-level threat actors taking advantage of the tools, then middle-of-the-road groups. And now? We’re seeing low-level spammers and phishing operators use vibe-coded landing pages and email scripts.
Below is an example of part of a campaign we’ve seen doing the rounds across several environments and in other companies globally. It uses an LLM-generated web interface that’s far cleaner than the typical credential-harvesting “View Document” scam we’re all used to seeing.
This is one of many campaigns we’ve observed over the past year, at a higher rate than in previous years.
AI is being used in many other, more sophisticated ways, as you will see from recent threat research – but this type of thing is going to be the most common.
Threat actors will use LLMs the way regular users are adopting them, to save time and have their ‘work’ done for them.
What are we doing about it?
That’s a fair question. How are we using AI to defend?
There are a million different ways to use LLM-powered technology to aid in SOC and Incident Response. But fundamentally, there are serious problems with the way LLMs operate regarding integrity and reliability, and, more generally, with how they approach investigations.
The (all-too-familiar) problem is that you can’t rely on AI for hypothesis generation, analysis, or investigative tasks in incident response. Why not? Because it’s a people pleaser: it will always agree with you unless carefully prompted, and it feeds on your bias to stick to certain narratives.
That said, we use AI in several practical ways, leveraging its strengths. Think: time-consuming tasks, processing, deobfuscation, summarising and collating data, searching and collecting data from the internet, and so on.
How do we use AI in our SOC and IR engagements?
We use AI to:
• Summarise tabulated data from SIEM queries
• Search Twitter, Mastodon and BlueSky for news, emerging threats, and things we need to know about
• Generate simple scripts and workflows for data processing
• Deobfuscation of malicious code (to an extent)
• Automate data entry
In data processing, working with custom or unusual data sources during an IR engagement often means writing your own processing scripts and rules to handle them. LLMs make that process considerably less tedious and speed up our ability to respond.
Malware analysis isn’t something we leave to ChatGPT, but working with highly obfuscated scripts and payloads can be time-consuming. AI drastically slashed the time it takes, even allowing for human scrutiny of its output.
In addition to standard threat intelligence collection and monitoring, agentic AI can help crawl and retrieve information from disparate data sources, such as social media, and summarise it cleanly for integration into a standardised threat intelligence platform (TIP).
Where to next?
The industry has moved a long way on this type of thing over the past few years, with many open-source MCPs and projects becoming available for anyone to use.
Every day, someone comes up with a novel way to use AI, or AI improves a bit more, but the reality is that it isn’t likely to change the balance between the good guys and the bad guys.
Yes, we’ve been able to save time, build new capabilities and become more efficient in tackling incidents.
But the bad guys have also been able to save time, build new capabilities, and become more efficient in causing incidents. Importantly, the barrier to entry gets lower when AI is used.
AI or not, we are still getting woken up at 2 am by our on-call system! The only consolation is that the bad guys probably are too.
