All your codebase are belong to us – Learning from the “AI malware” s1ngularity hack

When trust fails

In the cult 8-bit video game Zero Wing, the alien invaders open the gameplay with the taunt that ALL YOUR BASE ARE BELONG TO US.

The deep-space shoot-em-up that follows involves saving the Milky Way Federation from a ruthless gang of space pirates.

That ungrammatical but memorable opening salvo has become an widely-quoted meme, used sometimes for boasts, but more punchily for counter-boasts when audacious and aggressive bullies who took triumph for granted meet their match.

The cybersecurity tale that follows has the same ring of piratical near-disaster followed by rescue, with a dramatic dollop of AI mixed in as well.

This story has become known as the s1ngularity incident, after the name that the attackers used for the GitHub repository to which they exfiltrated stolen data.

The singularity, in popular jargon, refers to the dystopian moment in many science fiction stories when an artificial intelligence system that’s already close to human levels of ability gets into a runaway state of self-improvement.

At this point, the AI not only exceeds human abilities for the first time, but also continues to advance at high speed, leaving us behind intellectually forever, and turning itself into our eternal robot overlord.

We’re assuming that the cybercriminals chose this name because AI is intriguingly intertwined into the attack:

• An AI-generated code change in the affected product introduced a remote code execution bug that was, for better or worse, disclosed publicly via Twitter rather than privately to the project maintainers.
• The malware ultimately implanted by the attackers included a script to find files worth stealing based on their names (cryptographic keys and cryptocoin wallets, basically). Instead of using a pre-programmed search that would be the same every time, the malware cheekily used a pre-programmed AI prompt with one of three different AI engines to generate a fresh search command on each computer it infected.
• The product that was compromised explicitly describes itself as software build tool created “for developers and AI agents.”

Poisoning the well

This was a classic supply chain attack, meaning that instead of targeting the software packages of hundreds or thousands of different vendors, the criminals targeted a single software project that thousands of vendors use to build their products.

Simply put, the criminals poured poison into the well instead of trying to infiltrate poison into every household in the village.

The afflicted product is a popular software build automation tool called Nx that aims to improve software development in organizations that practice what’s known as CI/CD, short for continuous integration/continuous development.

In this development model, you allow your programmers to make continuous changes, some large and some small, rebuilding and re-testing the product comprehensively and automatically after every change, perhaps many times each day.

Furthermore, by using what’s commonly referred to as agentic AI, speed-hungry software vendors can respond faster than ever to so-called change requests from bug-hunters, customers, and their business development team.

Change requests are usually known these days as PRs, or pull requests, after the pull command in source code control systems, notably Git, that approves a change and pulls it from a development siding onto the mainline software track.

Because the pull process for any group of proposed changes can be triggered by a single command of the form git-pull ..., the operation can easily be automated, for example by trusting an agentic AI to do it by itself without human approval.

What went wrong

The Nx team approved a pull request documented as Add GitHub Actions workflow to validate PR title, apparently created and approved by “Agent Ender and claude,” where Agent Ender is a coder’s nickname and Claude is Anthropic’s generative AI system.

Unfortunately, the change, rather obviously with hindsight, includes a command injection bug, similar in nature to the infamous Log4Shell bug from late 2021, where simple text messages sent to Java programs for reporting purposes could be hijacked to run programs while the message was being processed.

Claude’s, or perhaps Ender’s, coding glitch was to take the title of a proposed pull request, which ought to be treated as plain old text, and to feed it directly to the operating system shell, which is responsible for turning text into commands so they can be executed.

The problem here is that the shell doesn’t treat all text strings the same.

For example, the command․․․

$ echo My name is whoami

․․․runs the echo command, which prints out what follows, so you get the output My name is whoami.

But if you change things a bit and write this instead․․․

$ echo My name is $(whoami)

․․․then the $(...) characters trigger what is known as command substitution.

Tthe shell first strips out the magic sequence $(...), extracts the string whoami from the middle, runs it as a system command (it returns your username), and substitutes the text that comes back from the command instead, producing the output My name is duck, or whatever the username of the the current process happens to be.

In the Nx software, this sort of unsafe processing of otherwise innocent input meant that the system could be tricked into running a command sneakily chosen by the person who submitted the PR in the first place.

To be fair to the Nx project, the team spotted the bug report, started working on it within six hours of it being publicly disclosed on Twitter, and published a full-and-frank timeline of their security response.

(Companies that have in the past responded to breaches with vague, brief statements and reassurances based on hope and not evidence are strongly advised to study this disclosure.)

Sadly, we have to assume that cybercriminals saw the public message on Twitter as well, and used it as an incentive to start looking into the bug and the code surrounding it, hoping to find related holes that could be used for malevolent purposes until patches were created and rolled out.

If you know where to start hunting for security holes, the process of developing an exploit is often greatly simplified, in the same way that it is easier to find a needle in a haystack if someone tells you exactly which bale it’s in first.

The attack sequence

The s1ngularity attackers quickly launched an attack that went like this:

• Submit a booby-trapped PR to run a rogue command. This command couldn’t publish or authorize software updates itself, but it could set up a command to trigger web requests (HTTP POST commands) to arbitrary external servers.
• Send the Nx software’s npmToken to a free public datacatcher site. This token acts as a magic security pass to signal that your username, password, and 2FA code have already been verified and you are now duly authorized to access the system. The datacatcher used was webhook.site, part of a popular online service that provides free cloud-based collection for data transmitted using HTTPS, DNS or email.
• Use the stolen authentication token to modify and publish updates to several packages in the Nx repository.

Customers of Nx who used any of the poisoned packages in their own build systems, and whose servers were set to update automatically, quickly fetched these updated packages and installed them on their own servers or developers’ laptops.

As we often see these days in supply chain attacks, the poisoned packages didn’t directly try to sabotage all future software builds of their victims, which might have attracted extra attention.

Instead, the malware was injected into an existing and apparently harmless script called telemetry.js, which ran just once after a successful update, presumably so that the Nx team could keep track of which versions were in use where.

After the toxic update, the rest of the Nx software appeared perfectly normal, and any builds carried out by the software would come out correct and untampered, so the update seemed unexceptionable.

However, the malicious telemetry.js file does all its dirty work during the Nx update itself, where it:

• Gives up if the user is running Windows, thus focusing on servers and laptops running Linux/Unix or macOS. (Note that the Windows Subsystem for Linux, or WSL, is reported as linux, not as win32.)
• Generates a list of security-related data files on the computer, including a wide range of filenames including the text keystore, wallet, trezor, solflare, id_rsa, Local Storage, and IndexedDB. These cover common locations where cryptographic keys, cryptocoin wallets, and browser authentication tokens are stored.

Instead of directly using traditional Unix command line tools such as find or ripgrep, the malware looks for an installed AI engine that is one of Anthropic’s Claude, Google’s Gemini, or Amazon’s Q.

As mentioned above, instead of using a pre-programmed search command that would be the same every time, the malware uses a pre-programmed AI prompt to generate a fresh search command every time it runs.

One explanation for this roundabout approach is that the attackers aimed to end up with a slightly different search script each time, and thereby to evade basic threat monitoring tools that are programmed to look out for specific commands used in very specific ways.

However, the pre-programmed AI prompt doesn’t rewrite itself every time, and is as much of a tell-tale as a pre-programmed search command would be.

It’s therefore also possible that the attackers were simply showing off here, using AI to ensure dramatic media headlines when the story broke.

After building a list of files of interest, the malware:

• Reads in all the files found, and base64-encodes the data three times over as a crude anti-analysis and anti-detection trick.
• Uploads the data to a now-defunct GitHub repository with the name s1ngularity-project. Stolen data could include an eclectic mixture of GitHub keys, SSH keys, browser cookies and authentication tokens, as well any cryptocurrency wallets lying around. Presumably the attackers didn’t expect to find cryptocurrency stashes on company servers, but assumed they might get lucky whenever their telemetry.js malware ran on a developer’s laptop.

The long and winding road

At you can see, the attackers weren’t directly poisoning their victims’ own software builds, or their build servers.

Instead, they were collecting access tokens that would let them back in for much more general acts of sabotage and criminality next time.

They also grabbed cryptocoinage if they could, although password-protected wallets wouldn’t be much use unless the attackers found and stole the relevant decryption keys at the same time.

According to some researchers, a second wave of attacks soon followed, using GitHub access credentials stolen in the first attack to log in to victims’ own GitHub repositories.

These follow-up attacks rather crudely and obviously renamed victims’ private repositories and made them public under names of the form s1ngularity-repository-XXXXX, where X denotes a random character.

As you can imagine, this attack was easy to detect and remediate once it was known, given that the hacked repositories had obvious names and were made public.

Until the follow-up hack was reported, however, not only the attackers but anyone else who happened to know what to look for could find the private data and copy it at will.

Whether this part of the attack was carried out by the same criminals for their own ends, or by other attackers acting out of spite or just to show off is not known.

What to do?

• Take a look at the Nx team’s breach disclosure documents. Even if you don’t use their products, and even if you aren’t involved in software development, there’s plenty to learn from the refreshingly honest and useful nature of the response.
• If you run software from a vendor that uses Nx products, consider asking them what they have done to detect and remediate any exploitation that may have happened because of this attack.
• If you create software yourself, using Nx products or any other CI/CD tools, consider reviewing your own processes to identify weak points where risky decisions are most likely.
• If you deliver software updates automatically yourself, consider adopting Nx’s newly-added precaution that requires human authorization at key stages, ideally involving some form of 2FA that can’t be automated and thereby creates a checkpoint to help limit compromise.
• Don’t be sidetracked by the AI angle in the malware. Remember that malware infection means that an attacker can do almost anything that you could, including running the same AI tools that you do, stealing your cryptocoins, posting to your social media accounts, reading and sending emails in your name, and rummaging through any private or company-related information that you can access yourself.
• Get help if you need it. Stay on top of cyberthreats without distracting staff from your core business. Sign up with SolCyber to do it for you, human style!

Why not ask how SolCyber can help you do cybersecurity in the most human-friendly way? Don’t get stuck behind an ever-expanding convoy of security tools that leave you at the whim of policies and procedures that are dictated by the tools, even though they don’t suit your IT team, your colleagues, or your customers!

More About Duck

Paul Ducklin is a respected expert with more than 30 years of experience as a programmer, reverser, researcher and educator in the cybersecurity industry. Duck, as he is known, is also a globally respected writer, presenter and podcaster with an unmatched knack for explaining even the most complex technical issues in plain English. Read, learn, enjoy!