XZ Utils backdoor
The XZ Utils backdoor was not a coding mistake. Someone who had become a trusted maintainer hid malicious code in two releases of the xz compression library, aimed at SSH. It was caught before it reached most stable Linux releases.
- Affected
- xz 5.6.0 and 5.6.1
- Fixed in
- go back to 5.4.6 (CISA)
- Published
- 2024-03-29
What the backdoor is
The release packages of xz 5.6.0 and 5.6.1 contained hidden code. On some Linux systems liblzma is loaded into the SSH server, and the code tampered with authentication so a holder of the attacker's key could get in.
Spot it in the logs
Which logins point to the backdoor?
No network signature was published, and no exploitation was reported. A Microsoft engineer noticed SSH logins on Debian unstable taking about 0.8 seconds instead of 0.3, and errors under valgrind, and traced them to xz.
How the attack works
- 1Over two years, someone becomes a trusted maintainer of xz
- 2Hidden code ships in the xz 5.6.0 and 5.6.1 release packages
- 3On some Linux systems, liblzma loads into the SSH server
- 4The attacker's key would unlock SSH
Why the XZ backdoor was so severe
NVD and Red Hat both score CVE-2024-3094 10.0, the highest possible CVSS score. It needs no login and no user action, works over the network, and its effects reach beyond the xz library itself, which is why the scope is marked as changed. The goal was the SSH server, the program administrators use to log in to Linux machines remotely. Researchers who studied the code showed that the holder of the attacker's private key could make a backdoored SSH server run any command as root, before any login, and that a successful use left no SSH log entries at the normal levels.
The backdoor was also built to stay hidden. The malicious parts were not in the readable source code: the release packages contained a changed build script, m4/build-to-host.m4, that was missing from the project's git repository, and it pulled a hidden object file out of two binary "test" files during the build. It only built on x86-64 Linux with GCC, during a Debian or RPM package build. At run time it checked that it was running inside sshd and stopped when it detected some debugging setups. Commands had to be signed with an Ed448 key, so nobody else could use the backdoor even after finding it.
XZ Utils is a common Linux compression library. OpenSSH does not use it directly, but Debian, Fedora and several other distributions patch sshd to talk to systemd, and libsystemd loads liblzma. Kaspersky wrote that the operation almost succeeded in compromising SSH servers on a global scale. It was caught first: Debian stable, Red Hat Enterprise Linux, SUSE Linux Enterprise and openSUSE Leap never shipped it.
Is your version affected?
- Linux and macOS
xz --version - Debian and Ubuntu
dpkg -l xz-utils liblzma5 - Red Hat and Fedora
rpm -q xz xz-libs
Affected. Upgrade to 5.4.6.
How to detect the XZ backdoor
The backdoor leaves almost nothing in logs, so detection starts with versions and files, then looks at how sshd behaves.
- Check the installed xz and liblzma versionOnly 5.6.0 and 5.6.1 carry the backdoor. Ask the package manager: Kali advises apt-cache policy liblzma5 and openSUSE advises rpm -q liblzma5. Distribution rollback packages, such as Debian's 5.6.1+really5.4.5-1, are clean even though the number starts with 5.6.1.
- Find which processes load liblzmaOnly systems where sshd loads liblzma, usually through libsystemd, were exposed. Akamai published queries that list every process with liblzma mapped in memory and every installed xz or liblzma package. Vegard Nossum's detect.sh script, linked in Freund's report, checks whether the installed sshd looks vulnerable.
- Scan binaries and source treesBinarly's free scanner checks binaries for the tampered ifunc resolvers, Kaspersky published a YARA rule for the backdoor's code, and Microsoft Defender reports names such as Backdoor:Linux/XZBackdoorBuild. In source trees and build caches, look for the injected line in m4/build-to-host.m4, which only the release tarballs had, and the payload files tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-large_compressed.lzma, which were also committed to git.
- Measure SSH login timeThis is how the backdoor was found. On an affected machine, a failed SSH login took about 0.8 seconds instead of about 0.3, and sshd used more CPU than usual. A sudden slowdown in SSH after an xz update deserves a closer look.
- Watch what sshd startsA command run through the backdoor is started by the sshd process itself, outside any normal login session. Akamai and the xzbot research both point to this: the process tree differs from a normal SSH session, and nothing is logged at the INFO level or above. Endpoint tools that baseline sshd's child processes can flag it.
Check a log line
Search the logs yourself
Which SSH logins were slow? The backdoor added time to each one.
Fix it, in this order
- 1Find every copyCheck the installed xz on every machine. Only 5.6.0 and 5.6.1 carry the backdoor.
- 2Go back to a clean releaseDowngrade to 5.4.6, as CISA advises.
- 3Narrow the exposureKeep the internet away from SSH on those machines, then hunt for anything unusual.
Related vulnerabilities
Timeline
- 2024-03-29Andres Freund reports the backdoor after noticing slow SSH logins
- 2024-03-29It had reached Fedora 40 beta, Fedora Rawhide and Debian unstable
- 2024-05-29Upstream publishes new, clean releases
How the XZ backdoor attack unfolded
None of the published analyses report the backdoor being used against a real system, and Akamai said it saw no scanning for it. The attack that did happen was against the project itself, and it lasted more than two years. A person using the name Jia Tan sent a first small patch to the xz mailing list on 29 October 2021. From April 2022, accounts named Jigar Kumar and Dennis Ens sent messages complaining that patches were slow and that the project needed a new maintainer. The first commit authored by Jia Tan was merged in June 2022, the account was added to the project's GitHub organization in October 2022, and it built its first release, 5.4.2, in March 2023.
In July 2023 Jia Tan turned off ifunc support in the project's builds for OSS-Fuzz, an automated testing service, saying it clashed with a memory checker. Ifunc is a mechanism that picks a function's implementation when a program loads, and the backdoor later hooked in through it. Jia Tan released 5.6.0 on 24 February 2024 and 5.6.1, with a revised backdoor, on 9 March, then pushed distributions to adopt them: an email asked a Fedora maintainer to update Fedora 40, an account named Hans Jansen asked Debian to move to 5.6.1, and Jia Tan filed an Ubuntu request on 28 March, the same day Freund reported the backdoor privately.
The releases reached several fast-moving distributions. Debian had the bad packages in testing, unstable and experimental, versions 5.5.1alpha-0.1 to 5.6.1-1. openSUSE Tumbleweed and MicroOS shipped them from 7 to 28 March 2024, and Kali Linux users who updated between 26 and 29 March got them. Fedora Rawhide was affected, and the Fedora 40 beta contained 5.6.0 builds, although Red Hat said Fedora 40 did not appear to be affected by the actual exploit. On 15 April 2024, the OpenJS Foundation reported similar pressure emails asking for maintainer rights on JavaScript projects, and said the XZ attack may not be isolated.
Who found the XZ backdoor
Andres Freund, a Microsoft employee, found it by accident. Over several weeks on Debian unstable he noticed SSH logins using a lot of CPU and errors from valgrind, a memory checking tool. A failed login that used to take about 0.3 seconds now took about 0.8. He traced the delay to liblzma, reported it privately to Debian security and the closed distros mailing list on 28 March 2024, and published his analysis on the oss-security list on 29 March. Red Hat assigned CVE-2024-3094 the same day, and CISA warned users to go back to an uncompromised version.
The original maintainer, Lasse Collin, did not create the bad releases. He says the 5.6.0 and 5.6.1 tarballs were made and signed by Jia Tan, a co-maintainer. GitHub suspended both accounts and disabled the repository. Collin's account was restored on 2 April, the repositories returned on 9 April, and new clean releases came out on 29 May 2024, together with version 1.0 of his review notes.
Lessons from the XZ backdoor
The XZ backdoor showed that a project can be attacked through its people. A patient attacker, helped by accounts that complained the project needed a new maintainer, earned the trust of its maintainer over two years. The OpenJS and OpenSSF alert that followed lists the warning signs: unknown people pressing hard for maintainer rights, endorsements from other unknown accounts, binary blobs in pull requests, changes to build steps and manufactured urgency. Their main advice is to fund and support maintainers, and to have a second person review every change, even from maintainers.
It also changed what defenders check. The release tarball did not match the git repository, so building from the reviewed source, or comparing the two, would have exposed the injected build script. Binary test files and switched off test features hid the payload. And a distribution's own patch, linking sshd to libsystemd, is what brought a compression library into the login path. Knowing what each critical service actually loads, and keeping an inventory of package versions, is what answers the question "are we affected?"
The case
XZ Backdoor: The Build Servers
- Find every backdoored copy
- Which one faces the internet?
- Roll it back
Questions
Was the XZ backdoor ever used in an attack?
None of the published reports describe exploitation in the wild. It was caught before most stable distributions shipped it.
Why is the fix an older version?
The backdoor came with the 5.6.0 and 5.6.1 releases themselves, so going back to 5.4.6 removes it.
What is the XZ backdoor?
It is malicious code hidden in versions 5.6.0 and 5.6.1 of XZ Utils, a common Linux compression library, tracked as CVE-2024-3094. On some Linux systems liblzma is loaded into the SSH server, and the hidden code changed how SSH checked keys. Someone holding the attacker's private key could then run commands on the server.
Who is Jia Tan?
Jia Tan is the name used by the co-maintainer who built and signed the backdoored 5.6.0 and 5.6.1 releases. The account contributed to xz from October 2021 and gained maintainer rights over the following year. The analyses cited here do not identify who is behind the name.
Has the XZ backdoor been patched?
Yes. Distributions replaced 5.6.0 and 5.6.1 with clean code within days, for example Debian's 5.6.1+really5.4.5-1, and upstream published new clean releases on 29 May 2024. CISA advises moving to an uncompromised version such as 5.4.6.
How do I check if I am affected by the XZ backdoor?
Check the installed xz or liblzma package with your package manager, for example apt-cache policy liblzma5 or rpm -q liblzma5. Only 5.6.0 and 5.6.1 are affected, and mostly on fast-moving distributions such as Debian testing and unstable, Fedora Rawhide, openSUSE Tumbleweed and Kali. Stable releases such as Debian stable and Red Hat Enterprise Linux never shipped them.
Is the XZ backdoor still a threat?
The risk today is low, since the bad versions were removed in March 2024 and no use in the wild has been reported. It can remain on systems built from rolling distributions during those weeks and never updated. openSUSE advised reinstalling exposed Tumbleweed systems and rotating their credentials, because a past compromise is hard to rule out from the machine itself.
Was macOS or Windows affected by the XZ backdoor?
The backdoor only built on x86-64 Linux using GCC during a Debian or RPM package build, according to Andres Freund's analysis. Other systems could install xz 5.6.x without getting a working backdoor. Updating to a clean version is still the simple fix.