xz / CVE-2024-3094 — detect, root, and re-image
Malicious xz builds backdoored sshd via a hidden test fixture. You can't 'patch' a backdoor — confirm exposure, then re-image from a known-good source and rotate everything.
01 The Big PictureFor leadership · no jargon
Okay, so this one's a bit special — not a buffer overflow, not a use-after-free, but someone deliberately poisoning the well. xz is a tiny piece of free software that compresses data, and it sits inside almost every Linux computer on earth, including the servers that run your business. Think of it like a lock manufacturer whose deadbolts are installed in nearly every office building by default. In March 2024 it emerged that a volunteer who had spent roughly two years earning the project's trust had used that position to slip a hidden master key into the product.
The master key fits SSH — the front door your engineers use to control servers remotely. On a machine running the tainted versions, the attacker could open that door with no password and no valid key, landing on the server as the most powerful user the machine has, before any login is recorded. In business terms that's full access to whatever the machine holds — customer data, source code, signing keys — plus a launchpad to attack partners and customers, or to plant ransomware. From the inside, it would look like a perfectly normal login. Yikes.
Now, the genuine good news (there is some, I promise): there is no confirmed evidence the key was ever used. A Microsoft engineer stumbled onto it within weeks because his logins were taking half a second too long — which has to be one of the most expensive performance investigations in history — and the tainted versions had only reached the fast-moving, pre-release editions of Linux, weeks away from the stable releases most production systems run. Treat that as a near miss, not a victory: any machine that *did* run the tainted versions must be treated as fully compromised.
Who acts: anyone running bleeding-edge Linux between late February and late March 2024. And here's the part people get wrong — fixing is not a patch. For exposed machines it means rebuilding from scratch using a trusted clean copy, and replacing every password, key, and token the machine ever touched. You cannot scrub a master key out of a lock you no longer trust. For everyone else it's a one-command version check: minutes per server, no downtime.
02 Technical BreakdownFor engineers
Let's get into the nitty-gritty. CVE-2024-3094 is a supply-chain backdoor in xz utils, not a conventional memory-safety bug. A persona ("Jia Tan") accumulated co-maintainer rights over roughly two years, then inserted a two-stage payload that exists only in the release tarballs for 5.6.0 and 5.6.1 — the public git tree was clean, so reviewing the repository showed nothing. A doctored m4/build-to-host.m4 in the tarball executes an obfuscated script at the end of configure, which decompresses data hidden in two committed "test" files (tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-large_compressed.lzma) and injects a prebuilt object into the liblzma build. Hiding your payload in the test fixtures is, I'll grudgingly admit, pretty neat. Horrible, but neat.
So what does the injected object actually do? It hijacks the glibc IFUNC resolvers for liblzma's CRC functions, installs an audit hook in the dynamic linker, and waits for sshd to resolve RSA_public_decrypt — then repoints that function's PLT entry at its own code before the GOT is remapped read-only. Worth noting: sshd does not link liblzma directly. It arrives via libsystemd on distributions that patch OpenSSH for systemd notification, which is most enterprise distros. The build-time preconditions are deliberately narrow to frustrate investigators: x86-64 Linux, glibc, gcc with the GNU linker, and a deb or rpm package build.
Mechanically, exploitation happens during SSH public-key authentication: the hook inspects the RSA modulus presented by the client for a payload verified against an Ed448 key embedded in the backdoor. A valid payload can run a command via system() or rewrite the authentication decision — inside the pre-auth sshd process, which runs as root, leaving no log of a session that never legitimately started. Because the trigger is cryptographically keyed to its author, mass exploitation by third parties was never the risk; targeted, deniable root access was.
Why no in-place fix? Because the payload lives in the compiled library, and whatever implant it may have dropped is unknown. The only sound remediation is to rebuild affected hosts from sources that predate or excise the backdoor — distributions immediately reverted to the 5.4.x series, and upstream shipped a cleaned 5.6.2 weeks later — and to rotate every credential that transited the host during the exposure window. Downgrading the package stops future exposure; only re-imaging evicts an implant that may already be there.
03 Affected
xz / liblzma 5.6.0 (released 2024-02-24) and 5.6.1 (released 2024-03-09), disclosed 2024-03-29. Distributions that shipped them: Fedora Rawhide and Fedora 40 beta, Debian testing/unstable/experimental, Kali Linux (updates installed 2024-03-26 through 2024-03-29), openSUSE Tumbleweed and MicroOS (2024-03-07 through 2024-03-28), and Arch Linux installation media and container/VM images from that window. The hook only fires on x86-64 glibc systems where sshd loads liblzma — in practice, systemd-patched OpenSSH builds.
Stable releases were never exposed: RHEL and its clones, Debian stable, Ubuntu LTS, Amazon Linux, and Alpine all shipped 5.4.x or older. If every host reports 5.4.x, no action is required beyond recording the check. Congratulations, you get to do the paperwork instead of the incident response.
04 Detection
Right, let's find out where we stand. Start here:
xz --versionReport the installed version. 5.6.0 or 5.6.1 means tainted; 5.4.x is clean.
rpm -q xz xz-libs || dpkg -l xz-utils liblzma5Show the exact package versions on disk — remember, the payload lives in the library package (xz-libs / liblzma5), not the CLI.
ldd /usr/sbin/sshd | grep liblzma Confirm the precondition: sshd loads liblzma via libsystemd. Output like liblzma.so.5 => ... means the hook had a path in; no output means sshd never loaded the library.
strings `which xz` | grep -E '.*\.so|ssh|auth'Heuristic tripwire: a clean xz binary has no business referencing ssh, auth, or injected shared objects — any hit is a red flag.
rpm -V xz xz-libs || debsums xz-utils liblzma5Verify tools and library against the signed manifests. Any modified file in the library package is a finding.
05 Remediation
01 Confirm exposure — do not patch in place
Establish the version and the exposure window first, and resist the urge to just downgrade and call it done: if a tainted liblzma ever served internet-facing sshd, an implant may already be on the host, and a package swap leaves it sitting there.
dpkg -s xz-utils liblzma5 | grep -E '^(Package|Version)'Debian/Ubuntu; check both the tools and the library — the payload lives in liblzma5.
rpm -qi xz | grep VersionRHEL-family; 5.6.0 or 5.6.1 on an affected distro — assume compromised and proceed to re-image.
02 Re-image from a known-good golden image
The only trustworthy eviction is a fresh host built from an image that predates the backdoor or carries the cleaned rebuild. If you've got immutable infrastructure, this is where all that upfront faff pays for itself: a root compromise becomes a routine replacement.
terraform taint aws_instance.bastion && terraform applyForce replacement from a pre-backdoor AMI; immutable infra makes this cheap.
knife bootstrap {{node}} -x rootRe-provision onto a clean xz build — pin the base image or package to <5.6.0, or a distro rebuild that excises the backdoor.
03 Rotate every secret that touched the host
sshd ran the payload as root, so anything on disk or in memory during the exposure window is burned: host keys, user keys, deploy credentials, cloud tokens. Rotate from a clean system — never from the suspect host, for hopefully obvious reasons.
vault kv patch secret/ssh/{{host}} key=newRotate host keys, deploy creds, and tokens present during the exposure window.
ssh-keygen -ARegenerate all SSH host keys on the fresh image — never copy the old host's keys forward.
06 Verification
xz --versionConfirm the re-imaged host reports 5.4.x, or a distro rebuild that explicitly excises the backdoor.
debsums xz-utils liblzma5 && echo CLEANDebian/Ubuntu equivalent; debsums stays silent on unmodified files, so the CLEAN marker means no findings.
rpm -V xz xz-libs && echo CLEANIntegrity of tools and library passes against the signed manifest; silence from rpm -V followed by the CLEAN marker means no modified files.
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pubPrint the new host-key fingerprint and compare it to the pre-incident record — it must differ, proving both re-image and key rotation.
07 Rollback
There is no rollback. A compromised host is re-imaged, not downgraded — reverting the package does not remove an implant. If the replacement host misbehaves, roll forward to an older known-good golden image, never back to the suspect one.
08 References
- NVD — CVE-2024-3094
- Andres Freund's original disclosure (oss-security)
- CISA alert — XZ Utils supply chain compromise
- XZ project incident page (tukaani.org)
Spotted something wrong, or got a question about your specific setup? Shout — corrections very welcome :)
Ship the fix. Prove it’s closed.
If this runbook doesn’t fit your stack, send the CVE — an operator writes a verified path, usually within a business day. Or contribute your own and get co-authored credit.