flawopen.com/Incidents/XZ Utils backdoor

The XZ Utils backdoor: the supply chain attack that took three years

Critical — CVSS 10.0 CWE-506: Embedded Malicious Code Discovered 29 March 2024
ELI5

A volunteer runs a small but essential town service single-handed, unpaid, for years. A helpful stranger appears and starts assisting. Meanwhile, a stream of other strangers write angry letters complaining the service is too slow and demanding the volunteer accept help. Worn down, the volunteer hands the helpful stranger a set of keys. Two years later, the stranger uses them to install a lock that only they can open. The letter-writers and the helper were the same person.

Key terms on this page
liblzma
The compression library shipped by the XZ Utils project. It is a dependency of an enormous amount of Linux software, often indirectly.
release tarball
The packaged source archive distributions actually build from — which is not byte-identical to the project's git repository. The gap between the two is where this backdoor hid.
IFUNC
An ELF feature allowing a function's implementation to be chosen at load time. The backdoor abused it to hook into the resolution process and redirect a cryptographic routine.

What happened

In late March 2024, Andres Freund — a PostgreSQL developer — was benchmarking on Debian unstable and noticed that SSH logins were taking around 500 milliseconds instead of the usual figure, along with unusual Valgrind errors. He traced the anomaly into liblzma, part of XZ Utils, and found deliberately planted malicious code. He reported it on 28 March 2024; the issue became CVE-2024-3094, scored CVSS 10.0.

The backdoor was present in XZ Utils versions 5.6.0 and 5.6.1. It targeted OpenSSH servers on systems where sshd is linked against liblzma via systemd integration, and was designed to let an attacker holding a specific private key execute commands remotely — a pre-authentication backdoor into a very large share of Linux servers.

It was caught largely by accident, in a pre-release distribution, weeks before the affected versions would have reached stable enterprise releases. Had the timing been slightly different, this would likely rank as the most damaging supply chain compromise ever executed.

The campaign, not the code

The remarkable part of this incident is not the payload. It is the patience of the operation that delivered it.

1. Identify a critical, under-resourced project

XZ Utils was maintained essentially by one volunteer, who had publicly mentioned struggling with the workload. The software was a dependency of an enormous amount of infrastructure. That combination — maximum reach, minimum maintainer capacity — is the target profile.

2. Build trust over roughly two years

An account using the name "Jia Tan" began contributing legitimate, useful patches and gradually took on more responsibility, over a period long enough that no reviewer was thinking in terms of a long-running operation.

3. Apply social pressure using sockpuppets

Separate personas filed complaints and feature demands, pressing the original maintainer over the project's slow pace and pushing for an additional maintainer to be added. The pressure and the solution came from the same source. The maintainer eventually granted co-maintainer status.

4. Hide the payload outside the source code

The malicious logic was not visible in the git repository. It was assembled at build time from binary files disguised as corrupt compression test fixtures, activated by obfuscated modifications to the autoconf build machinery that were present only in the release tarballs — and only triggered during distribution-style package builds, not a clean checkout.

That fourth step is the technical heart of it. Anyone auditing the GitHub repository would have seen nothing. The malicious code existed in the artefact that distributions actually compiled.

WHY REVIEWING THE REPO WASN'T ENOUGH
# What a reviewer reads:
git clone .../xz && less src/liblzma/...   # clean

# What a distribution builds:
tar xf xz-5.6.1.tar.gz && ./configure && make

# The tarball contains build-to-host.m4 changes
# absent from git, which extract a payload from
# files shipped as "corrupt test data" and link
# it into liblzma during packaging.

The lessons that actually transfer

FAQ

Was I affected?

Exposure was concentrated in bleeding-edge and rolling distributions carrying XZ Utils 5.6.0 or 5.6.1 — Fedora Rawhide, Debian testing/unstable, and some others. Most stable enterprise releases never shipped the affected versions. Distributions reverted within roughly a day of disclosure.

Would an SBOM have caught it?

No. A software bill of materials records which components you use, not whether a legitimate component's latest release contains a backdoor. This incident is an argument for build reproducibility and artefact verification, which are different controls.

Has the actor been identified?

No attribution has been publicly confirmed. The sophistication, patience and operational discipline led many analysts to describe it as consistent with a well-resourced actor, but that remains inference rather than established fact.

Related reading

Sources