SUNBURST: when the build pipeline was the vulnerability

SUNSPOT altered Orion source during builds, placing SUNBURST in signed updates. A valid signature proved origin, not code integrity.

💡 Plain English Explainer (ELI5)

Think of a trusted locksmith that sends sealed, stamped service kits to thousands of buildings. Someone breaks into the factory where the kits are assembled and adds a small extra part to each one before the seal and stamp go on. Every building receives a kit that is genuinely from the locksmith and installs it. The seal was never broken, because the tampering happened before the sealing.

Core Concepts & Subsystem Terms

SUNSPOT: The build-server implant CrowdStrike analysed. It watched for MSBuild.exe building Orion and replaced a source file with the backdoored version for the length of the build.
SUNBURST: The backdoor compiled into SolarWinds.Orion.Core.BusinessLayer.dll. It stayed dormant for up to two weeks, then contacted attacker infrastructure over HTTP.
Code signing: A cryptographic signature showing that a binary came from the key holder. It says nothing about whether the build that produced it was tampered with.
Reproducible build: A build that produces bit-identical output from the same source, so independent builders can check each other's results.
CWE-506: Embedded malicious code. Here the malicious code was inserted during the vendor's own build, not in the source repository.

Framework mappings

CWE reference for the weakness described in this guide: CWE-506 CWE-829
D3-OTF Outbound Traffic Filtering — This guide's outbound network restriction limits which destinations the service or host can reach.

Incident Timeline

2019-09-04

Attackers access SolarWinds' environment; on 12 September they inject test code into an Orion build to check the method works.

2020-02-20

SUNBURST is compiled into Orion builds through SUNSPOT.

2020-03-26

SolarWinds starts shipping Orion hotfixes that unknowingly contain SUNBURST; affected versions include 2019.4 HF5 and 2020.2.

2020-06-04

The attackers remove SUNSPOT from the build environment.

2020-12-13

FireEye publishes its SUNBURST analysis and CISA issues Emergency Directive 21-01 ordering federal agencies to disconnect Orion.

Step-by-Step Attack Flow

Step 1

The build server is compromised

The attackers gained access to SolarWinds' build environment and installed SUNSPOT on the server that compiled Orion.

Step 2

A source file is swapped mid-build

When SUNSPOT saw MSBuild.exe building the Orion solution, it replaced InventoryManager.cs with a backdoored version and restored the original after compilation.

Step 3

The genuine key signs the result

The compiled SolarWinds.Orion.Core.BusinessLayer.dll went through the normal signing step, so the backdoored DLL carried a valid SolarWinds signature.

Step 4

Customers install a trusted update

Fewer than 18,000 customers installed affected updates. The attackers then chose a much smaller set of victims for follow-on intrusion.

Source Code: Flaw vs. Secure Implementation

Illustrative C#, not SolarWinds' pipeline. The vulnerable job trusts a single, long-lived build server and signs whatever it produces, which is exactly what SUNSPOT exploited. The hardened gate requires two builds of the same approved commit on separately administered, freshly provisioned builders, compares their SHA-256 digests in constant time and refuses to sign on any difference. An implant on one builder then produces a mismatch instead of a signed release; it only fails if the attacker controls both builders, which is why they must not share credentials or images.

UNPATCHED FLAW

using System;
using System.Diagnostics;

public static class ReleasePipeline
{
    // Builds from the shared, writable working copy on one build server and signs
    // whatever comes out. Anything that edits the tree mid-build gets signed too.
    public static void BuildAndSign(string project, string outputDll)
    {
        using var build = Process.Start("msbuild.exe", $"\"{project}\" /p:Configuration=Release");
        build!.WaitForExit();
        if (build.ExitCode != 0)
            throw new InvalidOperationException("build failed");

        Signer.Sign(outputDll);   // HSM-backed signing service client
    }
}

HARDENED SECURE PATCH

using System;
using System.IO;
using System.Security.Cryptography;

public static class ReleaseGate
{
    // Two isolated builders compile the same pinned commit from fresh, read-only
    // checkouts with /p:Deterministic=true. The gate signs only identical output.
    public static void VerifyAndSign(string builderAOutput, string builderBOutput,
                                     string builtCommit, string approvedCommit)
    {
        if (!string.Equals(builtCommit, approvedCommit, StringComparison.Ordinal))
            throw new InvalidOperationException("build is not from the approved commit");

        byte[] a = SHA256.HashData(File.ReadAllBytes(builderAOutput));
        byte[] b = SHA256.HashData(File.ReadAllBytes(builderBOutput));
        if (!CryptographicOperations.FixedTimeEquals(a, b))
            throw new InvalidOperationException("independent builds differ; refusing to sign");

        Signer.Sign(builderAOutput, Convert.ToHexString(a));   // digest is logged with the signature
    }
}

Engineering & System Hardening Checklist

✓Protect CI runners like production: they hold source, secrets and signing keys and produce what customers execute, so give them the isolation, access control and audit logging of a production database.
✓Build releases twice on separately administered, freshly provisioned builders with deterministic compilation, and fail the release if the artefact digests differ.
✓Do not treat a valid signature as proof of safety; it proves provenance, which is only as trustworthy as the environment that held the key.
✓Restrict egress from privileged agents. SUNBURST had to reach its command-and-control servers, so monitoring tools that can talk to arbitrary internet hosts should be an explicit, reviewed decision.
✓Size each vendor agent's privileges against what an attacker would gain if it were fully compromised, not against what is convenient to deploy.
✓Keep telemetry the attacker cannot edit: off-host, append-only logs are what make it possible to reconstruct a months-long intrusion afterwards.