An interesting article has been making the rounds recently: Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies. The author describes how he “hacked” a bunch of companies using a technique dubbed “dependency confusion.”

In this article, I’ll share how we laughed off a “dependency confusion” attack on our supply chain, and how you can completely prevent these sorts of attacks on yours.

This particular dependency chain attack isn’t all that new or novel, but it’s certainly clever. It’s an attack aimed at the so-called “software supply chain”—specifically the components (packages and libraries) that are used to build applications.

In the physical world, a supply chain attack might be like breaking into a safe by first going after an easier target: the company that manufactures the hinges on the door. Sneak a vulnerability that you know how to exploit into the component and then next thing you know, open sesame.

What does a “dependency confusion attack” look like?

I can give you a real-life example of this “dependency confusion attack”: someone actually tried the dependency confusion attack on us! Here’s what they did:

1. Discover the name of an internal library; the fact that we use InedoLib is already public information, and it might take a bit more effort to find the name of an internal library at another company
2. Create a fake version of that library; it’s pretty easy to create a compromised InedoLib, since anyone can just download a real version and then construct a fake one based on that; it’s a bit more effort at a different company
3. Upload your compromised library to NuGet.org; someone created an account and uploaded InedoLib v999.0.0, which is a fake version of the real InedoLib that looks like an update
4. Hope that a developer is duped by the package; this is where the attack totally failed against us; it was basically like a squirt gun against a tank

The fake InedoLib we were “attacked” with was innocuous. It’s an empty package with a readme.txt containing a “dependency confusion test” and some links to an article. But it could have contained a modified version of a real InedoLib with some really nasty code in it.

How did we fare against the dependency confusion attack on InedoLib?

Being on the receiving end of a “dependency confusion” attack felt a bit baffling, but it was mostly entertaining. Kind of like that feeling when you get a spam email from a Nigerian prince offloading his inheritance.

It made me ask a lot of questions though.

Would anyone actually fall for this? What developer in their right mind would trust NuGet.org for internal library packages? Who would just arbitrarily upgrade InedoLib to v999 without seeing what changed?

And then I realized… apparently the developers at Spotify fell for it. As did Apple. And PayPal. And then a whole bunch more. Whoops.

Really smart developers do really dumb things

How could such brilliant developers—world-class, in fact—fall for such obvious attacks as these? It’s the same way that intelligent people get tricked by social engineering attacks: they lack basic training.

It may be common sense to you that alex_inedo_com@gmail.com is not my email address. But to untrained employees, it might seem like it’s the CEO’s emergency email account. And that he’s using it when he’s on the road and his laptop broke. And that he needs urgent help replacing a gift card for a customer who won a prize at an event.

Recognizing that InedoLib v999 is fake requires just as much skill as it does to recognize a gift card scam. Not a lot of skill, but not zero.

How to completely prevent supply chain attacks

I was pretty shocked that these big companies with these world-class developers fell for such a basic supply chain attack. There are three things those companies (and your organization) can do right now to prevent these sorts of supply chain attacks from happening: Training, Policy, and Enforcement.

1. Training on Packages and Dependency Management

This is obviously a pretty big topic, and we’ve done our best to help our customers train their engineers on package management practices. ProGet incorporates many of these practices and encourages their use.

That being said, if I had to pick two key lessons that completely prevent “dependency confusion” attacks at Inedo, or anywhere else, they would be these:

Training lesson #1: There’s no inherent benefit to updating dependencies

You should not update your libraries (dependencies) unless you have a good reason to. And no, “because Visual Studio told me there’s a new version available” is not a good reason. In fact, new versions of libraries should be a cause for skepticism, not action.

This isn’t your smartphone or personal computer. Upgrading your libraries will not suddenly give your application new features.

Think of it this way: would you accept a merge request for a giant block of unreviewed and untested code? That’s exactly what happens when you arbitrarily update your packages.

Just like your users don’t care if a particular module in your codebase hasn’t been edited in a decade, your users won’t care if the third-party library your application uses hasn’t been updated in as many years.

Customers want features, stability, performance, etc., and arbitrarily updating your libraries bring none of those things. In the absolute best case, an “update for update’s sake” will do absolutely nothing.

Arbitrarily updating your libraries will much more likely introduce bugs in hard-to-diagnose places. Those now are bugs you have to fix, instead of delivering value to your customers.

You may also find yourself with new and undiscovered security vulnerabilities. At least on old versions of libraries, the security vulnerabilities are known, and you can safely work around them. It’s unknowable what vulnerabilities might be shipped in a new version of a library and how those vulnerabilities might impact your codebase. But if you wait for a while—a year, two years—chances are someone else will find out what those vulnerabilities are.

Training lesson #2: Evaluating packages and package versions require additional training

Just like accepting a merge request into your codebase is more appropriate for senior developers with organizational experience and proper training, so is evaluating and using new packages.

And just like code review, package review is a process that can be facilitated with tools (ProGet has Package Promotion Pipelines for this), but it’s still a judgment-based, human-driven process. Deciding which packages to start using is well beyond the scope of this article, and it involves everything from the quality of code in the library to the license agreement to the quality of the library’s software engineering process.

Evaluating package updates is a bit easier. You just have to carefully evaluate what changed between versions, understand how those changes might impact the application, and then assess the risks/benefits of the new versions. Only then do you update, and only after you have a clear test plan.

2. Create Policies Surrounding Packages and Dependency Management

If these bigger companies had the simple policies that we have in place at Inedo, they would have prevented this “dependency confusion” attack many times over. At Inedo, because of our rules, there was no chance of it ever working, not even close.

While all organizations need to develop their own policies, I’ll share the rules we have in place at Inedo.

2a. Inedo’s Basic Policies

The first rule is simple and should be nearly universal to all organizations: Developers don’t decide on which packages to use nor when to upgrade packages in use.

That’s it! Now, you can just build policies based off of that. This really depends on how the organization is structured, but it could be as simple as the policies we use.

If developers need to use a new package or upgrade an existing package, they need to ask a senior developer for a package review.

Senior developers will have specialized training on both the package approval process and package management, and they can apply that, along with their knowledge and experience of the organization and its systems.

Developer’s workstations (Visual Studio, etc.) are connected to ProGet and only ProGet.

If developers can’t find the package they want to use in ProGet, then it means the package hasn’t been approved for use, and they will need to ask a senior developer for a package review. See above rule.

Software is meticulously versioned with changes carefully tracked.

We’ve publicly documented the Inedo Change Control Process, and InedoLib and our products are included in this process.

Library updates are software changes. Which are carefully tracked.

When a library needs to be updated in a product, it receives a change control issue just like all other code changes. It has a special format and might look like “InedoLib: Upgraded from 950.1 to 950.2”.

All changes are reviewed before merging and again before shipping.

Before releasing a new version of our product, we evaluate all code changes and make sure they are tied to a change control process.

2b. Policies protect and accelerate our development

We didn’t design our policies with “dependency confusion attacks” in mind, but they certainly protect us from it and all other supply-chain attacks. They may seem bureaucratic, but they actually let us develop faster, better, cheaper… and safer!

The proof is in our delivery: we’re a very small team of developers, and yet we release three different enterprise-grade products twice a month. Sometimes more. And we even make time to laugh off “dependency confusion attacks” and then write a long blog article about it.

I’ll let you in on another secret: at Inedo, “release engineer” is a part-time, rotating job that takes just a few hours every couple of weeks. It’s clicking some buttons, reviewing checklists, and then enjoying a nice single-malt scotch while watching BuildMaster do its thing.

There’s obviously a lot of human judgment involved in our release engineering work, but beyond that, it’s an easy job. The process enables that, and we invested a lot in building a great process.

It’s simply astonishing that these large, world-class companies didn’t design policies like we did, but hopefully, their mistake will help you develop your own policies. Let us know if we can help!

3. Strict Policy Enforcement

There are so many policy gates in place at Inedo that there’s no way a fake InedoLib version could have even found its way on a developer’s workstation, let alone be shipped in a product, because:

• developers can’t find InedoLib v999 because it’s not in ProGet
• even if they didn’t follow the policy, a senior developer would have rejected the code change to packages.config, because there’s no corresponding issue
• even if there was an issue, we would have never approved the issue because there’s no reason for upgrade
• even if the code was accepted, there would have been a failure on the build server because the package isn’t in ProGet
•  

But there’s just one problem: our processes are powered by people, and unlike computers, humans don’t do exactly what they were told to do. It’s in our nature, and that’s a good thing—except for when it’s not. Like following policy; humans aren’t always great at that.

So how can make people follow policy? What about saying something like this?

“It’s simple. This is a company policy. Period. It’s literally your job to follow company policies, and when you don’t do your job, there are consequences. Follow the policy or get fired. That’s it.”

Ha! If only it worked that way. We obviously don’t communicate to employees that way—nor should any employer. We’ve invested a lot of effort in developing Chōwa – Our Culture of Balance and have learned a much better way to ensure policies are enforced. It’s a balance of training and technology.

3a. Train on why policies exist

Without training, policies seem arbitrary to most otherwise intelligent and capable people.

“What do you mean I can’t choose my own packages? Those are my tools to do my job. What kind of ridiculous package dictatorship is Alex running over here at Inedo? Will he also demand my desktop color is Inedo Dark Blue #152C53?”

And by training, I don’t mean step-by-step instructions on how to follow the policy. That’s not real training; that’s just a reinforcement that you have arbitrary policies and an invitation to not follow them.

If nothing else, take what I wrote in the Training section about Packages and Dependency Management, and adapt it.

3b. Let technology automate enforcement

Long before DevOps and CI/CD became buzzwords, I designed BuildMaster to do the work of a fulltime “build master,” a poor fellow whose job was part paperwork, part builds/deployment, and part putting out fires.

As organizations automate the “build master” role with technology, they often forget about the paperwork part of the job, the process enforcement. It’s really important, as many of these big companies learned, and a real “build master” would have never been confused by InedoLib v999.

Neither would our BuildMaster software, because we’ve configured it and ProGet to enforce the policies I described.

There’s a lot that goes into that, which is well beyond the scope of this article, but in summary:

• BuildMaster’s Pipelines have a mix of automation and approvals so that “release engineer” can be a part-time job with minimal process training
• Our libraries are configured to follow the CI/CD for NuGet Packages pattern
• ProGet’s Connector Filters can be configured to block or allow packages by namespace, so that we a package like “InedoLib” could never come through a NuGet.org connector
• ProGet’s Package Promotion Feature allows us to move packages from unapproved to approved feeds, so we can speed up the approval process for developers.
• Using Prerelease Packages & Repackaging Feature we can see that InedoLib v999 has never gone through the repackaging process, and was never built on our servers

Other “dependency confusion” prevention measures to consider

There are also a few small steps that you can take on top of training, policies, and enforcement.

1. Register internal library namespaces

NuGet.org allows package owners to reserve and protect their identity by using ID Prefix Reservation.

Actually, we already did this with the “Inedo.” namespace because of public Inedo.SDK package. We didn’t register the prefix for security purposes, but in retrospect, it would have been a good reason to register the namespace.

By simply owning the “Inedo.” prefix, no one can make a fake version of our other internal libraries like Inedo.ServiceMessenger. We don’t have NPM packages, but if we did, we would try to register the @inedo scope for the same reason.

2. Publish your own fake internal libraries

As a preventative measure, you could simply just create dummy packages with your internal library names and publish them. Use version 999.0.0 just for fun and see how many developers fall for “dependency confusion.” It’s some good, safe, hands-on training.

3. Use consistent Internal Library Names

Only Inedo developers would know this, but “InedoLib” is our oldest library, and it’s the only one that doesn’t follow our naming or versioning conventions. That’s one of the reasons the version number is so high: it wasn’t semantically versioned at first.

But if it was named properly (“Inedo.Core”) then someone couldn’t have published a fake version of it because we already own “Inedo.”

We will do this someday, but it’s a lot of effort to rename libraries. It’s more important for going forward.

4. Work with NuGet.org and Galleries to Remove Packages

If you do find a fake version of one of your internal packages online, you should contact the package gallery and ask them to remove it. Then take ownership of the name by uploading your own fake version of the package.

We don’t plan on doing this, mostly because we think it’s funny to have a fake library out there. Plus, knowing that kpi58021 (the anonymous and fake InedoLib 999 owner) could inject malicious code at any time keeps us on our toes. Like actual security threats, we don’t know if or when they’ll happen, but we need to be prepared, nonetheless.

Learn more about supply-chain attacks

I’ve many learned these lessons over the years through research and from working with our software users. I’d like to share these experiences, and I’m considering putting together a guide on preventing, detecting, and fixing supply-chain attacks like dependency confusion. Contact us if you want to learn more.