Since the early 2000s, open-source software, accessible to all without licensing agreements, has captured an ever-growing slice of the world’s software supply. Estimates vary, but according to Synopsis, 96% of all codebases contain open-source software, with 76% of all code originating from open-source.
According to Harvard Business School researchers, open-source software has fueled a high-tech economic growth engine that has saved firms around $9 trillion in costs. However, as most cybersecurity defenders know too well, the freedom to build new projects using open-source packages comes with security costs.
As Darren Meyer, staff research engineer at Endor Labs, tells CSO, open source is “a crazy, crazy benefit. But the flip side is, things that are convenient for legitimate developers are also convenient for adversaries, and that’s the central problem of adopting open source safely.”
Most open-source packages come from code repositories such as npm, PyPI, NuGet, crates.io, RubyGems, Golang, StackExchange, and the giant developer platform GitHub. Rarely does a week go by without software security firms discovering that threat actors are spreading malicious Python, npm, jQuery packages, and even JPEG files to millions of developers via these repositories.
Open-source threats are up 1,300% since 2020
The rate at which “poisoned” code packages appear on open-source repositories is snowballing. ReversingLabs detected a more than 1,300% increase in threats circulating via open-source package repositories between 2020 and 2023, including a 400% increase in threats found on the PyPI platform in 2023 alone.
Although there is no single source for the number of poisoned code packages, a project called OpenSSF Malicious Packages, maintained on GitHub by several software security experts, aims to be a comprehensive, high-quality, open-source database of reports of malicious packages published on open-source package repositories.
Even as these experts devote themselves to flagging and helping the repositories remove malicious packages, only a tiny fraction of the malicious packages that end up in software resources are ever identified. And despite the repositories’ responsiveness to researchers, there are no fundamental safeguards to weed out packages.
Experts say the best strategy for CISOs to address this issue includes installing scanning mechanisms that identify poisoned packages and rigorous scanning of all software, both as a publisher and consumer. CISOs should also steer their organizations clear of using recently uploaded packages. Most importantly, CISOs should foster a culture of awareness for software developers who often download packages without fully appreciating the dangers.
The volume of tainted OSS packages is overwhelming
The biggest challenge in tackling tainted open-source packages is the sheer volume that organizations use. “Big organizations can have thousands or tens of thousands of projects without any real difficulty, especially if they’ve acquired companies because, typically, they’re not merging projects; they’re just sort of letting them be and letting them continue to be maintained,” Meyer says. “It’s very easy to end up with tens of thousands of projects if you are at all large, and hundreds is very common for medium-sized businesses.”
Moreover, it’s challenging to identify how many packages in the repositories contain malware. Tomislav Pericin, chief software architect and co-founder at ReversingLabs, tells CSO his firm currently estimates that about 15,000 packages on the top three platforms, npm, PyPI, and RubyGems, contain malware, including lower-level malware such as spam. He says other sources say the number runs into the hundreds of thousands by including multiple versions of the same package.
“We do find about a dozen or so malicious packages per day,” Pericin says. But that’s “basically a needle in a haystack” given that developers upload about 50,000 packages daily to those platforms.
Louis Lang, co-founder and CTO of Phylum, tells CSO that open-source package poisoning “is the biggest problem that exists in security today. When you look at these open-source package repositories and distill them down to their core component, it’s random people on the internet ostensibly [putting] remote code execution in your production builds because you’re pulling their code in, and you include it in your product. If that person is malicious or compromised, which we see happening often, you have a completely unpatchable remote code execution vulnerability in your infrastructure and your product. And if you’re not looking, it’s just a massive gaping hole.”
Lang illustrates how large that hole can be and how most organizations can’t deal with the problem by manually checking their software. “It is a kind of intractable problem to do as a manual endeavor. For one organization, we saw that one of their projects had 134,000 packages, and those packages, on average, have something to the order of 30 to 60 code files.”
No fundamental safeguards are in place
Because all software, including malicious software, is simply code, it’s almost impossible to determine which part of the code downloaded from repositories is malware without security researchers conducting time-consuming analysis.
“It is very difficult to automatically determine or to determine at scale the difference between legitimate and malicious code,” Endor Labs Meyer says. “If you look at the recent XZ Utils compromise, that got noticed only because a Microsoft engineer was bothered by a performance change. If you glanced at the code, you would never notice anything wrong. The tools out there capable of looking for malicious code all missed that. It’s very difficult because the code is relatively neutral. It’s the intent behind the code that is malicious or not.”
Moreover, there are no safeguards at the repository level to detect bad packages. “Anyone can write a piece of code and just upload it to those platforms,” Yehuda Gelb, research engineer at Checkmarx, tells CSO. “For instance, in Python, you can just create a Python package and upload it, and there’s no one really in PyPi that says, ‘okay, you can’t upload this’ unless someone like us catches them, and then we report it to them, and they take it down.”
The code repositories do what they can to screen out bad packages, but ensuring that the tens of thousands of packages they receive each day are malware-free is not their job. “The problem is that content uploaded to open-source registries are not vetted,” Jossef Harush, head of software supply chain security at Checkmarx, tells CSO. “
“If I want to publish a GitHub repository, I can do that,” Harush says. “It’s going to be public in a snap. I don’t have any filters doing so. If someone reports my GitHub repository as containing malware, then the GitHub security teams would get involved. It would take them time, and most likely, after that, the malware package would get removed or hidden from the public. But that relies on the community flagging those contributions as bad.”
How CISOs can grapple with the problem
CISOs face many problems in dealing with poisoned packages, starting with a shortage of personnel capable of identifying problematic packages in their organizations’ vast software collections.
“I think there’s a shortage of security professionals. We’re talking about people who must have very broad and deep skill sets to approach this in any meaningful way,” Phylum’s Lang says. “Some of those individuals can start writing scripts to help automate some of these components and whatnot. But the vast majority of organizations do not have that kind of talent.”
A quicker solution is to put various scanning solutions or firewalls in place to ensure that bad packages don’t end up in the software they use or develop. “You need to ensure that the open-source stuff that you’re bringing in has good quality, doesn’t have vulnerabilities, and doesn’t have malicious code,” Pericin says.
“In an ideal world, you would scan it on both ends, as a software publisher before you ship it, and then as a consumer before you use it, and the scan should be done on the exact file as it is.” Pericin emphasizes that software publishers should scan all code, even proprietary code, or “you could be the next SolarWinds.”
Phylum’s Lang recommends that CISOs be leery of code that has only recently been uploaded to repositories or features obfuscation elements. “I’ve talked to CISOs, and one of the things they do is not allow any package published in the last N number of days to be brought into the organization. Anything that has obfuscated code should not be allowed to be brought into the organization.”
Harush thinks that CISOs should foster a culture of awareness regarding downloaded packages’ threats, particularly given how frequently software developers turn to code repositories for quick and easy solutions to their coding challenges. “Threat actors are having a lot of success tricking developers because developers just don’t think about it enough,” he says. “They’re on autopilot consuming content from strangers, and this culture needs to change. CISOs in big companies need to educate developers to tell them that there are bad guys in open source and give them some examples.