Open Source: The coding threat that shook Red Hat and the open source world
I’m Brian Gordon, tech reporter for The News & Observer, and this is Open Source, a weekly newsletter on business, labor and technology in North Carolina.
Code running as written is not the same as code running as designed. It’s a distinction executives at Raleigh’s open-source software giant Red Hat encountered the last week of March when they learned a person (or persons) had stealthily introduced a secret backdoor into the components of the world’s most-used operating system, Linux.
“Code was introduced, and it wasn’t easily apparent that it was attackable,” Pete Allor, head of Red Hat’s product security, said in an interview this week. “And it had multiple stages in there, which meant that someone had planned.”
Linux software is pervasive, powering major companies, banks, government agencies and likely the technology you’re using to read this newsletter. The plot against it, now known as the XZ security incident, was two years in the making. And experts say it got too close to succeeding.
Under the alias Jia Tan, the perpetrator spent more than 24 months cultivating a reputation as a legitimate developer contributing to open source projects. Trust underpins the development and maintenance of Linux; the operating system is supported by tens of thousands of code libraries, each overseen by a team. Some teams are big. Some are small. Some developers are paid by large tech companies. Others are volunteers and weekend hobbyists. Each check their teammates’ contributions.
XZ Utils is a relatively unflashy but critical library. It compresses data and is present in nearly all Linux installations, including Red Hat’s. Allor said fewer people manage it compared to other libraries. Tan introduced code that gave them unauthorized access to any system that ran the util.
“This is what we call a supply chain attack,” said Dan Goodin, security editor for the tech news outlet Ars Technica. “You’re getting the malicious software distributed in a trusted package. It’s not unique to open source. Supply chain attacks happen in open source. They happen in closed source. It only takes one Jia Tan to get this malware in.”
Over time, downstream Linux distributors like Red Hat pull updates into their enterprise software packages — Red Hat Enterprise Linux, or RHEL, being the world’s biggest enterprise Linux platform. But RHEL doesn’t incorporate updates directly from Linux. First, new code gets added to a community distribution known as Fedora, which has both a main stable “build” and multiple other “builds” that get updated daily. Jia Tan’s infected code had reached the latter build when, two months ago, a Microsoft engineer named Andres Freund made a fortuitous discovery.
Freund says he uncovered the malicious code with some luck; he saw an operation wasn’t performing normally and traced the cause to the XZ Utils. On March 27, a Wednesday, he alerted his finding to security personnel at Debian, another Linux-based operating system. Debian then informed Red Hat’s Information Security.
“At this stage, there were two major fronts: The most urgent was to determine to what extent, if any, the malware had infiltrated our products,” wrote Red Hat’s principal security program manager, Rodrigo Freire, in a blog post a month later. “The second was to analyze the malicious nature of the package and better understand its cleverly concealed malicious payloads.”
Red Hat determined the compromised code had infected a rolling build called “Fedora 40 beta nightly,” which Freire called “the leading edge of Linux innovation.” But neither RHEL nor the main stable Fedora build were impacted. This was important because while niche early-adopters pull from Fedora’s rolling builds, many more people run the stable Fedora.
That Thursday morning, Red Hat’s security team briefed company and parent IBM executives about the incident. Red Hat also filed a vulnerability ticket with the U.S. Cybersecurity and Infrastructure Security Agency and took the unique step of submitting a security advisory to vendors known as a CVE (which stands for common vulnerabilities and exposures). Red Hat assigned the CVE a maximum score of 10.0 — meaning most critical.
The next day, Good Friday, Red Hat went public about the threat. In a post, it assured customers RHEL was unaffected and that the handful of distributors who took infected code from the Fedora 40 build had removed it. To his knowledge, Allor said no companies or organizations ran the compromised XZ Utils.
Reflecting two months later, Allor says the incident shows the open source ecosystem working as designed, that more eyeballs on the code leads to greater security. But he acknowledged that open source providers, especially large ones like Red Hat, need to ensure all corners of Linux have sufficient resources to identify malicious actors. If a certain code library is maintained by a smaller group of volunteer developers, perhaps the Red Hats, Googles and Microsofts of the world should pitch in more manpower.
So how close was the compromised XZ Utils code from getting into RHEL and other Linux-based enterprise operating systems like Debian and Ubuntu? Red Hat stresses many steps separate the code in a rolling Fedora build from making it into RHEL. “Depending on the size of the feature, it can be anywhere from a year to three years,” Red Hat spokesperson John Terrill told me this week.
Terrill disagrees with Goodin (and other tech journalists) who say Tan’s code was nearly installed in Linux-based production systems. He asserts the malicious XZ Utils was likely to have been caught prior to infiltrating stable Fedora (let alone RHEL). “It’s still pretty far from getting into anything that would typically be used in an actual production system,” Terrill said.
Network security expert HD Moore, who founded the Austin, Texas-based firm runZero, agrees it likely would have taken a few years for the nefarious code to reach stable Fedora, before which someone would have detected it. Had the code made it, though, Moore called the consequences “catastrophic.”
He noted there is a middle stage between a “rolling” distribution and “stable” distribution called “latest” distribution, which gets fewer updates than rolling and less long-term support than stable. Tan’s code might have gotten into the “latest” distributions of major providers within a couple months, Moore said, which would have been “horrible” for developers — though still would have largely spared regular customers.
Whatever the immediate scale of the XZ threat, crisis was avoided. This time. How the industry responds will be the next question. In his blog post on the Red Hat website, Freire said the experience will linger, writing “March 29, 2024 is a day that will hardly be forgotten by the open source community.”
Clearing my cache
Google Fiber has begun to install 20-gigabit internet access in local homes. Installations began last month, the company says, and the Triangle is among the first markets to get 20G residential service. At 20 times the bandwidth of standard Google Fiber, the internet service costs $250 a month. It probably isn’t for everyone, though, both financially and technically.
VinFast’s only U.S. electric vehicle model is under investigation by the National Highway Traffic Safety Administration after a one-car crash killed four in California last month. More bad news for the company that’s looking to build its first North American factory in North Carolina’s Chatham County.
The Durham semiconductor chipmaker Wolfspeed is growing and hiring ... and it’s lost half its stock market value since last summer. Now, a prominent activist investor has increased its stake in the company and raised the prospect of a sale. What gives? I spoke to three analysts who cover Wolfspeed to find out.
On Thursday, the Durham REI union rallied outside its store to demand the outdoor apparel company “negotiate in good faith” and reach a fair contract in 2024. It’s been a year since the workforce unionized. The United Food & Commercial Workers International Union says REI hasn’t bargained in good faith.
In an email, the company said “that is simply not true.”
“The collective bargaining process — especially when negotiating a first contract — can be lengthy,” an REI spokesperson said. “Both parties have been engaged in numerous negotiations and have reached tentative agreements on various topics.”
North Carolina Attorney General (and gubernatorial candidate) Josh Stein sued Pactiv Evergreen this week, demanding the paper company repay $12 million for allegedly violating a 2014 grant agreement after Pactiv shut down its factory in the Western North Carolina town of Canton. Pactiv’s exit last year was immense for the community nicknamed “Paper Town.” Smoky Mountain News has more on the lawsuit and a response from Stein’s challenger, Lt. Gov. Mark Robinson.
National tech happenings
OpenAI hit a nerve with a lot of people, actress Scarlett Johansson among them, when the company used a voice that sounded a lot like Johansson’s for its new artificial intelligence assistant, Sky. OpenAI has since pulled the voice. Its CEO, Sam Altman, denies his company copied Johansson’s voice, even though:
OpenAI had asked the actress to be the assistant’s voice (she declined).
Johansson provided the voice to an AI assistant in the 2014 movie “Her.”
On May 13, Altman tweeted a one-word message, “Her.”
High-end Teslas can be stolen with about $100 worth of radio equipment, researchers found.
Rideshare apps and the state of Minnesota reached a truce via new legislation that sets greater minimum wage levels for drivers while superseding an even higher wage ordinance passed by the city of Minneapolis (where the vast majority of the state’s ridesharing occurs). Uber and Lyft had threatened to leave Minnesota over the pay debate.
Thanks for reading!
