April 12, 2024

How a volunteer prevented a backdoor from exposing Linux systems worldwide

Linux, the most widely used open source operating system in the world, narrowly escaped a massive cyber attack over the Easter weekend, all thanks to one volunteer.

The backdoor was inserted in a recent release of a Linux compression format called XZ Utils, a tool little known outside the Linux world but used in almost every Linux distribution to compress large files, making them easier to transfer . Had it spread more widely, an untold number of systems could have remained compromised for years.

And if Ars Technica As noted in the extended summary, the perpetrator had been working on the project in public.

The vulnerability, which was inserted into Linux’s remote login, exposed itself to only a single key, allowing it to hide from scans of public computers. As Ben Thompson writes Stratechery. “The majority of the world’s computers would be vulnerable and no one would know about it.”

The story of the discovery of the in upstream xz/liblzma leading to ssh server compromise.”

Freund, who volunteers as a “maintainer” for PostgreSQL, a Linux-based database, noticed a few strange things while running tests in recent weeks. Encrypted logins to liblzma, part of the XZ compression library, used a lot of CPU. None of the performance tools he used revealed anything, Freund wrote on Mastodon. This immediately made him suspicious, and he recalled a “strange complaint” from a Postgres user a few weeks earlier about Valgrind, the Linux program that checks for memory errors.

After some detective work, Freund eventually discovered what was wrong. “The upstream xz repository and the xz tarballs have been moved behind the door,” Freund said in his email. The malicious code was in versions 5.6.0 and 5.6.1 of the xz tools and libraries.

Shortly afterwards, the open source software company Red Hat issued an emergency security alert for users of Fedora Rawhide and Fedora Linux 40. Ultimately, the company concluded that the beta version of Fedora Linux 40 contained two affected versions of the xz libraries. Fedora Rawhide versions probably also received version 5.6.0 or 5.6.1.

IMMEDIATELY STOP USING FEDORA RAWHIDE INSTRUCTIONS for work or personal activities. Fedora Rawhide will soon be rolled back to xz-5.4.x, and once that is done, Fedora Rawhide instances can be safely redeployed.

Although a beta version of Debian, the free Linux distribution, contained compromised packages, the security team acted quickly to roll them back. “At this time, no stable versions of Debian are known to be affected,” Debian’s Salvatore Bonaccorso wrote in a security alert to users on Friday evening.

Freund later identified the person who submitted the malicious code as one of the two main xz Utils developers, known as JiaT75 or Jia Tan. “Given the activity over several weeks, either the committer was directly involved or there was a pretty serious breach of their system. Unfortunately, the latter seems the less likely explanation, as they communicated on different lists about the ‘fixes’ mentioned above,” Freund wrote in his analysis, after linking several fixes created by JiaT75.

JiaT75 was a household name: they had been working side by side with the original developer of the .xz file format, Lasse Collin, for some time. As programmer Russ Cox noted in his timeline, JiaT75 began sending out apparently legitimate patches to the XZ mailing list in October 2021.

Other branches of the plan unfolded a few months later, when two other identities, Jigar Kumar and Dennis Ens, began emailing complaints to Collin about bugs and the project’s slow development. However, as noted in reports by Evan Boehs and others, “Kumar” and “Ens” have never been seen outside the XZ community, leading leading researchers to believe that both are fakes that existed solely to help Jia Tan get into position to deliver backdoor code.

“I’m sorry about your mental health issues, but it’s important to be aware of your own limits. I understand this is a hobby project for all contributors, but the community is hungry for more,” Ens wrote in one post, while Kumar said in another post, “Progress will not happen until there is a new maintainer.”

Amid this back and forth, Collins wrote, “I haven’t lost interest, but my ability to care for me is quite limited, mainly due to long-term mental health issues, but also due to a number of other things,” and suggested that Jia Tan would do the following: on a bigger role. “It’s also good to keep in mind that this is an unpaid hobby project,” he concludes. The emails from “Kumar” and “Ens” continued until Tan was added as a maintainer later that year, able to make changes and try to get the backdoor package into Linux distributions with more authority.

The xz backdoor incident and its aftermath are an example of both the beauty of open source and a glaring vulnerability in the Internet infrastructure.

A developer behind FFmpeg, a popular open-source media package, highlighted the problem in a tweet, saying: “The xz fiasco has shown how dependence on unpaid volunteers can cause major problems. Trillion-dollar companies expect free and urgent support from volunteers.” And they brought receipts detailing how they handled a “high priority” bug affecting Microsoft Teams.

Despite Microsoft’s dependence on its software, the developer writes: “After politely asking for a support contract from Microsoft for long-term maintenance, they instead offered a one-time payment of a few thousand dollars in…investments in maintenance and sustainability are not sexy and probably They won’t get a middle manager promotion, but pay out a thousandfold over many years.”

Details about who is behind “JiaT75,” how they carried out their plan and the extent of the damage are being unearthed by an army of developers and cybersecurity professionals, both on social media and online forums. But that happens without direct financial support from many of the companies and organizations that benefit from using secure software.

Leave a Reply

Your email address will not be published. Required fields are marked *