We need to figure out financing of open-source software

I recently had a discussion over coffee with my colleagues about the various open-source projects we use at work every single day and how much our work actually profits from open source. Safe to say (and this doesn’t only go for us) we use a lot of open source, as does anyone else in software development. And unfortunately, also safe to say, the issue of financing this open-source software, is not yet solved¹.

The issue of open-source financing has been around for a while: James Turner has written about it and more often than not, the funding problems do get solved, but only after there has been a problem as well.

Heartbleed

Following the discovery of the Heartbleed bug, what I like to describe as the security bug with the best marketing strategy, the question of financing also came up. In fact, before the discovery of the bug, the project is supposed to only have received about USD 2000 a year before the vulnerability has been discovered. It appears that this amount has greatly increased now, and there is some more structure to this critical project as well.

But still, addressing a issue after it arrives is not really a smart, or for that matter, sustainable way.

This brings us to the current affair

I was scrolling through my mastodon, as I do every once so often, and came across this mastodon post:

AndresFreundTec @AndresFreundTec@mastodon.social

I accidentally found a security issue while benchmarking postgres changes.

If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.

https://www.openwall.com/lists/oss-security/2024/03/29/4

oss-security - backdoor in upstream xz/liblzma leading to ssh server compromise

29 Mar 2024 17:10

In this post and the linked email, Andres desired how he kind of accidentally found a purposefully introduced security bug in a system calls xz, which is used in many Linux distributions. It is unclear how much effect this backdoor has had, and it is also unclear how this arrived in the code.

Evan Bohens has a blog post where they are tracking the current developments of this story with a lot of technical content. From this storyline, it appears that this is a very targeted, highly sophisticated supply chain attack.

From what it appears, this attack was only possible because it was targeting the creator of xz specifically and attacking them in a quite targeted way:

Kevin Beaumont @GossiTheDog@cyberplace.social

Back in 2022 a host of characters appeared and basically bullied the creator of the XZ project to hand it over to somebody else - at the time the guy cited mental health issues around not updating the project quickly.

At the time he was already talking about maybe handing over to the account who years later introduced the backdoor.

In mid 2023 said account introduced a change to Google’s OSS Fuzzer to weaken detection for XZ.

Somebody played a years long game of Jenga and lost.

30 Mar 2024 08:39

In fact, a pseudonymous, assumed to be a socket puppet, account was pressuring the maintainer of this open-source project forcefully with the following passive-aggressive message:

Over 1 month and no closer to being merged. Not a suprise.

— Jigar Kumar, 27 May 2022 10:49:47 -0700 on the xz-devel Mailing list

The repository maintainer replied with a sombre message:

I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. […] It’s also good to keep in mind that this is an unpaid hobby project.

— Lasse Collin, 08 Jun 2022 03:28:08 -0700 on the xz-devel Mailing list

The same account then pressures Collin to add another maintainer:

Progress will not happen until there is new maintainer. XZ for C has sparse commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this.

— Jigar Kumar, 07 Jun 2022 09:00:18 -0700 on the xz-devel Mailing list

Shortly thereafter, Jia Tan is added as a maintainer and given full access to the code repository and release infrastructure. Sometime thereafter, after some additional steps, the malicious code is added to the xz utility. Luckily, it hasn’t made it to any stable distributions. But the [severity of this] has made it such that numerous places rolled back the version of xz. Especially the message from Red Hat is quite daunting:

PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES

This is a systemic issue

Of course, these two examples are not the only time something like this has happened. In fact, it is generally known, that almost all software depends on some other open-source software, that may or may not be supported properly. There is even the well-known xkcd comic representation of this.

Often people are maintaining important and critical software libraries out of the good of their hearts. They are maintaining them because it’s essential and because they want to. They are maintaining critical parts of our daily lives in their free time, volunteering. Mostly they are not paid, some even pay themselves to do the work they do.

Gabe Kangas @gabek@social.gabekangas.com

My heart goes out to xz. A single maintainer, who was clearly in a rough place with mental health, screaming out to the world for some help and additional contributions, and somebody shows up wanting to help. Could you imagine how happy that maintainer was? They were no longer alone.

And it turns out the only reason somebody wanted to help them was nefarious. I can’t imagine how they feel right now as everyone is blaming them. I hope they’re ok.

29 Mar 2024 21:58

So we really need a system for how to deal with such significant libraries. We require some way to ensure they are being maintained and being maintained by people with no ill intentions. Something which is not quite that easy to do.

But we can start by talking about money.

Money talks

Money is indispensable, after all, this is what affords every one of us to survive in one way or another. Many people like supporting Open Source because it makes them feel like they are doing some good. I would tend to agree with that opinion. But over time, all these open-source projects require some kind of money, require some kind of way to survive. And there have been more or less successful ones.

Homebrew is a project that is entirely volunteer run, but is so important that most developers on Apple machines use it pretty much daily. In fact, homebrew would be a superb attack vector² if someone had access to that repository, they could smuggle in infected software into the system. And while I have seen some security advisories, the project appears so far to be unaffected and run well on the donation supported model.

In a similar vein works the VLC media player, which also seems to be run entirely on donations and which appears to be financially stable (and an awesome product as well). The Vue.js frontend framework seems to work in a similar vein. All of these seem to somehow work, it’s unclear to me how stable and sustainable they are.

Then we also have the large frameworks which are “sponsored” by their main developers and users, such as React, Angular and Flutter. I would also put the Java programming language and corresponding ecosystem into the same pot, while understanding that the ownership history is far too complicated for anyone with a mere computer science degree to understand³.

And then we have one final group that has financing figured out. The Python and Linux of the world, which work because there is a foundation in the background that manages to stay financed by convincing the people using these open-source software products to also contribute back to the projects. It works for the big fishes, but this model does not scale to smaller projects that are just as critical (see OpenSSL above).

And yes, I know I left out the dual licenses and other ideas where the main part is open source, but there is an additional-paid license such as Redis or MongoDB. While that might work in the short term, this always leads to multiple financial conflicts. For one, cloud providers used under the hood can always offer the open-source part as part of their cloud offering for less. And secondly, there is a certain tension of where to put features, in the money making part or in the idealistic open-source part.

So, what’s the solution

I don’t see a single workable solution for everything, but I do see the need for a solution. And a quick one at that. Open-source software is system critical in lots of places, and we should treat it as such.

Firstly, we should not change the systems that already work. The Linux kernel works, both from a financial perspective and from a governance standpoint. So we would not like to mess with that. Similarly, I don’t think there is a big need to address the big-company-produced open-source software, React and Angular work in the way they currently should, there is no point in changing that.

But we do need a solution for the system critical, smaller projects that need to be sustainable somehow and projects that are important for everybody should also be financed by, you guessed it, everybody.

Mandatory financing of open source

In Switzerland, making copies of digital files, even copyrighted ones, is legal⁴, as long as you only do it in a private manner. To counteract some lost revenues, there is a tax on every storage medium produced, or in the case of Switzerland, mostly imported. This tax then goes to the collective rights management organisations, from where the money is then distributed to artists.

This system is not perfect, far from it. I have probably paid some of these storage taxes on some devices, even though I then haven’t used these devices to privately copy music or other copyrighted material. And the distribution of the money is left to the collective rights management organisations, which whom sometime some artists are upset as well. And finally, I see the logic of how it doesn’t make sense to pay for storage somewhere which I’m going to use to host my photos. But anyway, the system kinda works, and it is for sure better than the alternative of not having anything.

And this is also what I propose for open-source software. If you sell some kind of software (something that is my daily bread), there should be a small percentage that goes into a collective rights organisation which then distributes it to different open-source projects. This would enable the projects to be financially well and would make them more sustainable. If this is done for everyone, this would also mean that there are no competitive disadvantages, it’s just everyone’s tax of doing business, and it would help a lot in the broader ecosystem. Easy and simple.

Coda: This will not fix everything

Of course, this is not a magic bullet. For example, this only works on an international level since software, especially open-source software is by its very nature international. This is also not a magically fair system, and I’m sure it will also leave some people out. Very private people that don’t want to deal with a government, or close to a government entity, might not get their fair compensation. The same goes for some less tangible open-source contributions. But it goes a step in the right direction.

And open-source software is too critical for all our systems, to not at least try.

Post script on the xz situation

This attack seems very targeted and well executed to me, it might have happened either way, with or without financing. But I do hope that proper financing would help a long way to appease the loneliness of some maintainers, and might even help them to set up trusted organisations that support them, who knows.

Also, I think the way this was discovered way largely accidental, but then again it was discovered. The findings were briefly discussed online, and the correct action was taken pretty quickly. I’m sure more will come up with time, but this has been my favourite take so far:

VessOnSecurity @bontchev@infosec.exchange

So, kids, what's the moral of the XZ story?

If you're going to backdoor something, make sure that your changes don't impact its performance. Nobody cares about security - but if your backdoor makes the thing half a second slower, some nerd is going to dig it up.

30 Mar 2024 15:13

Stay safe on the interwebs!