Heartbleed Exposes a Problem With Open Source, But It’s Not What You Think

BY CHRISTINA WARREN

(Image: Mashable)

A week after the Heartbleed OpenSSL vulnerability wreaked havoc across the web, the conversation is shifting from reaction to reflection. The discussion is no longer about what to do now, but what can be done to prevent another Heartbleed from happening in the future. In other words, we’re entering the blame game chapter in this saga.

So who is to blame for Heartbleed?

If OpenSSL, the software package at the root of the vulnerability, were a piece of commercial software, we could blame the company behind the app. In fact, when Apple released an emergency patch for its own SSL/TLS bug back in February, the company was scrutinized by security experts, programmers and pundits a like.

But OpenSSL isn’t a commercial program. It’s an open-source project maintained by a small group of committers and volunteers.

Because OpenSSL is open source, there isn’t an immediate figure or organization to blame. Are we really going to blame unpaid software engineers who commit to a project that 66% of the Internet uses for free?

So if you can’t blame an entity — the first recourse for some — is the model of open-source software itself to blame?

Linus’s Law Didn’t Fail

Over the last 15 years, open-source software has developed a reputation for being secure and reliable. Open-source code can actually be more reliable than proprietary (or, closed-source) code because there are more users looking at it to find bugs and security holes.

In Eric S. Raymond’s seminal essay on open source, The Cathedral and the Bazaar, he defines Linus’s Law (named for the father of the Linux kernel, Linus Torvalds), which states that “given enough eyeballs, all bugs are shallow.” In other words. If enough users are looking at the code, bugs and problems will be found.

Raymond argues that this distinction is one of the reasons that open-source software is inherently safer than proprietary code. After all, it has more people looking at it, capable of finding and repairing bugs.

In the wake of Heartbleed and other open-source security failures, some have questioned whether Linus’s Law still holds true.

The argument is that if this was a commercially backed project with a well-funded development team of full-time professionals, rather than the volunteers and committers that currently make-up OpenSSL, the coding process and auditing process would have better.

But we can’t know that. Apple’s SSL/TLS bug (which was much smaller than the Heartbleed bug in both scope and in threat), existed for more than a year before Apple engineers found the bug and released patches. Yes, that library was also open source, but it was maintained by Apple employees, and Apple had to approve any outside contributions.

I summarized my thoughts on what Heartbleed means for open source on Twitter last week:

Although I disagree with Raymond that open source is inherently more secure than other types of code, I agree with him that Heartbleed does not refute Linus’ Law. Yes, the Heartbleed bug went unidentified for more than two years, but it was caught — and caught by two different parties — because the code was open.

Heartbleed is not a failure of open source, at least not the way you may think. There is absolutely no reason to believe that Heartbleed happened because it was an open-source project powered by volunteers.

But if Heartbleed doesn’t refute Linus’s Law, that doesn’t mean that this situation still doesn’t put the spotlight on what really can hamper an open-source project: Lack of resources.

Open Source

One of the hallmarks of open-source software — in fact, a big part of its appeal — is that it is free to use and modify.

Open-source software isn’t always “free as in beer” (as Free Software Foundation leader and GNU founder Richard Stallman likes to say), but more often than not, companies don’t have to pay anything to access or implement open-source code into their projects.

In fact, it’s this “free” aspect that has led to the mass adoption of open-source in various industries. Although proprietary software (or software that is a mix of proprietary and open-source) still dominates many industries — open source has really found its place on the web.

Microsoft might be the world’s largest software maker, but most web servers don’t run Windows; they run Linux and a web server application such as Apache or Nginx. Why? Because Microsoft charges for its software. For a small business, running Linux might translate into relatively small savings. As a business grows, however, those savings can become quite substantial.

If Facebook or Google had to pay a license for every server or virtual server it operates — and another license for its web server — its business plan would be completely different.

This is even more true when open-source software — especially related to servers and security — has a track record that is as good or better than proprietary solutions.

Beyond that, the great thing about open source is that anyone can take a project or part of a project and build something else on top of it. And depending on the license, those changes often have to be shared with everyone else. That often leads to even better results.

But of course, nothing is really free. Maintaining and supporting software — open source or not — has a cost. Major (as in popular in size and in scope) open-source projects are generally funded in three ways:

1. Donations from individuals, volunteers (by time or coding abilities) and non-profits.
2. The project is funded and steered by a commercial entity or entities.
3. Corporations who use and benefit from the project hire employees who are dedicated to working on the project full-time.

The Debian Linux distribution is funded by volunteers and by non-profit donations. It is one of the few completely community driven projects without a corporate sponsor.

Ubuntu, another Linux distribution (which is based on Debian), is sponsored by Canonical. Volunteers still make up a big part of the project, but Canonical ensures that full-time employees are paid.

Red Hat was one of the first companies to build a successful business off of open-source and free software. Although it makes the source code for its projects available to everyone, it sells software and service contracts for its flagship Red Hat Enterprise Linux and other products. Red Hat also sponsors community projects such as Fedora and CentOS.

Companies including Red Hat and IBM also donate employees to work full-time on important projects, including the Linux Kernel.

The WebKit project, which is the basis for the Safari browser (and until 2013, Google Chrome), is sponsored and maintained by Apple.

Over the last 15 years, most companies that benefit tremendously from open source — including but not limited to Amazon, Google, Facebook, IBM, Cisco and Twitter — also give back code, employee time and money to the projects that are most important to their business and product.

But not every project gets the type of attention or funding that it needs.

OpenSSL: Used By Many, Supported By Few

As the defacto SSL/TLS cryptographic stack on the web, it might be easy to think that OpenSSL has tons of support.

After all, as we’ve learned from Heartbleed — it’s not just web servers that use OpenSSL. Routers (big, expensive, high-end routers), firewalls, smartphones and other connected devices all use OpenSSL.

If the number of people that relied on a project — and its importance to the overall web — was proportionally related to the amount of support a project has, OpenSSL would be well-funded and have a heft of full-time paid employees and maintainers.

It’s not.

OpenSSL, a project that runs on 66% of all web servers, has just one full-time employee. One.

It gets worse. In the five years since the OpenSSL Software Foundation (OSF) was created — as a way to help sustain the OpenSSL project — this important project has never received more than $1 million in gross revenue a year.

Pure donations to the project are almost non-existent. Steve Marquess, the OpenSSL contributor who handles the business aspects of the OSF, addressed the current situation on his blog. According to Marquess, the foundation typically gets just $2,000 a year in donations.

Marquess writes (emphasis ours):

Even if those donations continue to arrive at the same rate indefinitely (they won’t), and even though every penny of those funds goes directly to OpenSSL team members, it is nowhere near enough to properly sustain the manpower levels needed to support such a complex and critical software product. While OpenSSL does “belong to the people” it is neither realistic nor appropriate to expect that a few hundred,* or even a few thousand, individuals provide all the financial support. The ones who should be contributing real resources are the commercial companies and governments who use OpenSSL extensively and take it for granted.*

The rest of the money — that under $1 million figure — doesn’t come from volunteered employees or corporate stewardship or even support contracts. It comes from work-for-hire contracts.

That is, companies pay members of the OpenSSL team (there are six core committers — only one of which is able to make OpenSSL his full-time job) $250 an hour to work on a project related to OpenSSL for that company.

Even at $250 an hour, the fact that very few OpenSSL team members exist (which is partially due to the high skill requirements and lack of guaranteed income) means that existing contract work is often unstaffed, and thus, unpaid.

So what’s the solution? Well, Marquess would rather have OpenSSL funded via support contracts. In the footnotes, he writes:

Here’s a plug for one of the most effective ways your corporation can not only support OpenSSL but also receive something of tangible value in return: a software support contract. We have a formal contract with the fine print that lawyers love, and your accounts payable people won’t be all flummoxed at the bizarre notion of giving money away as they’re used to paying for expensive commercial support contracts for proprietary software. Someday you may even encounter an issue with your mission critical use of OpenSSL that could benefit from direct and prompt attention from the people who wrote that code.

He also makes mention of the fact that lots of big companies are already taking advantage of OpenSSL in their commercial products and not contributing back.

I’m looking at you, Fortune 1000 companies. The ones who include OpenSSL in your firewall/appliance/cloud/financial/security products that you sell for profit, and/or who use it to secure your internal infrastructure and communications. The ones who don’t have to fund an in-house team of programmers to wrangle crypto code, and who then nag us for free consulting services when you can’t figure out how to use it. The ones who have never lifted a finger to contribute to the open source community that gave you this gift. You know who you are.

So What’s the Solution?

On Hacker News, some commenters criticized the OSF’s approach of doing contract work as a way to fund the project. I think this is a fair assessment.

A project of the nature of OpenSSL really should be funded by support contracts or by corporate sponsors agreeing to pay the salary of cryptography experts to work on OpenSSL full-time.

With no disrespect intended towards any member of the OpenSSL team or the OSF, part of the problem also appears to be that the project lacks strong leadership — at least in the way that could put a plan in motion to ensure that resources required to keep the project running successfully (and not the part-time gig for a few individuals and the full-time job for one person) are in place.

The OSF itself could also do a better job being transparent about who its sponsors are (and the nature of the projects it works on), as well as doing larger calls for funding and support contracts.

Having said that, because this is not a commercial endeavor, its unfair to hold the project to the same standard as we would a commercial entity. Maybe if companies can’t donate employees to work on the code, they can donate people to help with some of the public-facing and fundraising aspects of the foundation.

I do hope that the largest companies that benefit from OpenSSL — especially those who use the software in their commercial hardware products and security consoles — will see Heartbleed as a wake-up call. Not to abandon OpenSSL and move to a paid solution — but to do a better job giving back to the project and community.

More support could also mean more improvements to the code itself — and to the whole process. Yes, it’s possible that even with a team of well-paid engineers, code auditors and support staff, Heartbleed could still have happened. I would imagine finding this bug would be like trying to find a typo in Ulysses.

The difference is that a better-staffed project would mean that making changes and improvements to the code-auditing and code-review processes would be more feasible. That makes for a better end product, which means that everyone using OpenSSL would have the advantages of a safer, more stable and feature-rich product.

Heartbleed didn’t happen because OpenSSL is open source, it happened because the project wasn’t given the support it needed. Let’s hope that changes. And soon. This project is too important to too many.

Source: Mashable