Heartbleed: A Wakeup Call for Open Source Users (Updated)

I suspect that we'll remember Heartbleed as one of the few security vulnerabilities to have its own logo and URL. But we should remember it for the wake-up call it sounded for users of open-source software (OSS).

Heartbleed, as everyone who hasn't been camping in the wilderness for the past month knows, is a serious vulnerability in the OpenSSL cryptographic software library. OpenSSL is popular and widely used, so the vulnerability, which the Web site states, "allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions," was a big news even outside tech media. We're talking hundreds of thousands of Web servers and products potentially affected. Security analyst Bruce Schneier characterized the seriousness of Heartbleed on his blog this way: "On a scale of one to 10, this is an 11."

The bug was reported to the OpenSSL team by security engineers at Codenomicon, a company that builds software for pre-deployment testing of business-critical products, and Neel Mehta, a researcher on the Google Security team. But it had apparently in the code since 2011. According to the Sydney Morning Herald, the bug was introduced  by a German software developer and OpenSSL code contributor, who missed a bounds check in the handling of the TLS heartbeat extension that could be used to reveal up to 64k of memory to a connected client or server. (Thus the name, "Heartbleed.")

Once they knew about the vulnerability, the OpenSSL guys fixed it, fast-surprisingly fast, given the relatively small size of this particular open source community, which, according to Steve Marquess, president of the OpenSSL Software Foundation, lacks "the manpower levels needed to support such a complex and critical software product."

Writing on his blog, Marquess pointed to funding as a big part of the problem. His organization has been supporting this critical piece of software with about $2,000 a year in "outright donations," as well as some commercial support contracts and "work-for-hire" consulting. In the five years since the foundation was created, it has never taken in more than $1 million in gross annual revenues, he said.

In the week following the Heartbleed disclosures, the foundation received about 200 donations, Marquess said, "along with many messages of support and encouragement." Most of those donations were for between $5 and $10, and they amounted to about $9,000. (You can still donate.)

"Even if those donations continue to arrive at the same rate indefinitely (they won't), and even though every penny of those funds goes directly to OpenSSL team members, it is nowhere near enough to properly sustain the manpower levels needed to support such a complex and critical software product," Marquess wrote. "While OpenSSL does 'belong to the people' it is neither realistic nor appropriate to expect that a few hundred, or even a few thousand, individuals provide all the financial support. The ones who should be contributing real resources are the commercial companies and governments who use OpenSSL extensively and take it for granted."

So the arrival of a badass bug that seemed initially to repudiate the OSS mantra "given enough eyeballs, all bugs are shallow," actually reinforced that notion, while raising some serious questions about the true responsibility of OSS users. I like the way Bruno Borges, principal product manager for Oracle Fusion Middleware in Latin America, put it on his blog. Although, he acknowledges, the developer and QA have responsibilities here, it is "The whole IT industry (companies and developers of all kind; FOSS or not) who uses OpenSSL for free but does not pay anything for it, and although being Open Source, don't look at it, don't review commits. Just expect it to work without bugs. These are the most to be blamed. (including myself)"

Jonas Falck, CEO and co-founder of Halon Security, weighed in on the subject in an e-mail: "The Open Source community has received a bad rap for the OpenSSL exposure," he wrote, "but the community has rallied together to patch the issue quickly. If anything, the Heartbleed issue has shown how reliant the Internet as a whole is on Open Source, so if corporations can give back to the Open Source community after taking advantage of OpenSSL for so long, there will be more eyeballs spotting vulnerabilities earlier in the future."

Crowdsourcing platform Bugcrowd has initiated a clever, if short-term solution to the Heartbleed problem with a Crowdtilt crowdfunding campaign to secure OpenSSL.

As cool as that campaign is (very cool), it's doesn't address the underlying problem. I thought Walker White, CTO of BDNA, which makes the Technopedia categorized repository of information on enterprise software and hardware, summed up the situation nicely during a recent conversation. (His company uses the repository to put all the hardware and software assets within an organization into a common, easily taxonomy.)

"It's the money-for-nothing syndrome," Walker said, "and it's not just OpenSSL. You have all these great products-Tomcat, Maven, Gradle, Jenkins, Git, all these development tools and frameworks. It is truly incredible what the open-source community has done. But vendors can't treat open source as something they can just plug in. We have to take responsibility for it-especially vendors who are embedding it into commercial solutions-and not just punt to the community when something goes wrong."

UPDATE:
Shortly after I filed this story, news broke that the Linux Foundation is launching a three-year, multi-million-dollar project to fund open source projects that are "in the critical path for core computing and Internet functions-including OpenSSL. The funds for the project, called The Core Infrastructure Initiative, will be administered by the Linux Foundation and a steering group made up of backers of the project and "key open source developers and other industry stakeholders." Support will come in the form of funding for fellowships for key developers to work full-time on open source projects, security audits, computing and test infrastructure, travel, face-to-face meeting coordination, and other support, the foundation said.

"We are expanding the work we already do for the Linux kernel to other projects that may need support," said Jim Zemlin, executive director of The Linux Foundation, in a statement. "Our global economy is built on top of many open source projects. Just as The Linux Foundation has funded Linus Torvalds to be able to focus 100 percent on Linux development, we will now be able to support additional developers and maintainers to work full-time supporting other essential open source projects. We are thankful for these industry leaders' commitment to ensuring the continued growth and reliability of critical open source projects such as OpenSSL."

Along with the Linux Foundation, the list of "founding backers" of this effort includes Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Rackspace and VMware.

Posted by John K. Waters on April 28, 2014