Linux supporters fiddle while OpenSSH burns

Once again, the OpenBSD project is asking for donations to keep its operations in motion. It doesn’t ask for much — U.S. $100,000 (small potatoes in the operating system development industry) — yet it provides so much to the software world. Even if you don’t use OpenBSD, you’re likely to be benefiting from it unknowingly. If you’re using Solaris, SCO UnixWare, OS X, SUSE Linux, or Red Hat Enterprise Linux, chances are you’re using the OpenBSD-developed OpenSSH for secure shell access to remote machines. If so many are using this software, why are so few paying for it? Official responses (and non-responses) from Sun Microsystems, IBM, Novell, and Red Hat are below, but if you’re one of the freeloaders who hasn’t contributed to OpenBSD or OpenSSH, what’s your excuse?

OpenSSH: you’ll miss it when it’s gone

“Bigger than OpenBSD, our big contribution is OpenSSH,” OpenBSD project leader Theo de Raadt told me in a 2004 interview. “It is now included in pretty much every non-Windows operating system made. It is included in network switches, in half of Cisco’s products, and who knows where else. It is used by everything from Arrecibo to the Greek Army to who knows where else. And what have we gotten for it in return? Pretty much nothing at all.”

While there are other, proprietary SSH implementations, OpenSSH is by far the most widely used. And while the proprietary competition is charging $150 per workstation license, OpenSSH is charging nothing.

Other projects could theoretically fork OpenSSH, but shouldn’t a network communications program that has as much power as OpenSSH be developed by programmers who live for greater software security? OpenSSH isn’t some cheap utility like telnet or BSD Mail — it’s the only secure way for most server operating systems to securely communicate with a sysadmin’s client terminal over a TCP/IP connection. Even if you don’t regularly use OpenSSH, a program that you rely on (the scp command, for instance) may need OpenSSH to create a secure tunnel over a network.

OpenSSH also achieves a more secure codebase and more security-related features because the programmers who work on it also work on OpenBSD.

“People seem to think, ‘OpenBSD is not what I run, so I don’t need to help them.’ I worry that this is what holds people back from doing the right thing, which is to fund OpenSSH, and thus OpenBSD will survive and improve, and then any improvements in OpenBSD will drive improvements in OpenSSH.

“Like when OpenBSD got so much address space randomization and propolice, but that magic day when we realized that every OpenSSH sshd process was still an address-space-clone of the parent. That is because every connection you make causes the parent sshd to fork, and this new process has the same propolice cookie, the same address space layout, the same random stack gap at the top, and even the same malloc layout. That is when we re-architectured OpenSSH so that it instead does a fork + execve, so that the new processes would be dissimilar to each other. That kind of approach would never have come out of any other development group.”

Some anti-BSD zealots have privately entreated OpenSSH programmers to split OpenSSH from the OpenBSD project in order to protect it, but OpenBSD’s stewardship is not the issue. While funding for OpenBSD has dwindled below critical levels, OpenSSH will not go down with the OpenBSD ship, so to speak. The issue is that OpenSSH, regardless of which programming team maintains stewardship of it and despite its critical importance to system administration, is not being monetarily supported by the companies and users that rely on it. Without full-time programmers working on it, OpenSSH’s legendary security could suffer.

All take, no give

Some of the OpenSSH freeloaders, like Apple Computer and The SCO Group, are notorious for reaping financial rewards from selling open source software bundled with their proprietary products. It’s no surprise that both of these corporations include OpenSSH in their operating systems without giving back to the programmers who make it all happen, but what about companies that are vocal in their support of open source software?

When asked what Novell would do if OpenSSH were no longer an option, and how much the alternatives would cost, company representative Bruce Lowry had this to say:

“As I know you’re aware, Novell is an active and constructive member of the open source community. We participate significantly in more then 30 open source projects including AppArmor, Hula, Gnome, KDE, Mono, OCFS2,, Samba, YaST and XEN. We participate in many different roles, in some cases sponsoring the whole project, in others employing key maintainers or giving back enhancements and bugfixes to the community. We acknowledge that openSSH is an important piece of our operating system. But the SUSE Linux distro also includes around 1000 other open source projects.”

“Instead of supporting projects with financial resources, our policy is to give and share code. This is largely how the community works and this is what our customers and users expect from us. Projects which think they need more then code exchanges to survive have generally had to look at establishing a business plan and structure to support long term viability.

“With regard to your specific questions, these are all speculative, so we can’t really reply concretely. If openSSH halted development, Novell would evaulate next steps. It’s possible that Novell and a consortium of Linux vendors would agree to continue the work. We would cross that bridge when we came to it. On question two, it’s difficult to estimate to possible costs of a project that is not currently planned. Successful open source projects are very organic, and are driven by community and vendor interests. If work on the openSSH project were to cease, it’s likely that other community members would step up and help keep the development going. It’s impossible to guess what Novell’s role in a new openSSH project might be.”

Since the release of Solaris 10, who has been a larger open source software cheerleader than Sun Microsystems? I asked Sun representatives what they would do if OpenSSH were to disappear. The only response I got was that there are parts of Solaris that compete with OpenSSH, and that because of this, the company would rather not comment further on the issue. Presumably Sun is referring to SunSSH, an OpenSSH derivative included with Solaris, though it’s likely that the Sun no-commenters were not aware of SunSSH’s heritage.

Upon learning of Sun’s competitive view of OpenSSH, Theo de Raadt told me, “People who care about having the best SSH on their Solaris machine immediately replace SunSSH with OpenSSH, because SunSSH is based on a 5 year old version of OpenSSH. Even more scary, Sun disabled our privilege separation security code for the pre-authentication phase (i.e. in the most risky part of the software). SunSSH was heavily tweaked to support Trusted Solaris, but in the process they totally demolished it.”

International Business Machines (IBM) is also a public supporter of open source software — primarily GNU/Linux — but are they all hat and no cattle when it comes to supporting actual open source developers? IBM includes OpenSSH in z/OS, AIX, and OS/400, which in turn control the company’s most expensive and powerful machines. But when pressed for comment on what they would do in the event that OpenSSH should slow development, no one seemed to have an answer for me. My questions were passed from employee to employee, never finding someone who knew what OpenSSH was or what AIX and z/OS would do without it. At the time of this article’s publication, IBM did not have any comments to offer. Perhaps they were too busy punting their customer support complaints to the OpenSSH programmers:

“As a side note,” said de Raadt on an OpenSSH mailing list, “earlier today IBM Support actually sent an energy company with whom they have a multi-million [dollar] support contract to our private development mailing list saying we had to fix a customer bug. I was shown an extensive set of IBM support emails with the customer where they were refusing to take responsibility for the issue, and finally told their customer that OpenSSH was responsible for fixing their problem. I say shame you, IBM, SHAME ON YOU. You take their money and want us to make your customers happy.”

Like IBM, Red Hat passed my questions around from desk to desk, eventually telling me that they had no comment on what they would do if OpenSSH were to cease development. Perhaps it’s just too difficult a task to find an engineer who can comment on one of the most important networking tools in the operating system your company is selling. This could just be a coincidental, collective PR failure by several companies that, for the most part, generally have no trouble commenting on highly technical software issues. Perhaps, though, there is more to it — Sun, IBM, Red Hat, and Novell all sell Linux-based operating systems that compete with OpenBSD. Do they have an interest in watching OpenBSD suffer and fail, even if it means losing OpenSSH in the process? Such an attitude could be the biggest case of nose amputation the face of the operating system world has yet seen.

Other OpenBSD contributions

OpenBSD’s contributions to the larger software world are not limited to OpenSSH, and you don’t have to use OpenBSD to benefit from them. Many of the technologies developed for OpenBSD are ported to other operating systems, such as the packet filter (pf), an advanced firewall framework. Both FreeBSD and NetBSD include this OpenBSD-authored software in their base system. pf is often combined with another OpenBSD creation, Common Address Redundancy Protocol (CARP), for firewall failover protection.

Among OpenBSD’s greatest contributions to programming are the strlcpy and strlcat C libraries. These make the process of copying and combining strings in C programs more secure. Since being incorporated into OpenBSD several years ago, they have also been added to Solaris, NetBSD, FreeBSD, and OS X.

OpenBSD programmers also squash important bugs in ancillary software such as In one recent example, OpenBSD helped discover and fix bugs in’s pixmap library that had been there for ten years:

“The bugs we fixed are indeed a good illustration of how the work we do on OpenBSD benefits a larger community,” said Mark Kettenis of the OpenBSD project. “We were able to catch the bug because we now have a memory allocator that is very unforgiving, and have it turned on by default. That made many people see X11 crashing and eventually provided us with enough clues to fix the bug. Other systems have a similar memory allocator, but since it’s not turned on by default, most people run without it. The interesting thing about these particular bugs is that they don’t disappear when you use a more forgiving memory allocator; that would just make X11 crash less often. For desktop users, having X11 crash is just as annoying as having the entire machine crash, so fixing these bugs is important even if they wouldn’t have an impact on security of the system.

“Once we find bugs, we usually go through our entire base system to fix similar bugs. Since we have several other pieces of software in our base systems that are also used on other operating systems (gcc, sendmail, perl, apache, etc.), we fix bugs in those packages too. In most cases we send our fixes to the maintainers of that software, so in the end everybody using the software will get the fixes. And in many cases, even before they’d have noticed the problem themselves.”

How you can help

If the big corporations won’t help support the software that they rely on, perhaps it is up to the users to take action. So how can you help keep OpenSSH and OpenBSD going? The easiest way is to make a donation to the OpenSSH project (or, if you prefer, to OpenBSD). But if you want something more than mere satisfaction, you might consider buying an OpenBSD CD set. Aside from helping to support OpenSSH and OpenBSD development and the general benefit to computer software security that the continued development of these projects provides, it’s also the best way to get one of the world’s most interesting operating systems onto your computer.

About the Author:

Leave A Reply

Your email address will not be published. Required fields are marked *