I'm long past the point of being bothered by people who write angry letters to me (or in today's world, writing blog posts about me) because I didn't favorably review their preferred operating system or software program. If I'm emailed with an assault on my credibility, sometimes I write back and explain my testing methods, my experience, and how I reach my conclusions. Rather than continue to repeat myself, I've decided to take the time today to write this article so that I can link to it in the future.

Review testing

First I will explain my testing procedure, because this seems to be a frequent hangup. When evaluating an operating system, program, or hardware component, there are three distinct levels of testing:

1. Basic functionality: does it install correctly and work as expected/advertised?
2. Compatibility: does it work only with a small number of machines, or can I run it on pretty much anything?
3. Hackability: how hard is it to make this do what I want it to?

If I can't get something to work at all, I usually do not publish a review of it.

I purposefully use hardware that I think will be problematic because I assume that older mainstream hardware will work perfectly. People read reviews because they want to know if something will work with what they have, and if it will do what they want it to do. If an operating system does not work on a Core 2 Duo system, readers need to know that. If the 64-bit version of a program doesn't work as well as the 32-bit (or if there is no 64-bit binary available), readers need to know that too. So I test on two architectures with opposite brands (AMD/ATI in one machine, Intel/Nvidia in the other), and on laptop computers.

Problem solving and bug reports

If an operating system or program can't be installed without having to work around programming bugs and hardware problems, then that is a huge negative point. If unincluded documentation or prerequisite knowledge is required to install and configure your product, then that also counts against it in a big way. The target userbase -- the people for whom the software is designed -- should have access to all necessary documentation at all times and without a network connection. So rarely is that the case these days, with most documentation being relegated to hidden FAQ sections on Web sites, how-to guides that are many versions old, and wikis that are grossly incomplete. Online documentation is no documentation because all too often, someone who is having trouble cannot get online to learn how to fix the problems he's having. Even if he can get online, how is he supposed to know which site to go to, or what to put into Google to see the right articles? Mailing lists and forums do not count as documentation, either. If your software doesn't work as intended or advertised, and I am required to search some mailing list to find the fix, then expect me to say that your product stinks.

Having said that, requiring documentation to install or use the software is not necessarily a bad thing as long as the documentation is included with the software and is easy to understand and follow. This has nothing to do with the complexity of the installation or configuration.

I do not file bug reports for the growing number problems I discover in everything I evaluate. Filing reports on all of them would take hours, but it's not just the time investment that shoos me away -- I actually used to file bug reports. I stopped because I found that in the staggering majority of cases my bug reports were quickly closed or disregarded. I was usually told that whatever problem I reported had been fixed in CVS already (even though no previous bug report existed), that it's an "upstream" problem with a dependent piece of software, that my year-old (or older) hardware is "too bleeding edge," or that my operating system was not supported (despite the fact that binaries were provided for it). If I was using the "stable" release of a product, I was told that the problem was fixed in the development release and I should use that or wait a few months for the next stable release; if I used the development release, I was told that problems were to be expected and that I shouldn't file a bug report unless I am also submitting a patch to fix the problem. Why bother wasting my time on bug reports that are destined to be tossed out by lazy programmers?

I now leave bug reports for the people who actually write and submit patches, or people who want to deal with ornery programmers who defend their product against all bug reports, foreign and domestic. The best you're going to get out of me is an article or review that contains a brief description of my hardware, an explanation of the problem, and my best guess as to what might be causing it. You want a bug report? File it yourself. My duty is to readers, not to programmers or hardware and software companies. I'm not obligated to make other people's projects or products better; all I'm obligated to do is to tell you, the reader, what you can expect from them. Having said that, I gladly respond to programmers who ask for more details on the problems I've written about. At least in those cases, I have a reason to believe that I'll be taken seriously.

No operating system should ever be released with known bugs. Somehow, though, many reviewers have gone soft -- especially on GNU/Linux distributions and free software/open source programs. They think that advocacy of free software is more important than exposing a program's or software distribution's warts. Just because your product is open source or free-as-in-rights doesn't mean that I or any other journalist owes you a glowing review, and it also doesn't mean that you deserve any slack when comparing your project to a proprietary competitor. At the risk of offending some of my colleagues, I will say that there is no shortage of professionally- and unprofessionally-managed Web sites and media networks on the Internet that will gladly write a puff piece on your open source program no matter how grievously it sucks. I am proud to say that mine is not one of them.

I remember the good old days when you didn't release software that you knew to be broken. I think that philosophy got trashed with the notion of "release early, release often" that extreme programming recommends. The "release notes" or "errata" file, which was once used for good and noble purposes, is now firmly within the domain of evil. Instead of some introductory documentation, configuration tips, or warnings about strange hardware incompatibilities, the release notes file is now too frequently used as the one and only place where the development team tells users about the bugs that they didn't feel like fixing. Heaven forbid you don't read that file, which gets longer and more tedious with every release as programmers increasingly concentrate on exciting new "rock star" features instead of squashing bugs and writing documentation.

Reader threats: the good, the bad, and the silly

Writing technology articles is a fascinating study in human behavior. I'm frequently amused by the lengths that readers will go to to try to undo what they perceive as bad PR from my reviews or articles. One disgruntled reader was so convinced that I was lying about hardware compatibility problems with his favorite operating system that he started an anti-Jem Web page (sadly it is gone now) that had more words than my review did. Pamela Jones of Groklaw was so offended by this editorial that I wrote that she reportedly threatened (behind my back) to "humiliate" me with her blog, called for editorial censorship of my article, and scolded Newsforge editors for not toeing the party line -- all because I said that most users don't value the four freedoms of the GPL because they take the freedom they need via software piracy. (What stopped her, I wonder? Maybe Maureen O'Gara "humiliating" PJ with a ramshackle exposé made her think twice about threatening to do hatchet-jobs on innocent people over the Internet?)

High-handed bloggers aside, some blogless readers want their reactionary message to get out to as many people as possible, so they will go around like Paul Revere and copy-and-paste their review of my review to every site that links to it. Presumably they are hoping that the snarky, misspelled comments they spent 15 minutes writing will change people's minds about findings, data, and analysis that took days to collect and publish, or that I will personally read what they wrote and shake my fist in anger.

I've had people call me all kinds of unusual names, swear at me in several different languages, threaten me with physical harm, make interesting guesses as to my sexual preference, make fun of my name, make fun of the way I look, telephone me in the wee hours of the morning while I'm on vacation, tell me that they make more money than I do, accuse me of lying, assess my IQ, analyze my writing skills, threaten to sue me, and make recommendations for potential future careers for me. For the first year that I wrote reviews, the negative comments would bother me because I took them seriously. Now, though, I eagerly look forward to rants because they're most often presented in such a clownish fashion and I'm curious to see what feelings and assumptions people project onto me based on one article that they did not agree with. It's also amazing to me that a few particularly motivated people get so upset over one journalist's honest product assessment that they'll spend a great deal of time and energy trying to cook up some sort of retaliation.

All of the people who attack reviewers share a common thread: they label an article "bad" if they either don't agree with it or if it says something that they feel should not be widely known (airing dirty laundry), and label an article as "good" if it supports their cause -- even if that support is untrue or misleading. That's a lovely fantasy, but this is not how it works in reality. In the real world, a good review is one that lauds every significant strength and exposes most or all of the weaknesses of a product from the frame of reference of the target market. In rare cases, a product can have so many negative points that the positive points don't matter. A bad review fits so many disparate qualities that I don't think I could list them all. But to fans and zealots, the only good review is one that says nice things about their product of choice; reviews that expose weaknesses or criticize poor design or development are bad and must be destroyed.

Few people respect the amount of time I spend evaluating computer hardware and software (and books now, too). This is my full-time job, and I spend anywhere between 50 and 80 hours a week testing things, reading manuals and licenses and books, and writing about what I find. I'm not a "blogger" -- I'm a real journalist who publishes online (previously for a big tech media company, but now I'm on my own), and a real author of several books and electronic PDF guides (many of which have not yet been released). I do have a blog, and like everyone else's, it is really lame.

I got into this business four years ago because I was sick of being misled by crappy computer hardware reviews that didn't tell me about the weaknesses and flaws in products. I figured I could do better myself, so I reserved a domain name, bought a shared hosting account, and started posting reviews of computer parts I had. I do a lot more than that now, but my mission is still the same: to publish the best technology articles on the Internet. So if you, the reader, are really that motivated to silence me or any other journalist, the absolute last thing you should do is sit around and whine about it on your lame blog. All that does is bring us more traffic and validate the very opinions you're attacking. Instead of wasting your time trying to think of a name I've never been called before or an insult that will get to me when thousands of others could not, close your browser window, open up your IDE and work on coding better software, take my review findings and file your own bug reports on them, join my forum and patiently help people with computer problems, or offer to help me with future articles and reviews. Send an email and say, "hey, next time you do a review on this, let me know. I'd love to help you work through any problems so that we can make our product better." Offer to do a technical edit of a review or article, or if you are a company or project representative, offer to make some meaningful on-record comments that can be quoted positively for future articles. Try taking positive action; perhaps you will find it rewarding.

Discuss this article or get technical support on our forum.

Copyright 2006 Jem Matzan.