The problem with tech reviews

I’m a pretty compulsive comparison-shopper (that is, a compulsive comparer — I don’t actually buy very much, as seen by my 4-year-old Creative Zen). I’m also a wannabe tech geek. So I read a fair number of reviews of TVs, digital cameras, MP3 players, printers, etc. And I’d say a good three-quarters of them are infuriating — because they barely discuss the one or two key aspects of a product that normal consumers care about.

Take two recent reviews from PC Magazine and PC World. PC Mag gave four stars (out of five) and an Editor’s Choice award to the Westinghouse TX-52F480S 52-inch LCD. I still have an old 32-inch CRT set, so I’m always on the lookout for good flat-panel tellies to file away for when we’re ready to upgrade. But despite the rating, this review was absolutely no help in my mental TV search.

The first paragraph says “this set has picture controls that help its more than two million pixels deliver crisp, detailed images” and calls it “a terrific value for a large-screen 1080p HDTV.” The rest of the review consists of lists of features I don’t really care about or techno-geek discussions of image quality.

First there are four paragraphs about the bezel, dimensions, remote, A/V ports, and input detection. The first real mention of image quality comes six paragraphs in, noting

a handy picture-scaling option that eliminates overscan when displaying 720p and 1080i/p HD video. This scaling can be achieved over component video input as well as through HDMI. Standard-definition (SD) images were overscanned by an acceptable 6 percent.

I don’t know or care what picture-scaling and overscanning are. Judging by this mention, I think I should care — but the review doesn’t explain why or what these terms mean in plain English.

Next we have paragraphs about color settings, brightness, and contrast ratio. From reading up a bit on TVs, I at least know what these terms are. But this means nothing to me: “dark-video black levels coupled with a correctly calibrated picture produced an average contrast ratio of 1186:1, a new record for an LCD in a darkly lit room.” What does that mean for actual human beings watching the TV in their living rooms?

Finally we get two paragraphs that talk image blur and jaggedness in terms of real-world use, with examples of Indiana Jones’ hat and Darth Vader’s helmet. Those are interspersed with more lab-test-speak like:

Despite some stumbles on the HD HQV Benchmark test, performance with HD material was good. As with the SD version, the HD benchmark test highlighted the TX-52F480S’s inability to eliminate jagged edges along moving bars on the suite’s video-reconstruction test.

The review concludes, “Overall, the TX-52F480S sets the current standard for what you should expect in an LCD TV priced at around $2,000.” Overall, I would never buy this TV set because to an average reader like meself, the review in no way reflects this conclusion.

A good review wouldn’t go down the checklist of TV features and talk about each in turn — just as a good movie review doesn’t go down the checklist of acting, directing, script, sound, cinematography, set design, costume design. A good review would explain in real-human terms why this TV’s image quality — the only feature that actually matters — is better than other TVs’. And it would do so at the top of the review, not eight paragraphs in.

Moving on to digital cameras, PC World’s review of the Canon PowerShot SD1100 IS has the same flaws as the PC Mag review. There’s a mention of “outstanding image quality” in the first paragraph, but then we get four paragraphs about the lens, “burst capability,” rounded edges, buttons and menu, white balance, and face detection. The sixth and penultimate paragraph is this:

In our lab tests, the SD1100 scored higher on overall image quality than nearly all of its competitors. On the sharpness scale, only a handful of the cameras we tested beat the SD1100 (including Kodak’s EasyShare V1253, Fuji’s FinePix F50fd, and Casio’s EX-Z1080), but they all cost more. To combat camera shake in low-light situations, Canon added its Optical Image Stabilizer feature, but I found its presence pretty unnoticeable–as it likely would be on most point-and-shoots.

Image quality is the one thing that should matter to average camera users. Yet this paragraph talks only about lab tests. It isn’t accompanied by examples of actual photos taken with this camera, juxtaposed with other cameras’ shots so readers can compare the two.

I realize these are both “enthusiast” magazines/Web sites, and readers are assumed to both understand and geek out over these technical terms and features (CNet’s review of the Canon camera has the same problems). But even a tech-savvy reader looking for a camera or TV isn’t going to buy a product because of the remote or menu buttons.

This reviewing style has filtered out of the enthusiast press. In his recent review of small high-def camcorders, the New York Times’ David Pogue spends 745 words talking about the video-cams’ features and descriptions before getting to the first big flaw — that they have no wide angle, so are useless for non-zoomed shots. It takes him 922 words before he gets to the most important part of any video camera: the image quality. And surprise! It turns out these cameras kinda stink:

When you hear “high definition,” you expect what you see in the TV stores: breathtaking sharpness, stunning color.

Unfortunately, “high definition” refers only to the number of pixels in the picture — not how good they are. On these cameras, they’re not very good.

Why is this not the lede of the review? Who cares about how small the things are, what sort of “sharpish edges” they have, if the cameras fail in their main function?

Here’s my philosophy for tech reviews: Don’t spend more than one paragraph on specs or anything else readers can find on the company’s Web site or press release (unless certain specs are relevant to your overall point). Don’t focus on lab tests and jargon; instead, use lab benchmarks as a supplement to real-world use and real-world language to evaluate a product’s primary functions. Most important, focus your review on the purpose of the product. If it’s a printer, the most important thing is print quality. For a scanner, camera, videocam, or TV, it’s image quality. The rest is techie noise.

(You can decide for yourself how well I’ve followed my own advice in these reviews of the PSP, a virtual-reality headset, and the Xbox 360 — though time has made my initial 360 criticisms moot.)