It all began when I saw @euanlindsay tweet:
This followed with a healthy discussion on the state of product reviews and how their credibility is diminishing. Are they just paid advertisements? Does advertising spend ensure a positive review? Do you duped by most reviews these days? Why do you never see a negative review? I have so much to say on this topic now that I’ve seen all three sides of the fence (the publisher, the advertiser, and the reader). It’s far too much to write in 140 characters in twitter so I thought it deserved some attention of it’s own.
I was going to write something but I first had the conversation with the CTech editor Matt Wikstrom about his views. He has some good insights so I’ll leave this one to him, and even more importantly I’m keen to hear your comments on what you think.
A Fair Review
by Matt Wikstrom
Buying bikes, cycling paraphernalia, or components can be expensive, so we often look to product reviews to help us with our decision. But where can we find a fair and impartial review? In this post, I argue that there is actually no such thing as an impartial review.
I think we’re all well aware of the sales agenda that underlies any product placement in popular media. So it goes with any product that appears in the articles of a cycling magazine or website, even when it is being tested or reviewed. Wade has always been concerned about this perceived conflict of interest for Cyclingtips, so much so, he appointed a tech editor. But the reality is, the only reason we receive products to review is because the company involved wants some exposure. It’s a symbiotic relationship because both parties benefit from the exchange. [CT: many product review on this site are also done because of the simple fact that I own them and I've been keen to tell you about my experiences]
So where can we find an honest review? Professional riders can’t always be trusted because they’re paid to represent their sponsors, and besides, riders win races, not their bikes. We all know about salespeople too, they’ve got to move their stock, so they’ll say anything to close the sale. The same applies to anyone that works in the industry, regardless of the gossip or insider knowledge they might be willing to share. Magazines and websites are going to protect their advertising revenue, so that pretty much leaves your riding buddies, and while you may trust their opinion, they probably lack the depth of experience to provide a fair review.
What about the guys in the workshop? As a source, mechanics are a fine choice. One of the reasons shop owners keep them in the workshop is that they can’t sell to save their lives. They’re also likely to have plenty of experience with a wide range of gear, both new and used, so they’ll be able to provide you with an honest assessment of the reliability and serviceability of a product. However, mechanics have their preferences and they can be exploited to represent a brand too, if only because they’ve just been offered a good price on a new groupset.
At this point, all we’ve got left are customer reviews, a hit-and-miss affair depending on the customer’s experience and their ability to articulate their thoughts on the product in question. Honestly, the best you can hope for after plumbing the depths of customer reviews is to gauge a product’s popularity.
So is there no such thing as an honest review? After all, magazines and websites are free to publish what they please according to any agenda they choose.
Perhaps the answer lies in the reliability and reproducibility of a scientific approach? If we can’t trust a human to reliably and impartially report on the performance of a cycling product, can an instrument do the work for us? If you’ve been a regular reader of RIDE Cycling Review for the last few years, you’ll have witnessed the evolution of this thinking in its pages. The test-rider still provides an opinion on the bike in question, but now it is complemented by hard data on some of the characteristics of the frame. This includes measures of deflection under load and most recently, telemetry-like output on road shock through the frame. I find this stuff absolutely fascinating, but I can’t work out how it translates to my experience on the bike in question. I’m not critical of the measures per se, but ultimately, their interpretation depends upon a human’s opinion of the data, which brings us full circle. Moreover, if you compare the test-rider’s impressions with the data across a number of bikes, then you will find instances where the rider’s impressions defy the predictions of the data, and vice versa.
Once again, I’m forced to conclude that in absolute terms, there is no such thing as an impartial review. But does that does that mean all reviews are worthless? Absolutely not. In fact, I find that thoughtful and informative reviews contain some of the stuff of cycling life. They feed my interest in all things bike.