Product reviews are a big part of what we do here are CyclingTips so it’s important that the system we use to score products is consistent, easily understandable and transparent.
While our existing system has served us well for the past few years, we’ve spent some time developing a new way of rating the products we test. This rating system can be used for any product, making it easier than ever to compare products against one another.
What you get
At their heart, product reviews are always going to be subjective. Different people find different characteristics appealing, and we all have different ideas about how a product should work or what features it should have. So when you read a CyclingTips product review, you know you’re getting a subjective view on the merits and failings of that product.
If you’re looking for a whole raft of empirical measurements about a particular bike (bottom bracket flex, for instance), then you’re probably looking in the wrong place. Here at CyclingTips we’re all about how the product performs. We’d rather tell you how something feels rather than how far it flexes.
We also prefer to give you a sense of what it’s like to live with a product rather than just use it once or twice. We spend many hours assessing each product, using it over and over, giving it a decent run under a variety of conditions.
Unfortunately any long-term assessment is generally beyond the scope of our reviews since the products have to be returned, but reader’s reviews can often provide extra insight on this issue. If you already own the product in question, please leave your own review so everyone can see how the product performs in the long term.
And finally, we always try to be as independent and objective as possible when putting our reviews together. Sure, we might have commercial arrangements with some of the companies that supply us with products, but we don’t let that get in the way of an honest and up-front review.
How we rate products
Each review ranks a product against five criteria: function, form, appeal, marketing claims and serviceability. The product is given a rating of between 1 and 10 for each of those five criteria.
After reviewing a variety of products for many years, it’s clear that very few products fail to perform at all. There have been a few exceptions, but the market is too competitive for defective products.
For our review process, a product will be given an automatic pass mark (5/10) provided that it’s not defective and unfinished, falsely promoted, poorly manufactured, unsafe to use or it malfunctions. We’re not afraid to criticise a product for its failings but we also need to be fair.
The criteria
Here’s an explanation of what we mean by each of the five scoring criteria and what a particular score means:
Function
When we rank a product’s ‘function’ we ask ourselves: “Does the product do what it’s supposed to do?”
1-4: product doesn’t do what it’s supposed to do
5: The product didn’t fail
6: Generally performs well but has weaknesses
7: Performs well; excels in some regards but has weaknesses
8: Strong performer; excels in many regards but has a weakness
9: Excels in most regards
10: Exceptional
Form
Is the product visually appealing?
1-4: product is so spectacularly ugly that you would be embarrassed to use it
5: Finished adequately/not ugly
6: Looks mildly interesting
7: Interesting
8: Eye-catching
9: Attractive
10: Thing of beauty
Appeal
A product’s appeal is about more than it’s aesthetic beauty: does this product make you want to buy it and use it?
1-4: Not appealing in the slightest. Do not want.
5: Neutral
6: Mildly appealing
7: Moderately appealing
8: Interesting product
9: Outstanding
10: Will sell a body part for it
Marketing claims
Does the product live up to its marketing hype and claims?
1-4: The marketing claims are over the top and not nearly achieved
5: An acceptable level of hype
6: Generally true, though some overemphasis
7: True with little overemphasis
8: All claims are true
9: True with some acknowledgement of limitations
10: Honest account of all strengths and weaknesses
Serviceability
If the product breaks or falls into disrepair, how hard is it to get it fixed? The products that score highly here are the ones that everyday cyclists can fix on their own, without any specific knowledge.
0-4: product malfunctions/fails
5: product is extremely difficult to install, service or operate, but it can be done
6: product is difficult to install, service or operate but it gets easier with practice
7: Product is a little more difficult to service, install or operate than similar products
8: Product is no easier or harder than similar products to service, install or operate
9: Product had features that makes it much easier than normal to service/install or operate
10: Product needs no servicing or is extremely easy to service, install, or operate
Weighting
We recognise that these five criteria aren’t as important as one another. Whether a product is easily serviceable, for example, is not nearly as important as whether it actually does what it’s supposed to do.
With that in mind, we’ve spent plenty of time working out how each of the five criteria should be weighted when calculating the product’s final score (out of 10):
Function: 40%
Form: 20%
Appeal: 20%
Marketing claims: 15%
Serviceability: 5%
To get to a product’s final score (out of 10) we take each criteria score, multiply by its weighting, then add all the criteria scores together, rounding to one decimal place.
Show your working
Take the case of the Storck Aernario, for example. We gave it the following scores:
Function: 10
Form: 9
Appeal: 9
Claims: 9
Serviceability: 6
Therefore the bike’s total score is calculated as follows:
Total score = (10 x 40%) + (9 x 20%) + (9 x 20%) + (9 x 15%) + (6 x 5%)
= 4 + 1.8 + 1.8 + 1.35 + 0.3
= 9.25 (rounded to 9.3)
Revisiting old products
We’ll be going through all of the published reviews when we get the time, and updating them to the new rating score. This will allow you to compare any product against another.
Questions or comments
If you’ve got any questions or comments about the new rating system, you can contact us at [email protected]
-
Sam