What’s Wrong with Wine Ratings?

Question marksThere are two sides to every fence, to every coin, to every argument.  This also holds true for wine ratings.  I am on the side of not giving a wine I taste a rating on a 100 point scale.  I prefer to describe the wine I tasted, it’s style, if it may pair with some food, etc.  Other people prefer to only buy wines that a wine reviewer / critic gives 90 points or more.  Why spend money on lower rated wines?  But could you tell the difference between an 89 point wine and a 90 point wine?  How much more money would you be willing to pay for the extra 1 point?

What are Nominal, Ordinal, Interval, and Ratio Measurements?

Besides these musings, I have some, more statistically related issues to point systems that I’d like to share with you.  What do Nominal, Ordinal, Interval, and Ratio measurements have to do with Wine Rating Systems?  Before I answer this question, I’d like to briefly outline these 4 measurement scales, then tell you how the wine rating systems fit in, or not.

apple and orangeThe Nominal scale is based on comparing items by “name”.  When we talk about fruits for example, we can list oranges, apples, and grapes.  Each fit on the fruit nominal scale but you cannot say that one item in the scale is greater than or less than another item.  You can only say that two items are either equal, as in 2 oranges, or not equal as in an orange vs a grape.

hot scaleThe Ordinal scale provides a descriptive difference between items you are measuring.  A typical ordinal scale is “bad, good, better, best“.  We know that one is better than the other, but there is not quantitative measurement between the scale items.  You can’t measure the difference between bad and good. It is subjective; it is not 5 units or 10 units better.  And you cannot say that the difference is the same amount between the scale items.  Maybe there is a larger measurement gap or distance between “bad vs good”, compared to “good  vs better”.  There is no specific measure of magnitude of difference.

fahrenheit and celsius scalesThe Interval scale provides a numerical measurement between the items that are being measured, but does not have a natural zero point.  The interval type allows for the degree of difference between items, but not the ratio between them. An example of this kind of measurement scale is the Fahrenheit temperature scale.  Zero degrees Fahrenheit is an arbitrary zero point, as you can still have negative degrees as well as positive degrees.  Because of this you cannot say that 20°F cannot be said to be “twice as hot” as 10°F. You can measure differences between values; for example, there is a 10 degree difference between 20° F and 10°F.

speed limit signsThe Ratio scale provides the most sophisticated method of measurement as it has a natural zero point, from which all measurements are based.  An example is driving speed.  When you are stopped you are at 0 km/hour.  When you start driving you can be at 50 km/hr for example.  You can say that 20 km/hr is twice as fast as 10 km/hr.  The other feature is you cannot have negative values, e.g. you cannot be driving -20 km/hr.  Everything starts from zero. With a ratio measurement, you can measure differences, as well as ratios.   40 km/hr – 20 km/hr = 20 km/hr makes sense.  Also 40 km/hr is twice as fast as 20 km/hr also makes sense.

Wine Rating Systems and Measurement Scale

So based on the above 4 measurement scales, where do the 100 point-based Wine Rating systems belong?  I would hope on the Ratio scale, but my guess many are on an Interval scale.  If the Wine Rating systems were on a Ratio scale, they all should have a natural zero point.

Should zero = something neutral like water? OR should something like water be given a rating of 50, and zero would be some wine that has all possible flaws, thus tastes utterly horrible?

I have noted on eRobertParker.com that they rate wines from 50 – 100 points, which would put it into the Interval scale.

eRobert Parker Point Rating  Rating Description
96 – 100 An extraordinary wine of profound and complex character displaying all the attributes expected of a classic wine of its variety. There are few wines that actually make it into this top category because there are few truly great wines.
90 – 95 An outstanding wine of exceptional complexity and character.  In short, these are very good wines.
80 – 89 An above average to very good wine displaying various degrees of finesse and flavour as well as character with no noticeable flaws.
70 – 79 An average wine with little distinction except that it is soundly made.  In essence, this covers straightforward, innocuous wines.
60 – 69 A below average wine containing noticeable deficiencies, such as excessive acidity and/or tannin, an absence of flavour, or possible dirty aromas or flavours.
50 – 59 A wine deemed to be unacceptable

WineAlign.com also uses eRobertParker.com’s wine rating scale.

Jancis Robinson uses a 20 point scale, but I did not notice what a zero point wine would be, so I would put it in the Interval system.  Her ratings are:

  • 20 – Truly exceptional
  • 19 – A humdinger
  • 18 – A cut above superior
  • 17 – Superior
  • 16 – Distinguished
  • 15 – Average, a perfectly nice drink with no faults but not much excitement
  • 14 – Deadly dull
  • 13 – Borderline faulty or unbalanced
  • 12 – Faulty or unbalanced

Wine Spectator uses a 100 point scale (Interval):

  • 95-100 Classic: a great wine
  • 90-94 Outstanding: a wine of superior character and style
  • 85-89 Very good: a wine with special qualities
  • 80-84 Good: a solid, well-made wine
  • 75-79 Mediocre: a drinkable wine that may have minor flaws
  • 50-74 Not recommended

For Decanter.com, I have seen 3 rating systems; one being a 100 point scale, a 20 point scale, and a 1-5 star rating.  The star rating would fall into the Ordinal rating system, which makes sense to me, as it gives a reader an indication of quality of the wine, without putting it on an absolute scale.  I think most people that just want to buy a good wine for a dinner party, for example, can better understand the difference between a 4 vs 5 star rated wine, than between a 89 vs 90 point wine, and make a quicker decision.

Finally I found a website, www.delongwine.com that made a nice PDF summarizing the different systems of the above plus others.

The Problem

The problem with the above Interval scaled numerical rating systems is that for most people, you cannot necessarily compare the same number between different systems.  Is an 80 point Wine Spectator wine equal to an eRobert Parker 80 point wine?  We cannot say as we do not quantitatively know what the zero point (in this case 50 point) means between the different systems, as they use a description, such as “Not recommended” or “Unacceptable“. If we can get the different systems to agree on a starting point, such as water = 50 points, or some simple wine with specific aroma and flavour profile, then maybe we have a chance to compare.

Also with more and more wine bloggers putting fingers to the keyboard, they may also be using a point scale, but how did they decide on the scale to use?  Are they using one of the above systems, AND do they really understand the limitations of those systems?

The other problem, is should there be separate rating scales for red, white, rose, sparkling, and dessert wines?  Each could in theory have their own zero point.  I think I will let this go for now.

If we cannot agree on a zero point, or if each critic / magazine, does not specifically state what their zero point is, I think we should not use a numerical scale, as it implies a level of precision that does not exist.  I think an Ordinal scale, such as 1-5 stars, or A-F, would be more appropriate.  Comments?

Drink Good Wine. That is my motto and I really want to help you drink good wine. What is good wine? That can be a different thing for each people. Some people prefer red wines. Some only like Cabernet Sauvignon. MyWinePal was started by Karl Kliparchuk, WSET. I spent many years with the South World Wine Society as the President and then cellar master. I met many great wine makers. I love to travel around the world, visiting wine regions and sharing my passion for food & wine with you. Come live vicariously through me, and enjoy all my recommended wines.