Wine magazines that conduct blind tastings using experienced wine tasters offer valuable and unbiased information to their readers as to the quality of wines. However, the value of such ratings are often questioned, as it is common for different ratings to be given to the same wine across different wine tastings. So why does this occur?

We can answer this by first exploring the circumstances under which the quality ratings given would always be the same. If we had a pool of tasters who perceived the sensory quality of the wines identically, interpreted quality parameters identically, and used exactly the same criteria for assessing quality, then yes, we would achieve identical quality ratings. However, quality is not absolute. It varies according to the context in which its is being viewed, the criteria being applied when viewing it, and to some extent by who is doing the viewing.

A typical quality assessment panel consists of a three of four experienced wine tasters. The individual tasters first assess the quality of a group of wines independently (and always blind) and then briefly discuss them before reaching a consensus. The anonymity of the wines is maintained until a decision is reached.

The evaluation of the wines is partly dependent on the combined performance of the panel's sensory equipment. The actual degree of taster variation is not widely appreciated. Sensory scientists use a rule of thumb that typically the most sensitive five percent of the population will be 250 times more sensitive to an odour than the bottom five percent. Some tasters have up to 30 times the number of taste receptors than others, and the volume and composition of saliva produced, which affects the way in which were perceive tastes and astringency varies even more. What this means is that some panels may decide on the existence of a fault, or see a wine as being out of balance, while another panel may not. Discussion by tasters assists in evening out many of these differences, but this act itself can introduce variation through the panel dynamics that arise.

The most important factor in explaining the paradox is that of the use of different quality criteria. This is best explained by example. Assume that we have a full-bodied Cabernet Sauvignon that has only moderate flavour levels, but distinct varietal character, and a typical level of acidity for a Cabernet (but a higher level than the average red table wine). Assume that the wine was tasted in a line up of straight Cabernet varietals. In this context, the judges would appreciate its varietal expression and although being 'highish' in acidity, it is considered to be within the acceptable and expected limits for a Cabernet . Using these criteria, the judges would rate the wine highly.

Assume that the same judges are invited back, but this time to assess a set of wines as full bodied dry reds. In this context the wine described above would be viewed as being only moderately flavoured with an inappropriate acid balance. As such, it would be rated lower than previously. Nothing sinister here, only that there was an appropriate change in perspective the part of the judges.

To compound matters all wine tasters, no matter how experienced have slightly different interpretations as to what constitutes quality. Sure, all judges would agree on the general parameters that underlie quality: aroma and flavour intensity, complexity, balance, structure, attractive mouthfeel and length. However, judges weigh the importance of these differently depending on their experience, training and personal philosophy. Therefore, even with everything else being equal, different panels of judges will not always exactly concur in their opinion as to the relative quality of a set of wines.

Another factor that affects quality perception and ratings across different occasions concerns relativity between the wines being assessed. As humans are excellent at comparing things, but are far less able to relate things in an absolute sense. As a result, we find it difficult to assess the quality of a wine given in a line-up completely independently of the wines around it. For example, there is a tendency to rate a good wine presented among a a set of poor wines a little higher than if it were assessed among a set of other good wines. Individualistic wines tend to be rated slightly higher in the case where sets of very good, but uniform wines are presented. Furthermore, subtle wines tend to be underrated if they are tasted among bigger tannic wines particularly when palate fatigue has set in. There are other examples of these types of context effects, so it is no wonder that the same wine in a different line-up is rated differently at times.

So what is the take home message? If a wine is a consistent star rater, being seen positively by a variety of judges using different quality criteria among different lineups, then in all likelihood it is an excellent wine. Matched with a gold medal or two from respected wine shows, it is difficult to go wrong.