The Absolute Sound’s Review Methodology: First Principles

[Ron Resnick]

The TAS Staff published today a very interesting and thoughtful essay, styled as answers to questions about, among other things, objective versus subjective reviews, objective observations versus subjective feelings, the value of objective measurements of audio components and issues with double-blind testing.

The part distinguishing subjective reviews from "observational, objective reviews" interested me the most:

Question: Why do you do subjective reviews?

Answer: We don’t. Or for the most part we try not to make that the core of our reviewing. We aim to do observational, objective reviews. Now, there is some confusion about terminology in which “quantification” is “objective” whereas human “observation” is “subjective”. But this is wrong. That notion incorrectly glosses over a critical distinction. “Subjective” in the dictionary means human reactions that primarily involve feelings. But humans are also capable of observing objectively.

. . .

A simple example may help make some sense of this important distinction. If your car is parked next to your house and we ask “which is taller?” you will observe that your house is taller than your car. It isn’t that you feel your house is taller, it is that you are fully capable of objectively observing the height differences.

Question: You talk about objective observation as a superpower, but how is your opinion objective?

Answer: To use observation as a meaningful measurement technique, you must have a reference standard. This is the case with quantified measurements too, just as it is the case with objective observation. Comparing what we (or you) hear to a reference gets us out of the realm of opinion (subjective feelings). As we said above, it isn’t your opinion that your house is taller than your car. It is an observable fact. It is an observable fact whether a guitar sounds like a guitar, and if it doesn’t, to what degree and in what way.

In music audio, we use the sound of real music and real musical instruments (the absolute sound) as the reference standard. An audio system that can reproduce a guitar or a singer or a jazz band or a symphony so that it sounds believably real, will tend to be more satisfying for most listeners most of the time. This latter point is our experience from over 50 years of listening to live music and audio reproduction of music across hundreds of reviewers.

The Absolute Sound’s Review Methodology: First Principles — the absolute sound

Q:I’m looking for a reviewer that reviews audio components like an engineer would. While I’m not an engineer, I’m fascinated by technical details and measurements. Is there anyone out there that reviews audio components who not only describes what they hear but breaks down the components i.e.

apple.news

---------------------------------------------------------------------------------

We always -- certainly I always -- talk about subjective reviews. But is this an incorrect characterization?

Are TAS reviewers stating observational, objective sonic impressions rather than subjective sonic impressions? Are these observational, objective impressions "facts"?

Is the example of visually observing objectively that a building is taller than a car a valid analogy to reporting on the sound of audio components?

Is the sound of an acoustic guitar an objective reference? Does a reference of the absolute sound of an acoustic guitar allow an observation to rise above a mere subjective impression and become an observable, objective fact?

What do you think?

 

[sbnx]

I believe that humans are certainly capable of objective observation. Scientists did this for 1000's of years. The ancient Greeks were exceptional at it.

If someone is making an objective observation then it needs to be somehow quantifiable by the human. This should involve some sort of check list of items. Then this checklist should be included in every review. The measuring device is a reference track of music and the ears. Each reference track would represent one or more items on the checklist. Let me give an example.

I could rate a components noise floor based on the first 30 seconds of a very quiet, well recorded piece of muisc that has a large dynamic range. In the review I could say I would give this component a 4 out of 5 for noise floor. This would mean that it is not the best I have ever heard but it is still very, very good. Of course the issue becomes that someday I find my new "5 out of 5" and then all of the preceeding components would effectively need to be decremented by 0.5 or 1 or whatever.

For some reviewers this could get a little mundane as they would have to use the same tracks over and over. But that is the nature of observational based evaluation.

 

[Rt66indierock]

Ron, fire up a pristine mono copy of Pet Sounds from the sixties with a mono cartridge. Then tell me what you hear.

When I get back from a brief celebration of the end of tax season, I’ll tell you a couple of things I listen for.

There are more than a few things that fall into a factual category.

 

[Tim Link]

This issue of how to separate objective from subjective has always interested me. When it comes to subjectivity, I think the term is unfortunately used in two different ways, leading to confusion. There's the issue of perception of physical facts, like perceiving that one thing is taller than another, and then there's the issue of emotional value judgements, deciding that something is good or bad. The fact that in both cases there can be disagreement leads some to conflate them together, so we end up arguing past each other because we haven't defined our terms well enough.

 

[Ron Resnick]

perception of physical facts, like perceiving that one thing is taller than another,

and then there's the issue of emotional value judgements, deciding that something is good or bad.

Hi Tim,

I think the second point goes further than it needs to. We're not trying to figure out if a component sounds good or bad, necessarily. Let's stipulate that we simply are trying to evaluate its sound.

Does this evaluation come closer to the perception of physical facts, or is this evaluation wholly subjective and divorced from any objective reference, or is it, as TAS suggests, something that is observably objective?

 

[Tim Link]

Hi Tim,

I think the second point goes further than it needs to. We're not trying to figure out if a component sounds good or bad, necessarily. Let's stipulate that we simply are trying to evaluate its sound.

OK, I agree. They were clear about that right at the beginning.

Does this evaluation come closer to the perception of physical facts, or is this evaluation wholly subjective and divorced from any objective reference, or is it, as TAS suggests, something that is observably objective?

I think it's obvious that up to a point we can make direct perceptual evaluations that are accurate and not divorced from objective reference - which I would take to mean an alternate method of observation involving instruments and techniques that will correlate the perception to other perceptions. The case of the height of the car vs the house is one where it's adequately different enough for us to easily detect correctly which one is taller. If we decide to use a measuring instrument to confirm our perception, we can be confident of what we're going to percieve from the instrument. Trying to evaluate whether two reasonably competent DACs sound more or less like real musical instruments or voices is a much more demanding task, and enters the realm where other sensory information and preconceptions can pull some powerful tricks on our perception. Even if we're tricked, I would not call the perceptual evaluation wholly subjective. It's an objective perceptual evaluation that is in error in terms of the nature of the soundwaves themselves, assuming the evaluator insists on the wrong explanation for their perception. Still, the evaluation of the perception itself is accurate. If it sounded more like the real thing to the evaluator, they can accurately report that as a truth about their perception. And, it may be that a lot of other people will get a similar perception under similar sighted evaluations. Really, it's the perception that we're after, and whatever it takes to get it, even if it involves tricking our ears using various techniques such as artful application of noise and distortion, or employment of other senses and preconceptions, it is ok by me. It just needs to reliably work for enough people.

 

[tima]

The TAS Staff published today a very interesting and thoughtful essay, styled as answers to questions about, among other things, objective versus subjective reviews, objective observations versus subjective feelings, the value of objective measurements of audio components and issues with double-blind testing.

The part distinguishing subjective reviews from "observational, objective reviews" interested me the most:

Question: Why do you do subjective reviews?

Answer: We don’t. Or for the most part we try not to make that the core of our reviewing. We aim to do observational, objective reviews. Now, there is some confusion about terminology in which “quantification” is “objective” whereas human “observation” is “subjective”. But this is wrong. That notion incorrectly glosses over a critical distinction. “Subjective” in the dictionary means human reactions that primarily involve feelings. But humans are also capable of observing objectively.

. . .

A simple example may help make some sense of this important distinction. If your car is parked next to your house and we ask “which is taller?” you will observe that your house is taller than your car. It isn’t that you feel your house is taller, it is that you are fully capable of objectively observing the height differences.

Question: You talk about objective observation as a superpower, but how is your opinion objective?

Answer: To use observation as a meaningful measurement technique, you must have a reference standard. This is the case with quantified measurements too, just as it is the case with objective observation. Comparing what we (or you) hear to a reference gets us out of the realm of opinion (subjective feelings). As we said above, it isn’t your opinion that your house is taller than your car. It is an observable fact. It is an observable fact whether a guitar sounds like a guitar, and if it doesn’t, to what degree and in what way.

In music audio, we use the sound of real music and real musical instruments (the absolute sound) as the reference standard. An audio system that can reproduce a guitar or a singer or a jazz band or a symphony so that it sounds believably real, will tend to be more satisfying for most listeners most of the time. This latter point is our experience from over 50 years of listening to live music and audio reproduction of music across hundreds of reviewers.

The Absolute Sound’s Review Methodology: First Principles — the absolute sound

Q:I’m looking for a reviewer that reviews audio components like an engineer would. While I’m not an engineer, I’m fascinated by technical details and measurements. Is there anyone out there that reviews audio components who not only describes what they hear but breaks down the components i.e.

apple.news

---------------------------------------------------------------------------------

We always -- certainly I always -- talk about subjective reviews. But is this an incorrect characterization?

Are TAS reviewers stating observational, objective sonic impressions rather than subjective sonic impressions? Are these observational, objective impressions "facts"?

Is the example of visually observing objectively that a building is taller than a car a valid analogy to reporting on the sound of audio components?

Is the sound of an acoustic guitar an objective reference? Does a reference of the absolute sound of an acoustic guitar allow an observation to rise above a mere subjective impression and become an observable, objective fact?

What do you think?

Sounds pretty much in line with my own views.

As I've said before: An audio review is expository writing and the root of 'expository' is 'expose'.
Having a reference allows observational comparison against that reference to be objective for the observer. Imo broad exposure to live music is a key to having a reference -- it is an accumulation of facts.

 

[Ron Resnick]

If it sounded more like the real thing to the evaluator, they can accurately report that as a truth about their perception.

I'm just not clear on how this isn't merely a subjective perception by the evaluator, even if the evaluator believes he/she is benchmarking to the "reference" sound of an acoustic instrument. How is the evaluator's perception not merely the evaluator's own (subjective) "truth"?

 

[Ron Resnick]

Having a reference allows observational comparison against that reference to be objective for the observer. Imo broad exposure to live music is a key to having a reference -- it is an accumulation of facts.

Thank you for your reply.

Here is my same question to Tim Link in terms of your post. You talk about the observer having a reference which allows observational comparison. But isn't that reference merely a reference in the mind of the observer in question? Since we perceive sound from live music differently, even though it seems like we are all listening to the same acoustic guitar, aren't we taking away from that experience different "references"?

Back to the TAS analogy, the reason I don't consider the height of the house versus the height of car observation to be analogous is because while different observers from different distances and different elevations themselves might perceive differently the height of the car versus the height of the house, ultimately we can walk up to the house and walk up to the car and measure their heights objectively and definitely.

We don't have with audio the same ability to measure objectively the reference in question. So the reference really isn't a general reference after all. It is, as you say, only a reference in terms of the composite sound in our individual minds. It is maybe objective to the observer, but it is objective only to the observer. This makes it an individual, subjective reference; not an objective reference of general applicability and of general measurement.

 

[A-Line]

I've always used my reference solo classical piano & pipe organ recordings as a primary reference for evaluating all analog or digital components.
Classical guitar is format that I love but lacks the dynamics that piano or pipe organ can produce.

Some will use full symphony recordings as their reference or some jazz or blues format. Whatever works for a particular individual and their listening habits. Selection of components is a personal one with no right or wrong choices unless hindered by cost and/or space restraints.

Just my opinions.

 

[Ron Resnick]

I've always used my reference solo classical piano & pipe organ recordings as a primary reference for evaluating all analog or digital components.

Okay, but the question we are discussing here is whether this reference you describe for yourself is a subjective reference or an objective reference.

Is the sound of your reference solo classical piano recording an objective reference which is analogous to different people looking from different distances and from different elevations at the height of a house compared to the height of a car?

 

[Tim Link]

I'm just not clear on how this isn't merely a subjective perception by the evaluator, even if the evaluator believes he/she is benchmarking to the "reference" sound of an acoustic instrument. How is the evaluator's perception not merely the evaluator's own (subjective) "truth"?

The fact that it's a personal perception does not make it subjective. There has to be an emotional bias of some sort to enter the subjective realm. Otherwise, in principle, it's objective. All perceptions are personal. We're stuck with this limitation, which leads to philosophical skepticism. With the house and car, we can walk up and measure them, but once again, we're stuck with our perceptions as we observe the measuring apparatus. The hope is that with enough methods to bring various perceptions into harmony, the vast majority of people will agree and there will be a consensus of what to call facts.

We can use a microphone to measure the sound of the real instrument. This can be used to change the sound waves into visual information. We can then observe that same signal after sending it through various reproduction devices like ADCs and DACs. We can do blind testing on people to determine what they can and cannot hear, and do further testing where we intentionally trick them by doing "sighted" testing, where what we show them isn't really what they are listening to. A lot of people will invalidate this kind of testing in principle, or at least nit pick it to death to make any particular instance invalid. But it looks to me that a strong consensus has already been reached, although there are niche groups that never accept the results.

As I said earlier, if you can test enough personal perceptions and enough people report the desired perception, then the technique is valid, whatever methods it uses to induce the reliable perception. I think a good example is 3 color printing and lighting methods. I'm looking at a screen right now that can perceptually create an incredible pallet of colors, while in reality it only produces 3. It perceptually fools most of us because most of us are just tri-chromats. For the few of us that are tetrachromats it doesn't look right at all. Paul Klipsch hired people to be blind tested to see what he could get away with. He found some individuals who he called golden ears because they were remarkably capable of hearing and properly identifying differences that the average person, including himself, couldn't. These people were generally not musicians, or hifi enthusiasts, or even particularly interested in sound or music. They could just hear really well, and consistently could prove that. If they couldn't tell a difference, then nobody else could either. So he used them as a standard, perhaps an excessive standard. If we made our TVs and print material for tetrachromats we'd be over-doing the color accuracy for a few at the expense of resolution for everybody else.

 

[Elliot G.]

I believe that humans are certainly capable of objective observation. Scientists did this for 1000's of years. The ancient Greeks were exceptional at it.

If someone is making an objective observation then it needs to be somehow quantifiable by the human. This should involve some sort of check list of items. Then this checklist should be included in every review. The measuring device is a reference track of music and the ears. Each reference track would represent one or more items on the checklist. Let me give an example.

I could rate a components noise floor based on the first 30 seconds of a very quiet, well recorded piece of muisc that has a large dynamic range. In the review I could say I would give this component a 4 out of 5 for noise floor. This would mean that it is not the best I have ever heard but it is still very, very good. Of course the issue becomes that someday I find my new "5 out of 5" and then all of the preceeding components would effectively need to be decremented by 0.5 or 1 or whatever.

For some reviewers this could get a little mundane as they would have to use the same tracks over and over. But that is the nature of observational based evaluation.

Asking for a friend If this is the case for the way that they handle reviews why do none of the reviewers listen to each others systems and use the same anything including music?
So as currently structured we have just personal observations in a vacuum no?

 

[A-Line]

Okay, but the question we are discussing here is whether this reference you describe for yourself is a subjective reference or an objective reference.

Is the sound of your reference solo classical piano recording an objective reference which is analogous to different people looking from different distances and from different elevations at the height of a house compared to the height of a car?

Exactly. What is reference or subjective? For me it's piano and pipe organ and based upon all of the dynamics of my listening space and other needs.

The audio dealerships in my area already know this about me and have always made accommodations for me with regards to my requirements. So that the demo at the dealership will be similar, but not exact, to my listening space. I've also been fortunate that these dealerships have a very open mind with regards to their customers needs. So while the dealership may not agree with how a individual may interpret sound they try their best to match components with what the sound that customer desires.

Concerning the glossy mags and their reviews:
They're useful at posting what's new, features, tech specs for those that are EE's, pricing, and glossy photos.
Their reviews traditionally are done by the same people and their reviews are always positive regarding the components that they choose to review.

Realistically it's difficult if not impossible to review hifi components in a subjective bias free way that everyone, or at least most, will agree upon.
Hifi forums, like this one, will take what those reviewers from the glossy mags present, like The Absolute Sound, and give their own reviews in the form of threads and posts.

At the end of the day it is what it is. The glossy mags hold value for some and others not so much.
For me I stopped purchasing glossy mags back when Audio Magazine was still in publication.

Just my opinions.

 

[sbnx]

Asking for a friend If this is the case for the way that they handle reviews why do none of the reviewers listen to each others systems and use the same anything including music?
So as currently structured we have just personal observations in a vacuum no?

Agree. I simply state that it is possible. Not that what reviewers are doing today is correct.

 

[Ron Resnick]

The fact that it's a personal perception does not make it subjective.

I am stuck right here. Are you sure about this in every case?

The personal perception that a yardstick is longer than a one foot ruler I agree is not subjective while looking at both at the same time on the table.

But what about the personal perception that one bourbon has more of a vanilla aftertaste than another bourbon. Is this personal perception subjective or objective?

There has to be an emotional bias of some sort to enter the subjective realm. Otherwise, in principle, it's objective.

I don't know about this. Every perception that doesn't have an emotional bias is by definition objective?

 

[sbnx]

I am stuck right here. Are you sure about this in every case?

The personal perception that a yardstick is longer than a one foot ruler I agree is not subjective while looking at both at the same time on the table.

But what about the personal perception that one bourbon has more of a vanilla aftertaste than another bourbon. Is this personal perception subjective or objective?



I don't know about this. Every perception that doesn't have an emotional bias is by definition objective?

For some reason the sense of hearing seems to be downgraded to "subjective". The human sense of hearing can be objective just like our other senses. For example, I can evaluate how something looks. If I state something is blue is that subjective? I could even make finer observations about the color and say that it is light blue or Turquoise. I could say that it is luminescent or pearlescent. When I say these words most can immediately visualize what I mean by these observations.

The same thing can be applied to sound. One of the issues is lack of a common vocabulary. We all grew up with a box of crayons so we all generally agree what each color name represents. Unless someone was a musician, concepts of sound were not part of their education. IMO, we also lack what I consider a meaningful vocabulary. For example, what does "warm sounding" mean? Is this just a euphemism for time smearing? I believe HP tried to establish a vocabulary that people could relate to very early on. Words like imaging and soundstage. However, both of these elements are mostly a function of how well the speaker are matched and placed. One could say "It was possible to achieve very precise imaging with speaker A when I placed it in my room." This would tell me that the speaker manufacture is putting in effort to closely match the left and right speaker.

It doesn't seem that the body of reviewers are organized in any real way. They are simply employed by the media publisher to write about products. Like I said in the other post about reviewers, the first criteria is that they are good writers and not necessarily good listeners. We all assume they are, but... I read far fewer reviews these days but one trend I have noticed is that much more of the review is spent on describing the physical attributes of the product (size, shape, buttons, look and feel, etc.) and much less space dedicated to how it sounds. How it goes about making music.

 

[Elliot G.]

For some reason the sense of hearing seems to be downgraded to "subjective". The human sense of hearing can be objective just like our other senses. For example, I can evaluate how something looks. If I state something is blue is that subjective? I could even make finer observations about the color and say that it is light blue or Turquoise. I could say that it is luminescent or pearlescent. When I say these words most can immediately visualize what I mean by these observations.

The same thing can be applied to sound. One of the issues is lack of a common vocabulary. We all grew up with a box of crayons so we all generally agree what each color name represents. Unless someone was a musician, concepts of sound were not part of their education. IMO, we also lack what I consider a meaningful vocabulary. For example, what does "warm sounding" mean? Is this just a euphemism for time smearing? I believe HP tried to establish a vocabulary that people could relate to very early on. Words like imaging and soundstage. However, both of these elements are mostly a function of how well the speaker are matched and placed. One could say "It was possible to achieve very precise imaging with speaker A when I placed it in my room." This would tell me that the speaker manufacture is putting in effort to closely match the left and right speaker.

It doesn't seem that the body of reviewers are organized in any real way. They are simply employed by the media publisher to write about products. Like I said in the other post about reviewers, the first criteria is that they are good writers and not necessarily good listeners. We all assume they are, but... I read far fewer reviews these days but one trend I have noticed is that much more of the review is spent on describing the physical attributes of the product (size, shape, buttons, look and feel, etc.) and much less space dedicated to how it sounds. How it goes about making music.

I agree and have said on this forum a few times that the audio "words" have been prostituted to become meaningless. The popular terms are used to describe and critic everything as these have become safe and warm blankets.

 

[Ron Resnick]

For some reason the sense of hearing seems to be downgraded to "subjective". The human sense of hearing can be objective just like our other senses. For example, I can evaluate how something looks. If I state something is blue is that subjective? I could even make finer observations about the color and say that it is light blue or Turquoise. I could say that it is luminescent or pearlescent. When I say these words most can immediately visualize what I mean by these observations.

The same thing can be applied to sound.

This is a very interesting point.

I would say blue is not subjective because every, or almost every, human with normally functioning eyes would confirm it is blue. The unanimous consensus authenticates individual perceptions which I think can be characterized fairly as "objective."

Is there as much consensus about observations about hearing?

Maybe different senses due to their nature have different benchmarks of consensus before we can agree that individual perceptions can rise to the level of being characterized fairly as "objective."

Maybe the TAS analogy of the height of a house and the height of a car is inherently defective because their heights are too obviously different. Maybe -- if the analogy is valid at all -- we should pick two items whose heights are much more similar and subtle -- like the subtleties in perceptions of soundstage depth staging from different stereo systems.

Maybe the visual example should be the difference in height between a Ford Explorer and a Toyota 4Runner? I bet there would be far less consensus on the observable perception between those two heights.

 

[A-Line]

Imagine if the glossy mags started doing double blind controlled listening sessions for all audio components.
Just saying...