The Absolute Sound’s Review Methodology: First Principles

[sbnx]

This is a very interesting point.

I would say blue is not subjective because every, or almost every, human with normally functioning eyes would confirm it is blue. The unanimous consensus authenticates individual perceptions which I think can be characterized fairly as "objective."

Is there as much consensus about observations about hearing?

Maybe different senses due to their nature have different benchmarks of consensus before we can agree that individual perceptions can rise to the level of being characterized fairly as "objective."

Maybe the TAS analogy of the height of a house and the height of a car is inherently defective because their heights are too obviously different. Maybe -- if the analogy is valid at all -- we should pick two items whose heights are much more similar and subtle -- like the subtleties in perceptions of soundstage depth staging from different stereo systems.

Maybe the visual example should be the difference in height between a Ford Explorer and a Toyota 4Runner? I bet there would be far less consensus on the observable perception between those two heights.

I believe part of it has to do with the level of training. As I mentioned we all had the box of crayolas. We all colored the sky blue. (well, most of us did. And as an off topic side note we all colored the sun yellow even though it is white.) We, as humans in the modern era, focus much attention on the our sense of vision. We don't really train our ears intentionally from a young age. WIth the exception of those that play musical instruments.

I guess I am saying humans can make objective observations using their sense of hearing. I believe it takes some degree of training to do this. If the question is, "Are magazines such as TAS being objective in the reviews?", then I am not sure that is the case. As Eliot pointed out they have not even spent time together to agree on what to call or how to describe "X". One would think that at least within an organization there would be consnesus on what "X" is. I thik everyone just assumes that everyone else knows what they mean when they say somethig about "X". This is a very bad assumption and leads to consufion. Perhaps at this point that ship has sailed. Getting all of the reviewers together to listen, comment together, and agree on "X" might be the right thing to do but likely not a great business decision.

 

[Ron Resnick]

I thik everyone just assumes that everyone else knows what they mean when they say somethig about "X". This is a very bad assumption and leads to consufion.

I agree!

 

[Elliot G.]

I believe part of it has to do with the level of training. As I mentioned we all had the box of crayolas. We all colored the sky blue. (well, most of us did. And as an off topic side note we all colored the sun yellow even though it is white.) We, as humans in the modern era, focus much attention on the our sense of vision. We don't really train our ears intentionally from a young age. WIth the exception of those that play musical instruments.

I guess I am saying humans can make objective observations using their sense of hearing. I believe it takes some degree of training to do this. If the question is, "Are magazines such as TAS being objective in the reviews?", then I am not sure that is the case. As Eliot pointed out they have not even spent time together to agree on what to call or how to describe "X". One would think that at least within an organization there would be consnesus on what "X" is. I thik everyone just assumes that everyone else knows what they mean when they say somethig about "X". This is a very bad assumption and leads to consufion. Perhaps at this point that ship has sailed. Getting all of the reviewers together to listen, comment together, and agree on "X" might be the right thing to do but likely not a great business decision.

Todd that is the point. We are trying to understand what truthfully is not understandable since the basic premises aren't agreed upon. The methodology and terminology are different and non standard. You as well as anyone ones the changes that occur during a real set up. If there is not standard then what does this all mean? It means there are a host of different opinions.
There are standards for color temperatures. There are standards for judging wine. There are standards and tests for many different fields that have critics and opinions. We have lists of opinions. No judgement just a statement.

 

[Tim Link]

I am stuck right here. Are you sure about this in every case?

The personal perception that a yardstick is longer than a one foot ruler I agree is not subjective while looking at both at the same time on the table.

But what about the personal perception that one bourbon has more of a vanilla aftertaste than another bourbon. Is this personal perception subjective or objective?

Yes. I'm sure about it in every case. Right, it's not subjective when comparing a yardstick to a 1 foot ruler, but neither is it when comparing two yardsticks that are nearly identical. If some see the left one as shorter, while others see the right one as shorter, it's perceptual disagreement. We might be able to resolve it using other methods. Lining them up with the fingers might lead to consensus when vision doesn't.

In the case of perceiving that one bourbon has more vanilla aftertaste than another, that's definitely an objective observation. If two people disagree on that, we can start looking at the vodkas to see what all is in them, and maybe isolate something that one person is more capable of detecting than the other.

The subjective example would be when nearly everybody agrees that one has more vanilla aftertaste than the other, but some feel that it's not good to have that much vanilla aftertaste, while others think it's better to have that. We're not going to be able to look at the vodka anymore, or even people's perceptual organs to understand why this is. We're going to have to look at the state of their brain and mind to understand the preference difference. That's what makes it subjective. The behavior is based on things that are completely internal, although these may have been influenced by recent external inputs.


I think there's an important distinction to make between the state of mind and the state of the sensory apparatus and its relation to the mind. Remember all the buzz about the picture of the dress where people couldn't agree on the colors?

We Finally Know Why People Saw “the Dress” Differently

The science behind why people saw different colors.

slate.com

To understand what's happening here, why there's a huge disagreement between lots of people about what they are perceiving, we have to look at where the viewer's perceptual settings are when they see the picture. It turns out to be a case of ambiguity that forces the visual system to make an assumption about the lighting conditions in the picture. We're likely to move our assumption to whatever lighting arrangements we are most accustomed to.

A funny thing happened at work once. A guy who rides his bike to work walked in to the room, and there was an odor wafting through the air with him. Rene, the office manager, told me when she first smelled it she thought it was horrible because she assumed it was his body odor. When she turned around to talk with him she saw it was his hot bowl of soup, which included onions. Immediately it smelled really nice to her!

I once tried Salvia. The stuff knocked me out of the real world for about 8 minutes. In that time I felt a horrible thing that was torturing me. I hated it so much it's hard to describe but the salvia prevented me from understanding what it was. As the salvia's effect wore off I continued to feel that same feeling, which didn't change at all, but my opinion changed dramatically. It was just the feeling of the cool floor on my bare feet, which I thought felt very nice!

 

[Kal Rubinson]

All these posts and no mention of unconscious bias. There are 2 for emotional bias and 1 for subjective bias.

 

[Ron Resnick]

All these posts and no mention of unconscious bias. There are 2 for emotional bias and 1 for subjective bias.

What is your take on the subjective evaluation versus TAS' observable objective evaluation question?

 

[Kal Rubinson]

What is your take on the subjective evaluation versus TAS' observable objective evaluation question?

I do not percieve a difference but I am probably biased.

 

[Tim Link]

All these posts and no mention of unconscious bias. There are 2 for emotional bias and 1 for subjective bias.

I don't even know what unconscious bias is. Is that when we are not aware that we prefer one thing over another? If only I knew that I really liked Pepsi more, I'd quit buying Coke.

I just looked it up. Seems to be always used in relation to group identities and behavior towards others, not so much about objective vs subjective evaluations of products. But maybe it applies in odd ways. If we don't think the company behind the a product has anybody with guru status, and we don't see that it costs enough, we might perceive its sound qualities in a negative way.

Regarding TAS method, I would say they would do no harm to test the products in a double blind fashion. The results might be interesting. Ah, but it's hard, and perhaps only mildly interesting to a few nerdy types. Not enough to keep the publication going.

But, if what they're evaluating is the total of their perception, including all biases conscious and unconscious, emotional and subjective, then other people who are similar to them might find that they have similar perceptual experiences, so the information could be useful.

 

[sbnx]

Perhaps a form if unconscious bias is when you know how expensive or inexpensive a piece of gear is if a preamp is $40k your assumption is that is must be good. Or if a preamp is $5k then it can’t be good.

 

[ssfas]

What is your take on the subjective evaluation versus TAS' observable objective evaluation question?

It’s taken two pages or less for the subjective versus objective argument to tie everyone in knots. No one has yet used the word empirical. Objective empirical evidence may be used to formulate a theory, prove a theory or disprove a theory. It ready depends on your personal choice of philosopher. My personal choice is the latter, on a purely subjective basis because it relates to the philosopher Sir Karl Popper, who resided in our department when I was at university. If I’d studied with the Harman people in Toronto, I might take a different view.

Hopefully most of us make objective empirical observations and base our consumer decisions on those. They are unlikely to be subjective, I don’t know how that would apply, but they could very easily be biased, for example having been preconditioned by reading reviews, conscious of the price or the quality of the coffee served by the audio dealer. Due to the prevalence of these factors, I suspect it is very difficult to make unbiased (objective) empirical observations.

Personally, I think the whole thing is undermined by this concept of there being any specific reference. As my interest is more music than audio equipment, I was recommended to pop over to the Steve Hoffman forum yesterday. I spent most of the night listening to Chopin recordings, often the same music (waltzes, fantasias, mazurkas and, impromptus) chosen by the performer and recording venue. I mostly listen to recordings by performers I’ve heard live in venues I’ve attended. There is no way a reviewer can use an historic recording of a long since dead performer in a long since demolished venue, but they often do.

If you read reviews from music magazines, they will of course mostly be about the performance characteristics usually with an end note about the general recording quality and the contribution of the recording venue.

 

[sbnx]

I don’t understand your point. Of course reviews are based on observation and are therefore emperical. I have not seen any theoretical reviews.

from the dictionary:
Emperical: originating in or based on observation or experience
2
: relying on experience or observation alone often without due regard for system and theory

 

[ssfas]

Moreover, even with acoustic music in the same venue, there is no reference. An annual music series I’ve attended several times used to be hosted at the Hollywell music room in Oxford, a purpose-built recital venue (the oldest in Europe, built in 1734). Usually there are two performances each day, mostly classical music. Some years ago they arranged a performance by the Jazz Trio Phronesis, at night in complete darkness. The windows were boarded and curtained, all phones and watches removed, even the emergency lighting was covered up. If that wasn’t enough, we were given blindfolds. If all visual stimuli are excluded, the sound is elevated to the most extraordinary level, in audio terms you might say from mid-fi to high-end.

You might say, well, I listen with the lights well down, but the experience was that even the slightest amount of light perception ruins the effect.

Any listening “reference“ is dependent upon the listening conditions, as well of course the condition of the listener’s. I heard of one speaker reviewer working for stereofile who was doing reviews in his 80s. It is probably a relief that he is no longer with us.

 

[Ron Resnick]

ChatGPT:

Question:
If a listener has a good sense of what acoustic instruments sound like when played in real life does this convert his subjective impressions into an objective observation?

Answer:
A listener with a good understanding of how acoustic instruments sound in real life can certainly enhance their subjective evaluations of audio equipment, but this does not automatically convert their subjective impressions into objective observations.

Key Points:

1. Subjective Expertise:

• A listener who is familiar with the natural sound of instruments may have a more refined ability to articulate their impressions and preferences when listening to recordings. They might identify specific characteristics, such as timbre or tonal balance, that align closely with their real-life experiences.

• However, these evaluations remain subjective because they are still rooted in personal perception and emotional response, even if informed by experience.


2. Objective Observations:

• Objective observations are grounded in measurable and quantifiable data. For instance, using audio analysis tools to assess frequency response or distortion provides an objective framework for evaluating sound quality.

• A listener’s assessment, regardless of their expertise, does not replace the need for empirical measurements that can be universally agreed upon.


3. Bridge Between Subjective and Objective:

• While subjective impressions can be informed by expertise and may align with objective measurements (e.g., a system reproducing sound accurately as measured), they do not inherently become objective.


Question:
If two people are looking from a distance at a car and the roof of a house next to the car, and both feel that the roof of the house is higher than the roof of the car, are these observations by these people subjective or objective?

In the scenario where two people observe a car and a house from a distance and both believe that the roof of the house is higher than the roof of the car, their observations are primarily subjective.

Reasons for Subjectivity:

1. Personal Perception: Their conclusions are based on personal perceptions and judgments rather than measurable data. They may interpret the visual information differently based on their viewpoints, experiences, or biases.

2. Lack of Measurement: Without measuring tools (like a ruler or laser rangefinder) to provide objective data about the actual heights of the car and house roofs, their observations remain subjective.

Objective Aspect:

• If the heights of the car and house roofs were measured and found to be different, then those measurements would constitute objective observations. In this case, any conclusions drawn based on these measurements would be considered objective because they are based on quantifiable evidence.

Conclusion:

Thus, while both individuals may agree on their observations, those observations are inherently subjective as they rely on personal interpretation rather than empirical evidence.

 

[ssfas]

I don’t understand your point. Of course reviews are based on observation and are therefore emperical. I have not seen any theoretical reviews.

from the dictionary:
Emperical: originating in or based on observation or experience
2
: relying on experience or observation alone often without due regard for system and theory

I just don’t think objective and subjective are the appropriate parameters. There’s a website that calls itself objective, but it is riddled with bias.

I think all we can hope for is a record of the unbiased empirical assessment of the reviewer, plus a complete set of measurements. I doubt that happens very often, if at all.

If reviewers recorded a set of observations and said they like a particular piece of equipment because of certain things, and disliked it because of other things, we might be getting somewhere. From what I understand, if the review is negative, often it will not get published. This is one of the most obvious forms of bias.

Another experiment I once did was to go to a top performance venue in London, Kings Place, and listen to the band Emily Barker and the red Clay Halo play their album Dear River live. I bought the album, a superb Linn recording, rushed home and listened to it and also listened to the 24/192 download. I live about 25 minutes from the venue. I was hoping to draw some conclusions from listening to a performance live and the recording almost immediately thereafter. It just didn’t work and was quite frustrating. Our memory just doesn’t work like that. I can listen to recordings made in familiar venues and the general sonic quality will be familiar, but there is bias because I know what I’m expecting to hear. Nevertheless, it makes for an enjoyable musical experience.

I agree entirely with your post 21 above. To put it more bluntly, I’ve come to the conclusion that the listening observations, more in sonic than musical terms, have no real practical value to the consumer.

 

[Tim Link]

ChatGPT:

This is why I don't trust ChatGPT too much. This and plenty of other examples show that it organizes its response well but doesn't understand the underlying concepts.

 

[Kal Rubinson]

I don't even know what unconscious bias is. Is that when we are not aware that we prefer one thing over another? If only I knew that I really liked Pepsi more, I'd quit buying Coke.

No, it is preferring one thing over another without being aware of why. It is sometimes referred to as "sighted bias." For example, when you compare two speakers, side by side, and one looks much more appealing than the other, there are neural mechanisms which enhance your enjoyment of the better looking one, regardless of the sound. It may be enough to bias your choice or not but it exists and cannot be eliminated............except by blinding.

This is an easy, obvious example. Any perceptible cue can trigger similar neural unconscious biases, and does.

 

[A-Line]

This is why I don't trust ChatGPT too much. This and plenty of other examples show that it organizes its response well but doesn't understand the underlying concepts.

ChatGPT is a AI bot that lacks the human element. It can only respond to what it's been programmed to respond to by some programmer(s). ChatGPT is not a self aware sentient being however it can learn about human behavior and responses nonetheless, just like a terminator from the 1984 film...

 

[Tim Link]

Any perceptible cue can trigger similar neural unconscious biases, and does.

Indeed, I've experienced this many times. The effect can be very powerful, making it hard to believe that bias has anything to do with the perception. I know you don't like Youtube, but the McGurk effect is demonstrated very well for me in this video:

If we can't stop confusing a 'b' for an 'f' just because of what we are seeing, how much can we trust in any sighted listening test?

Well, if the visual effect always works, or at least reliably works on a lot of people, then we can reasonably rely on the desired perception being conjured. Unless we're going to be using the product in a blind listening setting, perhaps the effect of the look of the speaker should be included in the evaluation. Others would say we should somehow train ourselves to override the effect. But, as the guy in the video says, after years and years of trying, he can't defeat the visual override on his hearing perception.

 

[AndrewChen]

But what about the personal perception that one bourbon has more of a vanilla aftertaste than another bourbon. Is this personal perception subjective or objective?

You hit the nail on the head!

The point is that when differences border on the threshold of human auditory perception, as so much of modern electronics do, then measurements are the only objective means. Not human perception.

In my opinion, TAS is talking nonsense.

 

[Rt66indierock]

ChatGPT:

Question:
If a listener has a good sense of what acoustic instruments sound like when played in real life does this convert his subjective impressions into an objective observation?

Answer:
A listener with a good understanding of how acoustic instruments sound in real life can certainly enhance their subjective evaluations of audio equipment, but this does not automatically convert their subjective impressions into objective observations.

Key Points:

1. Subjective Expertise:

• A listener who is familiar with the natural sound of instruments may have a more refined ability to articulate their impressions and preferences when listening to recordings. They might identify specific characteristics, such as timbre or tonal balance, that align closely with their real-life experiences.

• However, these evaluations remain subjective because they are still rooted in personal perception and emotional response, even if informed by experience.


2. Objective Observations:

• Objective observations are grounded in measurable and quantifiable data. For instance, using audio analysis tools to assess frequency response or distortion provides an objective framework for evaluating sound quality.

• A listener’s assessment, regardless of their expertise, does not replace the need for empirical measurements that can be universally agreed upon.


3. Bridge Between Subjective and Objective:

• While subjective impressions can be informed by expertise and may align with objective measurements (e.g., a system reproducing sound accurately as measured), they do not inherently become objective.


Question:
If two people are looking from a distance at a car and the roof of a house next to the car, and both feel that the roof of the house is higher than the roof of the car, are these observations by these people subjective or objective?

In the scenario where two people observe a car and a house from a distance and both believe that the roof of the house is higher than the roof of the car, their observations are primarily subjective.

Reasons for Subjectivity:

1. Personal Perception: Their conclusions are based on personal perceptions and judgments rather than measurable data. They may interpret the visual information differently based on their viewpoints, experiences, or biases.

2. Lack of Measurement: Without measuring tools (like a ruler or laser rangefinder) to provide objective data about the actual heights of the car and house roofs, their observations remain subjective.

Objective Aspect:

• If the heights of the car and house roofs were measured and found to be different, then those measurements would constitute objective observations. In this case, any conclusions drawn based on these measurements would be considered objective because they are based on quantifiable evidence.

Conclusion:

Thus, while both individuals may agree on their observations, those observations are inherently subjective as they rely on personal interpretation rather than empirical evidence.

All well and good but did you listen to a mono copy of Pet Sounds?