The Absolute Sound’s Review Methodology: First Principles

[Kingrex]

Have you ever thought about sight perception. Walk a person in a room, then take them out. Ask them to tell you all they saw in the room. You can get wildly different answers. Some notice people. Others decorations. Others colors and structure. Some people remeber a whole lot more than others. Some can't remember a thing outside the name of the person they talked to. They won't even remember the clothes the person had on.
Sound perception is exactly the same. We hear the same stuff like were all in the same room. But our focus heavily impacts how we perceive and remember it.

 

[Kingrex]

Per post 61, anyone can validate what I tell you I saw by going back in and looking. With sound, no one can reenter the room. Once the door is closed, the reality has ended and its all just memories from different people that all had different take away from the experience.

The ability to validate what I saw by looking back in the room allows me to say my response was objective.

With audio sounds assesed the same way, someone could say they are being objective. The issue is you can't validate it. And we have no idea how and what the perceptions of any reviewer really are. They could be the person that only remembers a name. Or they could be like my wife and tell you what everyone was wearing with great accuracy and all the decorations in the room. Reviewers have to be entertaining writers. Does that mean they are highly perceptive of the environment around them????? I don't see a corelation.

 

[Tim Link]

I do not think it is possible for a human being to evaluate sound objectively. Most of the energy used to design successful scientific experimentation is about barricading feelings and biases from the results. The complexity involved in architecting proper experiments is based on this. This is analogous to why “legalese” is such a complicated form of syntax and grammar, since the ideal is that the words are so clear that they deflect interpretation and force only the meaning that is intended

We are, after all, talking about reviews. Reviewing a product implies value. Value is subjective. Even if product A has measurably more bass than B that doesn’t make it better, especially when someone has a small room like me and more bass is usually a negative. There is no way you will convince anyone that the price tag also doesn’t bias a listener (Otherwise why do every single review in the magazine always verify that the more expensive product higher up the line is better?) Claiming that “our opinions are objective because we say so” is the opposite of the scientific method and simply would never hold up in any serious discussion of what objectivity means. The fact that they have a reference system, or have been to many live concerts and are comparing the sound of the reproduction to the original is irrelevant. There are way too many biases for such a process to be considered objective by any scientific standard. Even the type of music is relevant, for example. Maybe I like rock and you like rap. The type of system that will make listening to either more or less pleasurable is an opinion.

You could hand TAS to any serious academic on earth, whether they know anything about audio or not, and they would quickly recognize that it is a bunch of opinions, not objective facts.

Even if their reviews are objective and not biased by advertising, cost, music, etc., my OPINION is that they are boring. They need to start publishing a much wider range of articles for me to spend any of my cash on them. (PVP and I came up with several interesting topics in a short brainstorm, so surely they could find much more if they really wanted to.) What’s more, these publications are by and for a very specific demographic and one that may not have longevity, though time will tell on that front.

I’d rather read forums or just listen to music and forget about the gadgetry making it happen. The more of the latter in my life, the better for my wallet and my well being. Instead of PRAT or other cliched descriptives, I like rating gear by how quickly, deeply and for how long do I completely forget gear in favor of music…. That is why I am a Lampi fan, for example.

I totally agree with the spirit of what you are saying, but an observation doesn't become subjective just because it is inaccurate due to human bias. Using evaluation methods that don't properly account for human biases is not a good way to do science, I agree. But it doesn't make the effort a subjective one. It makes it a poorly controlled attempt to state objective facts, which are likely not going to be accurate.

Subjectivity is a different realm, and by its very nature you can't test the objects of a subjective opinion to prove that opinion right or wrong.
If a house is blue, and most agree that it is, you can check the wavelength of the light to verify that it's blue. That's objective. But if someone says that house looks good in blue, while another says it looks bad in blue and ought to be red, that's subjective. What are you going to test about the house to see who's right?

 

[ssfas]

I do not think it is possible for a human being to evaluate sound objectively. Most of the energy used to design successful scientific experimentation is about barricading feelings and biases from the results. The complexity involved in architecting proper experiments is based on this. This is analogous to why “legalese” is such a complicated form of syntax and grammar, since the ideal is that the words are so clear that they deflect interpretation and force only the meaning that is intended

We are, after all, talking about reviews. Reviewing a product implies value. Value is subjective. Even if product A has measurably more bass than B that doesn’t make it better, especially when someone has a small room like me and more bass is usually a negative. There is no way you will convince anyone that the price tag also doesn’t bias a listener (Otherwise why do every single review in the magazine always verify that the more expensive product higher up the line is better?) Claiming that “our opinions are objective because we say so” is the opposite of the scientific method and simply would never hold up in any serious discussion of what objectivity means. The fact that they have a reference system, or have been to many live concerts and are comparing the sound of the reproduction to the original is irrelevant. There are way too many biases for such a process to be considered objective by any scientific standard. Even the type of music is relevant, for example. Maybe I like rock and you like rap. The type of system that will make listening to either more or less pleasurable is an opinion.

You could hand TAS to any serious academic on earth, whether they know anything about audio or not, and they would quickly recognize that it is a bunch of opinions, not objective facts.

Even if their reviews are objective and not biased by advertising, cost, music, etc., my OPINION is that they are boring. They need to start publishing a much wider range of articles for me to spend any of my cash on them. (PVP and I came up with several interesting topics in a short brainstorm, so surely they could find much more if they really wanted to.) What’s more, these publications are by and for a very specific demographic and one that may not have longevity, though time will tell on that front.

I’d rather read forums or just listen to music and forget about the gadgetry making it happen. The more of the latter in my life, the better for my wallet and my well being. Instead of PRAT or other cliched descriptives, I like rating gear by how quickly, deeply and for how long do I completely forget gear in favor of music…. That is why I am a Lampi fan, for example.

I just spent $10 for 12 months of stereofile, so at least I could now say I have a subscription to an audio rag. And what a rag it is! Every single product is highly recommended, including a streamer that has software that isn’t fit for purpose and the thing still costs $30,000. The review is just pages of waffle, an endless stream of marketing comments given to the reviewer over the telephone by people associated with the manufacturer and then a load of subjective listening comments.

He gives them the benefit of the doubt, saying “ like some other server manufacturers, ideon has some software work to do“. Well, I’m sorry, this isn’t the case. I’ve been streaming for 15 years and whilst there may have been a bit of latitude back then, now it is a given that any streamer at any price should have effectively faultless software. This thing had fundamental problems with cover art, booklets and can only connect to one NAS drive at a time. To its credit, it can’t process MQA files.

Who would recommend buying a $30,000 streamer with faulty software? An unbiased review would suggest that you wait on this product until they fix the software. The conclusion states “never before have I reviewed a standalone streamer/server so accomplished in the hardware department, yet so behind the best in software implementation. … one way or another you’ve got to give (product name, too long to type) a spin.”

There are reviews of three more affordable products, but two of them are listed in the EISA awards at the back of the magazine, implying that they have already been widely reviewed as I understand those awards are based on the opinions of a range of international audio journalists. One of the products has been widely reviewed going back at least to December 2023. So this magazine seems to be a bit late to the party.

I wonder if there is a shortage of people willing to write reviews. Someone mentioned the UK publication where most of the reviews were written by one person.

 

[Rt66indierock]

I don't have Pet Sounds.

Ron, it going to be hard to discuss TAS’s review methodology or what John Atkinson calls sighted subjective evaluation without talking about the basics. Pet Sounds has two things that are the basics.

 

[the sound of Tao]

Sounds pretty much in line with my own views.

As I've said before: An audio review is expository writing and the root of 'expository' is 'expose'.
Having a reference allows observational comparison against that reference to be objective for the observer. Imo broad exposure to live music is a key to having a reference -- it is an accumulation of facts.

Benchmarking is absolutely critical in assessment… one of the challenges in our pursuit of sound fidelity is that as fabulous as synthetic/electronic music can be (I love electronic guitar/keyboards and am listening to jazz marvels Grant Green and Jimmy Smith as I write) but its simply much harder to benchmark against the sound of electronic instruments as opposed to an acoustic instrument recording.

Good recordings and a mix of varying styles and scales into more complex and wider frequency acoustic instrument recordings serve generally much better guides to the authenticity of your sound. For me the better the performance and recording the better as a guide to represent a systems ability to convey context like balance, timbre, dynamic shading, nuance, energy and beyond ultimately into spirit in music and performance. You can learn a whole lot from a reviewer’s choice of reference music in their assessing.

 

[Sampajanna]

Good recordings and a mix of varying styles and scales into more complex and wider frequency acoustic instrument recordings serve generally much better guides to the authenticity of your sound. For me the better the performance and recording the better as a guide to represent a systems ability to convey context like balance, timbre, dynamic shading, nuance, energy and beyond ultimately into spirit in music and performance. You can learn a whole lot from a reviewer’s choice of reference music in their assessing.

I love this! “Spirit of the music” is what I’m after. For me, that is more important than its ability to sound like live music, which no system has ever done for me…. That doesn’t mean clarity/fidelity isn’t important, but only as a part of that spirit. If even half the articles in a magazine were about the things discussed in this thread, and especially this paragraph, I would happily subscribe and read avidly. I love hifi and reading. These are my favorite things in life. (I even reread good books again and again. There are books I have read several times.) If your magazine turns me away, you are doing something seriously wrong….

I totally agree with the spirit of what you are saying, but an observation doesn't become subjective just because it is inaccurate due to human bias. Using evaluation methods that don't properly account for human biases is not a good way to do science, I agree. But it doesn't make the effort a subjective one. It makes it a poorly controlled attempt to state objective facts, which are likely not going to be accurate.

Subjectivity is a different realm, and by its very nature you can't test the objects of a subjective opinion to prove that opinion right or wrong.
If a house is blue, and most agree that it is, you can check the wavelength of the light to verify that it's blue. That's objective. But if someone says that house looks good in blue, while another says it looks bad in blue and ought to be red, that's subjective. What are you going to test about the house to see who's right?

You are smart and have obviously thought this through well. I like your perspective. I agree that this is all a spectrum and recognize the definition of subjective vs. objective which you are positing. On the quantum level, the observer affects the experiment anyway . I still think TAS’s attempt to come off as objective is not a good look. They won’t appeal to hardcore “objectivists” who will just argue that they should do it all blind then, and with controls. And subjectivists (and I would count myself among them) don’t really care that their reviews are biased attempts to be objective by your thought or more subjective by my account. Even if I agree that they are making “poorly controlled attempts to state objective facts” they do then turn those “facts” into subjective opinions when they conclude by rating the product, no? Any time they tell me how “good” the (insert cliche term like PRAT) is, they have slipped into what you are calling subjective also ( in effect saying the “house looks good in blue”). Maybe I haven’t read enough (very possible), but I haven’t seen a single TAS article that didn’t include at least one of the “house looks good in blue” statements. In fact, as others have suggested, it is my experience that authors over there live in neighborhoods where all the houses look good (if you catch my drift)…

Just got an original pressing of GO…. So, old man’s salute and off to find that spirit….

 

[Tim Link]

I agree that this is all a spectrum and recognize the definition of subjective vs. objective which you are positing.

I'm not saying it's a spectrum, as if there are degrees of subjectivity and objectivity.

On the quantum level, the observer affects the experiment anyway .

That's a purely objective problem.

I still think TAS’s attempt to come off as objective is not a good look. They won’t appeal to hardcore “objectivists” who will just argue that they should do it all blind then, and with controls. And subjectivists (and I would count myself among them) don’t really care that their reviews are biased attempts to be objective by your thought or more subjective by my account.

I think people are always wanting a good subjective experience. If I go to TAS as a resource in my quest for that experience, I'm hoping they'll give me useful information that will lead to success without spending more than I need to. It doesn't seem useful to divide people into categories of "subjectivist" and "objectivists." There are objective claims, and subjective assertions. People can assert their subjectivity with certainty. Objectivity always needs some element of skepticism and doubt. Claims are never subjective. I think people get called "objectivists" when they notice and complain about objective claims that are not well supported.

Even if I agree that they are making “poorly controlled attempts to state objective facts” they do then turn those “facts” into subjective opinions when they conclude by rating the product, no?

If they give their emotional opinion on it, then yes. If they are just scoring it as more accurate or less accurate, but not how much they personally enjoy or prefer it, then no. If someone is rating things by how tall they are, and they eyeball one to be taller, and rate it better on tallness, that is an objective assessment .

Any time they tell me how “good” the (insert cliche term like PRAT) is, they have slipped into what you are calling subjective also ( in effect saying the “house looks good in blue”).

If they define PRAT in objective terms, and then by ear determine that a speaker scores high in this area, then that is an objective assessment. Their metric may be how much the sound causes them to tap their toes or uncontrollably move to the beat. This opens an interesting opportunity to very carefully measure that in blind testing rather than just go by their recollection of how much they think they're tapping their toe. Saying the house looks good in blue is subjective. But the person could further define something about trim thickness and roof angles and color, making an equation of some sorts for scoring a house, and they could say that with blue that house scores high. That becomes objective. The subjective part is whether or not a house should score high on that metric, or whether a speaker or soundsystem should score high on the PRAT metric, whatever that may be.

Maybe I haven’t read enough (very possible), but I haven’t seen a single TAS article that didn’t include at least one of the “house looks good in blue” statements. In fact, as others have suggested, it is my experience that authors over there live in neighborhoods where all the houses look good (if you catch my drift)…

Just got an original pressing of GO…. So, old man’s salute and off to find that spirit….

It may be that they include a subjective assessment, and it's always a positive assessment or they don't write the review at all. In that case, the lack of a review on a widely available product is an implicit bad rating.