Objectivists, Harman Testing, Reviewers, and Reality

[mep]

Amir has spilled countless pages of digital ink on the virtues of Harman's methods for designing and testing loudspeakers, why reviewers fail miserably at writing reviews he feels he can learn from, and wanting to raise the bar for what customers should expect both from companies and reviewers. Meanwhile back at the ranch, we know all good card-carrying objectivists love them some DBTs and measurements. Making any decisions on purchasing audio products with your eyes involved is a strict no-no because now you have sighted bias and you are buying with your eyes and not your ears.

First of all, let's take a look at the total number of high end products available for purchase in the marketplace. We need to count every speaker in every line that each manufacture offers for sale. Ditto for amps, preamps, phono sections, tonearms, cartridges, CD players, DACs, streamers, and all the other digital doo dads including DSP/DRC. IF you add them all up together, what percentage of them do you think we actually have measurements for? How many audio publications actually take measurements and publish them and how many products do they measure a year? Add them all up and please tell me how close we are coming to hitting all of the products for sale. I think it's fair to say we are only reviewing a fraction of the gear available for sale and the products that are measured are a fraction of that fraction.

If we focus just on speakers for a second, how many of those speakers have a full set of measurements performed on them from a second party? Even for companies that measure their own speakers, how many of them are using anechoic chambers? Let's get back to Harman. How many audiophiles outside of the members of WBF are even aware of the Harman testing and training methods(WBF members can't escape this knowledge )? How many would care if they knew? Besides Amir, Harman, and some of the objectivists on this forum, who thinks that Harman's methods of designing and testing speakers is the gold standard that all speaker companies should aspire to? How many people are lining up to buy Harman speakers vice speakers made from every other company throughout the world (and that includes all of you card carrying objectivists)?

In the real world and not somebody's fantasy version of it, audiophiles have damn few measurements to go by when they make purchasing decisions. Audiophiles have even less of a chance of participating in real scientific DBTs for products they are interested in than they do of having measurements of something they are interested in purchasing (spec sheets don't count). Many of the objectivists that like to beat their chests and shout out how much they love DBTs and measurements never buy their equipment with either. Some never even have the chance of hearing their speakers (or other components) before they buy them because they simply aren't available in their area and they lack the will or means to travel to hear them. So what do they do? They go and try to find some reviews on line and read what reviewers had to say about them. That's right, those same reviewers who are clueless and just write fancy poetry and flowery words as Amir says are who these very same objectivists turn to for guidance in their purchasing decisions. That's the real reality even if it's denied.

If you take a look at Stereophile's model where JA does a damn fine job of measuring gear, the reviewers never get to see what the measurements are before they submit their review. Why? Because if you knew how something measured, it could influence what you think you hear (sound familiar?) and how you would write the review without bias? Reviewers describe what they hear in their systems in their rooms without knowledge of how a piece of gear measures on the test bench. And getting back to hearing, seeing, and Harman, I'm somewhat fascinated that under double baffled conditions, everybody hears the same. Yes, Harmanites rank speakers much more harshly than untrained ears, but it doesn't change the order of ranking. When people have one of their senses back (eyesight), suddenly everyone doesn't hear the same anymore and the preferences do change. This is all of course attributable to your eyes playing tricks on you, but I'm not sure it's that simple and I think there are others who feel the same way. The only thing that is proven to me is that when your eyesight is taken away from you, suddenly everyone hears the same which should give people pause because many of us believe we don't hear the same.

 

[rbbert]

I must confess that I have only seen a comparison of 4 speakers referenced in all these posts, and I guess it's not too surprising that the relative rankings for those 4 speakers under the test conditions described would be consistent. And as I posted elsewhere, the Harmon method of evaluation is not specifically meant to be a speaker "review" or a recommendation for purchase. Nevertheless, I find JA's measurements of most components useful as a means of comparing other components of the same type, something reviewers are rarely able to do for more than one or two other components, except by memory. Note that "useful" doesn't mean definitive information.

As far as purchasing, I'm never going to buy another piece of audio equipment without an in-home trial (unless it is very inexpensive), and these days it's a little hard to see why anyone would.

 

[TheMadMilkman]

In the real world and not somebody's fantasy version of it, audiophiles have damn few measurements to go by when they make purchasing decisions.

Damn straight we don't. We should be demanding more information.

If you take a look at Stereophile's model where JA does a damn fine job of measuring gear, the reviewers never get to see what the measurements are before they submit their review. Why? Because if you knew how something measured, it could influence what you think you hear (sound familiar?) and how you would write the review without bias?

I agree with this.

Reviewers describe what they hear in their systems in their rooms without knowledge of how a piece of gear measures on the test bench.

Subject to their own individual biases (visual appeal, bias towards/against certain designs, etc.), yes.

The only thing that is proven to me is that when your eyesight is taken away from you, suddenly everyone hears the same which should give people pause because many of us believe we don't hear the same.

I agree that this should give us pause, but I believe that the question in our minds should be "how different is our hearing, really?"

I would be very interested to sit in Harman's listening room to compare, say, the Revel Salon2 against the Magico S3. The retail prices are within spitting distance of each other. Both measure extremely well, and are produced by designers who maintain that good measurements are extremely important. If you put a bunch of forum members together to do a blind preference test, which would win out, and why?

In other words, I think the "everybody hears differently" would come down to fairly minute differences in preference, rather than large-scale differences in hearing.

And, as a matter of investigation, I would like to repeat that same listening preference test using the same listeners, but with the speakers set up in a stereo configuration, rather than the mono configuration that Harman utilizes. I would be curious to see how the scoring and preference would differ, if at all.

 

[kevinkwann]

The only thing that is proven to me is that when your eyesight is taken away from you, suddenly everyone hears the same which should give people pause because many of us believe we don't hear the same.

My great uncle was a bomber pilot in WWII. He flew countless sorties over Germany and Italy in P-51 Mustangs. They knew nothing about "hearing protection" back then. Of course now my great uncle is stone deaf although the two hearing aids he wears do occasionally allow him to hear his wife yelling in his better ear. Did I mention he's virtually blind now too?

Given that "everyone hears the same when your eyesight is taken from you" makes your acuity, by definition, the same as my great uncle's. That must make your reviewing chores difficult indeed.

 

[TBone]

My great uncle was a bomber pilot in WWII. He flew countless sorties over Germany and Italy in P-51 Mustangs. They knew nothing about "hearing protection" back then. Of course now my great uncle is stone deaf although the two hearing aids he wears do occasionally allow him to hear his wife yelling in his better ear. Did I mention he's virtually blind now too?

nice read, i've similar stories ... spitfires, wildcats, hellcats, 109, 190, zero ... then the jets ... had all those models as a child.

 

[PeterA]

Nice thread, mep, and excellent OP. Somehow the tone seems different than that on the other Harman thread. I think one is about what could be and the other is about what is, at least it is so far. I too would like to take part in a DBT of the Salon 2 and Magico S3 at Harman's listening room. It would also be interesting to learn the differences between what the single speaker, centrally located listening evaluation tells us versus the properly positioned, two speaker stereo evaluation. I will grant that the same sufficiently powerful (300-400 watt, 4 ohm), low distortion, Class A SS amplifier could be used.

This is a big tent. There is plenty of room for both meaningful measurements, rigorous testing, well written, informative, subjective reviews and actual listening auditions. The irony is that at this developed point in my audio quest, I am very pleased with my system and am unlikely to change it much in the future.

 

[Joe Whip]

There is no question that we hear differently when our sight is not involved. I am reminded of that video where we are tricked by our eyes into hearing the person say something he really wasn't. When we see the lips move one way and add different audio, we hear what we see his lips mouthing but not what is on the audio track. So clearly, there is some connection between sight and hearing. I for one would love to participate in just that sort of DBT. Should prove to be interesting. There is no doubt that our preferences play a role in what we think sounds better. I don't like bass shy speakers, I want that low end impact. Others want a lush midrange others those highs. I like a nice balance but without the bass, not interested.

 

[NorthStar]

... I too would like to take part in a DBT of the Salon 2 and Magico S3 at Harman's listening room. It would also be interesting to learn the differences between what the single speaker, centrally located listening evaluation tells us versus the properly positioned, two speaker stereo evaluation. I will grant that the same sufficiently powerful (300-400 watt, 4 ohm), low distortion, Class A SS amplifier could be used.

This is a big tent. There is plenty of room for both meaningful measurements, rigorous testing, well written, informative, subjective reviews and actual listening auditions. ...

Some professional audio reviewers of solid reputation and of high caliber and integrity can give us pretty much what there is sufficient to know about what you just posted above.

 

[Phelonious Ponk]

By pointing out how little science is evident in the industry, I think you're making Amir's case. There appears to be little due diligence on behalf of reviewers, little accountability on behalf of high-end manufacturers. Amir is right. It's time for this industry - manufacturers and reviewers alike - to step up and serve the customer. And you know what? I'll bet a whole lot of those high-end manufacturers have measurements. They're just not sharing them with you.

Tim

 

[TBone]

I don't like bass shy speakers, I want that low end impact.

being primarily into R&R, maybe that's the reason I prefer equipment that limits smearing instrumental impact, at any frequency, esp in reference to any live reference/experiences.

 

[jkeny]

Good post mep.
I too believe that the reality is pretty different to the simple models that seem to guide objectivist thinking.

Firstly, let's not forget the objective of this hobby - to create the most realistic illusion to our auditory perception. Note, I didn't say accurate - our auditory perception is not concerned with accuracy. I had my thinking about this prodded recently when I was introduced to Donald Hoffman & his thoughts about perception. I had lazily assumed that our perceptions evolved to provide us with an accurate view of our physical world & that more accurate perception = better adaptability = superior survival statistics. His concepts are much more cognisant of our limitations & the limitations of all species. We have finite & limited processing capabilities - our brains. Evolutionary advantage is conferred on those who make the most fitting use of this limited processing i.e those who come to the most fitting conclusions, quickest. In other words perception is not about accuracy, it's about finding the best tricks to interpret the real world & be able to negotiate it safely, find food, find a mate, etc. He calls it the "interface theory of perception" & I have to say it makes great sense.

This makes sense on so many levels - we are not measuring machines - we all know that. We are limited in our auditory processing. It is widely agreed that the "perceptual load theory of selective attention" is correct. In other words a person' attentional resources will be fully deployed in the processing of any incoming sensory information. In other words, as the perceptual load increases, our field of attention narrows & the selectivity occurs earlier in the auditory processing.

I relate this to different types of listening as follows. In relaxed, casual listening, we allow more of the elements of the soundfield through to the higher levels of our auditory perception - we have better access to more sound elements. In more focussed, concentrated listening we limit the soundfield elements available to our attention & they are eliminated earlier in the processing steps so less accessible - change blindness, anyone? I gave this link before but maybe it was in an obscure area - try it & see what you think https://www.youtube.com/watch?v=zGKADgFCoeU&feature=youtu.be

The idea that we all hear differently is just more lazy thinking but I believe it is correct in one aspect - we have different levels of working memory & this working memory is what determines our ability to focus our attention - so those with lower working memory span find it harder to filter out irrelevant perceptual information than those with a higher working memory span. I wonder is this at the heart of the diminution in our abilities, as we get older, to listen to & follow a conversations in noisy environments?

Anyway, this says to me that DBTs are only suitable to formally conducted tests in research labs & not suitable to home tests. The great damage that has been perpetrated & continues to be perpetrated by such blind tests is encapsulated in the statements which you can witness on every audio forum that goes like "I could hear definite differences between X & Y sighted but when I did a blind test I couldn't hear any difference so now I know the truth - there is no difference" I've seen people adding to this "even though I still can hear this difference I now know that it is due to my sighted bias"

This is such a sad state of affairs - that people have been led into distrusting their auditory perception (which they use everyday & trust it's information) because they are testing their perception in a way that is fraught with so many error-traps.

As regards measurements - they are mostly just technical titillation when it comes to auditory perception. Until we start to get a handle on what's important to auditory perception & how it works, we are just pretending to do anything of scientific importance (except show how good our equipment is). I'm reminded of a L. Cohen phrase (he's always good for such snippets of wisdom) - "I'm guided by the beauty of our weapons" I'm also of the strong opinion that when we better understand the workings of auditory perception, we will have a more realistic measure of what the limits of our auditory perception (rather than the limits of hearing). For instance, Meridian mentioned one of two factor that echoed my thinking - the importance of the absolute stability of the noise floor. I'm of the opinion, At the moment anyway, that when this is investigated further it may well turn out that we are more sensitive to a very low level noise floor that is fluctuating in the presence of music - lower than the currently established threshold limits that are based on pure tones or pink/white noise tests.

 

[Joe Whip]

Very nice post jkeny. Much food for thought.

 

[amirm]

Amir has spilled countless pages of digital ink on the virtues of Harman's methods for designing and testing loudspeakers, why reviewers fail miserably at writing reviews he feels he can learn from, and wanting to raise the bar for what customers should expect both from companies and reviewers. Meanwhile back at the ranch, we know all good card-carrying objectivists love them some DBTs and measurements. Making any decisions on purchasing audio products with your eyes involved is a strict no-no because now you have sighted bias and you are buying with your eyes and not your ears.

Hi Mark. Let me commend you from the point of view of your writing skills. Certainly far better than mine, with drastically less grammar error. I am unclear as to the central message though. Are you saying that since there is not much data out there, we need to bend to adjust to it because that is reality? Seems like a very defeatist point of view instead of using our influence on manufacturers and reviewers. How about next time you are at a show and going room to room as, "how does the off-axis measure on this? Is it smooth and similar to on-axis?" As a minimum, it would make you look like you know something about loudspeaker design .

That aside your statement above is partly false. As I have said repeatedly, on this issue of loudspeaker preferences and double blind testing, you are unified with objectivists on forums. Their schtick is to prove to you everything sounds the same. But they know loudspeakers don't sound the same so they never talk about blind testing of loudspeakers. Indeed, they mount a heavy defense and fight against relying on double blind testing of loudspeakers: "they sound different so we don't need blind testing." So this is one time the two camps are unified.

I explained all of this in the other thread so you must have skimmed over all the digital ink there .

Also, sighted evaluation does not automatically mean wrong conclusions. I used the word "not trustworthy." You could tell me A is better than B in sighted evaluation. You could be right and reflecting true fidelity you heard. Or you could be influenced by looks, price, marketing, prior knowledge, peer pressure, etc. into having a modified observation. We can't separate these things in your conclusions. In that regard, the outcome is not reliable and can't be counted upon. Certainly if this were life and death situation we wouldn't remotely take your word for it that what you observed was purely based on the one factor. We are human after all and such an ability eludes even the most skilled and trained listeners (although not nearly as frequently as it does for casual observers).

First of all, let's take a look at the total number of high end products available for purchase in the marketplace. We need to count every speaker in every line that each manufacture offers for sale. Ditto for amps, preamps, phono sections, tonearms, cartridges, CD players, DACs, streamers, and all the other digital doo dads including DSP/DRC. IF you add them all up together, what percentage of them do you think we actually have measurements for? How many audio publications actually take measurements and publish them and how many products do they measure a year? Add them all up and please tell me how close we are coming to hitting all of the products for sale. I think it's fair to say we are only reviewing a fraction of the gear available for sale and the products that are measured are a fraction of that fraction.

Dr. Toole laments that when a person goes shopping for multi-thousand dollar piece of audio equipment, he has far, far less information than on the side wall of a $50 tire:

As you say, none of that information is available to us for vast majority of equipment we buy. As I said at the outset of the reviewer's threads, we have ourselves to blame. We let, heck with threads like this, encourage manufacturers to tell us less. Tim is very right. A lot of information exists than is provided to us. If we don't ask or value it, then the less they tell us, the more apt we are to buy on touchy, feely words than reliable information.

It is not like you can go to the corner store and see all of these loudspeakers in one place either. They cost so much, and manufactures so selective, than you can't do any kind of comprehensive comparison by any method yourself. It is in that light that I find it so surprising to not support better information conveyance to us. Look, you can always ignore the data. But you can't manufacture it if it doesn't exist.

If we focus just on speakers for a second, how many of those speakers have a full set of measurements performed on them from a second party? Even for companies that measure their own speakers, how many of them are using anechoic chambers? Let's get back to Harman. How many audiophiles outside of the members of WBF are even aware of the Harman testing and training methods(WBF members can't escape this knowledge )? How many would care if they knew? Besides Amir, Harman, and some of the objectivists on this forum, who thinks that Harman's methods of designing and testing speakers is the gold standard that all speaker companies should aspire to? How many people are lining up to buy Harman speakers vice speakers made from every other company throughout the world (and that includes all of you card carrying objectivists)?

A lot more should be lining up but because they are sold on marketing messages instead of engineering, they don't do that. Case in point is a high-end customer of my company. He wanted a bunch of in-wall loudspeakers. We supplied Revel in-walls powered by ordinary commercial amplifiers. He outfits the one room himself with B&W loudspeakers and Moon amplifier. He calls us yesterday and says he wants to rip out all the B&W's and Moon amp and replace it with Revels! Replacing in-wall speakers is big deal as it rips out the walls and such. But that is what he wants. Why? Because even that few hundred dollar in-wall Revel loudspeaker is not allowed to be released until such time that it beats its competitors in double blind testing.

Now, I wish my company did not have a business relationship with Harman and I was not friends with many of the people I talk about who are behind this research. Maybe then you would believe what i am saying as being less biased. But something tells me you wouldn't regardless. A negative bias is set in there as you say, "who buys Revel loudspeakers???"

It is a hard switch to flip. I know I am shaking the very foundation we all, yes, all of us whether objectivists or subjectivists, walk on. It didn't come easily to me to believe it either. So appreciate you keeping the topic alive as it gives us a chance to put all the data on the table. All I hope is that a seed is planted in you ear. That one day you too will see it of value if not for anything but to have a lexicon with which to speak to your loudspeaker designer. One can only hope .

 

[jkeny]

I believe that it's a fallacy to think that manufacturers have either "secret" measurement techniques that they are unwilling to release or that they have flattering measurements of their equipment which they are not releasing.

Just look at Harman - they have released their measurements - other manufacturers could use their findings to develop speakers which can compete head-to-head with them - yet they didn't keep these measurement techniques secret - why? - because it gave them a commercial advantage

 

[mep]

By pointing out how little science is evident in the industry, I think you're making Amir's case. There appears to be little due diligence on behalf of reviewers, little accountability on behalf of high-end manufacturers.

Exactly what due diligence are reviewers supposed to bring to the table? We don't have any more information about the products we review than you do. All we have to go on is the same spec sheets everybody can see from the websites of the manufacturers. What reviewers try and do is tell you how a given component or speaker sounds in the context of their system and their room. Based on comments by people on this very forum who admitted they have bought products based on reviews and were quite happy with their purchase, you have to surmise there is merit to the method.

Amir is right. It's time for this industry - manufacturers and reviewers alike - to step up and serve the customer. And you know what? I'll bet a whole lot of those high-end manufacturers have measurements. They're just not sharing them with you.
Tim

If they truly have them, beyond the information they print on their spec sheets, they are not sharing them with you either-never mind the reviewers.

 

[TBone]

I wonder is this at the heart of the diminution in our abilities, as we get older, to listen to & follow a conversations in noisy environments?

that wouldn't sound nearly as mean; if it wasn't so potentially true. :b

 

[jkeny]

that wouldn't sound nearly as mean; if it wasn't so potentially true. :b

Don't get me wrong - I'm not talking about diminished comprehension ability as we age - I'm talking about our diminished ability to pick out a conversation thread from the background noise of a room full of chattering people, for instance.

 

[amirm]

Good post mep.
I too believe that the reality is pretty different to the simple models that seem to guide objectivist thinking.

Once more, nothing about this topic relates to objectivists as defined by the term in audio forums. They are completely and equally situated to subjectivists in disbelief and requirement for controlled testing. Indeed I made mortal enemies with more than on objectivist on AVS because I discussed this very work with them. They are adamant that because there are large differences here, bias cannot possibly play a role. When I show them the research, they become anti-DBT in an interesting turn of events!

Much of what you describe does not apply to blind testing of loudspeakers. Most of the stress and worry of taking blind tests when differences are small or non-existent go out the window when testing loudspeakers. They radically sound different. Yes, you do have to develop a rating system. I remember the first time I heard a loudspeaker I didn't know what number to give it. So I wrote something down. Then I heard the second one and immediately started to get calibrated in my mind of one versus the other. I also had to develop a mental method of analysis, separating bass performance from voices and high frequencies and generating a composite index for all of them. So not saying test is the same as not testing. But what on earth do you do when you compare speakers yourself? You ask all the same questions, no?

Firstly, let's not forget the objective of this hobby - to create the most realistic illusion to our auditory perception. Note, I didn't say accurate - our auditory perception is not concerned with accuracy.

It is actually. Say some word naturally and then cup your hand around your mouth and say it again. One would sound more natural. It is this type of evaluation we do. A natural sense of what is the right sound, and what is the wrong sound, with nary of a reference in sight. Is loudspeaker A more real than B? And real we define in our minds as accurate even though again, we have no reference.

This makes sense on so many levels - we are not measuring machines - we all know that.

But we can be turned into one with far more reliability than random attempts.

We are limited in our auditory processing. It is widely agreed that the "perceptual load theory of selective attention" is correct. In other words a person' attentional resources will be fully deployed in the processing of any incoming sensory information. In other words, as the perceptual load increases, our field of attention narrows & the selectivity occurs earlier in the auditory processing.

Again, all of this is there in both cases. What do you do when you audition a new loudspeaker? You are turning into a measurement machine. You are measuring fidelity in your mind. You will be devoting all of your brain power, constantly analyzing the highly different sound against some other reference you think you have memorized.

The type of testing I am talking about makes all of this easier. In 4 seconds you can hear the other loudspeakers. You can go forward, or backward as many times as you like. You can focus on the vocals once, the bass in the next cycle. Heck of a lot easier than remembering either one of these from another loudspeaker in different time and location.

I relate this to different types of listening as follows. In relaxed, casual listening, we allow more of the elements of the soundfield through to the higher levels of our auditory perception perception - we have better access to more sound elements.

I am sorry John but none of this resonates with me. When I sit in a demo room at CES, I am not having a relaxed time. I am stressing to take in all that there is, hoping to identify better or worse sound than my reference. I see people sitting there so attentively doing the same. Oh so they pretend half the time . I say that half jokingly because I suspect they are struggling to assess if this is a better sound than the room next door. So many variables changed from loudspeaker to room and music. Yet they attempt to rationalize it all. How is that walk in the park and AB test within 4 seconds hard???

I will stop here but I really think you are mixing this type of testing with that of electronics when differences are large. Were you not the person who said DBTs are good when differences are large anyway?

 

[TBone]

... like a room/system, in which to communicate w/the person siting right nxt, you need scream to be understood, even during less dynamic passages.

 

[amirm]

Just look at Harman - they have released their measurements - other manufacturers could use their findings to develop speakers which can compete head-to-head with them - yet they didn't keep these measurement techniques secret - why? - because it gave them a commercial advantage

I don't see how that is a marketing advantage when hardly anyone understands them. And ironically, even Harman doesn't release all that they have. I am able to get that data from them but they don't distribute them openly because of what I just said: it doesn't help sell loudspeakers.

BTW, as I have said time and time again, hardly any of this research is from "Harman." The research comes from when Dr. Toole, Olive, etc. worked at the Canadian National Research Council (NRC). At Harman, the size of their toys, I mean tools , got bigger. But the nature of research and message therein has not changed. I provided all the references to the papers from NRC in the other thread.