Objectivists, Harman Testing, Reviewers, and Reality

[jkeny]

...
I will stop here but I really think you are mixing this type of testing with that of electronics when differences are large. Were you not the person who said DBTs are good when differences are large anyway?

Yes, Amir, I wasn't limiting my reply to just speakers & Harmon's testing - I was addressing a wider aspect. I obviously failed in may attempts at being as clear as I hoped.

 

[jkeny]

I don't see how that is a marketing advantage when hardly anyone understands them. And ironically, even Harman doesn't release all that they have. I am able to get that data from them but they don't distribute them openly because of what I just said: it doesn't help sell loudspeakers.

BTW, as I have said time and time again, hardly any of this research is from "Harman." The research comes from when Dr. Toole, Olive, etc. worked at the Canadian National Research Council (NRC). At Harman, the size of their toys, I mean tools , got bigger. But the nature of research and message therein has not changed. I provided all the references to the papers from NRC in the other thread.

Ok, Amir, but again I was addressing Tim's statement that "I'll bet a whole lot of those high-end manufacturers have measurements. They're just not sharing them with you."

OK, the research came from Toole & Olive but Harman did differentiate themselves as a speaker company by adopting this research - making them unique - a marketing advantage, I believe?

Without giving away any secrets, can you tell us if the measurements that Harman don't release are of any substantial importance?

 

[amirm]

Ok, Amir, but again I was addressing Tim's statement that "I'll bet a whole lot of those high-end manufacturers have measurements. They're just not sharing them with you."

Sorry, I misunderstood .

OK, the research came from Toole & Olive but Harman did differentiate themselves as a speaker company by adopting this research - making them unique - a marketing advantage, I believe?

It is a small part of their consumer messaging although a larger one in private communications to dealers.

Without giving away any secrets, can you tell us if the measurements that Harman don't release are of any substantial importance?

It is the so called "spin data." It is a 72 point measurement system where they capture the radiation of the loudspeaker in 3-D space in anechoic chamber. They then take all of those measurements and combine them into three graphs that based on special weighting can to good degree predict user preference. Here is an example of them providing it for the JBL M2:

They have this data for loudspeakers from $150 in-wall to $60,000+ JBL. But with some exceptions, they are not put out in public. They are shown in private dealer training though.

 

[rbbert]

John, don't you have measurements of your products you don't put on a spec sheet or Web page? I would be very surprised if manufacturers openly shared all their measurement data, for the simple reason that few if any people would know how to use that information. In all likelihood the manufacturer isn't even sure of the relevance of at least some of it.

 

[JackD201]

Out of curiosity I wondered if Harman's marketing guys are in line with what their tech guys are doing regarding their more expensive products.

http://www.revelspeakers.com/productdetail/~/product/studio2.html

http://ph.jbl.com/tl_files/catalog/...ification Sheet - Studio 590 (English EU).pdf

http://www.jblsynthesis.com/products/floorstanding-speakers/everest-dd67000/

http://www.jblpro.com/ProductAttachments/M2_Brochure_Jan2013.pdf

http://www.jblpro.com/ProductAttachments/7Series_Datasheet_120414.pdf

Only the M2 has Spectral Polar Response, none have the SPL, Frequency, Directivity Index (Spinorama) charts we saw in Dr. Toole's video. Note that the Studio 2 and the Everest surely predate the 2013 standard. Whether they've been "spinned" or not we don't know. What we do know is that the charts are absent in their online material. The 7 series speaker in the link provided however does not. It is a new speaker.

Thoughts:

Either Harman chooses not to include the data or not all their products have been submitted to their research arm for measurement the same way as the M2. Maybe it is a mix of the two. What is more likely? I don't know. Why the originator of a new AES standard doesn't include the standard for their own products I also don't know.

As I submitted this post amir posted and said that they do have the data down to the little in-walls. I suppose the next question is why this is shared with dealers during training but not to the general buying public.

 

[Bruce B]

Dr. Toole laments that when a person goes shopping for multi-thousand dollar piece of audio equipment, he has far, far less information than on the side wall of a $50 tire:

AFAIK, no one has ever been killed using an incorrect piece of high-end equipment! Me thinks that is the reason why....

 

[jkeny]

John, don't you have measurements of your products you don't put on a spec sheet or Web page? I would be very surprised if manufacturers openly shared all their measurement data, for the simple reason that few if any people would know how to use that information. In all likelihood the manufacturer isn't even sure of the relevance of at least some of it.

By & large, my experience is that we aren't rational beings - we have decided something emotionally & then rationalise our decision. Objectivists demonstrate this all the time. I recently experienced it on the Squeezebox forum in a thread asking about AKM4420 Vs AKM4396. Of course it was pointed out by the objectivists that nothing on the two datasheets show any audible difference. Some who poste dthat they heard differences between devices that had these two DAC chips in them - they were asked about DBTs to "prove" this.

Later some measurements from Sean Adams (the Squeezebox developer) were also posted showing comparisons between SQ transporter & other DACs. Comments by Archimago - "Wow great post by Sean Adams". When I pointed out that Sean Adams introduced these measurements with ""if we look closer at some of the characteristics that we think will determine subjective sound quality, there are some pretty substantial differences between transporter and any other DACs I've tested." & all measurements were below the accepted levels of audibility (according to objectivists) ), Archi went quiet & others started to defend the measurements as technical measurements.

So we have the same group having two different & polar opposite reactions to basically the same information (measurements below "audibility") but that information coming from Sean Adams on one side & a normal forum member on the other side. As i said - rationalisation of a position already decided - not based on rational thinking.

I believe that good sounding products are recognised by people given an extended period of casual listening in their own home systems!

 

[kevinkwann]

AFAIK, no one has ever been killed using an incorrect piece of high-end equipment! Me thinks that is the reason why....

Astute observation!

 

[mep]

Hi Mark. Let me commend you from the point of view of your writing skills. Certainly far better than mine, with drastically less grammar error. I am unclear as to the central message though. Are you saying that since there is not much data out there, we need to bend to adjust to it because that is reality?

No, what I'm saying is that is the reality we live in. If you think Meridian is going to have a problem growing legs with MQA in the marketplace, your utopian vision of where you would like the audio world to be doesn't have a chance of growing feet in the marketplace.

Seems like a very defeatist point of view instead of using our influence on manufacturers and reviewers.

First of all, I'm just being a realist vice being a defeatist. What incentive do manufacturer's have to spend more money to take more measurements and publish them when the reality of the marketplace is that the majority of their customers wouldn't understand the measurements anyway and certainly aren't demanding them?

How about next time you are at a show and going room to room as, "how does the off-axis measure on this? Is it smooth and similar to on-axis?" As a minimum, it would make you look like you know something about loudspeaker design .

How many rooms have you been in at shows and the people who are running the gear while you are there only have knowledge of the component they have in the system? Speaker designers aren't standing in every room that is using their speakers waiting to answer your questions. And even if they were, do you really expect them to say anything damning about their speakers?

That aside your statement above is partly false. As I have said repeatedly, on this issue of loudspeaker preferences and double blind testing, you are unified with objectivists on forums. Their schtick is to prove to you everything sounds the same. But they know loudspeakers don't sound the same so they never talk about blind testing of loudspeakers. Indeed, they mount a heavy defense and fight against relying on double blind testing of loudspeakers: "they sound different so we don't need blind testing." So this is one time the two camps are unified..

I think you just sang some of my song Amir. Where I disagree is that some objectivists on this forum have been cheerleaders in public on DBT testing of speakers in general and Harman's method in particular. They just don't practice what they preach if for no other reason that it's really not possible. It just sounds like a good idea so they salute it.

Also, sighted evaluation does not automatically mean wrong conclusions. I used the word "not trustworthy." You could tell me A is better than B in sighted evaluation. You could be right and reflecting true fidelity you heard. Or you could be influenced by looks, price, marketing, prior knowledge, peer pressure, etc. into having a modified observation. We can't separate these things in your conclusions. In that regard, the outcome is not reliable and can't be counted upon. Certainly if this were life and death situation we wouldn't remotely take your word for it that what you observed was purely based on the one factor. We are human after all and such an ability eludes even the most skilled and trained listeners (although not nearly as frequently as it does for casual observers).

Did I just read that you said sighted evaluation does not automatically mean wrong conclusions? I think if you read your entire response to me that we have many areas of agreement.

Dr. Toole laments that when a person goes shopping for multi-thousand dollar piece of audio equipment, he has far, far less information than on the side wall of a $50 tire:

Again, we are singing from the same sheet of music.

As you say, none of that information is available to us for vast majority of equipment we buy.

I'm glad you agree with me again.

As I said at the outset of the reviewer's threads, we have ourselves to blame. We let, heck with threads like this, encourage manufacturers to tell us less. Tim is very right. A lot of information exists than is provided to us. If we don't ask or value it, then the less they tell us, the more apt we are to buy on touchy, feely words than reliable information.

Reviewers aren't going to change those facts Amir. We are a drop in the bucket compared to their non-reviewing customers and their non-reviewing customers aren't asking for more or better measurements.

It is not like you can go to the corner store and see all of these loudspeakers in one place either. They cost so much, and manufactures so selective, than you can't do any kind of comprehensive comparison by any method yourself. It is in that light that I find it so surprising to not support better information conveyance to us. Look, you can always ignore the data. But you can't manufacture it if it doesn't exist.

Again we are in total agreement. That's why many people rely on reviews to help them make their purchasing decisions.

A lot more should be lining up but because they are sold on marketing messages instead of engineering, they don't do that. Case in point is a high-end customer of my company. He wanted a bunch of in-wall loudspeakers. We supplied Revel in-walls powered by ordinary commercial amplifiers. He outfits the one room himself with B&W loudspeakers and Moon amplifier. He calls us yesterday and says he wants to rip out all the B&W's and Moon amp and replace it with Revels! Replacing in-wall speakers is big deal as it rips out the walls and such. But that is what he wants. Why? Because even that few hundred dollar in-wall Revel loudspeaker is not allowed to be released until such time that it beats its competitors in double blind testing.

But yet your customer bought your speakers not because of the measurements, but how the speakers sounded.

Now, I wish my company did not have a business relationship with Harman and I was not friends with many of the people I talk about who are behind this research. Maybe then you would believe what i am saying as being less biased. But something tells me you wouldn't regardless. A negative bias is set in there as you say, "who buys Revel loudspeakers???"

It's not that I don't believe what you are saying. As evidenced by the things you have said in your post, we agree on many things. I never said "Who buys Revel loudspeakers" because I know that plenty of people do. No company can stay in business without customers to buy their products unless the owner(s) of the company are rich and are running a hobby and not a business.

It is a hard switch to flip. I know I am shaking the very foundation we all, yes, all of us whether objectivists or subjectivists, walk on. It didn't come easily to me to believe it either. So appreciate you keeping the topic alive as it gives us a chance to put all the data on the table. All I hope is that a seed is planted in you ear. That one day you too will see it of value if not for anything but to have a lexicon with which to speak to your loudspeaker designer. One can only hope .

I hear what you are saying-trust me. At the end of the day, most people are buying on blind faith or their ears. Trying to educate the masses on not just the importance of measurements but how to read and interpret them is a long, hard slog. Most people are content to make their decisions with their ears because in the end, that is what they have to please.

 

[amirm]

How many rooms have you been in at shows and the people who are running the gear while you are there only have knowledge of the component they have in the system? Speaker designers aren't standing in every room that is using their speakers waiting to answer your questions.

I have never ever had trouble speaking to designers at any of these rooms, sans a few standoffish ones like Wilson room. Vast majority of the rooms are actually presented by the designers. And the ones that are not, as soon as I ask questions, they introduce me to the designers. Since you mentioned Meridian and MQA, I listened to the demo, asked a technical question and they took me to Bob Stuart immediately, not knowing of course that we knew each other already. This is at CES but maybe other shows are different.

Indeed access to designers is one of the pleasures of dealing with high-end companies. They are small so the principals are almost always there. As sharp contrast to say, Apple, Samsung, Sony, etc where you can't even get past the actors on stage let alone anyone technical.

Now I must admit that I walk around with a badge that says "Buyer" because of my company so maybe I am getting special treatment. Some also know me. But the people are definitely there.

This is by far the most enjoyable part of the show for me. I get to chat with my peers, get inside scoop, background on the design, etc.

 

[amirm]

AFAIK, no one has ever been killed using an incorrect piece of high-end equipment! Me thinks that is the reason why....

Not killed but good number have committed suicide virtually over someone not agreeing with their audio views online.

 

[Bruce B]

touche

We've had 2 this week already!!

 

[esldude]

And, as a matter of investigation, I would like to repeat that same listening preference test using the same listeners, but with the speakers set up in a stereo configuration, rather than the mono configuration that Harman utilizes. I would be curious to see how the scoring and preference would differ, if at all.

I understand wanting to do this to convince yourself. But that testing has already been done at Harman. You get the same preference ranking among a group of speakers in stereo as in mono. The range of rankings is smaller, somewhat like trained vs untrained listeners. It seems less discriminating in stereo than in mono. Again, far different than most audiophiles expect. Their data is counter to those expectations. Harman does mono testing because their data indicates it works better than stereo testing. If stereo was the better more revealing way they would use it instead. Another one of those results I would not expect, but again one we should be happy with as it is good news. Makes comparison testing quicker, easier, and more doable if we wish to do such things ourselves.

 

[TheMadMilkman]

That it's been done before is no reason for me to not want to personally experience it again.

And I would like to see the results among forum members who state that they find imaging and soundstage to be at least as important as tonality. Especially given the claims by some that Revel speakers lack in the soundstage department.

I should also state that I do have some bias in this discussion. I will be purchasing a pair of Revel M105s as soon as my current speakers sell.

 

[Rodney Gold]

The tyre analogy is not that good , the specs of a tyre do not predict its performance , the unsprung weight , suspension , power delivery , traction aids etc and the tyre determine how it performs. It's a system.. all you can do with the tyre info is select the right fitting tyre..

At any rate , a HUGE part of my buying decision re the G1's were the looks.. and I have done my room to be aesthetically pleasing to me.
I eat using my eyes , my sense of smell , taste and texture and all of them combine.
It is the same with humans , we use our senses but not in isolation.. why should we do so with Hifi.

And furthermore , even if a difference does not exist in real life , it might be perceived , and if so it will be TRUE for the person that perceives it.

One has to investigate how the various segments in the hifi world buy.. this forum would not be relevant , cos it maybe only represents the top 5% of the 10% of music lovers that care about hifi more than just playing music.

 

[esldude]

The tyre analogy is not that good , the specs of a tyre do not predict its performance , the unsprung weight , suspension , power delivery , traction aids etc and the tyre determine how it performs. It's a system.. all you can do with the tyre info is select the right fitting tyre..

At any rate , a HUGE part of my buying decision re the G1's were the looks.. and I have done my room to be aesthetically pleasing to me.
I eat using my eyes , my sense of smell , taste and texture and all of them combine.
It is the same with humans , we use our senses but not in isolation.. why should we do so with Hifi.

And furthermore , even if a difference does not exist in real life , it might be perceived , and if so it will be TRUE for the person that perceives it.

One has to investigate how the various segments in the hifi world buy.. this forum would not be relevant , cos it maybe only represents the top 5% of the 10% of music lovers that care about hifi more than just playing music.

Well, I think you are wrong about the tyre info. The speed rating predicts it speed capabilities. Its traction ratings predict what conditions it would be suitable. Its load rating can only be seen as a predictor of performance. Now if you don't use some sense to apply the tire, then you ignored the info made available.

 

[Rodney Gold]

I have raced cars , built cars , done drag racing , had a pimp your ride V8 shop and been into performance cars for 40+ years
I can tell you , you are wrong.. the ratings on a tyre cannot tell you how it will perform in real life situations.

 

[FrantzM]

The tyre analogy is not that good , the specs of a tyre do not predict its performance , the unsprung weight , suspension , power delivery , traction aids etc and the tyre determine how it performs. It's a system.. all you can do with the tyre info is select the right fitting tyre..

At any rate , a HUGE part of my buying decision re the G1's were the looks.. and I have done my room to be aesthetically pleasing to me.
I eat using my eyes , my sense of smell , taste and texture and all of them combine.
It is the same with humans , we use our senses but not in isolation.. why should we do so with Hifi.

And furthermore , even if a difference does not exist in real life , it might be perceived , and if so it will be TRUE for the person that perceives it.

One has to investigate how the various segments in the hifi world buy.. this forum would not be relevant , cos it maybe only represents the top 5% of the 10% of music lovers that care about hifi more than just playing music.

If their looks didn't match their performance I am sure you would have looked elsewhere.

I believe that standardized tests disturbs our audiophile mind very deeply. It is a bruise to a person psyche to discover how pedestrian a perception you imagined to be above average truly is. Thus anything that remotely suggests that is met with incredibly fierce resistance and sometime downright erratic behavior. Let's look at the "Believe you ears" mantra. Most here do know how unreliable our senses are. Proofs abound of how easy it is easy to foll them, yet this belief is very prevalent on this board and likely in the audiophile world at large.
The methodology advocated by Harman is not easy to pull off. I would however imagine that for someone who purports to review a technological items whose purpose is to replicate the signal that is fed to it, a certain amount of objectivity and a methodology to remove (some of) their biases for the review would be warranted. The Harman methodology would be at least a starting point. We have the contrary. Most audio reviews are similar to food reviews. Some will argue that they do get something right, a debate in itself but let's suppose that it is is so. I would counter with Bruce B. tag line "Even a blind squirrel finds a nut once in a while ". If it was the case to accept a sorry state of affair as "Reality" there would bre no High End Audio. it was a reaction in the late 70's to what was perceived as a lack of product of distinction in the marketplace. We must demand that things change. The status quo is not good... This must change. We must demand more and better reviews. Some are trying to. I would cite the works of Soundstage among other but also Stereophile. There may be others.

 

[JonFo]

While being an objectivist that finds the Harman speaker tests to be of great value, it does have its limitations. It really only works for monopole speakers, yet they put in one dipole ESL in a group and of course it got ranked really low. Not surprising as the rear wave of the dipole was completely unmanaged in the test rig. So testing methodology needs to align with product topology.

But in general, I for one, really value all the objective data a manufacturer can provide on their product, as it is a great predictor of a given piece of gears performance. I won't really know until I hear it, but I can sure see why product A might be preferable to product B.

Measurements alone won't tell me what will be best, but they do help me correlate what I hear (and prefer) as I change elements or setting in the system. I wish more audiophiles would learn to make use of measurements, as it is a great way to reliably 'see' the results of your changes.

 

[Phelonious Ponk]

Exactly what due diligence are reviewers supposed to bring to the table? We don't have any more information about the products we review than you do. All we have to go on is the same spec sheets everybody can see from the websites of the manufacturers. What reviewers try and do is tell you how a given component or speaker sounds in the context of their system and their room. Based on comments by people on this very forum who admitted they have bought products based on reviews and were quite happy with their purchase, you have to surmise there is merit to the method.

I don't expect the audio press, as it exists today, to bring anything more to the table. They're simply not equipped to do it.

If they truly have them, beyond the information they print on their spec sheets, they are not sharing them with you either-never mind the reviewers.

This is correct. Manufacturers of good audio equipment - high end or otherwise - have much more data than they're sharing. They couldn't develop their products without it. They're not sharing it because A) Much of it would probably not be very good marketing. B) The market is not asking for it. We're simply accepting stats, reviews based on the subjective impressions of people who know no more than we do, and the very limited amount of auditioning we can actually do. These limitations are well-accepted in the "high-end," but in the meantime, the mainstream seems to have abandoned the stereo system altogether. There are many factors influencing that abandonment, but I wonder, if there was a good market of great-sounding stereo gear available at reasonable prices, would stereo have done better? Even for watching TV and movies, I'd put a couple of thousand dollars worth of monitors and a sub up against a couple of thousand dollars worth of 5-channel any day. And it would be a lot less intrusive in the living space.

Tim