Why Some Audiophiles Fear Measurements

[Kal Rubinson]

Our ears and brain are more accurate than science for the tasks we perform on a daily basis.

If you mean tasks that have survival value, like predator detection, yes. I do not think there is any Darwinian value in having a "golden ear." That said, our brains contribute great and subtle mechanisms to auditory analysis but they confound that same task by being more specialized for other tasks. The best example, imho, is the rapid neural adaptation to environmental acoustics. (BTW, the visual system does the same for the same reasons.)

Kal

 

[Kal Rubinson]

I have a question for Jeff, Alan or Myles or anyone for that matter

When a speaker is sent for review or when a speaker is heard under ideal room conditions, has there ever been an instance where you "liked" the speakers only to now have a look at the measurements, which if "bad" creates some bias on your parts only to now "not like" the speakers?

IOW do you listen to the speakers before or after reading the measurements? If you read the measurements before listening and the measurements are not so good, does this introduce bias to your reviews?

By the same token if you see measurements that are really good, has there been an instance when after listening to such speakers you didn't like their sound?

I guess what I am really trying to ask is "if you read the measurements before listening, does this not add some degree of bias either way on your part?"

Not trying to stir up the pot but just a question that I feel is worth asking

First, at Stereophile, we do not get to see the measurements until we submit the copy of the subjective review and, as a result, the review comments are unaffected. Second, there have been a few cases where I have been very positive about a product and the measurements revealed a troubling flaw. In his commentary, JA will tactfully point this out for the reader.

BUT..... although it cannot bias my review, it does influence my feelings, as it will the reader's.

Kal

 

[Kal Rubinson]

I would like to ask this next question to everyone who listens first and then looks at the measurements second, have you ever pondered why, what you heard does NOT jibe with the measurements

Of course. It precipitates a lot of self-analysis.

 

[Jeff Fritz]

For instance, what if a reviewer stated the following:

"I have to say that the . . . loudspeaker system is the best that there is, the best that there ever was, and I suspect the best that there will ever be."

Would you expect there to be some type of objective element to supplement that listening impression? Would it be helpful? Or is that statement all you need?

Myles, if these speakers were measured and they revealed that the product was not the "best that there is," would you question the measurements or the reviewer? In other words, would we be measuring the wrong things?

 

[Kal Rubinson]

Well currently, my magazine does not provide any measurement at all. I polled the readers about this soon after I took over and as we already have News, Choice, Critic and World that measure, Plus was looked upon as the 'light relief'.

However, from my own opinion and prior experience:
If a product sounds good and measures good... highest recommendation
If a product sounds good and measures bad... recommendation
If a product sounds bad and measures good... no recommendation
If a product sounds bad and measures bad... no recommendation

In my current magazine, current policy dictates the latter two would be returned to the manufacturer unreviewed. This highlights a large flaw in one aspect of the reviewing process - a significant proportion of current readers of magazines seem to read the magazine to confirm their buying decisions after buying, rather than view the review as a part of the pre-purchase buying decision. Our policy to only speak good of audio products is predicated on lots of angry 'cancel my subscription' email that follows every negative review, from disgruntled owners of the equipment that received the negative review.

Interesting. This is not Stereophile's policy. Every product accepted for review must be reviewed even it falls into the latter two categories. Of course, we are subject to the same comments "from disgruntled owners of the equipment."

 

[silviajulieta]

This is part of the problem, Myles. It assumes the reviewer's hearing is infallible.


Dear jeff: As you well said it this is an important issue of the problem: all us " Golden Ears " gurus are infallible and this is not true, nothing is perfect.

What worried me about is that sometimes a " new kid " in the reviewer system under test sounds " bad " ( even if has good specs and measures. . ) and instead to look around the system to find out if the " new kid " is so good that " disclose " a lesser quality performance elsewhere in the system the reviewer rank that " new kid " like a non recommended audio item.

Because there is not only that we found " infallible ears " but we found too " infallible system and infallible system set up ".

Regards and enjoy the music,
Raul.

 

[steve williams]

The reviewer doesn't see the measurements until they are published with the review.

Now, question for you Steve: When you see poor measurements of a product that you like, do you question your hearing, ever?

I definitely question my hearing Jeff but to ask you the same question, have you ever heard speakers which in your impression sounded good but measured poorly

 

[Jeff Fritz]

I definitely question my hearing Jeff but to ask you the same question, have you ever heard speakers which in your impression sounded good but measured poorly

Yes.

 

[steve williams]

So did you check your hearing or undergo self analysis as Kal suggests. How did you account for your perceptions when compared to the measurements

 

[Jeff Fritz]

So did you check your hearing or undergo self analysis as Kal suggests. How did you account for your perceptions when compared to the measurements

Self analysis is the key. I've since become a better listener, and now when I hear the same speaker(s), I have a greater comprehension of what I am hearing. And since that time I've heard much better, more neutral speakers, so my pool of experiences has broadended considerably.

 

[MylesBAstor]

Myles, if these speakers were measured and they revealed that the product was not the "best that there is," would you question the measurements or the reviewer? In other words, would we be measuring the wrong things?

Depends on many things Jeff. What measurement would make the speaker "not the best there is?"

Many things enter into the picture when evaluating reviews. For instance, what is my opinion of the reviewer and does he/she have the experience to make such a statement! I find too often people become reviewers nowadays and all they want to review is expensive gear. Years ago, reviewers, esp. with TAS, had to prove themselves with entry level gear before moving onto more cutting edge designs. There's a lot to be said for a process driven program, rather than a results driven methodology. Then, I have to say my BS meter is turned on when someone would say these speakers (or for that matter any component) are best there ever will be HP is wonderful at leaving himself wiggle room; others are not and they tend to be the component of the month. Also, I just don't think that there is such as thing as a "best" speaker. All speakers have tradeoffs; planars/electrostatics will do some things that box speakers won't and vice versa. Some may come close to bridging that gulf but never totally eliminate it. Lastly, I'd like to hear these speakers for myself in my system before drawing any conclusion.

 

[silviajulieta]

I definitely question my hearing Jeff but to ask you the same question, have you ever heard speakers which in your impression sounded good but measured poorly

Dear Steve: IMHO I think that you are going to unreal hipothetic speaker issues in a review. What means " measured poorly "? against manufacturer specs? against which standards? against which environment? which the references?

Speakers design in today times are really good and normally we don't find " poorer measures or specs " for what the speaker was designed. The speaker competence for the market is " fierce " and every manufacturar in this market is fighting very hard to take his portion, so they are entilted to make good speaker designs. This not happen in the same " level " in other audio areas. If you take a look how many amplifier or preamps or TT's exist against the speakers manufacturers number you will see that the speaker manufacturer number is really really high against other audio products.

Maybe is better to talk in a stage more real.

Regards and enjoy the music,
Raul.

 

[steve williams]

But Raul didn't Jeff post a link to a SS review where the mfrs measurements suggested a 94 Db efficiency but SS only found it to be 86db

 

[silviajulieta]

But Raul didn't Jeff post a link to a SS review where the mfrs measurements suggested a 94 Db efficiency but SS only found it to be 86db

Steve: The Exception confirm the rule. You could find that kind of cases but in that speaker case efficiency is only important on the amplifier choice but the speaker quality performance stay the same.

Anyway, my meaning is that I think we all have to find out answers in the thread that could help us. The thread already touched several audio subjects and my hope is that does not end with out " conclusions ", rational conclusions. This is my honest hope because the majority of the threads in forums end in almost " nothing " to any one, ends like started.

regards and enjoy the music,
raul.

 

[steve williams]

I definitely agree with you but does suggest that independent tests are necessary for proper evaluation.

 

[Alan Sircom]

Thanks Alan

I must also ask the question ( I know the answer) as to why you send back components that will get a negative review rather than just print your review and have the brass cajones to stand by it

IMO if a manufacturer sends a product for review, that manufacturer has hopefully done due diligence on his product, so much so that once the product is sent the manufacturer agrees that a review will be published with no guarantees that it will be a favorable one

Unfortunately it comes down to your magazines subscriptions and membership. IMO there is a big difference between reviews in an online publication such as jeff's Soundstage versus a hard journal such as yours

As I said, it comes down to the reaction from the readership. Having worked on magazines that don't have an all-sugar policy, the biggest influx of 'incandescent with rage' letters and emails come from owners of the disliked product who wanted reassurance of the wisdom of their buying decision and now feel the resale price of that product has just gone down the toilet. I've even been pinned to a wall by an angry reader over 'slaughtering' his beloved cartridge (four out of five star review) at a show. The manufacturers - by and large - are not happy with a bad review, but accept it.

I have been struggling to change this in the current magazine, but as ever it's not well received by the readers. Some of this might be UK specific, in fairness. We had a decade of incredibly dogmatic magazines like The Flat Response that might have been fun to read, but made those outside the magazine's little clique feel like they were some kind of audio leper. Readers who have been with audio for a couple of decades or more are perhaps over-sensitive to negativity in reviews, as it's like opening an old wound.

Bear in mind also that the rejection process depends on whether a product is unsuitable for use or unsuitable for that reviewer. Giving a reviewer with a listening room the size of Ohio a pair of LS3/5a for review (it's been done) or a pair of Wilson Alexandrias to someone with a room 7'x10' (it's not been done.. yet) will not give results indicative of how typical buyers will use that product. Similarly, if I give a Jadis amp to someone who starts every review with 'I hate tubes almost as much as I hate the French', their own bias will cloud the issue. In those cases, I would prefer a reviewer reject a product, or pass it over for another reviewer, than unfairly destroy a product. This covers something close to 80% of the products that get rejected. Some - most - will recirculate products through the rest of the reviewers in a magazine's stable, but there are some who don't and some companies who get upset when the product is handed to reviewer X and reviewed by reviewer Y. Those ones get sent back.

A greater concern is those products that appear unsuitable for any reviewer, but still fit for purpose. The decision then is to whether run with the product or reject it. My usual reaction is to run with the product - if you can't find anyone to like it, that usually happens for good reason - but fortunately this is a relatively rare situation. Usually, this would involve a conversation with the manufacturer to try to determine 'why' the product was so poorly received. These are usually tested, but on a case-by-case basis, as they are usually the ones reviewers are most likely to be perceived to get 'wrong'.

Products that are just plain unsuitable don't come along that often, because the free market tends to cut off their oxygen long before they make it to review. Generally, such products tend to come from small start-ups that lack the R&D budget, but such reviews are seen as the magazine is picking on the little guy. More often than not, having talked the newcomer designer through the design and manufacturing processes, marketing and how a review will not necessarily and instantly kick-start their company to superstardom, fame and fortune and why sticky tar is not a good cabinet finish, I will decide not to review the product because my review will extinguish the start-up before it has a chance to start-up. In one case - that is approximately one case out of more than 2000 products - I decided not to write the review because there was no way I could word it without being sued for libel. I returned the product with a list of books on loudspeaker design that I felt the 'designer' should consider reading should he wish to continue in business. I subsequently discovered I was not the only reviewer to do that. You might view this as suppressing the little guy and protecting the industry titans (the designer certainly did), but I guess the world simply isn't ready for a crossoverless 12-way.

I would hope for due diligence on products received for reviews, but I reject more products because someone lashed together a last-minute sample for review that doesn't work than I do reject products on the basis of poor performance. Once a company submits a product for review, the company has no outcome on the nature of the review, or indeed when or whether it runs in the magazine. Some try.

 

[jazdoc]

Well this thread is likely to go on for many more posts and I suspect that in the end, no one's mind will be changed.

I do think that the speaker example is relevant for both schools of thought in that you aren't likely to have satisfactory subjective results trying to mate an inefficient speaker with a 6 wpc SET amp. This is an example were the measurements can help guide the audiophile. I dare say that we all use a combination of the subjective and objective approaches in choosing equipment and assembling a system. Don't we all gravitate towards equipment that we find appealing to look at?

I would propose the following fun challenge:
The objectivists could put together a system from an extensive equipment list, blinded to everything except objective numbers. The subjectivists would do the same; assembling a system from the same equipment list, except blinded to everything except how the system sounds. Each camp could set up their system to the best of their ability at a show and have the public decide which sounds better.

 

[steve williams]

Thanks Alan

We all appreciate the candor

 

[Jeff Fritz]

What measurement would make the speaker "not the best there is?"

My guess? Pretty much any one of them.

We need checks and balances. Measurements, editors, etc. . . . that's really the point.

 

[Alan Sircom]

Interesting. This is not Stereophile's policy. Every product accepted for review must be reviewed even it falls into the latter two categories. Of course, we are subject to the same comments "from disgruntled owners of the equipment."

I think the policy is not a good policy, but it's a policy I inherited. I am slowly changing it.

I think we can be oversensitive toward the 'angries', but I think this is more common in UK magazines as a reaction to the flattest of Flat Earth mags in the 1980s. If you spent nearly a decade being repeatedly told that you were some kind of lobotomized pond-scum for not owning a Linn Sondek, you tend to twitch at polemic.