Why Some Audiophiles Fear Measurements

[Steve Williams]

I ask that question for a simple reason..

IMO a reviewer should NOT look at the measurements before they listen to the speakers because would this not introduce bias to their evaluation. Also after listening and then looking at the measurements, if the measurements do not confirm what they heard does this not go back to what Mike Lavigne said about "preference"?

 

[FrantzM]

OK that's interesting and what Hi-Fidelity did before JGH left the magazine. But the disconnect one must ask is then how does 86 dB sound like 94 dB? Something in the measurements isn't telling the whole story.

Myles

at the same frequency? NEVER.. repeat NEVER .. on this planet .. NEVER

 

[Steve Williams]

I would like to ask this next question to everyone who listens first and then looks at the measurements second, have you ever pondered why, what you heard does NOT jibe with the measurements

 

[FrantzM]

I ask that question for a simple reason..

IMO a reviewer should NOT look at the measurements before they listen to the speakers because would this not introduce bias to their evaluation. Also after listening and then looking at the measurements, if the measurements do not confirm what they heard does this not go back to what Mike Lavigne said about "preference"?

So Steve

We are hereby admitting that Psycholofy (bias) does play a role in our testimony of perception ... W could go on and ask the reviewer not to know the brand .. We are moving toward Blind testing here .. aren't we?

 

[Steve Williams]

Not at all Frantz

I was merely wondering how reviewers can avoid bias themselves when reviewing and reading the measurements whether this be before or after they listen?.. Nothing more

 

[Alan Sircom]

But please answer my question.....


Has there been an instance when after reading the measurements that your opinion was swayed based on what you heard?

Not unless the original review was rushed through by an editor insisting on scoop-chasing. I've missed a few chuffing ports down to having minutes instead of weeks in front of a product, that labs picked up and I've subsequently heard it on later listening. No-one's perfect.

 

[Steve Williams]

Alan

Please understand I am not trying to be confrontational as I don't know what steps and in what order a reviewer does his evaluation

I do agree that we are humans after all and no one is perfect

 

[Johnny Vinyl]

Good morning Steve! I was wondering if you were a Hornets fan?

John

 

[Steve Williams]

I have, like others, been reading this thread with great interest..

I was merely trying to ask the questions from a different perspective

And Frantz, by no means was I ever trying to suggest DBTs

 

[Alan Sircom]

A sincere question for you Alan!

When you've done the review and are ready to compose it into words for publication, do you make a recommendation or provide an opinion on it's sound or on its measurement data?

John

PS: Steve....that was a wonderful question.

Well currently, my magazine does not provide any measurement at all. I polled the readers about this soon after I took over and as we already have News, Choice, Critic and World that measure, Plus was looked upon as the 'light relief'.

However, from my own opinion and prior experience:
If a product sounds good and measures good... highest recommendation
If a product sounds good and measures bad... recommendation
If a product sounds bad and measures good... no recommendation
If a product sounds bad and measures bad... no recommendation

In my current magazine, current policy dictates the latter two would be returned to the manufacturer unreviewed. This highlights a large flaw in one aspect of the reviewing process - a significant proportion of current readers of magazines seem to read the magazine to confirm their buying decisions after buying, rather than view the review as a part of the pre-purchase buying decision. Our policy to only speak good of audio products is predicated on lots of angry 'cancel my subscription' email that follows every negative review, from disgruntled owners of the equipment that received the negative review.

 

[Alan Sircom]

Alan

Please understand I am not trying to be confrontational as I don't know what steps and in what order a reviewer does his evaluation

I do agree that we are humans after all and no one is perfect

No confrontation seen, or responded to. It's a very fair question and more than deserving of an honest answer.

Pity then that as a reviewer, I wouldn't know how to make an honest answer, even if you gave me the instruction manual

 

[FrantzM]

I have, like others, been reading this thread with great interest..

I was merely trying to ask the questions from a different perspective

And Frantz, by no means was I ever trying to suggest DBTs

Nor was I saying that you were .. It is however clear to me that everyone here admits to our senses being subjected to our psychology IOW are unreliable. The reliability can however be increased by the applications of bias reducing protocols or procedures .. Not necessarily blind but repeatability would be nice . In that quest to be "nice", measurements are the only thing which can insure reliability/repeatability/determinism. Since we already know our senses are not that reliable..????? So even in our evaluation why ..why should some just reject measurements (euphemism for "being afraid of" ) ?

 

[Steve Williams]

If a product sounds good and measures good... highest recommendation
If a product sounds good and measures bad... recommendation
If a product sounds bad and measures good... no recommendation
If a product sounds bad and measures bad... no recommendation

Thanks Alan

I must also ask the question ( I know the answer) as to why you send back components that will get a negative review rather than just print your review and have the brass cajones to stand by it

IMO if a manufacturer sends a product for review, that manufacturer has hopefully done due diligence on his product, so much so that once the product is sent the manufacturer agrees that a review will be published with no guarantees that it will be a favorable one

Unfortunately it comes down to your magazines subscriptions and membership. IMO there is a big difference between reviews in an online publication such as jeff's Soundstage versus a hard journal such as yours

 

[Johnny Vinyl]

Thanks Alan

I must also ask the question ( I know the answer) as to why you send back components that will get a negative review rather than just print your review and have the brass cajones to stand by it
IMO if a manufacturer sends a product for review, that manufacturer has hopefully done due diligence on his product, so much so that once the product is sent the manufacturer agrees that a review will be published with no guarantees that it will be a favorable one

Unfortunately it comes down to your magazines subscriptions and membership. IMO there is a big difference between reviews in an online publication such as jeff's Soundstage versus a hard journal such as yours

I also have that question...although I don't yet know the answer.

John

 

[MylesBAstor]

Alan

Please understand I am not trying to be confrontational as I don't know what steps and in what order a reviewer does his evaluation

I do agree that we are humans after all and no one is perfect

Well the point is to remain neutral and objective when reviewing any component. The reviewer's ultimate goal should be to describe the sound of a component, without their own prejudices, so that the reader can in their mind recreate the sound of the DUT. That's why I though John Nork was an excellent reviewer. He had the ability to remove his preferences from his reviews.

1.) It's impossible to ignore manufacturers measurements since some of these are needed to make sure there's no unaccounted for interactions between components or speakers/amplifiers.

2.) It's also necessary to look at the measurements say in the case of a small speaker like the Alon Adriannas I reviewed years ago, as to whether they sound like they really extend down as far as the measurements might indicate. (and of course many factors, electronic and room enter into this picture.) I might also use something like the Goldline analyzer (http://www.gold-line.com/pdf/manual/pa_30m8.pdf) to check out the speaker/room interactions. Granted it's not the most sophisticated thing available, it does the job that I want.

3.) I think that reviewers on the whole spend more time and have more experience positioning speakers for the best sound than the average audiophile. A lot just comes down to time. Also, they may have the ability to try say more than one amp and speaker cables with a given speaker.

4.) If it measures good and sounds good, it is good; if it measures bad and sound good, then you're measuring the wrong thing.

5.) I'm always curious about the measurements but I don't let them sway my final conclusion. One thing though. It's great that SS and SP attempt to do more sophisticated speaker measurements; however, these measurements are rarely, if ever, done in the reviewers room. So there's always a disconnect between the measurements and the reviewer's perceptions. And as Jeff and Sean have pointed out, different speakers present different problems with room; how about an omni-directional speaker such as the MBL101s that probably excite the whole room. I don't think very many people have heard what these speakers can actually do eg. every time I've heard them at a show, they've sounded awful. And I might have concluded that's the way the sound, if I didn't hear them at a colleague's home where he had the right room for the MBLs. Again that's where the measurements will help in determining in the final estimations on whether a given speaker will work in a wide variety of rooms.

 

[silviajulieta]

Dear Steve: I understand that at STP JA don't shows the audio item measurements before is reviewed. What I don't know is if after the review and before is published the reviewer can make changes in the review due to those measurements that he now knows.

Regards and enjoy the music,
Raul.

 

[Jeff Fritz]

IOW do you listen to the speakers before or after reading the measurements?

The reviewer doesn't see the measurements until they are published with the review.

Now, question for you Steve: When you see poor measurements of a product that you like, do you question your hearing, ever?

 

[Johnny Vinyl]

The reviewer doesn't see the measurements until they are published with the review.

Now, question for you Steve: When you see poor measurements of a product that you like, do you question your hearing, ever?

Interesting.......

John

 

[Kal Rubinson]

Dear Myles an Kal: IMHO there is no rules about, that that is what is working for you only means and say: that's working for you and nothing else. Could it works for other people? sure could works but not for all.

I'm not a reviewer like you both but I have a process that I use to test audio items and this process works for me just from the beginning that I'm used it. First I don't trust in the long time ( two months. ) bbecause psychoacoustic brain effects: it is proven that our brain accustom/equalize to " long " sound reproduction time periods, the brain switch and makes the necessary adjustements to the new " sound " and then we lose efficiency to discern against the old audio signal.
So, IMHO the best is to go as fast as you can for not lose these critical/important first listening hours that tell you almost all ( maybe 90%-95% ) about the new signal/sound before the brain be equalized. That's why ( between other targets ) I attend to listen live music at least twice every week and try to heard other systems as often I can: to avoid " brain equalization ".

Believe me, I understand the issues of neural adaptation. However, there are things that one hears differently in different listening paradigms, A/B comparisons and long-term listening being the opposite poles. So, I agree that one needs "fresh air" periodically and that is why I do listen to live music and to multiple systems regularly. More significant, imho, is the different mental receptivity between active A/B listening for distinctions and long-term listening to the music (while the equipment is subjugated to a vehicle). All modes are useful but the latter, obviously, is closer to what the ultimate user/owner will be doing.

Kal

 

[Jeff Fritz]

4.) If it measures good and sounds good, it is good; if it measures bad and sound good, then you're measuring the wrong thing.

This is part of the problem, Myles. It assumes the reviewer's hearing is infallible.

This really comfirms, in my mind, what I said in the article: "One is that measurements can threaten an audiophile’s belief system. . . . along comes a set of measurements that directly contradicts what . . . he claims to hear. This situation can be embarrassing to the audiophile, even stressful, and the easiest way out of it is to dismiss the measurements altogether."