Poll: Objective Measurements versus Subjective Reviews

[PeterA]

Two recent threads discuss the merits of objective measurements and subjective reviews. Here is a poll to determine what the WBF membership values most when selecting audio gear.

 

[rockitman]

i rely on listening but won't ignore reviews and measurements when available.

 

[CGabriel]

There are entirely too many choices in all audio equipment categories. You really need some method of narrowing the field of possible components to try. I primarily rely on people that I know and have a level of trust in their perception and judgement. Then, I try the component in my own systems. If the manufacturer supplies specs I read them. If they provide measurements I read those also. And I do my own measurements for some equipment. For example I have measured several speakers including the KEF LS50 which has exceptionally smooth frequency response. But in the end the final arbiter of what I keep is determined by extensive listening tests.

 

[mep]

There are entirely too many choices in all audio equipment categories. You really need some method of narrowing the field of possible components to try. I primarily rely on people that I know and have a level of trust in their perception and judgement. Then, I try the component in my own systems. If the manufacturer supplies specs I read them. If they provide measurements I read those also. And I do my own measurements for some equipment. For example I have measured several speakers including the KEF LS50 which has exceptionally smooth frequency response. But in the end the final arbiter of what I keep is determined by extensive listening tests.

And so it is for the majority of people and even those who are full of puffery and claim to love that which they don't have (measurements) nor participated in (DBTs).

 

[rockitman]

And so it is for the majority of people and even those who are full of puffery and claim to love that which they don't have (measurements) nor participated in (DBTs).

I think the ardent measurementalists would like to delude themselves into thinking that an amp for example...that has the same wattage and distortion levels but costs $2,000 will sound a good as another with similar specs costing 5 times or more.

There is nothing wrong with affordable equipment. Everyone has a budget.

 

[amirm]

How do I vote if I highly value measurements but also do listening? You have all variety of "I rely mainly on listening..." but not the other way around.

And oh, the whole poll is invalid because all the choices end in colons not periods.

 

[amirm]

There is nothing wrong with affordable equipment. Everyone has a budget.

Don't say that please with that picture in your Avatar. What we call a "budget" in this forum, is the cost of dream homes for vast majority of people...

 

[BlueFox]

Don't say that please with that picture in your Avatar. What we call a "budget" in this forum, is the cost of dream homes for vast majority of people...

I agree 100%. On the other hand, inexpensive gear can sound very good, and I expect the measurements show it.

 

[mep]

I think the ardent measurementalists would like to delude themselves into thinking that an amp for example...that has the same wattage and distortion levels but cost $2,000 will sound a good as another with similar specs costing 5 times or more.

They are already there and think you are delusional because you think you do hear a difference-trust me. We have so many amplifiers of different classes available. Class A (which in Pure Class A form are really much more rare than all other classes), Class A/B (which has been the dominant class of amplifier available since the Jurassic age. Then we have Class G, and now Class D is taking root with the measurement crowd. Leaving out Class D for the here and now, there are different circuit topologies used in all of the above classes of amplifiers and of course different parts and layouts are used in their circuits. Some will have heaping helpings of negative feedback, and some will have no global feedback. Some of these amplifiers will measure more similar than different, yet some people will reasonably say that no one should be surprised if they sound different while objectivists who have never participated in a DBT will swear up and down that if only you would participate in a DBT, you couldn't hear any differences. And so it goes...

 

[amirm]

They are already there and think you are delusional because you think you do hear a difference-trust me. We have so many amplifiers of different classes available. Class A (which in Pure Class A form are really much more rare than all other classes), Class A/B (which has been the dominant class of amplifier available since the Jurassic age. Then we have Class G, and now Class D is taking root with the measurement crowd. Leaving out Class D for the here and now, there are different circuit topologies used in all of the above classes of amplifiers and of course different parts and layouts are used in their circuits. Some will have heaping helpings of negative feedback, and some will have no global feedback. Some of these amplifiers will measure more similar than different, yet some people will reasonably say that no one should be surprised if they sound different while objectivists who have never participated in a DBT will swear up and down that if only you would participate in a DBT, you couldn't hear any differences. And so it goes...

Why don't all the people who believe otherwise, find a few amplifiers that they all agree sound very different from each other, i.e. "any idiot can tell them apart; night and day difference and so on," and test them blind. Show that the blind test agrees with the sighted view and then we have some evidence to fight them with. Remember, if you are going to convince the other camp, you must use their laws of universe (i.e. bias controlled testing) to convince them. You can't use yours (sighted testing), or else, why bother.

Are you willing and able to do this and if not, why not?

Right now we only have one such test in existence: http://www.whatsbestforum.com/showt...ests-*did*-show-amplifiers-to-sound-different. It is a dusty old test. Love to see a fresh one.

 

[mep]

Why don't all the people who believe otherwise, find a few amplifiers that they all agree sound very different from each other, i.e. "any idiot can tell them apart; night and day difference and so on," and test them blind. Show that the blind test agrees with the sighted view and then we have some evidence to fight them with. Remember, if you are going to convince the other camp, you must use their laws of universe (i.e. bias controlled testing) to convince them. You can't use yours (sighted testing), or else, why bother.

Are you willing and able to do this and if not, why not?

Right now we only have one such test in existence: http://www.whatsbestforum.com/showt...ests-*did*-show-amplifiers-to-sound-different. It is a dusty old test. Love to see a fresh one.

From http://www.analogplanet.com/content/im-appalled-way-you-treat-your-readership:

And were I to take and pass your test, I am certain you'd do the same thing again. You'd move the goalposts. Or if a large group took the test and only I got it right, my result would be tossed and I'd be declared a "lucky coin," which is what happened in an amplifier double blind test I took at an AES.

The test was set up specifically for me to fail because someone challenged ME to a double blind test. Someone who claimed that all amplifiers that measure the same sound the same.

Five amplfiiers were chosen (not by me) and used for a test set up by someone else: the person who challenged me to take his test.

I got 5 of 5 identifications correct. Stereophile editor John Atkinson got 4 of five correct. But among the many members of the AES who took the test the results were statistically insignificant. Did that prove the contention that all amplifiers that measure the same sound the same?

Well, actually not because for some reason the test producer chose to include a Crown DC-300 known as an "ear-bleeder" and a VTL 300 a warm, vacuum tube amplifier. These amplifiers sound and measure very differently yet the AES members, most of whom were audio engineers could not distinguish between them!

Now if the test creator's point was to show that I'm a bad listener his test backfired, since I was easily able to hear those two.

Though I passed the test I got no credit and my detractors said "well there weren't enough samples. It wasn't a valid test. You don't understand statistics, etc." Of course I didn't create the test! And had I failed it do you think those same people would have said "There weren't enough samples, it wasn't statistically valid?" Of course not! They'd have said "You failed. All amplifiers do sound the same." That's how it works.

AES members couldn't tell a Crown DC amp from a VTL amp and yet Mikey picked 5 out of 5 amps correctly and JA got 4/5 correct. Did that satisfy anyone? Apparently not. The beatings continue.

 

[Johnny Vinyl]

Why was price even brought into this? Whether something costs $100G's or $1G they're both measurable pieces of equipment.

I listen and read reviews.

 

[Johnny Vinyl]

BTW...I really like reviews that go a little more in depth about build-quality.

 

[dan31]

I follow both camps and let my ears decide in the end. I really appreciate Amir's efforts to enlighten everyone. He has been putting a lot of effort into his posts.

 

[amirm]

From http://www.analogplanet.com/content/im-appalled-way-you-treat-your-readership:
AES members couldn't tell a Crown DC amp from a VTL amp and yet Mikey picked 5 out of 5 amps correctly and JA got 4/5 correct. Did that satisfy anyone? Apparently not. The beatings continue.

I have have been more specific: I don't want to include tube amps and such where we can clearly show load-dependent frequency response variations. Objectivists are actually on record (whether explicitly or not) that there are audible differences. Here is Meyer's often references article/blind tests on amplifiers, The Amp/Speaker Interface: Are Your Loudspeakers Turning Your Amplifiers Into a Tone Control? :

and

So we have this kind of data on amplifiers whose response clearly changes with loudspeakers. What we need is when this is not the case. Take two high-power solid state amplifiers that you all say sound very different and let's test them. And yes, you do need to have enough samples to have statistically valid results. You can predict the toss of a coin 4 or 5 times in a row. Need to have at least 8 to 10 trails for the results to stick.

 

[rockitman]

Why was price even brought into this? Whether something costs $100G's or $1G they're both measurable pieces of equipment..

That's the whole point..why pay $100k vs $7k when the specs closely match ? Because it sounds better...not necessarily measuring better.

 

[esldude]

Why don't all the people who believe otherwise, find a few amplifiers that they all agree sound very different from each other, i.e. "any idiot can tell them apart; night and day difference and so on," and test them blind. Show that the blind test agrees with the sighted view and then we have some evidence to fight them with. Remember, if you are going to convince the other camp, you must use their laws of universe (i.e. bias controlled testing) to convince them. You can't use yours (sighted testing), or else, why bother.

Are you willing and able to do this and if not, why not?

Right now we only have one such test in existence: http://www.whatsbestforum.com/showt...ests-*did*-show-amplifiers-to-sound-different. It is a dusty old test. Love to see a fresh one.

I don't know who decided to declare an Acoustat with that interface a relatively flat nominal 5 ohm load. They were incompetent to do so. I owned such speakers with that interface some 25 years ago. I measured both the impedance and phase of it at one point. The interface was a 200:1 and 50:1 step up xfmr. A crossover split those I am thinking around 750 hz. Then the outputs were combined on the panel. The impedance was rather high below 200 hz dropping to maybe 6 ohms where it jumped up a bit near 700 hz. Then it was almost a capacitor with just a bit of resistive component dropping steadily to something like 1.5 ohms at 20 khz. The latter dependent upon where you set the "Brilliance" pot. It was a good choice to show different amps would differ in their ability to properly drive it.

I had some MC 752 McIntosh SS amps 75 wpc that were fine within power limits. I had a Counterpoint hybrid that would go bonkers with that load. The old Carver Receiver would play them fine with no issues. A Mark Levinson ML9 had plenty of power, but sounded harsh. O-scope showed it on the verge of oscillation at times. Putting a .5 ohm resistor in series fixed that though it effected frequency response. Tube amps worked great within power limits. They usually bumped up the below 150 hz range nearly 2 db, and rolled the treble about the same amount. Which wasn't really a bad thing. So nothing like a nominal flat 5 ohm load.

 

[CGabriel]

Are you willing and able to do this and if not, why not?

Because no one at the polar extremes subscribe to the belief systems of the other. Neither thinks that the opposite side's methods are valid. No sense it getting too worked up about it. It has been that way for as long as I have been an audiophile and nothing is likely to change. And nothing said in this thread or the others is going to change anyone's mind. IMO of course.

I have experienced great sounding systems from both belief camps and I have heard terrible sound from both also. Great sounding systems are created by people talented in component choice and system setup. Some of them use measurements and others don't.

We start with subjectivity because human hearing is and will continue to be subjective until brain scanning technology advances. The vaunted Harmon test began with subjective tests and collecting a body of subjective data and then attempting to correlate one or more measurents with the subjective data. So subjective data lead to objective measurements with an apparently high degree of correlation. At least the data has a strong correlation in that particular room and with the electronics used. Good for Harmon for trying.

There are obvious issues with the testing but that is a consequence of trying to simplify and objectify a component (speaker) when what is actually being tested is a complex system with several dependencies. For instance they attempted to ensure relative volume levels to within .1 db. However in doing so one speaker that is less efficient than another will require more gain to achieve the same volume level. This will drive the amplifier into a different part of the gain curve level and potentially different output distortion or dampening factors. Some the speakers have a wide dispersion patten which would be advantageous in the wide large room used for testing. The ML which is a dipole has well known advantageous in "normal" untreated rooms where the cancellation effects tend to reduce side wall reflections. Would the Ml have performed better a smaller room. Who knows without trying it. But let's be serious - the Harmon guys are very smart. One of the spokesmen said that they setup the room and ran tests with competitive speakers repeatedly (while continuously modifying their speaker) until their speaker successfully tested above the others. Can't argue with the intent and the desire to be competitive. And finally NO ONE listens to mono. We listen to stereo (or multi-channel) systems because they have the potential to create an aural artificial reality that can transport you to various sonic venues. There are many speaker parameters that are nullified by doing a mono test and most of them you can name them so I won't belabor the point.

 

[amirm]

I don't know who decided to declare an Acoustat with that interface a relatively flat nominal 5 ohm load. They were incompetent to do so. I owned such speakers with that interface some 25 years ago. I measured both the impedance and phase of it at one point. The interface was a 200:1 and 50:1 step up xfmr. A crossover split those I am thinking around 750 hz. Then the outputs were combined on the panel. The impedance was rather high below 200 hz dropping to maybe 6 ohms where it jumped up a bit near 700 hz. Then it was almost a capacitor with just a bit of resistive component dropping steadily to something like 1.5 ohms at 20 khz. The latter dependent upon where you set the "Brilliance" pot. It was a good choice to show different amps would differ in their ability to properly drive it.

I had some MC 752 McIntosh SS amps 75 wpc that were fine within power limits. I had a Counterpoint hybrid that would go bonkers with that load. The old Carver Receiver would play them fine with no issues. A Mark Levinson ML9 had plenty of power, but sounded harsh. O-scope showed it on the verge of oscillation at times. Putting a .5 ohm resistor in series fixed that though it effected frequency response. Tube amps worked great within power limits. They usually bumped up the below 150 hz range nearly 2 db, and rolled the treble about the same amount. Which wasn't really a bad thing. So nothing like a nominal flat 5 ohm load.

Very good info. Thanks. BTW, I have that Carver "class T" receiver and I had good luck powering a friend's Maggies well beyond the loudness of his Byrston amp (this is circa 1982-1984). Had a nasty habit of passing through line noise though. Every time the burner on our stove cycled on and off, it produced a loud tick through the loudspeaker . Could not listen to anything while my wife was cooking!

Back to that test, story that Arny tells now is that they actually heard clicking and cracking with that amp and decided to run a double blind test. Gives all of us a bad name if true when someone tries to test a scenario where clear amplifier failures may be occurring. So no, I don't vouch for their knowledge, motivation and abilities in that regard. That they kept the article hidden for 30 years until I found it does not endear them to me either.

 

[amirm]

Because no one at the polar extremes subscribe to the belief systems of the other. Neither thinks that the opposite side's methods are valid. No sense it getting too worked up about it. It has been that way for as long as I have been an audiophile and nothing is likely to change. And nothing said in this thread or the others is going to change anyone's mind. IMO of course.

I have experienced great sounding systems from both belief camps and I have heard terrible sound from both also. Great sounding systems are created by people talented in component choice and system setup. Some of them use measurements and others don't.

We start with subjectivity because human hearing is and will continue to be subjective until brain scanning technology advances. The vaunted Harmon test began with subjective tests and collecting a body of subjective data and then attempting to correlate one or more measurents with the subjective data. So subjective data lead to objective measurements with an apparently high degree of correlation. At least the data has a strong correlation in that particular room and with the electronics used. Good for Harmon for trying.

There are obvious issues with the testing but that is a consequence of trying to simplify and objectify a component (speaker) when what is actually being tested is a complex system with several dependencies. For instance they attempted to ensure relative volume levels to within .1 db. However in doing so one speaker that is less efficient than another will require more gain to achieve the same volume level. This will drive the amplifier into a different part of the gain curve level and potentially different output distortion or dampening factors. Some the speakers have a wide dispersion patten which would be advantageous in the wide large room used for testing. The ML which is a dipole has well known advantageous in "normal" untreated rooms where the cancellation effects tend to reduce side wall reflections. Would the Ml have performed better a smaller room. Who knows without trying it. But let's be serious - the Harmon guys are very smart. One of the spokesmen said that they setup the room and ran tests with competitive speakers repeatedly (while continuously modifying their speaker) until their speaker successfully tested above the others. Can't argue with the intent and the desire to be competitive. And finally NO ONE listens to mono. We listen to stereo (or multi-channel) systems because they have the potential to create an aural artificial reality that can transport you to various sonic venues. There are many speaker parameters that are nullified by doing a mono test and most of them you can name them so I won't belabor the point.

Not much to argue with , other than saying I care . And Harman also performs stereo testing: http://www.whatsbestforum.com/showt...rs-and-Reality&p=325818&viewfull=1#post325818