Poll: Objective Measurements versus Subjective Reviews

[microstrip]

Because no one at the polar extremes subscribe to the belief systems of the other. Neither thinks that the opposite side's methods are valid. No sense it getting too worked up about it. It has been that way for as long as I have been an audiophile and nothing is likely to change. And nothing said in this thread or the others is going to change anyone's mind. IMO of course.

I have experienced great sounding systems from both belief camps and I have heard terrible sound from both also. Great sounding systems are created by people talented in component choice and system setup. Some of them use measurements and others don't.

We start with subjectivity because human hearing is and will continue to be subjective until brain scanning technology advances. The vaunted Harmon test began with subjective tests and collecting a body of subjective data and then attempting to correlate one or more measurents with the subjective data. So subjective data lead to objective measurements with an apparently high degree of correlation. At least the data has a strong correlation in that particular room and with the electronics used. Good for Harmon for trying.

There are obvious issues with the testing but that is a consequence of trying to simplify and objectify a component (speaker) when what is actually being tested is a complex system with several dependencies. For instance they attempted to ensure relative volume levels to within .1 db. However in doing so one speaker that is less efficient than another will require more gain to achieve the same volume level. This will drive the amplifier into a different part of the gain curve level and potentially different output distortion or dampening factors. Some the speakers have a wide dispersion patten which would be advantageous in the wide large room used for testing. The ML which is a dipole has well known advantageous in "normal" untreated rooms where the cancellation effects tend to reduce side wall reflections. Would the Ml have performed better a smaller room. Who knows without trying it. But let's be serious - the Harmon guys are very smart. One of the spokesmen said that they setup the room and ran tests with competitive speakers repeatedly (while continuously modifying their speaker) until their speaker successfully tested above the others. Can't argue with the intent and the desire to be competitive. And finally NO ONE listens to mono. We listen to stereo (or multi-channel) systems because they have the potential to create an aural artificial reality that can transport you to various sonic venues. There are many speaker parameters that are nullified by doing a mono test and most of them you can name them so I won't belabor the point.

Great summary Caelin. I tried to transmit the same idea - you did it much better than me.

 

[Phelonious Ponk]

F: None of the above.

Tim

 

[esldude]

F: None of the above.

Tim

+1

None of the choices actually fit well enough for me.

 

[CGabriel]

Hey I'm not criticising the effort nor the result. It is just that it should be taken into a proper context and not used as an end all exclusive determination of what's best. I happen to believe in and use every measurement tool available. Every measurement and every piece of test equipment has a set of limitations that must be understood by the engineer. And then there is NO test equipment that combines all the known audio tests together and aggregates or mathematically combines them into a single number or graph that simulate what a human interprets as music. We can perform many different tests such as FR, THD, IMD, DIM, phase and group delay and many others that have relevance. Then who makes the decision about which is more important and does one measurement overshadow the others in importance. Does one measurement obviate the need for any other? These are the areas of testing and research that I would like to see going forward into the future and I applaud anyone or group that is trying.

And then there are things that can be reliably heard that have no obvious correlation with the tests used in audio measurement today. And by heard I do mean blind testing not sighted testing. IMO again.

 

[mep]

I have have been more specific: I don't want to include tube amps and such where we can clearly show load-dependent frequency response variations. Objectivists are actually on record (whether explicitly or not) that there are audible differences. Here is Meyer's often references article/blind tests on amplifiers, The Amp/Speaker Interface: Are Your Loudspeakers Turning Your Amplifiers Into a Tone Control? :

and

So we have this kind of data on amplifiers whose response clearly changes with loudspeakers. What we need is when this is not the case. Take two high-power solid state amplifiers that you all say sound very different and let's test them. And yes, you do need to have enough samples to have statistically valid results. You can predict the toss of a coin 4 or 5 times in a row. Need to have at least 8 to 10 trails for the results to stick.

The point that you either missed or glossed over is that AES members couldn't tell the difference between a SS amp and tube amp under DBTconditions. Does that not say something? Does anybody really think a Crown DC amp and a VTL sounded identical?? Really?

I don't want to include tube amps and such where we can clearly show load-dependent frequency response variations.

So you don't want to include tube amps against SS amps and yet the AES brethren couldn't tell them apart under DBT conditions and now you want to drill down to SS against SS.

 

[rbbert]

Can you guys tell your left mono amp from the right? They do not sound the same, really they don't. Measurements can prove it easily.
bb
I don't think that all amps sound the same. But so what, the question is always, when someone says their amp sounds the best....that's when the chains start rattling....

Actually most dual mono stereo amps, as well as pairs of monoblocks, do sound and meaure, "the same" in the sense that differences are well below any threshold of audibility. That may not be true for every single amp out there, but in cases where Stereophile has found significant channel discrepancies (usually both audible and measurable) it's almost always due to a manufacturing defect and correctable.

 

[mep]

Actually most dual mono stereo amps, as well as pairs of monoblocks, do sound and meaure, "the same" in the sense that differences are well below any threshold of audibility. That may not be true for every single amp out there, but in cases where Stereophile has found significant channel discrepancies (usually both audible and measurable) it's almost always due to a manufacturing defect and correctable.

Don't let any facts get in the way of his reality.

 

[rbbert]

+1

None of the choices actually fit well enough for me.

So given a free-form choice, how would you phrase yours?

 

[amirm]

The point that you either missed or glossed over is that AES members couldn't tell the difference between a SS amp and tube amp under DBTconditions. Does that not say something?

Without knowing the test itself, it says very little. From the bit I read, they were testing the setup in a large room with hundreds of people in attendance??? I would need something more than that to know if it was or was not revealing.

Besides, I have shared parallel results for loudspeaker testings where reviewers couldn't remotely match the detection ability of trained listeners:

What was your walk-away position on this?

Does anybody really think a Crown DC amp and a VTL sounded identical?? Really?

Don't care what people "think." That doesn't resolve the disagreement. Data does. If you think they are that different, how about doing the test I suggested and showing how you can tell them apart in your sleep?

So you don't want to include tube amps against SS amps and yet the AES brethren couldn't tell them apart under DBT conditions and now you want to drill down to SS against SS.

I don't want to test what is already under agreement. That is, both sides agree that tube sounds can sound different. Want to test a scenario where there is not supposed to be a difference according to one camp, but there is night and day difference by the other. The AES test per above, is not documented well enough, nor had enough trials to be usable other than an anecdote.

Are you concerned you can't find two solid state amplifiers that show this kind of difference?

 

[mep]

Hey I'm not criticising the effort nor the result. It is just that it should be taken into a proper context and not used as an end all exclusive determination of what's best. I happen to believe in and use every measurement tool available. Every measurement and every piece of test equipment has a set of limitations that must be understood by the engineer. And then there is NO test equipment that combines all the known audio tests together and aggregates or mathematically combines them into a single number or graph that simulate what a human interprets as music. We can perform many different tests such as FR, THD, IMD, DIM, phase and group delay and many others that have relevance. Then who makes the decision about which is more important and does one measurement overshadow the others in importance. Does one measurement obviate the need for any other? These are the areas of testing and research that I would like to see going forward into the future and I applaud anyone or group that is trying.

And then there are things that can be reliably heard that have no obvious correlation with the tests used in audio measurement today. And by heard I do mean blind testing not sighted testing. IMO again.

Caelin-I agree with what you are saying because it's the truth as I know it. The flip side of that is people who claim to have extreme technical competence and want to say ad nauseam that everything we hear can be measured and therefore we know everything which is complete nonsense.

 

[PeterA]

How do I vote if I highly value measurements but also do listening? You have all variety of "I rely mainly on listening..." but not the other way around.

No poll is perfect Amir.

You should vote for the first choice: "I rely mainly on measurements:" That choice does not exclude other influences. As long as you rely at least 51% on measurements and 49% or less on other influences, that is the correct choice for you. The next three are nuanced versions of the same preference for listening. The last choice is for those who are mainly influenced by the reviews they read either by choice or because they can not actually hear a piece of gear before buying it. This is not that unusual, think tweaks, or cartridges, or tonearms etc.

You do not have to vote in this poll, but your vote might dramatically alter the total number of people who vote for the first choice.

As you discovered in the Analog vs Digital poll, that poll question was also flawed. That "simple poll" was confusing because some people thought it asked about personal preference, while others thought it asked about absolute accuracy, while others thought it asked which was listened to more often, while others thought something else. And what about all of those people who listen to both formats the same amounts, or different amounts? Or different budgets favor different formats. Etc, etc. That poll generated a fair bit of discussion precisely because it was not just a simple poll.

That Analog/Digital poll showed a 2:1 majority in favor of analog. This poll will also add a data point for us to consider. The result may or may not be very surprising.

 

[amirm]

Caelin-I agree with what you are saying because it's the truth as I know it. The flip side of that is people who claim to have extreme technical competence and want to say ad nauseam that everything we hear can be measured and therefore we know everything which is complete nonsense.

On scale 1 to 10, with 10 being that position, what is yours Mark?

 

[rbbert]

On scale 1 to 10, with 10 being that position, what is yours Mark?

Better clarify what is position 1 and what is position 10, because it's not at all clear from this post

 

[CGabriel]

Caelin-I agree with what you are saying because it's the truth as I know it. The flip side of that is people who claim to have extreme technical competence and want to say ad nauseam that everything we hear can be measured and therefore we know everything which is complete nonsense.

I know many of the foremost designers in audio working today. Virtually all of them use measurments in the development of products, of course. It would be ridiculous to not do so. However, many of them privately and some of them publically will acknowledge that we don't know everything and that there are certain design decisions that influence the end result that cannot be measured. At least not currently.

I am not aligned with an ideology of subjective or of objective. I am results orientated and choose to use whatever method is available to advanced the sound quality by whatever criteria. I agree with Amir in that the so called subjectivist need to at least be open to measurements that show a strong correlation to what is subjectively heard. It is also true that more manufacturers need to demonstrate that claimed differences can be means tested even it can't be measured successfully. If that means that we do more blind tests then let's do it. Then the so called objectivists need to relax their intractable position that ONLY measurements and DBT/ABX are valid. I believe that most of us are somewhere in the middle ground on this issue.

i would like to see us move beyond the finger pointing, accusations and innuendo that do nothing but obscure the underlying issues.

 

[amirm]

No poll is perfect Amir. You should vote on the first one: "I rely mainly on measurements:"

But I don't. I rely on examining the design of a system, reading research papers and white papers from company, discussing the technology with others in the industry, measurements, technical reviews, and listening tests. How does that summarize into "I rely mainly on measurements?" You are pigeonholing us due to your view of us, not reality. If you had also said, "I rely mainly on listening" as your other alternative, I could see it not be being biased but you gave so many shades of gray to that, but not this one.

That choice does not exclude other influences.

Of course it does. Why else did you include all the other considerations for the listening crowd?

As long as you rely at least 51% on measurements and 49% or less on other influences, that is the correct choice for you.

This would have eliminated all the other options in the "I rely on listening." You are giving them opportunity to describe the totality of their decision making process but not for people who come from engineering and science point of view.

The next three are nuanced versions of the same preference for listening.

Don't know what this means. You gave this choice: "I rely mainly on listening but also read measurements." I am asking why the reverse is not there: "I rely mainly on measurements but also use listening tests."

You do not have to vote in this poll, but your vote might dramatically alter the total number of people who vote for the first choice.

I don't plan on voting the way you have written it. You have just seen my process of evaluating loudspeakers. I have post extensively on results of listening tests, design approaches, and measurements but only to the extend they correlate with listening tests results. How would someone reading that first choice by me think this is what I do to evaluate a product???

As you discovered in the Analog vs Digital poll, that poll question was also flawed.

Sure, there are always critics of polls. But nothing like what is here. You have given 4 choices to one group, and one (1) to the other. When I created the other test, I didn't tilt things this way. It was a binary choice. Not four for one camp to fine grain their point of view, and a single one to stuff the other camp in.

That "simple poll" was confusing because some people thought it asked about personal preference, while others thought it asked about absolute accuracy, while others thought it asked which was listened to more often, while others thought something else.

Confusion is one thing: creating a poll which sets out in advance to position the results in a certain way is another. The lack of clarity if there, was not meant, and did not result in the outcome favoring me. Whereas in this test, your poll as written is designed to say, "look at these guys who only look at measurements" when in reality none of us are in that pure camp. We value measurements far more than the other camp. That is true. But you can't draw conclusions from that and create a poll based on your preconceptions.

And what about all of those people who listen to both formats the same amounts, or different amounts? Or different budgets favor different formats. Etc, etc. That poll generated a fair bit of discussion precisely because it was not just a simple poll.

Doesn't matter. It was a binary choice. It did not favor one camp over the other by giving them four ways to vote, while one for the other.

That Analog/Digital poll showed a 2:1 majority in favor of analog. This poll will also add a data point for us to consider. The result may or may not be very surprising.

You can consider it but you should also consider how to create polls where your intentions don't attempt to foreclose choices this clearly. You could have made this binary and I would have been fine with. I am not looking for perfection in the wording of the poll. I am looking for at least making a reasonable attempt at it being unbiased.

But sure, if the results are useful to you without me and others voting, all the power to you .

 

[mep]

Without knowing the test itself, it says very little. From the bit I read, they were testing the setup in a large room with hundreds of people in attendance??? I would need something more than that to know if it was or was not revealing.

I don't care if there were a 1,000 people in the room. MF got 5/5 right and JA got 4/5 right. If it was revealing enough for them, why not AES members?

Besides, I have shared parallel results for loudspeaker testings where reviewers couldn't remotely match the detection ability of trained listeners:

What was your walk-away position on this?

That I don't have enough information about how the test was conducted and what speakers were chosen and why this test was graded so different than other Harman tests which usually result in the Harmanites grading speakers way lower than the other test groups and now the Harmanites are smarter because they scored the results higher in this test.

Don't care what people "think."That doesn't resolve the disagreement. Data does. If you think they are that different, how about doing the test I suggested and showing how you can tell them apart in your sleep?


I don't want to test what is already under agreement. That is, both sides agree that tube sounds can sound different. Want to test a scenario where there is not supposed to be a difference according to one camp, but there is night and day difference by the other. The AES test per above, is not documented well enough, nor had enough trials to be usable other than an anecdote.

I don't care either Amir, but I care that MF and JA could hear things and make the correct choices in such a manner that eluded everyone else who took the test. If AES members couldn't tell the difference between a Crown DC amp and VTL tube amp, why do we need to worry about picking apart SS vs. SS? My point is that challenge tests like you are proposing have already been accomplished and they showed that reviewers can hear what other educated people couldn't hear. Why do we need to reprove it or try and disprove what has already been proven? Why don't we argue that planes can't really fly and redo the Wright Brothers' experiments?

Are you concerned you can't find two solid state amplifiers that show this kind of difference?

No. I'm concerned you keep wanting to reinvent the wheel and prove it's not really round. Are you really postulating that all SS amps sound alike or are you postulating that under DBT conditions all SS amps sound the same?? We have already shown that under DBT conditions that some people can't tell a tube amp from a SS amp so where does that lead us?

You remind me of the Energizer Bunny. You just keep going, going, and going until you wear people down with all of your charts, graphs, and arguments until they finally give up and cry "Uncle." Standing in the ring with you is mighty tough because your corner greased you up with an entire case of vaseline and nothing sticks to you. Like a boxer who has over taped his hands, you aren't against putting words into peoples' mouths they never spoke in order to make a new argument. And meanwhile you do have your cheerleader section 'filled' with those who sing the praises of DBTs even though they have never participated in them, and claim they love measurements even though we both know they really don't have any. If you want all reviewers to leave your forum because we are not worthy because all we write is flowery prose, we aren't brow beating OEMs to provide measurements, we aren't willing to spend $60k of our own money so we could take measurements, and we aren't making everyone who is interested in owning a stereo system understand and care how important measurements are, just say so.

 

[Mosin]

I know many of the foremost designers in audio working today. Virtually all of them use measurments in the development of products, of course. It would be ridiculous to not do so. However, many of them privately and some of them publically will acknowledge that we don't know everything and that there are certain design decisions that influence the end result that cannot be measured. At least not currently.

I am not aligned with an ideology of subjective or of objective. I am results orientated and choose to use whatever method is available to advanced the sound quality by whatever criteria. I agree with Amir in that the so called subjectivist need to at least be open to measurements that show a strong correlation to what is subjectively heard. It is also true that more manufacturers need to demonstrate that claimed differences can be means tested even it can't be measured successfully. If that means that we do more blind tests then let's do it. Then the so called objectivists need to relax their intractable position that ONLY measurements and DBT/ABX are valid. I believe that most of us are somewhere in the middle ground on this issue.

i would like to see us move beyond the finger pointing, accusations and innuendo that do nothing but obscure the underlying issues.

Great post. +1

People who argue on forums may be in it to win, but at the end of the day the winner can actually be the loser. It's up to those many lurkers to determine that. They don't actively participate, but they are the ones who really determine the winners, and it is they who decide who is the good guy, who is to be respected, who is smart, who is an idiot, who is a charlatan, and who is a pompous ass. I always try to keep that in the back of my mind.

 

[rbbert]

Mark, you are unfairly generalizing. While it's true that amir is persistent, it's also true that he has posted much data useful even to those of us who don't believe in ABX or other DBT's for audio, nor that "everything we hear can be measured". Just read many of the posts in this and other threads over the past couple of weeks.

 

[Rodney Gold]

I dont think using a whole slew of research and graphs to bludgeon one into a subjective preference is at all valid.. Any subjective preference based research really means nothing to me.. its MY preference that counts.

 

[amirm]

I don't care if there were a 1,000 people in the room. MF got 5/5 right and JA got 4/5 right. If it was revealing enough for them, why not AES members?

Where are their results so that I can see they did not get that kind of outcome? But sure, there are many people who lack critical listening skills much like many audiophiles. And don't think because someone goes to AES, they are supposed to be experts in anything much less critical listening. There is for example an exhibition area where manufacturers send countless people to set up booths and such. All of them can and do attend conference sessions. Does it matter if they can or cannot hear a difference?

This is why we do formal tests. We know the characteristics of listeners. We know the test setup. We can examine all of this to see if the results should have been revealing. Without it we are spitting in the wind.

That I don't have enough information about how the test was conducted and what speakers were chosen and why this test was graded so different than other Harman tests which usually result in the Harmanites grading speakers way lower than the other test groups and now the Harmanites are smarter because they scored the results higher in this test.

I know more than you, having had JA in attendance in the discussion. Here is what John has to say:

"It was neither a published paper nor a conference paper but the report of the conference. But that is not the point. Though he was not present at the 1988 AES Convention and was not involved in the test, Arnyk claims to know that the test was _not_ intended to be a test. Yet it was described as a test in the Convention prgram, the workshop on the final day of the Convention, where the results were presented, described it as a test, and the report on the Convention in the JAES described it as a test. Occam's Razor would suggest rejecting the outlier (Arnyk's wording) and accepting that the organizer intended it as a test.

Regarding the choice of just 5 trials for each listener, I have explained why that was the case and criticized it at that time for the same reasons that have been outlined in this thread. As I explained, scoring 5/5 identifications does not reach the 95% confidence level and I was saddened that it was not possible for either Michael Fremer or myself to take part in another set of 5 trials. (Note, BTW, that I had no connection with organizing the test and was involved purely as a listener, as a visitor to the Convention, which, as a member of the AES, I routinely attend.)

John Atkinson
Editor, Stereophile"

But none of this explains the full extent of the test, nor the results of how others did.

I don't care either Amir, but I care that MF and JA could hear things and make the correct choices in such a manner that eluded everyone else who took the test.

Again, we don't know if it eluded everyone else. And what it was that eluded them.

If AES members couldn't tell the difference between a Crown DC amp and VTL tube amp, why do we need to worry about picking apart SS vs. SS?

Because a forum post or two is not enough to make an argument that sticks. Heck, I have been quoting published, peer reviewed paper after paper, test after test, and you have refused to accept them. But in return, you want the other camp to care about a form post?

My point is that challenge tests like you are proposing have already been accomplished and they showed that reviewers can hear what other educated people couldn't hear.

Nope. You like to think that but it is not a test that I proposed. The test I proposed will have details that we lack here. It will have the benefit of repeatability and it would be current.

Why do we need to reprove it or try and disprove what has already been proven? Why don't we argue that planes can't really fly and redo the Wright Brothers' experiments?

Because to say we have proven something, means we don't even understand what it is that has been accomplished. As you see from JA, he also agrees that statistical significance cannot be attached to the results. Yet you think it was "proven?" As I said, you can predict a coin toss 5 out of 5 times. Doing that 10 times in a row gets hard though. That is why we need minimum number of trials. That is, if you want to convince someone who values this type of rigor. If you just want to convince yourself, then sure, why even bother with that test?

No. I'm concerned you keep wanting to reinvent the wheel and prove it's not really round. Are you really postulating that all SS amps sound alike or are you postulating that under DBT conditions all SS amps sound the same?? We have already shown that under DBT conditions that some people can't tell a tube amp from a SS amp so where does that lead us?

I am saying that you all make it like it is so easy to tell amplifiers apart by ear. So it should be easy work to show that blind. If it is not possible to show it blind, then it reasons that the differences are not as big as you think they are. Or possibly non-existent. Either way there is learning in it.

No "DBT" test is in front of us. All we have are forum posts. The other camp will feel some heat from it, hence the reason I brought it up when JA commented on it to support me , but that is it.

You remind me of the Energizer Bunny. You just keep going, going, and going until you wear people down with all of your charts, graphs, and arguments until they finally give up and cry "Uncle." Standing in the ring with you is mighty tough because your corner greased you up with an entire case of vaseline and nothing sticks to you. Like a boxer who has over taped his hands, you aren't against putting words into peoples' mouths they never spoke in order to make a new argument. And meanwhile you do have your cheerleader section 'filled' with those who sing the praises of DBTs even though they have never participated in them, and claim they love measurements even though we both know they really don't have any. If you want all reviewers to leave your forum because we are not worthy because all we write is flowery prose, we aren't brow beating OEMs to provide measurements, we aren't willing to spend $60k of our own money so we could take measurements, and we aren't making everyone who is interested in owning a stereo system understand and care how important measurements are, just say so.

I am tenacious. Not going to apologize for that any more than you can Mark for posting repeatedly just the same. The entire forum is available to you as I have mentioned to express your subjective reviews without me there or anyone else challenging you for that matter. You are choosing instead to focus your posts and energy in these couple of debate threads. And then go on to complain??? Present your best case if you want to debate. Don't ask us for sympathy. If your argument is weak, you are going to be pushed against the wall. Simply because I know all of your arguments. As I mentioned, I have used the JA/Fremer story myself. So don't think you can throw this stuff at me and sit back and expect satisfaction. It doesn't work that way. You need to be accountable for your arguments or choose as many members have, to post elsewhere.

And remember, you are in a forum that a) has twice as many analog lovers than digital and b) provides best protection of any forum in support of cordial discussions, your commentary here notwithstanding.