Great summary Caelin. I tried to transmit the same idea - you did it much better than me.
Poll: Objective Measurements versus Subjective Reviews
- Thread starter PeterA
- Start date
Great summary Caelin. I tried to transmit the same idea - you did it much better than me.
Hey I'm not criticising the effort nor the result. It is just that it should be taken into a proper context and not used as an end all exclusive determination of what's best. I happen to believe in and use every measurement tool available. Every measurement and every piece of test equipment has a set of limitations that must be understood by the engineer. And then there is NO test equipment that combines all the known audio tests together and aggregates or mathematically combines them into a single number or graph that simulate what a human interprets as music. We can perform many different tests such as FR, THD, IMD, DIM, phase and group delay and many others that have relevance. Then who makes the decision about which is more important and does one measurement overshadow the others in importance. Does one measurement obviate the need for any other? These are the areas of testing and research that I would like to see going forward into the future and I applaud anyone or group that is trying.
And then there are things that can be reliably heard that have no obvious correlation with the tests used in audio measurement today. And by heard I do mean blind testing not sighted testing. IMO again.
And then there are things that can be reliably heard that have no obvious correlation with the tests used in audio measurement today. And by heard I do mean blind testing not sighted testing. IMO again.
I have have been more specific: I don't want to include tube amps and such where we can clearly show load-dependent frequency response variations. Objectivists are actually on record (whether explicitly or not) that there are audible differences. Here is Meyer's often references article/blind tests on amplifiers, The Amp/Speaker Interface: Are Your Loudspeakers Turning Your Amplifiers Into a Tone Control? :
and
So we have this kind of data on amplifiers whose response clearly changes with loudspeakers. What we need is when this is not the case. Take two high-power solid state amplifiers that you all say sound very different and let's test them. And yes, you do need to have enough samples to have statistically valid results. You can predict the toss of a coin 4 or 5 times in a row. Need to have at least 8 to 10 trails for the results to stick.
The point that you either missed or glossed over is that AES members couldn't tell the difference between a SS amp and tube amp under DBTconditions. Does that not say something? Does anybody really think a Crown DC amp and a VTL sounded identical?? Really?
So you don't want to include tube amps against SS amps and yet the AES brethren couldn't tell them apart under DBT conditions and now you want to drill down to SS against SS.
Actually most dual mono stereo amps, as well as pairs of monoblocks, do sound and meaure, "the same" in the sense that differences are well below any threshold of audibility. That may not be true for every single amp out there, but in cases where Stereophile has found significant channel discrepancies (usually both audible and measurable) it's almost always due to a manufacturing defect and correctable.
Without knowing the test itself, it says very little. From the bit I read, they were testing the setup in a large room with hundreds of people in attendance??? I would need something more than that to know if it was or was not revealing.
Besides, I have shared parallel results for loudspeaker testings where reviewers couldn't remotely match the detection ability of trained listeners:
What was your walk-away position on this?
Don't care what people "think." That doesn't resolve the disagreement. Data does. If you think they are that different, how about doing the test I suggested and showing how you can tell them apart in your sleep?
I don't want to test what is already under agreement. That is, both sides agree that tube sounds can sound different. Want to test a scenario where there is not supposed to be a difference according to one camp, but there is night and day difference by the other. The AES test per above, is not documented well enough, nor had enough trials to be usable other than an anecdote.
Are you concerned you can't find two solid state amplifiers that show this kind of difference?
Caelin-I agree with what you are saying because it's the truth as I know it. The flip side of that is people who claim to have extreme technical competence and want to say ad nauseam that everything we hear can be measured and therefore we know everything which is complete nonsense.
No poll is perfect Amir.
You should vote for the first choice: "I rely mainly on measurements:" That choice does not exclude other influences. As long as you rely at least 51% on measurements and 49% or less on other influences, that is the correct choice for you. The next three are nuanced versions of the same preference for listening. The last choice is for those who are mainly influenced by the reviews they read either by choice or because they can not actually hear a piece of gear before buying it. This is not that unusual, think tweaks, or cartridges, or tonearms etc.
You do not have to vote in this poll, but your vote might dramatically alter the total number of people who vote for the first choice.
As you discovered in the Analog vs Digital poll, that poll question was also flawed. That "simple poll" was confusing because some people thought it asked about personal preference, while others thought it asked about absolute accuracy, while others thought it asked which was listened to more often, while others thought something else. And what about all of those people who listen to both formats the same amounts, or different amounts? Or different budgets favor different formats. Etc, etc. That poll generated a fair bit of discussion precisely because it was not just a simple poll.
That Analog/Digital poll showed a 2:1 majority in favor of analog. This poll will also add a data point for us to consider. The result may or may not be very surprising.
Better clarify what is position 1 and what is position 10, because it's not at all clear from this post
I know many of the foremost designers in audio working today. Virtually all of them use measurments in the development of products, of course. It would be ridiculous to not do so. However, many of them privately and some of them publically will acknowledge that we don't know everything and that there are certain design decisions that influence the end result that cannot be measured. At least not currently.
I am not aligned with an ideology of subjective or of objective. I am results orientated and choose to use whatever method is available to advanced the sound quality by whatever criteria. I agree with Amir in that the so called subjectivist need to at least be open to measurements that show a strong correlation to what is subjectively heard. It is also true that more manufacturers need to demonstrate that claimed differences can be means tested even it can't be measured successfully. If that means that we do more blind tests then let's do it. Then the so called objectivists need to relax their intractable position that ONLY measurements and DBT/ABX are valid. I believe that most of us are somewhere in the middle ground on this issue.
i would like to see us move beyond the finger pointing, accusations and innuendo that do nothing but obscure the underlying issues.
But I don't. I rely on examining the design of a system, reading research papers and white papers from company, discussing the technology with others in the industry, measurements, technical reviews, and listening tests. How does that summarize into "I rely mainly on measurements?" You are pigeonholing us due to your view of us, not reality. If you had also said, "I rely mainly on listening" as your other alternative, I could see it not be being biased but you gave so many shades of gray to that, but not this one.
Of course it does. Why else did you include all the other considerations for the listening crowd?
This would have eliminated all the other options in the "I rely on listening." You are giving them opportunity to describe the totality of their decision making process but not for people who come from engineering and science point of view.
Don't know what this means. You gave this choice: "I rely mainly on listening but also read measurements." I am asking why the reverse is not there: "I rely mainly on measurements but also use listening tests."
I don't plan on voting the way you have written it. You have just seen my process of evaluating loudspeakers. I have post extensively on results of listening tests, design approaches, and measurements but only to the extend they correlate with listening tests results. How would someone reading that first choice by me think this is what I do to evaluate a product???
Sure, there are always critics of polls. But nothing like what is here. You have given 4 choices to one group, and one (1) to the other. When I created the other test, I didn't tilt things this way. It was a binary choice. Not four for one camp to fine grain their point of view, and a single one to stuff the other camp in.
Confusion is one thing: creating a poll which sets out in advance to position the results in a certain way is another. The lack of clarity if there, was not meant, and did not result in the outcome favoring me. Whereas in this test, your poll as written is designed to say, "look at these guys who only look at measurements" when in reality none of us are in that pure camp. We value measurements far more than the other camp. That is true. But you can't draw conclusions from that and create a poll based on your preconceptions.
Doesn't matter. It was a binary choice. It did not favor one camp over the other by giving them four ways to vote, while one for the other.
You can consider it but you should also consider how to create polls where your intentions don't attempt to foreclose choices this clearly. You could have made this binary and I would have been fine with. I am not looking for perfection in the wording of the poll. I am looking for at least making a reasonable attempt at it being unbiased.
But sure, if the results are useful to you without me and others voting, all the power to you
.I don't care if there were a 1,000 people in the room. MF got 5/5 right and JA got 4/5 right. If it was revealing enough for them, why not AES members?
Besides, I have shared parallel results for loudspeaker testings where reviewers couldn't remotely match the detection ability of trained listeners:
What was your walk-away position on this?
That I don't have enough information about how the test was conducted and what speakers were chosen and why this test was graded so different than other Harman tests which usually result in the Harmanites grading speakers way lower than the other test groups and now the Harmanites are smarter because they scored the results higher in this test.
I don't care either Amir, but I care that MF and JA could hear things and make the correct choices in such a manner that eluded everyone else who took the test. If AES members couldn't tell the difference between a Crown DC amp and VTL tube amp, why do we need to worry about picking apart SS vs. SS? My point is that challenge tests like you are proposing have already been accomplished and they showed that reviewers can hear what other educated people couldn't hear. Why do we need to reprove it or try and disprove what has already been proven? Why don't we argue that planes can't really fly and redo the Wright Brothers' experiments?
No. I'm concerned you keep wanting to reinvent the wheel and prove it's not really round. Are you really postulating that all SS amps sound alike or are you postulating that under DBT conditions all SS amps sound the same?? We have already shown that under DBT conditions that some people can't tell a tube amp from a SS amp so where does that lead us?
You remind me of the Energizer Bunny. You just keep going, going, and going until you wear people down with all of your charts, graphs, and arguments until they finally give up and cry "Uncle." Standing in the ring with you is mighty tough because your corner greased you up with an entire case of vaseline and nothing sticks to you. Like a boxer who has over taped his hands, you aren't against putting words into peoples' mouths they never spoke in order to make a new argument. And meanwhile you do have your cheerleader section 'filled' with those who sing the praises of DBTs even though they have never participated in them, and claim they love measurements even though we both know they really don't have any. If you want all reviewers to leave your forum because we are not worthy because all we write is flowery prose, we aren't brow beating OEMs to provide measurements, we aren't willing to spend $60k of our own money so we could take measurements, and we aren't making everyone who is interested in owning a stereo system understand and care how important measurements are, just say so.
Great post. +1
People who argue on forums may be in it to win, but at the end of the day the winner can actually be the loser. It's up to those many lurkers to determine that. They don't actively participate, but they are the ones who really determine the winners, and it is they who decide who is the good guy, who is to be respected, who is smart, who is an idiot, who is a charlatan, and who is a pompous ass. I always try to keep that in the back of my mind.
Mark, you are unfairly generalizing. While it's true that amir is persistent, it's also true that he has posted much data useful even to those of us who don't believe in ABX or other DBT's for audio, nor that "everything we hear can be measured". Just read many of the posts in this and other threads over the past couple of weeks.
I dont think using a whole slew of research and graphs to bludgeon one into a subjective preference is at all valid.. Any subjective preference based research really means nothing to me.. its MY preference that counts.
Where are their results so that I can see they did not get that kind of outcome? But sure, there are many people who lack critical listening skills much like many audiophiles. And don't think because someone goes to AES, they are supposed to be experts in anything much less critical listening. There is for example an exhibition area where manufacturers send countless people to set up booths and such. All of them can and do attend conference sessions. Does it matter if they can or cannot hear a difference?
This is why we do formal tests. We know the characteristics of listeners. We know the test setup. We can examine all of this to see if the results should have been revealing. Without it we are spitting in the wind.
I know more than you, having had JA in attendance in the discussion. Here is what John has to say:
"It was neither a published paper nor a conference paper but the report of the conference. But that is not the point. Though he was not present at the 1988 AES Convention and was not involved in the test, Arnyk claims to know that the test was _not_ intended to be a test. Yet it was described as a test in the Convention prgram, the workshop on the final day of the Convention, where the results were presented, described it as a test, and the report on the Convention in the JAES described it as a test. Occam's Razor would suggest rejecting the outlier (Arnyk's wording) and accepting that the organizer intended it as a test.
Regarding the choice of just 5 trials for each listener, I have explained why that was the case and criticized it at that time for the same reasons that have been outlined in this thread. As I explained, scoring 5/5 identifications does not reach the 95% confidence level and I was saddened that it was not possible for either Michael Fremer or myself to take part in another set of 5 trials. (Note, BTW, that I had no connection with organizing the test and was involved purely as a listener, as a visitor to the Convention, which, as a member of the AES, I routinely attend.)
John Atkinson
Editor, Stereophile"
But none of this explains the full extent of the test, nor the results of how others did.
Again, we don't know if it eluded everyone else. And what it was that eluded them.
Because a forum post or two is not enough to make an argument that sticks. Heck, I have been quoting published, peer reviewed paper after paper, test after test, and you have refused to accept them. But in return, you want the other camp to care about a form post?
Nope. You like to think that but it is not a test that I proposed. The test I proposed will have details that we lack here. It will have the benefit of repeatability and it would be current.
Because to say we have proven something, means we don't even understand what it is that has been accomplished. As you see from JA, he also agrees that statistical significance cannot be attached to the results. Yet you think it was "proven?" As I said, you can predict a coin toss 5 out of 5 times. Doing that 10 times in a row gets hard though. That is why we need minimum number of trials. That is, if you want to convince someone who values this type of rigor. If you just want to convince yourself, then sure, why even bother with that test?
I am saying that you all make it like it is so easy to tell amplifiers apart by ear. So it should be easy work to show that blind. If it is not possible to show it blind, then it reasons that the differences are not as big as you think they are. Or possibly non-existent. Either way there is learning in it.
No "DBT" test is in front of us. All we have are forum posts. The other camp will feel some heat from it, hence the reason I brought it up when JA commented on it to support me
, but that is it. I am tenacious. Not going to apologize for that any more than you can Mark for posting repeatedly just the same. The entire forum is available to you as I have mentioned to express your subjective reviews without me there or anyone else challenging you for that matter. You are choosing instead to focus your posts and energy in these couple of debate threads. And then go on to complain??? Present your best case if you want to debate. Don't ask us for sympathy. If your argument is weak, you are going to be pushed against the wall. Simply because I know all of your arguments. As I mentioned, I have used the JA/Fremer story myself. So don't think you can throw this stuff at me and sit back and expect satisfaction. It doesn't work that way. You need to be accountable for your arguments or choose as many members have, to post elsewhere.
And remember, you are in a forum that a) has twice as many analog lovers than digital and b) provides best protection of any forum in support of cordial discussions, your commentary here notwithstanding.
Similar threads
