Is ABX finally Obsolete

[microstrip]

(...) I just think evaluating components blind - even if it is casual, with no pretense of statistical validity - is a couple of huge steps ahead of sighted listening, which is just a bad way to compare two pieces of gear, period.

Tim

I see I was loosing my time researching on ABX and DBT.

Anyway, if the evaluation is casual and has no pretense of statistical validity what can you conclude about it?
That you were influenced by the order in which you heard the components, your mood, the music you were listening or something else?

 

[terryj]

I am not going down this roadd. It was an insult and you know it. How could you possibly know what "research" I do on the subject? Even if I had guys like you always throwing it in my face. Citing papers that support my position? How dare I? Proponents of ABX never do that.

I'll decline to participate in ABX. But you know if Harmon International wanted to hire me to participate in thier product evalautions, I'll be happy to oiblige.

Trawl was an insult? Well then I DO apologise. I honestly meant no more than 'take the results that you wish', guess I could have called it cherry pick, but right now am at a loss to know what way to call it that would not be an insult.

I do know some of the research you are willing to do on the subject...you would refuse to partake in a properly conducted blind test. I know THAT much.

What better way to understand a subject than to experience a subject? You might find after all that a lot of these 'what if' questions you ask have been answered...by participating.

Till then, simply because you refuse to accept the results you are reduced to carping on the sideline. (to the mods...is that a personal attack?? I hope not, it is not meant personally but more an observation on 'methodology' if you will)

Ok, YOU want this (hmm, what were her words without subjecting myself to reading it again?) 'grotesque scientific parody known as ABX testing' ( that will do) to be tightened up? Good, stop complaining and help tighten it up. By participating, and making positive suggestions on how to improve it.

And one last point you did not address, and is a further insight into the 'research' you do on the subject, is WHY this farce was cited as somehow bolstering *your sides* opinion?

It was supposed to be a critique of a 'scientific' methodology, yet done in the most unscientific poorly argued manner conceivable.

I am at a loss why rather than being decried as an insult to your side (along the lines of 'Oh man, why has OUR position been so badly mauled and open to ridicule by such a lame article as THIS?') it is instead found worthy of a thread in it's own right and worthy of discussion?

 

[fas42]

Fascinating always is how the human element seems to be largely considered irrelevant to the audio ABX thing ...

I propose three ABX tests conducted in precisely the same way, with the same audio samples and setup, the only variable being the test group:

First is a group of citizens picked at random from the street or university campus or whatever, and told you're going to have to do this test or else.

Second is from visitors to an audio show, again picked at random, and ushered into a pleasant room, given coffee and cake as a reward for doing the procedure.

Thirdly, of volunteers from the "golden ears" brigade, who rate themselves highly on their critical facilities, and who are told that the best performer will be given an open order of $1000 to spend on music or other audio goodies, the second best a $500 open order, etc.

Who's willing to lay decent money on a bet that there won't be a statistical valid difference in the results from these 3 groups?

Frank

 

[amirm]

Such a test was done by Harman Frank and there was a difference. Here are the results:

 

[Phelonious Ponk]

I see I was loosing my time researching on ABX and DBT.

Anyway, if the evaluation is casual and has no pretense of statistical validity what can you conclude about it?
That you were influenced by the order in which you heard the components, your mood, the music you were listening or something else?

I can conclude whether or not I can hear a difference between two components in my system and, if yes, which I prefer, without any pre-conceived notions about price, brand, design or reputation influencing me. I am absolutely certain I am not immune to expectation bias, so I like to remove it sometimes. Of course sometimes I may not hear a difference because I expect not to hear a difference, but listening blind, at least I can be pretty sure any difference is pretty small. Sighted "testing?" Audiophiles often talk themselves into attributing to electronics changes that could only come from speakers and rooms.

Tim

 

[Phelonious Ponk]

Such a test was done by Harman Frank and there was a difference. Here are the results:

What were they listening to, Amir?

Tim

 

[amirm]

Speakers: http://seanolive.blogspot.com/2008/12/loudspeaker-preferences-of-trained.html

 

[Phelonious Ponk]

Speakers: http://seanolive.blogspot.com/2008/12/loudspeaker-preferences-of-trained.html

I would have guessed. I'll bet those bars would look very different if they were comparing electronics.

Tim

 

[RUR]

What were they listening to, Amir?

Tim

Loudspeaker preference. In context, the trained listeners reached the same conclusions (same preferences) as untrained, they just did it "faster".

 

[amirm]

Loudspeaker preference. In context, the trained listeners reached the same conclusions (same preferences) as untrained, they just did it "faster".

I should say my experience with trained listeners is different. They are able to hear things others do not no matter how much more time they take. Sean's key point is correct that trained listeners do not have a different taste so optimizing for them is good for general public.

 

[microstrip]

I can conclude whether or not I can hear a difference between two components in my system and, if yes, which I prefer, without any pre-conceived notions about price, brand, design or reputation influencing me. I am absolutely certain I am not immune to expectation bias, so I like to remove it sometimes. Of course sometimes I may not hear a difference because I expect not to hear a difference, but listening blind, at least I can be pretty sure any difference is pretty small. Sighted "testing?" Audiophiles often talk themselves into attributing to electronics changes that could only come from speakers and rooms.

Tim

Tim,
You are missing my point. I repeat - if, as you admitted "Anyway, if the evaluation is casual and has no pretense of statistical validity" what can you conclude about it? In other words, if the test has no statistical value why bothering with the expectation bias? IMHO, the only consequence will be that the result can probably be false, but your conscience will feel better and consider it as a valid test. Please note we are addressing the audibility, not the preferences.
BTW, what changes are you referring to?

 

[microstrip]

I should say my experience with trained listeners is different. They are able to hear things others do not no matter how much more time they take. Sean's key point is correct that trained listeners do not have a different taste so optimizing for them is good for general public.

Amir,
I understand training listeners can increase the efficiency, but how can we be sure that training someone will not "tailor" their preferences? If the Harman listeners were trained by David Wilson would the Harman sound be the same?

 

[RUR]

Amir,
I understand training listeners can increase the efficiency, but how can we be sure that training someone will not "tailor" their preferences? If the Harman listeners were trained by David Wilson would the Harman sound be the same?

I'm not Amir, but this study shows that the trained and untrained listeners preferred the same speakers i.e. training did not tailor preferences vs. untrained.

 

[amirm]

Amir,
I understand training listeners can increase the efficiency, but how can we be sure that training someone will not "tailor" their preferences? If the Harman listeners were trained by David Wilson would the Harman sound be the same?

Because they are not trained for preference. They are trained on artifacts and failures of fidelity. For compression systems for example, we teach them what the different artifacts sound like and once there, they are able to spot them a mile away. We play transients for them and teach them how the compression system fails to reproduce them and why.

If you run Sean's EQ training software, you see the same thing. They are training people to instantly hear elevated levels of very narrow bands of spectrum. Since Harman believes in flat frequency response, and their trained listeners are trained to hear such anomalies, the mission is accomplished.

That said, there is still a possibility for preference. For example when we developed our video compression at Microsoft, at high definition we put priority on resolution vs compression suppression which softened the video. Our competitor did the opposite. Sometimes there are two versions of truth and you have to pick one and hope that it is right .

 

[Robh3606]

I'm not Amir, but this study shows that the trained and untrained listeners preferred the same speakers i.e. training did not tailor preferences vs. untrained.

Why don't you try the training and decide for yourself if it will bias you in any way??

Rob

 

[fas42]

Interesting that no-one picked up on the concept of motivation level when doing ABX testing, which I was trying to emphasise with my thought experiment on using 3 differently focussed groups ...

Frank

 

[Robh3606]

Interesting that no-one picked up on the concept of motivation level

What does that have to do with the testing? It shouldn't matter, either you hear a difference or you don't and assuming all members of the three groups have normal hearing the result should be quite similar for all three groups. Actually going into it with the idea of having something to prove could very easily be your undoing. Interesting idea though.

Rob

 

[amirm]

He is referring to the hypothesis Sean Olive has on why audio salespeople did better than other groups. That they cared more about the outcome and perhaps spent more time and attention during the test.

In blind tests, it is very easy to give up and just vote randomly. The process is tedious and boring. After five minutes of you, I suspect many voters just want to get through it if the differences are hard to spot and vote randomly.

 

[Phelonious Ponk]

Tim,
You are missing my point. I repeat - if, as you admitted "Anyway, if the evaluation is casual and has no pretense of statistical validity" what can you conclude about it? In other words, if the test has no statistical value why bothering with the expectation bias? IMHO, the only consequence will be that the result can probably be false, but your conscience will feel better and consider it as a valid test. Please note we are addressing the audibility, not the preferences.
BTW, what changes are you referring to?

No, I get your point, I just disagree. I won't be getting a statistical result comparing two components blind, alone, at home, but I can listen to them without being prejudiced by their design, badge, etc. I find that very valuable. I had a very nice tube headphone amp in on loan for about a month a few years ago. I listened to it exclusively for a few weeks and I was absolutely convinced that it had the lush midrange and beautiful tonality that has always been attributed to to tubes, but perhaps a bit less slam and definition in the bass, compared to good SS amps. At the end of 3 weeks of listening and enjoying and thoroughly convincing myself that this was a wonderful choice of headphone amp that I absolutely had to buy, I compared it blind to what is arguably the most humble headphone amp in my house -- the headphone out of a re-conditioned, vintage Harman Kardon SS integrated amplifier.

I was unable to distinguish between them with any consistency at all. And this was back when my bias was most definitely in favor of there being a strong difference.

What could I conclude from that? I could conclude that after enjoying 3 weeks of expectation bias, as soon as I was unable to know I was listening to the beautiful black headphone amp with glowing amber tubes, that I had dutifully warmed up prior to listening, my listening experience changed dramatically. I could conclude that I am not immune to bias and that on that day, in that system, listening blind, I could not reliably differentiate between those two components.

Frankly, I suspect if I had taken a lot of time going back and forth between the two and listened very carefully for differences, and trained myself to hear the differences, I could have done much better. But I would never have gotten back to hearing that beautiful midrange I imagined before the revelation, I would have been reduced to comparing the subtlest of differentiators, mostly in their artifacts. So there's another conclusion that I could possibly reach from blind listening at home: Sometimes there may be differences that may be revealed when listening for differences but have absolutely nothing to do with listening to music.

What I concluded from that headphone amp experience, and a couple of similar experiences with other amps, and DACs, has changed my approach to the hobby, and greatly enhanced my ability to enjoy the music. Who needs statistics?

Tim

 

[terryj]

Interesting that no-one picked up on the concept of motivation level when doing ABX testing, which I was trying to emphasise with my thought experiment on using 3 differently focussed groups ...

Frank

Off you go frank, run with the ball, RUN with it man ! Leave aside the great reward of coffee and cakes tho...

I'll help you get started.

Which *group* is more or less likely to participate in a DBT? what conclusions can you draw about motivation.

Which *group* is more or less likely to deny results from said DBT's? what conclusions can you draw about motivations.

Which *group*, tho steadfastly and vehemently refusing to acknowledge or participate in said DBTs will cry 'foul' and find excuses (which may or may not exist...after all they don't know because it is all 'ivory tower' to them) for results from said DBT's? What conclusions can you draw about motivations for that.

Obviously, there are many moire variations of the above, so off you go. Discuss and examine.

Now, it seems many of your tweaks are of sufficient magnitude that you can hear the changes from anywhere in the house, seems an easy thing to test.

I am more than happy to drive to sydney and help you examine them and the blind test protocol.

THAT you can do it from another room makes it that much easier! No need for me or thee to leave the room or ensure things are hidden!

Or, you can be a bit more serious, I don't mind, and we do it in the listening room.

What are you up for, interconnects? Speaker cables?

Whatever, we'll finalise it as time goes by.

I will finally get to see if your memory is true or faulty, whether we have met as you have claimed or not. Either way a fun and educational time no?