Is ABX finally Obsolete

[Phelonious Ponk]

I agree regarding forum decorum. Too many discussions are shut down immediately with "do you have a DBT? If not, we are done and what you say is inaudible." I call it censorship for the lack of a better word to describe it. We come to these forums to explore ideas. And to have a good time discussing things. Such answers stifle both.

Even I agree with this in principle, and when I'm dealing with someone who is being reasonable, I try not to run to the "have you listened blind" argument, but jeez guys, some Audiophiles are better at hearing what they want to hear than a young girl in love, and manage to hear the most absurd, impossible things. Call it grumpiness, impatience, censorship or just the end of tolerance for the babbling of fools, there comes a point at which, "If you haven't heard it blind, we are done and it's inaudible" is a hell of a lot more polite than the justifiable alternatives.

Tim

 

[fas42]

"If you haven't heard it blind, we are done and it's inaudible" is a hell of a lot more polite than the justifiable alternatives.

Tim

Actually, the notion of listening blind is a valid one, and the LIAR test is quite excellent in that regard. If you can go another space, and focus on another activity, and part of your brain is still telling you that the sound is good then you're a long way there. Typically that is where I pick that there is some problem, and then check the system much closer up. On a side note, and this will please many, the humble HT has gone ta-ta's for the moment, looks like the control circuitry has gone funny on me, and just today no sound at all. If I can't fix it easily I might have to break out the big guns, the Yamaha battleship and the 200 watt Perreaux, and that might make a few more people happy ...

Frank

 

[Orb]

The problem is though Frank, how many who mention blind testing actually follow this for speakers; in my experience this is close to zero.
Speakers are also important not only due to the brand, but critically speaker location and this can affect whether we subjectively like what we hear.
So the same speaker in same position but blind in one instance can and is likely to give different subjective preference results, this came out as part of a Harmen test procedure.

However I would say 99% of all calls in forums to do blind testing never consider this type of behaviour and also applying the rules to speakers (going as far as not knowing-seeing the position of the speaker in the room).
But as you say, notion of blind listening is a valid one, but possibly and critically so is subjective long term and extensive listening preference (emphasis long term).
That said, biases can be overcome with a methodical and analytical approach also taking into account bias heuristics; some debiasing papers do exist.
It would had been interesting to see the results of say the wine tasting preference test paper done with those who are involved heavily with marketing-brand recognition services-etc, wondering how much the results would differ even though they have not necessarily applied debiasing approach themselves but work with consumer related biases.

Cheers
Orb

 

[fas42]

Speakers are also important not only due to the brand, but critically speaker location and this can affect whether we subjectively like what we hear.
So the same speaker in same position but blind in one instance can and is likely to give different subjective preference results, this came out as part of a Harmen test procedure.

Orb, I agree that speakers driven by conventional electronics and setups are a hairy area: the slightest change in position, method of driving them and way of listening to them screams at you because your brain has handle the extra load of the sound not being as good as it could be. An old technique of mine was to try and talk to someone while a system was wound up a bit, this immediately, subjectively, changed the sound because if your brain has to fight the speaker output at this point it then sounds much worse. In other words, the level of concentration applied to the sound changes the subjective impression of the sound ...

My way of dealing with this, of course, is to make sure the sound leaving the speaker drivers is as good as possible.

Frank

 

[Gregadd]

http://lsbaudio.com/publications/AES127_ABX.pdf

Not to math intensive. A few things we have talked about before.

I am currently looking for an article I think it was on AVS or Hydrogen Audio. It provides some support to memory being a problem in ABX. Initlally they failed almost unanimoulsy. Taking the test numerous times thier scores were like 7/8, 12/13, 14/17. Indicating the greater thier exposure the better thier score. Other explanations exist.

 

[Phelonious Ponk]

http://lsbaudio.com/publications/AES127_ABX.pdf

Not to math intensive. A few things we have talked about before.

I am currently looking for an article I think it was on AVS or Hydrogen Audio. It provides some support to memory being a problem in ABX. Initlally they failed almost unanimoulsy. Taking the test numerous times thier scores were like 7/8, 12/13, 14/17. Indicating the greater thier exposure the better thier score. Other explanations exist.

Yeah, that's what you hear - the greater the exposure the better the score. Except when you hear that the greater exposure fatigues the participants and invalidates the results. Except when you hear that no amount of quick-switching accomplishes anything anyway, you have to listen for weeks in your own home to hear the night and day difference.

You don't need to run a listening test at all to prove that Audiophiles hear, and believe, what they want to. All you have to do is discuss it on the internet.

Tim

 

[amirm]

My personal experience in blind test is that the smaller the difference, the smaller the switch over time needs to be. Here is an analogy. Let's say I have the same signal feeding an AM station and an FM station. Do you think you will remember the difference a few days later? I am sure everyone would say yes. So the larger the impairment, the less critical switching time needs to be. I have been through many blind tests and I can attest to this. When given the smallest differences, I need to have instantaneous switching time. This not only allows me to find the difference but also confirm that I have. I may do as much as 20 switches in 10 seconds to hone in on the key difference.

An inverse relationship exists between level of difference and amount of time one needs for the test. Have me sit through the AM/FM difference and I can in one or two seconds declare the winner. Make the difference small and I may need half of hour with just one clip to find the critical section and find the difference.

Related to above point, major part of the problem is the content not being revealing enough. If the critical section is provided, then testing time can be sharply reduced and reliability heavily increased. MPEG short clips for codecs do this, providing much better tool than picking random songs hoping they can show a difference.

 

[Phelonious Ponk]

My personal experience in blind test is that the smaller the difference, the smaller the switch over time needs to be.

I think that experience is pretty universal, Amir. The folks who say short switching times make it harder to discern subtle differneces are not using blind listening and sometimes are simply trying to discredit it.

Tim

 

[Gregadd]

Correct me if I am wrong. The brain prioritizes. The more exposure the better the memory. If I go throught the first test quickly I may miss the diffference. Just suppose I know I failed and the tester points out the differences. Now I am looking for the difference. My score improves. I have stopped trying to memorize the entire selection. Now I am looking for clues. My confidence grows. I began to know where the clues. are. In the test I am not talking about listening longer. They are merely taking the test again with a greater number of trials.

I think the most interesting thing that writer suggests that 2 out 10 may be as significant as 8 out of ten.. That is to say results that are inconsisitnet with guessing. 2 out of 10 is inconsistent with guessing.

 

[Gregadd]

I think that experience is pretty universal, Amir. The folks who say short switching times make it harder to discern subtle differneces are not using blind listening and sometimes are simply trying to discredit it.

Tim

Tim there may be some truth to that. On the other hand you have to discern a difference and remember it.

 

[Phelonious Ponk]

Just suppose I know I failed and the tester points out the differences. Now I am looking for the difference.

...and you no longer have a hope of an objective study. This is the X in ABX. Then, even the tester doesn't know what you're listening for or what plays when, so he can't inadvertently influence the outcome. In what you've described, he is deliberately influencing the outcome.

Tim

 

[Gregadd]

...and you no longer have a hope of an objective study. This is the X in ABX. Then, even the tester doesn't know what you're listening for or what plays when, so he can't inadvertently influence the outcome. In what you've described, he is deliberately influencing the outcome.

Tim

I think you missed the point Tim. Or I miststated the hypothetical. Your fialure is recorded. The tester did not say he told them anything. In any event Dr. Olive teaches people what to look for.

 

[Phelonious Ponk]

Tim there may be some truth to that. On the other hand you have to discern a difference and remember it.

You do. That's the reason for the short switching. Everyone, except for Audiophiles who don't believe in the methodology at all, seems to agree that quick switching is the only reliable way to differentiate subtle differences. And they base this not on speculation, but on what got the best results when they knew a difference existed.

Tim

 

[Phelonious Ponk]

I think you missed the point Tim. Or I miststated the hypothetical. Your fialure is recorded. The tester did not say he told them anything. In any event Dr. Olive teaches people what to look for.

I did miss your point. Sorry. Yes. Harman trains listeners. The also get the same statistical results from untrained listeners that they get from the trained ones. The trained listeners just help them reach conclusions faster, more efficiently. A very useful thing for a commercial endeavor.

Tim

 

[RUR]

...Related to above point, major part of the problem is the content not being revealing enough. If the critical section is provided, then testing time can be sharply reduced and reliability heavily increased.....

Makes perfect sense as the use of revealing material is one of the test conditions for ITU-R BS.1116-1, which is widely used as the basis for "practical" blind testing, e.g.

6 Programme material

Only critical material is to be used in order to reveal differences among systems under test. Critical material is that which stresses the systems under test. There is no universally “suitable” programme material that can be used to assess all systems under all conditions. Accordingly, critical programme material must be sought explicitly for each system to be tested in each experiment. The search for good material is usually time-consuming; however, unless truly critical material is found for each system, experiments will fail to reveal differences among systems and will be inconclusive.

It must be empirically and statistically shown that any failure to find differences among systems is not due to experimental insensitivity because of poor choices of audio material, or any other weak aspects of the experiment, before a “null” finding can be accepted as valid.

 

[microstrip]

I did miss your point. Sorry. Yes. Harman trains listeners. The also get the same statistical results from untrained listeners that they get from the trained ones. The trained listeners just help them reach conclusions faster, more efficiently. A very useful thing for a commercial endeavor.

Tim

Tim,
My main concern about training is that when you have to evaluate something you always have to choose what you consider the most important aspects and them score these aspects according to some weighting scheme.
The tests you refer only prove that for Dr. Olive evaluating system trained or untrained people get the same results in some specific tests. They also prove they can do it faster with increased reliability. Nothing else.

I have read some testimonies of people who proudly refer that after doing the Olive Sean training they are permanently identifying the faults and non natural things in recordings. I am not interested in becoming this type of listener - INHO exactly as are not interested in developing extra-sensitivity to some aspects audiophile praise a lot.

I am sure that after going through that training people appreciate more the excellent Revel speakers - they were developed to satisfy their new preferences.

 

[fas42]

Makes perfect sense as the use of revealing material is one of the test conditions for ITU-R BS.1116-1, which is widely used as the basis for "practical" blind testing, e.g.

Now curious, what is generally considered "revealing" material? For myself, I use "bad" recordings almost universally, it makes differences extremely obvious because the ear/brain has to work harder to process the recorded material, so any extra load caused by B not being quite as good sound as A, say, strikes the brain straight away. A good example, big band material of the 30's ...

Frank

 

[Phelonious Ponk]

Tim,
My main concern about training is that when you have to evaluate something you always have to choose what you consider the most important aspects and them score these aspects according to some weighting scheme.
The tests you refer only prove that for Dr. Olive evaluating system trained or untrained people get the same results in some specific tests. They also prove they can do it faster with increased reliability. Nothing else.

I have read some testimonies of people who proudly refer that after doing the Olive Sean training they are permanently identifying the faults and non natural things in recordings. I am not interested in becoming this type of listener - INHO exactly as are not interested in developing extra-sensitivity to some aspects audiophile praise a lot.

I am sure that after going through that training people appreciate more the excellent Revel speakers - they were developed to satisfy their new preferences.

I'll leave it to Sean to speak to what those people are trained for, but I wasn't under the impression that they were trained to hear artifacts and flaws, like Amir's codec listening skills. If they are, we're in agreement. The last thing I want to be is the kind of listener who listens to the weaknesses in recording and reproduction instead of the strengths in the music. Though I imagine I could be trained to hear those things and still set it aside and just listen to the music. I can certainly hear the limitations of many of my recording and still thoroughly enjoy them.

Tim

 

[Phelonious Ponk]

Now curious, what is generally considered "revealing" material? For myself, I use "bad" recordings almost universally, it makes differences extremely obvious because the ear/brain has to work harder to process the recorded material, so any extra load caused by B not being quite as good sound as A, say, strikes the brain straight away. A good example, big band material of the 30's ...

Frank

For most of us, Frank, it's the opposite. We use good recordings. The kind that have captured a lot of detail and managed not to obscure it with noise, distortion and extreme manipulations of the FR and dynamic range. We do this because only when the detail has been captured and not obscured do our systems have a chance of reproducing it and our ears have a chance of hearing it. It's a good way to test a system's ability to reproduce what was there. Your test, using bad recordings is, I suppose, a very good way to test your brain's ability to imagine that which was never on the recording in the first place. I can't imagine it being good for much else. Not in audio, anyway.

Tim

 

[Gregadd]

I did miss your point. Sorry. Yes. Harman trains listeners. The also get the same statistical results from untrained listeners that they get from the trained ones. The trained listeners just help them reach conclusions faster, more efficiently. A very useful thing for a commercial endeavor.

Tim

But the test was not invalid because they were told what to look for? Another person could administer the test ,thus keeping it DB?