Doing anything in a rigorous way is always harder than pulling stuff out of your butt, so that shouldn't be any surprise at all.
Is ABX finally Obsolete
- Thread starter Gregadd
- Start date
- Status
... doing it in a sloppy way, or doing it in a way which most likely will confirm your prejudices. The simple truth is, if you really, really want audio to work "properly", that is, as good as it is capable of being then you have to work hard at it, and if you likewise want to test it in such a way to truly understand what's going on then you have to work just as hard again.
Really good audio is hard, full stop. Slightly less than that is dead easy: as Tim would say, just buy some decent pro gear. The margin between the two is very small, and ABX will most likely never pick the difference. But that doesn't mean it's not there ...
Frank
I don't see any double talk there, Greg. Do you need interpretation? I believe that he said that there are some amps that color the sound, tubes in particular, but most, when used properly, from car audio up through midfi and beyond, don't. He goes on to say that he can't tested them all, but this is consistent with all he has tested. What part didn't you understand?
Tim
My position of using peer-reveiwed papers? What is that? I don't recognize any such thing.
Are you saying that since an rank amateur audiophile doing a sighted non-level-matched of equipment in questionable condition is providing anecdotal evidence and a degreed engineer doing a level-matched, DBT of equipment known to conform to its technical specs is providing anecdotal evidence, that their evidence has equal weight?
The above is 100% speculation on your part. The effects you are claiming are audible simply aren't. The issues that you are obsessiing over are simply figments of your imagination.
If you want to prove your point, first prove your subpoints!
Amir, I've answered this question any number of times for you and somehow my answer to you gets lost every time. If you want me to show respect for you by answering your questions, then show respect for me by admitting when I answer them.
The way I see it, there is considerable scientific opinion that sighted evaluations are a totally inadequate means for doing studies based on sensory perceptions. So, as soon as I see people taking that to heart, and abandoning that totally fallacious course of action, only then can we talk about alternatives. Far more logical alternatives exist.
What I now see is people en masse completely and totally ignoring the utter rediculousness of doing sighted comparisons of amplifiers, DACs, media formats, etc. If there is a desire for a rational solution, then there should be an admission of the irrationality of the status quo.
Arny, Frank doesn't do logic, much less proof. And the "discriminating listeners" are the members of this board who read his posts for more than amusement. Both of them.
Tim
I'm with this. The best argument for blind audio evaluations, regardless of the informality and/or lack of scientific rigor, is that no matter how sloppy they are, if they remain blind they are way ahead of sighted evaluations.
tim
Although it ignores what I mentioned relating to debiasing principles, which I have explained a little bit in this thread and other threads.
I want to stress that this does not mean there is no need for blind testing or even DBT depending upon the purpose of the test, but one should also consider the fact that all sighted evaluations are not equal (nor are blind tests-methodologies and their focus), especially when considering those with scientific-engineering backgrounds who also understand bias and in some parts the principles towards debiasing sighted evals.
It is not one or the other when it comes to such debates IMO, but about the factors towards both that should be considered and debated.
Thanks
Orb
I get all of that, Orb (well, except for the part about "debiasing sighted evaluations"), and perhaps the audio community can get to methodologies and scientific/statistical rigor at some point. Now? They're mired in something that is the equivalent of believing the earth is flat. A whole bunch of them seem to believe they can compare two pieces of audio gear more objectively when they can see them. They seem to think that removal of a high chance of bias creates a chance of bias. Good methodology designed to remove secondary influences, achieve statistically valid results and analyze the data is hardly relevant until we can get past that point. We're still not sailing east because we'll sail right off the edge of the earth.
Tim
Tim
It's not I who claim ABX is to hard. It's the people who do advocate it. Without exception they have all claimed it's too difficult to do with any consistency. Harmon pays other people to do it and does not do ABX. Some guy doing it in his basement is not scientific. Too many variables and the sample size is too small. I do not suggest that he would have to test every amp. There is a scientific basis for sampling. But that is a rigorous endeavor more demanding than a "spot test." AS John Atkinson said, it is just to much trouble to wind up with a null hypothesis or false negative.
If a statement is to have any meaning it has to have sufficient certantty.
"Some amps are colored some are not." Not only is it a contradiction of the previous statement, "all amps sound the same" it is so vague that any audiophile off the street could have said it.
If you are really committed to the scientific method, You would take a scientific sample,and put them under the rigorous ABX test you claim is the only one that is valid. The test defeats everyone that disagrees with you. But you don't need to because you already know the answer.
I'll put it back at you. Show me those test results.
Try doing a sighted listening test. It is time consuming and difficult. Try listening to the same music selections over and over looking for differences. It's not easy by any means.
If a statement is to have any meaning it has to have sufficient certantty.
"Some amps are colored some are not." Not only is it a contradiction of the previous statement, "all amps sound the same" it is so vague that any audiophile off the street could have said it.
If you are really committed to the scientific method, You would take a scientific sample,and put them under the rigorous ABX test you claim is the only one that is valid. The test defeats everyone that disagrees with you. But you don't need to because you already know the answer.
I'll put it back at you. Show me those test results.
Try doing a sighted listening test. It is time consuming and difficult. Try listening to the same music selections over and over looking for differences. It's not easy by any means.
So, I don't think we've concluded that ABX is obsolete. It is, after all, just a blind comparison with a control; pretty fundamental stuff that will be used again regardless of what we conclude. I'm not sure anyone invented it either, but that's another subject. And Greg seems to have convinced himself that it's too hard; that probably didn't take much convincing as I suspect he came to the party with that conclusion. So those of us who think we might be subject to sighted bias can find ways to compare gear blind - with or without the X - and the rest of you can carry on as before, comparing audio with your eyes and ears. That took 36 pages.
Tim
Tim
If we talk science it probably goes like this:
Do you have an experimental design?
No: it is no science
Yes: is the design apt for what you are going to investigate?
No: invalid experimental design
Yes: is the experimental setup correct?
No: invalid results
Yes: is the analyses correct?
No: invalid results
Etc.
To conduct a well-controlled experiment is not easy.
Anybody familiar with meta-analyses knows that often reports are dropped from the analyses because of methodological flaws.
If science was easy we all are scientist.
On the other hand; does our judgement improves when our perception is influenced by all kind of factors not relevant to where it is about: sound quality.
This pic from Sean’s blog demonstrate it nicely
In both cases the big floorstanders (G,D) are preferred compared with the two smaller ones.
However, in the unsighted test the differences are much smaller.
Shows you how easily our perception is influenced.
We simply use clues irrelevant to the task.
Unsighted testing removes the clues.
Of course we are not scientist so our experiment is not well controlled so not really valid.
But I prefer a non-controlled unsighted test over a non-controlled sighted one.
Saves you a judgement based on irrelevant clues.
Do you have an experimental design?
No: it is no science
Yes: is the design apt for what you are going to investigate?
No: invalid experimental design
Yes: is the experimental setup correct?
No: invalid results
Yes: is the analyses correct?
No: invalid results
Etc.
To conduct a well-controlled experiment is not easy.
Anybody familiar with meta-analyses knows that often reports are dropped from the analyses because of methodological flaws.
If science was easy we all are scientist.
On the other hand; does our judgement improves when our perception is influenced by all kind of factors not relevant to where it is about: sound quality.
This pic from Sean’s blog demonstrate it nicely
In both cases the big floorstanders (G,D) are preferred compared with the two smaller ones.
However, in the unsighted test the differences are much smaller.
Shows you how easily our perception is influenced.
We simply use clues irrelevant to the task.
Unsighted testing removes the clues.
Of course we are not scientist so our experiment is not well controlled so not really valid.
But I prefer a non-controlled unsighted test over a non-controlled sighted one.
Saves you a judgement based on irrelevant clues.
True. I have cited above an AES paper saying what you have to do. Not many takers, if any. So maybe you should stop bashing us "subjectivists" and get to the heavy lifting of performing your own ABX/DBT. I know it's not your job. It's to expensive or time consuming.
That's 20 tests at least. The more listeners the greater the confidence level.
You never heard a subjectivist back out like that. We are always up for a listening evaluation.
Last edited:
If we talk science it probably goes like this:
Do you have an experimental design?
No: it is no science
Yes: is the design apt for what you are going to investigate?
No: invalid experimental design
Yes: is the experimental setup correct?
No: invalid results
Yes: is the analyses correct?
No: invalid results
Etc.
To conduct a well-controlled experiment is not easy.
Anybody familiar with meta-analyses knows that often reports are dropped from the analyses because of methodological flaws.
If science was easy we all are scientist.
On the other hand; does our judgement improves when our perception is influenced by all kind of factors not relevant to where it is about: sound quality.
This pic from Sean’s blog demonstrate it nicely
In both cases the big floorstanders (G,D) are preferred compared with the two smaller ones.
However, in the unsighted test the differences are much smaller.
Shows you how easily our perception is influenced.
We simply use clues irrelevant to the task.
Unsighted testing removes the clues.
Of course we are not scientist so our experiment is not well controlled so not really valid.
But I prefer a non-controlled unsighted test over a non-controlled sighted one.
Saves you a judgement based on irrelevant clues.
But we do learn don't we Vince? Are we stuck with that example forever?
On to page 37...There's nothing to prove, Greg. Are you evaluating sound? If so, and all other things are equal, nothing is added by sight but the opportunity for bias (and in context, blind can mean something as simple as not knowing if it's the 320kbps file or the lossless one playing, so don't give me a bunch of nonsense about blindfolds and clinical labs and stress...). Nothing. Some sighted tests may get the same results. Some may occasionally even get more accurate results. But they can't possibly be more objective. If you can come up with some kind of sighted evaluation methodology that would would reduce the probability of our being influenced by what we see more than not actually seeing it, we'll have a conversation. Until then, this position is too dopey for reason, much less "proof."
Tim
Peer reviewed is far from perfect.
http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/85605.htm#a11
Or as they say: Birds of feather flock together but fools seldom differ.
- Status
Similar threads
