I do get your reasoning, I don't agree with it though. It's simpler to accept that the difference was caused by sight than accept that the null was caused by lack of sight
Conclusive "Proof" that higher resolution audio sounds different
- Thread starter amirm
- Start date
I do get your reasoning, I don't agree with it though. It's simpler to accept that the difference was caused by sight than accept that the null was caused by lack of sight
Of course! Did you not read Amir's description of him doing his test? One of the factors leading to him doing this test was a challenge by ArnyK. Winer is also famous for such tactics. In both these cases, Amir's training, expertise & pure doggedness was enough to overcome these obstacles. But I already gave an example of someone we both know, Vital who initially couldn't hear any difference in Winers loopback test even with 20 passes & after he persevered (because others got positive results) & broke through, he could then identify 5 passes. He admitted that he was suffering from both negative bias & anxiety which he stated was holding him back. He admitted that double blind testing brings it's own pressures - he stated that you have a tendency not to want to overly tax or delay the person doing the equipment changing.
I gave an example of an added factor introduced with blind listening that could give null results - there is no "as opposed to"
Last edited:
Well, that's progress, at least. I know it's simpler to accept the stock, received wisdom but it's often better to think these things through if time & willingness (& other factors) allow.
As I said to Max, Amir's description of his testing is very informative & honest. It should be noted by all & noticed how much the test (& influencing factors) tends towards producing a null result.
That post of Amir's alone should be made into a sticky, IMO
This tendency towards a null result is not unusual & Orb I believe, has stated this many, many times before - showing the difficulty of doing a valid blind test & the controls needed!
Last edited:
John, both Vital and I heard no differences between the 20-pass files and the original, sighted.
Then Adamdea posted results showing a perfect result - he aced it, using ABX via Foobar.
This actually put me under pressure to do the same, and I assume Vital too, so we both tried using ABX tools and then, under more pressure than before, we both aced it too.
This was a challenge that caused differences to be revealed, not masked!
Then Adamdea posted results showing a perfect result - he aced it, using ABX via Foobar.
This actually put me under pressure to do the same, and I assume Vital too, so we both tried using ABX tools and then, under more pressure than before, we both aced it too.
This was a challenge that caused differences to be revealed, not masked!
Just because the concept of bias is very simple and the limitations of auditory perception need some extra time and effort to explain.
Do you disagree that "challenge" can have an influence on the results then?
Do you disagree that it can have an influence in delivering the opposite result to your & Vitals' experience?
Do you disagree with my reporting of Vital's statements about influencing factors/pressures?
BTW, I believe you are interpreting your & Vital's experience incorrectly. Firstly, it's not any reflection on "how bad" sighted listening is - I've seen you state this before
John, assuming pressure is considered a variable (I don't believe so) - what controls do you think can account for it?
I do not consider 'challenge' to be a variable, no. It's just far too convenient, IMO.
Vital's experience in terms of the test was similar to mine, but what he concluded from it was very different to what I concluded.
Max, there's no point in me explaining this to you when you don't accept it as a factor - I'd just be wasting my time. I suggest if you are interested (but I doubt you are) that you need to go & read more about blind testing particularly for the testing of codecs (MP3 & others) that J_J was heavily involved in developing - the testing for barely perceptible differences.
Anyway, off to bed now
John, you'd be explaining for the house, not just me. You'd also be giving others a better chance to understand your overall position, IMO.
I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.
You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing
I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).
I doubt we'll ever move much from our respective positions
I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.
You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing
I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).
I doubt we'll ever move much from our respective positions
For large differences it is valid for me. For small differences, again for me, it is not.
If the differences are large then they would be picked up in short term testing too, so what does long term testing add?
John, you'd be explaining for the house, not just me. You'd also be giving others a better chance to understand your overall position, IMO.
I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.
You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing
I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).
I doubt we'll ever move much from our respective positions
The conclusion I came to several pages back. This thread has been running in weird self-abusive circles for a long time, but what it seems to come down to is that John is fully embracing "pressure," as a variable powerful enough to dismiss the entire blind listening methodology. He must want to dismiss it awfully badly, because right here, in the examples in this thread, sighted listening has returned a null result and blind listening, by both the trained and untrained, has accurately identified a difference. The example in front of us, the only real evidence in the discussion, supports exactly the opposite of what John says we should expect from the variable, "pressure." His only other argument is that the method that invites the biases of brand, beauty, engineering methodology, reputation, size, price, etc...in other words, the plethora of biases that are opened up with knowledge of what is being compared...is just as valid as one that eliminates all of those biases. Of course the evidence above counters that as well, but it hardly needs evidence. No one could buy it without a very strong desire to dismiss blind listening.
This was over a long time ago.
Sorry to single you out, John. It really is the position, not the person, I'm talking about. It's just that you're the last man standing on that position. Now I'll bow out again for the time being, as I'm sure this is not over.
Tim
Ultimately I knew this was fruitless but still I tried - just shows you how irrational human beings can be!
Tim, you choose to ignore Amir's description of the psychological pressure he felt & it's tendency to cause him to second guess himself & instead focus on what Max says were his reasons. Sure we all select the evidence that best supports our case. Here's what J_J says " ABX tests (or any kind of scientific test) should not be presented as personal challenges, either. They should be presented with as much support as possible given to those taking the test, including (but not limited to) training, feedback, comfort, appropriate equipment, audio material, and so on."
Hey, Tomelex, "how many times have I made an adjustment and thought that wow, that's going to do this or that to the sound, and then I hear no change when knowledge is removed, its damn frustrating but it keeps me honest." I do this sighted all the time & don't have a problem identifying a failed adjustment.
One last thing, I have done some listening comparisons between DSD & PCM (the same tracks in both- not remastered, AFAIK). There is some difference between them but I don't know what it is, can't put my finger on it, yet. Would I be able to A/B identify them, at the moment - I don't know. I will need some long-term sighted listening to work it out, when I have time. One doesn't need to be an inexperienced audiophile to be beguiled by what they hear initially - it happens quiet a lot. It takes a bit of experience to know that only over long-term listening can conclusions be reached.
Tim, you choose to ignore Amir's description of the psychological pressure he felt & it's tendency to cause him to second guess himself & instead focus on what Max says were his reasons. Sure we all select the evidence that best supports our case. Here's what J_J says " ABX tests (or any kind of scientific test) should not be presented as personal challenges, either. They should be presented with as much support as possible given to those taking the test, including (but not limited to) training, feedback, comfort, appropriate equipment, audio material, and so on."
Hey, Tomelex, "how many times have I made an adjustment and thought that wow, that's going to do this or that to the sound, and then I hear no change when knowledge is removed, its damn frustrating but it keeps me honest." I do this sighted all the time & don't have a problem identifying a failed adjustment.
One last thing, I have done some listening comparisons between DSD & PCM (the same tracks in both- not remastered, AFAIK). There is some difference between them but I don't know what it is, can't put my finger on it, yet. Would I be able to A/B identify them, at the moment - I don't know. I will need some long-term sighted listening to work it out, when I have time. One doesn't need to be an inexperienced audiophile to be beguiled by what they hear initially - it happens quiet a lot. It takes a bit of experience to know that only over long-term listening can conclusions be reached.
John, you'd be explaining for the house, not just me. You'd also be giving others a better chance to understand your overall position, IMO.
I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.
You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing
I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).
I doubt we'll ever move much from our respective positions
Maybe people should just re-read Amir's description of his running of his tests & examine the elements of pressure
I have highlighted almost all of his post as it all reflects the pressure he was under & what it was doing to his choices.
Are you really telling me that you can't see the effects of pressure in influencing the results towards a null result?
Because I can't tell the DSD Vs PCM differences immediately?
I'm not so sure that here is anything wrong - there were a group of us with varying opinions on DSD so no group think in action.
We mainly used a Meitner (which I know internally converts PCM to DSD) & somewhat ensured that the volume levels were matched.
We used a good PC playback source although this was identified as probably the weakest link in the chain & good playback software - for PCM we used MQN & for DSD we used Jplay. Good amp, Jadis & speakers Quad ESLs.
Anyway, as far as I'm concerned, the jury is still out although I do believe that there was some audible difference in that scenario - just what it was I'm not sure, yet!
Or, if you mean something is wrong with my whole argument then touche!
Anyway, no more time today - must go!
Actually I've done the opposite of that John. I've acknowledged the "pressure" in Amir's methods, and noted that the pressure didn't return a negative result in this case, while casual listening has. Which is the opposite of what you're saying the great "pressure" (which could be easily avoided, by the way) of unsighted listening should cause. You're position is full of holes, john, and in its relentless defense, you've been contradicting yourself for awhile now. it's way past time to give this one up.
Tim
Some pertinent quotes from J_J
"ABX tests (or any kind of scientific test) should not be presented as personal challenges, either. They should be presented with as much support as possible given to those taking the test, including (but not limited to) training, feedback, comfort, appropriate equipment, audio material, and so on."
"And, yes, you have to be comfortable, relaxed, and extremely familiar with everything involved in order to get a sensitive result. One does not just jump into the test willy-nilly."
I don't know why the idea of pressure in a test is being rejected as an influence? Just because Amir was able to overcome it does not mean that it's therefore of no significance. Just because Max couldn't differentiate the two samples sighted & at a later stage could do so in an ABX doesn't show us anything about pressure - it could very well be that at the time Max first tried the sighted test was when no one on the PFM forum (his peers) had returned a positive result, he felt pressure to conform. Later when positive results were returned by some of his peers he suddenly found positive ABX results - who knows, probably not even Max, knows - motivations & pressures can remain hidden to the individual.
These particular anecdotes do not show any evidence that contradicts my statement that "psychological pressure is an influencer in such tests". Do you really feel that it doesn't have an influence?
"ABX tests (or any kind of scientific test) should not be presented as personal challenges, either. They should be presented with as much support as possible given to those taking the test, including (but not limited to) training, feedback, comfort, appropriate equipment, audio material, and so on."
"And, yes, you have to be comfortable, relaxed, and extremely familiar with everything involved in order to get a sensitive result. One does not just jump into the test willy-nilly."
I don't know why the idea of pressure in a test is being rejected as an influence? Just because Amir was able to overcome it does not mean that it's therefore of no significance. Just because Max couldn't differentiate the two samples sighted & at a later stage could do so in an ABX doesn't show us anything about pressure - it could very well be that at the time Max first tried the sighted test was when no one on the PFM forum (his peers) had returned a positive result, he felt pressure to conform. Later when positive results were returned by some of his peers he suddenly found positive ABX results - who knows, probably not even Max, knows - motivations & pressures can remain hidden to the individual.
These particular anecdotes do not show any evidence that contradicts my statement that "psychological pressure is an influencer in such tests". Do you really feel that it doesn't have an influence?
Last edited:
From RECOMMENDATION ITU-R BS.1116-1
METHODS FOR THE SUBJECTIVE ASSESSMENT OF SMALL IMPAIRMENTS IN AUDIO SYSTEMS INCLUDING MULTICHANNEL SOUND SYSTEMS
It must be empirically and statistically shown that any failure to find differences among systems is not due to experimental insensitivity because of poor choices of audio material, or any other weak aspects of the experiment, before a “null” finding can be accepted as valid. In the extreme case where several or all systems are found to be fully transparent, then it may be necessary to program special trials with low or medium anchors for the explicit purpose of examining subject expertise.
These anchors must be known, (e.g. from previous research), to be detectable to expert listeners but not to inexpert listeners. These anchors are introduced as test items to check not only for listener expertise but also for the sensitivity of all other aspects of the experimental situation.
End of quote
As all blind tests concerning small differences (except Amir's tests ) resulted in the conclusion that the system was transparent, there is still a lot of work to be done!
METHODS FOR THE SUBJECTIVE ASSESSMENT OF SMALL IMPAIRMENTS IN AUDIO SYSTEMS INCLUDING MULTICHANNEL SOUND SYSTEMS
It must be empirically and statistically shown that any failure to find differences among systems is not due to experimental insensitivity because of poor choices of audio material, or any other weak aspects of the experiment, before a “null” finding can be accepted as valid. In the extreme case where several or all systems are found to be fully transparent, then it may be necessary to program special trials with low or medium anchors for the explicit purpose of examining subject expertise.
These anchors must be known, (e.g. from previous research), to be detectable to expert listeners but not to inexpert listeners. These anchors are introduced as test items to check not only for listener expertise but also for the sensitivity of all other aspects of the experimental situation.
End of quote
As all blind tests concerning small differences (except Amir's tests
) resulted in the conclusion that the system was transparent, there is still a lot of work to be done!Similar threads
