Conclusive "Proof" that higher resolution audio sounds different

[Whatmore]

Well, yes a new variable has been introduced by the blind "challenge" - is that difficult to understand?
I was asked for a specific example by Max & I used an example that he would be familiar with.
So, I'm not sure what you are saying?

I do get your reasoning, I don't agree with it though. It's simpler to accept that the difference was caused by sight than accept that the null was caused by lack of sight

 

[jkeny]

Thanks for clarifying. So pressure may mask differences?

Of course! Did you not read Amir's description of him doing his test? One of the factors leading to him doing this test was a challenge by ArnyK. Winer is also famous for such tactics. In both these cases, Amir's training, expertise & pure doggedness was enough to overcome these obstacles. But I already gave an example of someone we both know, Vital who initially couldn't hear any difference in Winers loopback test even with 20 passes & after he persevered (because others got positive results) & broke through, he could then identify 5 passes. He admitted that he was suffering from both negative bias & anxiety which he stated was holding him back. He admitted that double blind testing brings it's own pressures - he stated that you have a tendency not to want to overly tax or delay the person doing the equipment changing.

As opposed to there just being none, unless sighted, when pressure isn't a factor?

I gave an example of an added factor introduced with blind listening that could give null results - there is no "as opposed to"

 

[jkeny]

I do get your reasoning, I don't agree with it though. It's simpler to accept that the difference was caused by sight than accept that the null was caused by lack of sight

Well, that's progress, at least. I know it's simpler to accept the stock, received wisdom but it's often better to think these things through if time & willingness (& other factors) allow.
As I said to Max, Amir's description of his testing is very informative & honest. It should be noted by all & noticed how much the test (& influencing factors) tends towards producing a null result.

That post of Amir's alone should be made into a sticky, IMO

This tendency towards a null result is not unusual & Orb I believe, has stated this many, many times before - showing the difficulty of doing a valid blind test & the controls needed!

 

[maxflinn]

John, both Vital and I heard no differences between the 20-pass files and the original, sighted.

Then Adamdea posted results showing a perfect result - he aced it, using ABX via Foobar.

This actually put me under pressure to do the same, and I assume Vital too, so we both tried using ABX tools and then, under more pressure than before, we both aced it too.

This was a challenge that caused differences to be revealed, not masked!

 

[microstrip]

(...) It's simpler to accept that the difference was caused by sight than accept that the null was caused by lack of sight

Just because the concept of bias is very simple and the limitations of auditory perception need some extra time and effort to explain.

 

[jkeny]

John, both Vital and I heard no differences between the 20-pass files and the original, sighted.

Then Adamdea posted results showing a perfect result - he aced it, using ABX via Foobar.

This actually put me under pressure to do the same, and I assume Vital too, so we both tried using ABX tools and then, under more pressure than before, we both aced it too.

This was a challenge that caused differences to be revealed, not masked!

Do you disagree that "challenge" can have an influence on the results then?
Do you disagree that it can have an influence in delivering the opposite result to your & Vitals' experience?
Do you disagree with my reporting of Vital's statements about influencing factors/pressures?

BTW, I believe you are interpreting your & Vital's experience incorrectly. Firstly, it's not any reflection on "how bad" sighted listening is - I've seen you state this before

 

[maxflinn]

This tendency towards a null result is not unusual & Orb I believe, has stated this many, many times before - showing the difficulty of doing a valid blind test & the controls needed!

John, assuming pressure is considered a variable (I don't believe so) - what controls do you think can account for it?

 

[maxflinn]

Do you disagree that "challenge" can have an influence on the results then?
Do you disagree that it can have an influence in delivering the opposite result to your & Vitals' experience?
Do you disagree with my reporting of Vital's statements about influencing factors/pressures?

I do not consider 'challenge' to be a variable, no. It's just far too convenient, IMO.

Vital's experience in terms of the test was similar to mine, but what he concluded from it was very different to what I concluded.

 

[jkeny]

John, assuming pressure is considered a variable (I don't believe so) - what controls do you think can account for it?

Max, there's no point in me explaining this to you when you don't accept it as a factor - I'd just be wasting my time. I suggest if you are interested (but I doubt you are) that you need to go & read more about blind testing particularly for the testing of codecs (MP3 & others) that J_J was heavily involved in developing - the testing for barely perceptible differences.

Anyway, off to bed now

 

[maxflinn]

John, you'd be explaining for the house, not just me. You'd also be giving others a better chance to understand your overall position, IMO.

I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.

You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing

I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).

I doubt we'll ever move much from our respective positions

 

[amirm]

So is long term sighted testing valid or not?

For large differences it is valid for me. For small differences, again for me, it is not.

 

[Whatmore]

For large differences it is valid for me. For small differences, again for me, it is not.

If the differences are large then they would be picked up in short term testing too, so what does long term testing add?

 

[Phelonious Ponk]

John, you'd be explaining for the house, not just me. You'd also be giving others a better chance to understand your overall position, IMO.

I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.

You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing

I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).

I doubt we'll ever move much from our respective positions

The conclusion I came to several pages back. This thread has been running in weird self-abusive circles for a long time, but what it seems to come down to is that John is fully embracing "pressure," as a variable powerful enough to dismiss the entire blind listening methodology. He must want to dismiss it awfully badly, because right here, in the examples in this thread, sighted listening has returned a null result and blind listening, by both the trained and untrained, has accurately identified a difference. The example in front of us, the only real evidence in the discussion, supports exactly the opposite of what John says we should expect from the variable, "pressure." His only other argument is that the method that invites the biases of brand, beauty, engineering methodology, reputation, size, price, etc...in other words, the plethora of biases that are opened up with knowledge of what is being compared...is just as valid as one that eliminates all of those biases. Of course the evidence above counters that as well, but it hardly needs evidence. No one could buy it without a very strong desire to dismiss blind listening.

This was over a long time ago.

Sorry to single you out, John. It really is the position, not the person, I'm talking about. It's just that you're the last man standing on that position. Now I'll bow out again for the time being, as I'm sure this is not over.

Tim

 

[jkeny]

Ultimately I knew this was fruitless but still I tried - just shows you how irrational human beings can be!

Tim, you choose to ignore Amir's description of the psychological pressure he felt & it's tendency to cause him to second guess himself & instead focus on what Max says were his reasons. Sure we all select the evidence that best supports our case. Here's what J_J says " ABX tests (or any kind of scientific test) should not be presented as personal challenges, either. They should be presented with as much support as possible given to those taking the test, including (but not limited to) training, feedback, comfort, appropriate equipment, audio material, and so on."

Hey, Tomelex, "how many times have I made an adjustment and thought that wow, that's going to do this or that to the sound, and then I hear no change when knowledge is removed, its damn frustrating but it keeps me honest." I do this sighted all the time & don't have a problem identifying a failed adjustment.

One last thing, I have done some listening comparisons between DSD & PCM (the same tracks in both- not remastered, AFAIK). There is some difference between them but I don't know what it is, can't put my finger on it, yet. Would I be able to A/B identify them, at the moment - I don't know. I will need some long-term sighted listening to work it out, when I have time. One doesn't need to be an inexperienced audiophile to be beguiled by what they hear initially - it happens quiet a lot. It takes a bit of experience to know that only over long-term listening can conclusions be reached.

 

[jkeny]

John, you'd be explaining for the house, not just me. You'd also be giving others a better chance to understand your overall position, IMO.

I think where we're poles apart is that you consider every possible variable imaginable as equal in terms of importance, whereas I don't.

You appear to believe that any one of them can cause a null result if not accounted for, though apart from 'challenge', you have yet to explain hypothetically how any of the others may do so (barring the obvious two - K + L), rather it's just - one variable unaccounted for - test invalid and worse than even sighted testing

I on the other hand think that knowledge and levels (K + L) are the two variables that once accounted for, elevate any such test way beyond the anecdotal (sighted, long-term etc).

I doubt we'll ever move much from our respective positions

Maybe people should just re-read Amir's description of his running of his tests & examine the elements of pressure

Let me add some perspective on how it "felt" to be taking these tests:

1. It was so easy to forget which track was A or B. In other words, I would hear the difference yet attribute it to the wrong sample A or B.

2. Second guessing was deadly. Often I would find the difference, get it right 5 or 6 times in a row. Then I would wonder if I had correctly classified A or B and would change my mind only to guess 100% wrong from then on.

3. Despite finding the difference reliably, if I let myself, I could "undo" the differential I was hearing. That is, I could read the characteristics of the alternate track into the one I was listening to.

4. As I would get closer to 0% chance of guessing, I would get more and more nervous before hitting the button "next trial." The anxiety was that if I guessed one wrong, it would revert the probability stats and would force me to run a bunch more trials to get down to 0% chance. As the results show, I could identify with 100% certainty the difference yet the nerves would get in the way, causing me to second guess. And at any rate, not being in the best mood to run the tests.

5. I would pass the test 4 to 5 times in a row, only to then keep getting it wrong from then on. If I could not reverse the trend, I would know that I must have been imagining the difference. I would re-listen and still find the same difference. I would then imagine the difference not being there and it would vanish! Then on demand I could bring back the difference in my mind. I would usually abandon the segment as not revealing enough.

As differences get small, our ability to avoid these errors becomes more limited to none. Training and discipline helps but even there, I faced all of the above challenges as I was taking these tests. You can see examples of that in my initial failed attempts in some of these tests.

Put another way, there is something to the fact that when testing for small difference, and the outcome being some kind of race that is scored online, and needing to achieve perfect score, does create an abnormal situation relative to how we usually enjoy music without having to report on it. Instead of the test being purely about what we hear, a lot more gets involved beyond just the ear.

I have highlighted almost all of his post as it all reflects the pressure he was under & what it was doing to his choices.
Are you really telling me that you can't see the effects of pressure in influencing the results towards a null result?

 

[jkeny]

Something is wrong.

Because I can't tell the DSD Vs PCM differences immediately?
I'm not so sure that here is anything wrong - there were a group of us with varying opinions on DSD so no group think in action.
We mainly used a Meitner (which I know internally converts PCM to DSD) & somewhat ensured that the volume levels were matched.
We used a good PC playback source although this was identified as probably the weakest link in the chain & good playback software - for PCM we used MQN & for DSD we used Jplay. Good amp, Jadis & speakers Quad ESLs.

Anyway, as far as I'm concerned, the jury is still out although I do believe that there was some audible difference in that scenario - just what it was I'm not sure, yet!

Or, if you mean something is wrong with my whole argument then touche!

Anyway, no more time today - must go!

 

[Phelonious Ponk]

]Ultimately I knew this was fruitless but still I tried - just shows you how irrational human beings can be!

Tim, you choose to ignore Amir's description of the psychological pressure he felt & it's tendency to cause him to second guess himself

Actually I've done the opposite of that John. I've acknowledged the "pressure" in Amir's methods, and noted that the pressure didn't return a negative result in this case, while casual listening has. Which is the opposite of what you're saying the great "pressure" (which could be easily avoided, by the way) of unsighted listening should cause. You're position is full of holes, john, and in its relentless defense, you've been contradicting yourself for awhile now. it's way past time to give this one up.

Tim

 

[thedudeabides]

In my experience, assuming everything is burned in, one week, two max.

 

[jkeny]

Some pertinent quotes from J_J

"ABX tests (or any kind of scientific test) should not be presented as personal challenges, either. They should be presented with as much support as possible given to those taking the test, including (but not limited to) training, feedback, comfort, appropriate equipment, audio material, and so on."

"And, yes, you have to be comfortable, relaxed, and extremely familiar with everything involved in order to get a sensitive result. One does not just jump into the test willy-nilly."

I don't know why the idea of pressure in a test is being rejected as an influence? Just because Amir was able to overcome it does not mean that it's therefore of no significance. Just because Max couldn't differentiate the two samples sighted & at a later stage could do so in an ABX doesn't show us anything about pressure - it could very well be that at the time Max first tried the sighted test was when no one on the PFM forum (his peers) had returned a positive result, he felt pressure to conform. Later when positive results were returned by some of his peers he suddenly found positive ABX results - who knows, probably not even Max, knows - motivations & pressures can remain hidden to the individual.

These particular anecdotes do not show any evidence that contradicts my statement that "psychological pressure is an influencer in such tests". Do you really feel that it doesn't have an influence?

 

[microstrip]

From RECOMMENDATION ITU-R BS.1116-1

METHODS FOR THE SUBJECTIVE ASSESSMENT OF SMALL IMPAIRMENTS IN AUDIO SYSTEMS INCLUDING MULTICHANNEL SOUND SYSTEMS

It must be empirically and statistically shown that any failure to find differences among systems is not due to experimental insensitivity because of poor choices of audio material, or any other weak aspects of the experiment, before a “null” finding can be accepted as valid. In the extreme case where several or all systems are found to be fully transparent, then it may be necessary to program special trials with low or medium anchors for the explicit purpose of examining subject expertise.

These anchors must be known, (e.g. from previous research), to be detectable to expert listeners but not to inexpert listeners. These anchors are introduced as test items to check not only for listener expertise but also for the sensitivity of all other aspects of the experimental situation.

End of quote

As all blind tests concerning small differences (except Amir's tests ) resulted in the conclusion that the system was transparent, there is still a lot of work to be done!