The fallacy of Expectation Bias

[Gregadd]

It's a sign of the times. No one wants to talk about results only the method by which such decisions should be obtained. Even on Hydrogen Audio aactual scientific tests are far and few between. If the science is perfect then so would be the results. By the accounts of all the results are by all opinion imperfect.

That is to say no matter what your philosophy you reach an imperfect result in reproducing music .

I have said before that anyone who is willing to submit a scientific result in support of their argument will be well received by me at least. I am talking about an actual scientific test not an argument that scientific test is required.

The question is should we become Hydrogen Audio and require ABX /DBTtesting?

 

[Phelonious Ponk]

This may be true for research work in sound perception, and developments in audio where you are looking for specific aspects of sound, but not to to satisfy the "audiophile myth", as you say, of having listening pleasure using all the complicated methods that subjective perception and our brains allows us.

Yeah, like I said...doesn't apply to audiophiles.

Do you know of any loudspeaker manufacturer that develops speakers listening exclusively to three minutes periods in blind conditions?

a) The ability to discern audible differences between two components has nothing to do with developing speakers. b) The differences between speakers are rarely subtle. If you were trying to differentiate between two different designs, quick switching would not be necessary. Think DACs, amps...that sort of thing.

Tim

 

[MylesBAstor]

It's a sign of the times. No one wants to talk about results only the method by which such decisions should be obtained. Even on Hydrogen Audio aactual scientific tests are far and few between. If the science is perfect then so would be the results. By the accounts of all the results are by all opinion imperfect.

That is to say no matter what your philosophy you reach an imperfect result in reproducing music .

I have said before that anyone who is willing to submit a scientific result in support of their argument will be well received by me at least. I am talking about an actual scientific test not an argument that scientific test is required.

The question is should we become Hydrogen Audio and require ABX /DBTtesting?

I wonder if Mozart or Miles Davis believed in DBT

 

[Phelonious Ponk]

I don’t agree with that statement one whit. Just because information is encoded in the source material doesn’t mean you can hear all of it unless your system is up to the task. For instance, your speakers are missing at least 1 ½ octaves of bass as they are -6dB at 60 Hz. If you didn’t listen to headphones, you would have no idea how much information is there, being played, but not being reproduced.

We were talking about flez007's changing opinion of a preamp over time, Mark, not source material or the bass extension of my system, which I'm well aware of.

Tim

 

[mep]

We were talking about flez007's changing opinion of a preamp over time, Mark, not source material or the bass extension of my system, which I'm well aware of.

Tim

Tim-You responded to his post by telling him this:

If they are audible now, they were audible then, it is your perceptions, attitude, needs, taste...you...that has changed. The inanimate object, unless it is no longer functioning properly , is unchanged. Tim

The whole point is that Flez couldn’t hear what his preamp was doing in his former system because something else was holding it back. You pretty much stated that it was really being reproduced for him to hear when you stated: “if they are audible now, they were audible then…” My point is that his system was holding it back and it wasn’t audible until he improved his system. And that led me to my example of someone who has no real bottom end and has no idea of the information that is there and they never hear.

 

[Ron Party]

I'm sure DBT will work beautifully when the situation being assessed is static, that is, absolutely nothing varies in the state of the 2 situations, A and B: before the switching takes place, during the period of the changeover, and following the switching, for an indefinite period of time. Most research matters do deal in such situations, so there are no real problems.

But if you want to play science, I remember a very enlightening series of lectures where an academic who specialised in the field of measurement pointed out the extremely rocky road you go down if you want to be sure, absolutely sure that you're measuring exactly what you think you're measuring. Many times, he made clear, you were in fact measuring your limits of understanding of the matter, the subtle but extremely important inaccuracies in the equipment, and the fact the measuring process itself was impacting what you were measuring: you were contaminating the very process you were attempting to measure by your interaction with it ...

Frank

So what? If, and that's a big IF, there are further gains to be made in our knowledge base it will come through application of the scientific method. It will not come through application of astrology, aroma therapy, crystal therapy, tarot cards, and as is more germane to this board the egotistical claim of superhuman ability to consciously control one's biases.

 

[Phelonious Ponk]

Tim-You responded to his post by telling him this:



The whole point is that Flez couldn’t hear what his preamp was doing in his former system because something else was holding it back. You pretty much stated that it was really being reproduced for him to hear when you stated: “if they are audible now, they were audible then…” My point is that his system was holding it back and it wasn’t audible until he improved his system. And that led me to my example of someone who has no real bottom end and has no idea of the information that is there and they never hear.

Odd. I don't remember anything in his post regarding changes in his system. I do remember him agreeing wi th me, though.

Tim

 

[Ron Party]

No one wants to talk about results only the method by which such decisions should be obtained.

One without the other is unreliable and not falsifiable. If we don't have that, we don't have fact.

If the science is perfect then so would be the results.

Greg, it is not that science is perfect. Instead it is that the scientific method is THE method by which we study the perceptual world.

The question is should we become Hydrogen Audio and require ABX /DBTtesting?

I for one am not in favor of that. I am, however, not in favor of unsubstantiated opinion which disavows the application of the scientific method to our sense of hearing.

 

[fas42]

So what? If, and that's a big IF, there are further gains to be made in our knowledge base it will come through application of the scientific method. It will not come through application of astrology, aroma therapy, crystal therapy, tarot cards, and as is more germane to this board the egotistical claim of superhuman ability to consciously control one's biases.

I agree 100%, but from where I'm standing there appears to be quite a substantial, egotistically based, belief out there that we do know all there is to know about what needs to be done to give acceptable or even superlative sound quality. And I'm not pointing any fingers here !!

Of course, it is a hard call to claim that one has no biases: I certainly have them, as an example, I enjoy listening to old or poor recordings where the deficiencies are quite obvious; other people listening to the same material at the same time may only be able to take it for a short time. So, I "expect" to get enough out of the material, popcorn noise and scratchy violin tone included, to enjoy the replay, I'm biased towards wanting the sound to please me.

But if the defects pass a certain point, which I know from experience is due to the playback system rather than the recording being "faulty", then my bias will count for nought. In other words, I "expect" the system to work properly, but my "bias" is an extremely small margin of error of how far it can deviate from this level for it to be unacceptable ...

Frank

 

[fas42]

I think really the whole thing is that almost none of us listen to live music anymore, and almost none of us actually listen then to a recording of the live performance we attended, and none of the recordings will emulate what we heard at our particular spot and its all make belive and then we either can make belive to a more or lesser extent, and because we have to "make believe" in the first place, we are already down a slippery slope of expectation bias depending on our pshychologial mood at the moment and stuff like that.

Well, speaking for myself I always enjoy the chance to recalibrate my hearing. Managed to listen to an excellent round of talented, and otherwise, school age performers a few weeks back: vocals, piano, flute, electric guitar. Some of the flute work was excellent, beautifully ringing and "pressurising" the room as some people are fond of saying; and one of the male vocalists had a lovely rich tone - again, "big sound". But the piano? A nicely sized Yamaha grand - dear me, it definitely needed some tweaking: it was in tune technically, but, that extra level of tuning skill was missing, many of the notes were missing the rich harmonics that the instrument was certainly capable of, a person with the right skills was needed to give it some TLC. So, a curate's egg: some notes excellent, other just so-so, and definitely a case where a good system would give one much greater pleasure ...

But piano is tricky, certainly agree with that. Friend's system was humming nicely yesterday, but at first the piano on vinyl was very out of sorts. He had been playing with layers of platter interface materials, the mat; which changed the VTA. Finally got it right: chalk and cheese as many here would vouch. But there was not the slightest difficulty in having us both agree when it was right ...

FRank

 

[Whatmore]

And herein lies a real dilemma in this audio business. Tim is exactly right, that quick switching will identify real differences between 2 versions with genuine, but slight differences. What undoes the value of this is that the differences are not static, they depend upon the relatively long term stabilisation of the system to manifest properly, and the better the system the more this is relevant. You're caught between the devil, and the deep ...

Frank

If differences are not static then what is the point in doing any comparisons at all, ABX or otherwise?

 

[JackD201]

I wonder if Mozart or Miles Davis believed in DBT

Differences between instruments weren't audible to Beethoven in the latter part of his life, that's for sure. One can but wonder what his expectations were.

 

[FrantzM]

I wonder if Mozart or Miles Davis believed in DBT

They didn't have to .. They made music ... The role of our audio designer is to produce gears that reproduce music as close to the original as possible .. I would like these guys to have no bias .. I would want them to use method that help them reproduce music as well as possible without relying exclusively on their too fallible sense of hearing ...

Happy New Year to you ALL !!

 

[fas42]

If differences are not static then what is the point in doing any comparisons at all, ABX or otherwise?

The answer is, life is messy, live with it!

But seriously, it just means that you have to take a different approach. Which of course is what most audiophiles do, they check around what other people's experiences are, they read up a bit, then they take something on board over a period of time and see how it pans out. Maybe the change works for the better, and maybe it doesn't.

Of course, there are degrees in this, some things start out so obviously superior or inferior that it's made easy to decide whether to acquire or persist, or not. Other times, it's hard. I certainly know with my tweaking that it's all about hard, because I'm dealing with behaviour and characteristics that need time to stabilise and manifest; not because I want it to be like that, it's just the way it is! Quite often I need a whole day's solid running to know whether I'm ahead or not.

One "amateur" approach that is reasonably feasible is to thoroughly condition and stabilise the A version of the system with time and running, put on your test recordings and note how they perform, make the change to B, immediately try the test stuff again and note your impressions straight away, and then leave the system to stabilise again with time and running, and finally see how the test recordings perform yet again at the end of this period. Then, do the whole exercise again, starting with stabilising the B version, and then changing to and ending with the A version. This balances the whole process, and should give you some food for thought: maybe it will just prove that any change always sounds better, or nothing varies with time. Or, it just might give you some valuable new understanding ...

Frank

 

[microstrip]

Yeah, like I said...doesn't apply to audiophiles.

Sorry , I thought WBF is an audiophile forum ... Where happily people have some sense of humor .

a) The ability to discern audible differences between two components has nothing to do with developing speakers. b) The differences between speakers are rarely subtle. If you were trying to differentiate between two different designs, quick switching would not be necessary. Think DACs, amps...that sort of thing.

Tim

a) Do you think that developers do not have to prove they really hear what they are doing? I think their bosses do not agree, as they are paying for their salaries and the resources they use. One of the reasons Harman carries its scientific program of sound research is to become more efficient and reliable in their development. It is why they rely a lot on DBT methods.

b) In an ideal world every decision about a sound system should be carried blind to avoid expectation bias. However such tests are very complex and long, and the the simplifying we usually accept make them invalid for the main purpose we discuss (assembling an enjoyable sound reproduction system). Using their flawed results would lead us away of the audiophile desire - getting from stereo what stereo theoretically can not give. May be this can be the topic on another thread.

 

[Phelonious Ponk]

Sorry , I thought WBF is an audiophile forum ... Where happily people have some sense of humor .



a) Do you think that developers do not have to prove they really hear what they are doing? I think their bosses do not agree, as they are paying for their salaries and the resources they use. One of the reasons Harman carries its scientific program of sound research is to become more efficient and reliable in their development. It is why they rely a lot on DBT methods.

b) In an ideal world every decision about a sound system should be carried blind to avoid expectation bias. However such tests are very complex and long, and the the simplifying we usually accept make them invalid for the main purpose we discuss (assembling an enjoyable sound reproduction system). Using their flawed results would lead us away of the audiophile desire - getting from stereo what stereo theoretically can not give. May be this can be the topic on another thread.

We don't actually disagree on much here, micro. Really, the only place I'd differ is on the usefulness blind listening to audiophiles. It's usefulness is limited, but powerful. I would only use it to test things among which the differences are pretty subtle -- quality preamps, DACs, amps operating within their range, cables, digital sources... And formal AB/X methodology isn't necessary, as we're not trying to prove anything to anyone but ourselves. Simply find a way to remove the knowledge of which component is playing, and a way to switch between them fairly quickly. Rapid switching, while necessary for sound scientific methodology, isn't needed here.

Now listen. Can you identify which component is playing, or when the new component is in the signal chain? Can you do it consistently, and significantly more than the 50% that is random? Can you hear a qualitative difference between them? What is that quality?

You won't prove anything. Even a very carefully-conducted single study with multiple listeners, quick switching, etc. doesn't prove anything. But you will most definitely learn something. If this is a new component being added to the system, you will learn whether or not, in your system, to your ears, that component is transparent or additive. I personally think that's very valuable information. If you're comparing two components -- a potential new one with the existing unit, you'll learn whether or not the new component has an audible impact in your system, to your ears, at all.

I can't imagine how those learnings could not be very valuable. The problem, I'm afraid, is that many audiophiles, without trying it, consider it completely worthless. They are absolutely convinced of what they think they hear, and that such an exercise would be a waste of time.

They're wrong.

And yes, I know this is (sort of) an audiophile forum. I should have been more clear before; I was talking about a breed of audiophile, not all of them. That breed is well-represented here, but is not alone. On many forums they are so dominant that a discussion like this one would have been shouted down by dozens of voices in unison, long ago. I'm thankful for the perspective and balance of WBF. Here, we have a chance of learning something.

Sorry for my earlier humorlessness. Here:

Tim

 

[MylesBAstor]

We don't actually disagree on much here, micro. Really, the only place I'd differ is on the usefulness blind listening to audiophiles. It's usefulness is limited, but powerful. I would only use it to test things among which the differences are pretty subtle -- quality preamps, DACs, amps operating within their range, cables, digital sources... And formal AB/X methodology isn't necessary, as we're not trying to prove anything to anyone but ourselves. Simply find a way to remove the knowledge of which component is playing, and a way to switch between them fairly quickly. Rapid switching, while necessary for sound scientific methodology, isn't needed here.

Now listen. Can you identify which component is playing, or when the new component is in the signal chain? Can you do it consistently, and significantly more than the 50% that is random? Can you hear a qualitative difference between them? What is that quality?

You won't prove anything. Even a very carefully-conducted single study with multiple listeners, quick switching, etc. doesn't prove anything. But you will most definitely learn something. If this is a new component being added to the system, you will learn whether or not, in your system, to your ears, that component is transparent or additive. I personally think that's very valuable information. If you're comparing two components -- a potential new one with the existing unit, you'll learn whether or not the new component has an audible impact in your system, to your ears, at all.

I can't imagine how those learnings could not be very valuable. The problem, I'm afraid, is that many audiophiles, without trying it, consider it completely worthless. They are absolutely convinced of what they think they hear, and that such an exercise would be a waste of time.

They're wrong.

And yes, I know this is (sort of) an audiophile forum. I should have been more clear before; I was talking about a breed of audiophile, not all of them. That breed is well-represented here, but is not alone. On many forums they are so dominant that a discussion like this one would have been shouted down by dozens of voices in unison, long ago. I'm thankful for the perspective and balance of WBF. Here, we have a chance of learning something.

Sorry for my earlier humorlessness. Here:

Tim

And what pct of the time is the system actually operating w/in its normal range? Why is it that high power amps say of the 300+ range have a greater sense of ease at all levels than a 100 WPC amp? After all we're only talking 3dB more.

And please tell us what preamps you've compared. Oh one of the biggest differences say between a moderately priced and top preamp is dynamics. I can't understand why you don't hear this or don't you believe that the greater power supply of the top unit (among other things) doesn't make a difference. Try rebuilding a dynaco PAS-3 and use a bigger PS. And it's not subtle. And you're telling us that the differences between tube and SS pres is subtle? Hardly.

And differences between Dacs is subtle??? Wow I don't know what you've heard. Just whether a DAC is tube or SS based makes a big difference. Oh which of the leading dacs such Playback Designs, Wadia, DCS, Stahltech, Neodio, Weiss, Audio Note, etc. have you heard?

And all ABX shows is that short term memory isn't infallible. That's news? Not to mention the conditions and that the number of samples is purposely skewed to produce a null effect. Laughable if you know rudimentary statistics. Or does being born in a garage make you a car?

 

[Phelonious Ponk]

And what pct of the time is the system actually operating w/in its normal range? Why is it that high power amps say of the 300+ range have a greater sense of ease at all levels than a 100 WPC amp? After all we're only talking 3dB more.

I agree. And my guess is that the 50 or 75 or 100 watt amp doesn't have the headroom to hit the small transient peaks that are all over well-recorded music, and you hear the lack of many very small, very rapid incidents of clipping as a "sense of ease."

And please tell us what preamps you've compared. Oh one of the biggest differences say between a moderately priced and top preamp is dynamics. I can't understand why you don't hear this or don't you believe that the greater power supply of the top unit (among other things) doesn't make a difference. Try rebuilding a dynaco PAS-3 and use a bigger PS. And it's not subtle. And you're telling us that the differences between tube and SS pres is subtle? Hardly.

Perhaps we have a different personal evaluation of what is "subtle," Miles. I would definitely agree that the difference between most solid state and tube equipment is not subtle. Before we get into which preamps (comparing technologies from the same age), power supplies, etc. are and are not subtle in their differences, let's test the premise: Dynamics are a function of the noise floor and the output levels. These are quite measurable. You should be able to easily demonstrate your premise to me using two high quality pres from the same school, SS or tube. What I "hear" and what you "hear" can vary broadly. Good measurements will not and dynamic range will reveal itself.

And differences between Dacs is subtle??? Wow I don't know what you've heard. Just whether a DAC is tube or SS based makes a big difference.

Tubes and solid state again. I have no argument there. You don't believe the difference between good DACs is subtle. You're the reviewer, Myles. You have access to them. Pick two thats analog output measurements are very similar (a DAC designed to be colored is another issue altogether) that you think are unsubtly different and compare them without knowledge of which you're listening to when. See how you do.

Tim

 

[jazdoc]

Reviewing this thread, it strikes me that measurements themselves can create an expectation bias.

 

[microstrip]

I agree. And my guess is that the 50 or 75 or 100 watt amp doesn't have the headroom to hit the small transient peaks that are all over well-recorded music, and you hear the lack of many very small, very rapid incidents of clipping as a "sense of ease."

The "small transient peak hypothesis can be easily verified when playing digital recordings. After calibrating my system I found that in my experiences it was never relevant as I never approach the voltage clipping level of my amplifiers. I have been playing for last months with an ARC REF 150. Even choosing levels that are well within the non clipped power capabilities of the amplifier, it does not have the sense of ease of my old 450W VTL MB750s. But is as much better sounding amplifier ...

However as it is easier to use a blindfold that a voltmeter , some people go on referring to that magical transient as the only explanation why sometimes more powerful amplifiers sound better.