Do blind tests really prove small differences don't exist?

[arnyk]

That, and sighted tests allow you to continue to believe what you want to believe.

Since sighted evaluations (I can't even call them tests because they usually fail the dictionary definition of the word test) purposefully engage every bit of prior knowledge that you have about the UUT, they exactly help you to continue to believe what you want to believe.

Giving a customer a blind test demonstration of a new component is just about the stupidest thing I can think of for a salesman to do, short of brutally assaulting or killing the customer on the spot. ;-) It is also just about the stupidest thing I can think of for a magazine devoted to subjective audio opinion to do. What could be stupider? Including a letter bomb with the publication? ;-)

When I first started testing blind at home, it was pretty challenging. It's humbling to discover that you can't even consistently differentiate between two components when formerly you thought one (you had paid a heck of a premium for) was clearly superior.

Sic transit gloria mundo for one's belief in strong differences among audio products of basic or better technical quality.

 

[FrantzM]

Tim-Your mailbox is full and my head hurts from reading this thread.

Mark

lol

 

[FrantzM]

Hi

Reading some of the posts from this very interesting thread ... I am currently in Haiti on Business and consult the WBF infrequently .. Great thread ...
It has been repeated quite a bit here that Blind testing is difficult but I would surmise that a simple knowledge-removed test for cable is not ... One simply needs a few friends to dissimulate the cables behind/under something and listen .. Same for the strange and to me incredible claim of audible differences when the DAC is put upside down .. These tests are very easy to conduct you take cable A the 30,000 cable that is so out of this world and compare it to a 100 run of thick gauge cable and "see" if you can hear which one is in the system .. Straightforward ...
What is coming clearly to me is that there is a belief, a faith that little things should make a difference ... if such faith leads to more enjoyment .. fine! Enjoy your cable once yo have seen it is in the system .. but don't berate blind tests and claim them difficult and inconsistent .. on the co
ntrary they constantly tells us that many differences are not reliably perceived if perceived at all ... the truth is not always pretty .. Most of us Audiophiles do not possess the golden Hearing we would like to think we do ...

 

[Phelonious Ponk]

Since sighted evaluations (I can't even call them tests because they usually fail the dictionary definition of the word test)

I prefer to call them "staring lovingly at the gleaming billet aluminum faceplate of your long-desired and hard-earned new component and hearing every dime invested in it," but it is rather long and, I fear, may be a bit slanted by my personal views.

Sic transit gloria mundo for one's belief in strong differences among audio products of basic or better technical quality.

My Latin is pretty limited, but my Google skills are not. This one translates to "Thus passes the glory of the world," or "Worldly things are fleeting."

Indeed they are. I once had, all at the same time, a very nice tube headphone amp, a good SS model, a vintage SS integrated amp in which the headphone circuit was derived from a series of resistors on the mains, and a modern digital receiver thats headphone circuit is an unknown DAC and an unknown op amp. I was absolutely convinced that the tube amp was superior to all. I tested them blind. The cheap headphone circuit of the digital receiver was the only one of the four that distinguished itself. Then I listened sighted again. The difference between the new tube amp and the vintage (midfi) integrated was insignificant, at best. The modern SS headphone amp seemed a bit quieter than both of those, but not much. The receiver had the quietest background, the greatest clarity and detail, and the best bass control.

Totally unscientific blind A/B listening, mind you. I proved nothing to anyone but myself. I've had a few more nice headphone amps through here in the last few years, but the digital receiver still hasn't been beat. I'm just looking for its replacement, because it is such a cheap piece I expect it to fail on me at any moment.

Tim

 

[Phelonious Ponk]

Same for the strange and to me incredible claim of audible differences when the DAC is put upside down

Not incredible at all, Frantz. When I place my Apple Airport Extreme on its edge, it runs cooler and your posts become more fluid and poetic. Works every time.

Tim

 

[microstrip]

The worst thing that blind tests do to listeners is ensure that they are actually hearing a difference as opposed to randomly imagining that one is there. If it isn't about hearing things, why is it so important?

Even worst would be if the differences really existed and the methodology of testing would hide them.

I will not enter a discussion - happily other more credited people are seriously considering these points. But just wanted to present another view. My experience is opposed to yours - if it is not for listening fatigue, longer listening is needed to carry decent tests. Unhappily life is full of compromises. Happily one of my compromises was building a good high-end system.

 

[arnyk]

Even worst would be if the differences really existed and the methodology of testing would hide them.

True only if the differences were large enough to relate to differences in listening enjoyment. In the world of high end audio, there seems to be this almost phallic-like worship of *differences*. Differences come in many flavors, but in the world of high end audio it seems almost like *any* difference is automagically an improvement.

I will not enter a discussion

But you did! Statements like this almost seem like hit-and-run debating. An attempt to assert a point without wanting to be compelled to defend it.

happily other more credited people are seriously considering these points.

But so did you, and you obviously wanted your considerations to have weight, even if you don't want to be accountable for them.

But just wanted to present another view. My experience is opposed to yours - if it is not for listening fatigue, longer listening is needed to carry decent tests.

You again misunderstand me. There is actually a hierarchy of differences. The kinds of difference that lead to listening fatigue are generally quite large compared to JNDs.

If you think about it, the very idea that a difference near the JND would lead to listening fatigue is not logical.

Unhappily life is full of compromises. Happily one of my compromises was building a good high-end system.

So you say.

 

[Johnny Vinyl]

So you say.

Ouch!

 

[FrantzM]

Even worst would be if the differences really existed and the methodology of testing would hide them.

<snip>

And what would be an alternative methodology to uncover those minutes differences ? ..... Sighted "testing"? Highly unlikely!

 

[arnyk]

Not incredible at all, Frantz. When I place my Apple Airport Extreme on its edge, it runs cooler and your posts become more fluid and poetic. Works every time.

Tim

Not only that, but I've got a dozen sighted evaluations to *prove* it! ;-)

 

[microstrip]

True only if the differences were large enough to relate to differences in listening enjoyment. In the world of high end audio, there seems to be this almost phallic-like worship of *differences*. Differences come in many flavors, but in the world of high end audio it seems almost like *any* difference is automagically an improvement.

But you did! Statements like this almost seem like hit-and-run debating. An attempt to assert a point without wanting to be compelled to defend it.

But so did you, and you obviously wanted your considerations to have weight, even if you don't want to be accountable for them.

Apologies, I did not come to WBF to have a boxing fight . You won. (No icon, WBF still misses the clapping icon). May be WTF will have it.

 

[JackD201]

I don't take AP or MAR to be models of real world application of test results. I see that AP has a reasonable role as they enable people who produce audio products to make sure that their products are consistent with desired performance levels. But many extant peforamnce levels are orders of magnitude into overkill.

That I am being asked to explain real world usage mystified me. I think it is self evident. Real world usage is here you use audio gear to listen to music, spoken word, and drama. If an improvement in performance has meaning in terms of real world usage, then it will at least make a reliable change in the sound quality of those things.

I think that performance significantly in excess of JND levels is warranted because of the things we do in audio production. But performance that is 100 or 1000 times better than JND levels is a waste of everything done to obtain it.

Emphasis mine.

This is what I've been saying here for a year now. Blind tests are the best tests for finding out what's GOOD ENOUGH. Is there anything wrong with that? Hell no. It saves manufacturers and consumers valuable resources. Here's the catch though. I have to take the test. A DBT in a magazine review is just the findings on a test procedure and panel about whom I know nothing about. That to me is as useless as reading a review of someone whose biases I know nothing about.

My system continuous to involuntarily make my hair stand, dance and do fist pumps. I have a tertiary system that in all honestly I could live with. It is good enough. So yes my main system is overkill, but I love what it does to and for me. Anybody have a problem with that? Before calling me out go look in the mirror and ask yourselves if you've never gone overboard on anything because you felt you deserved to give yourself a treat.

 

[Ron Party]

True but a tad incomplete, no? Aren't they not only the best tests but the only reliable, repeatable tests?

Now you say "here's the catch", perhaps allowing room for inference that it is a bad thing. To those who deny that science applies to matters audio, i.e., faith trumps science, I suppose it is a bad thing. Yes you (or I) have to take the test. I don't see it as a catch. I see it as liberation. I see it as THE method to learn the truth, i.e., the truth for me. If I can or cannot pass a blind test in my room with my gear, I am wiser for the effort. It is the embodiment of rational thought. It is the essence of humility.

The fact of the matter is blind tests do far more than just establishing a baseline for what's good enough. For example, if one truly was interested in finding out if flipping a DAC upside down, demag-ing a CD, switching out a power cable, using cable elevators or a Clever Little Clock, etc., generates an improvement in sound quality, one could perform a blind test in one's own home at one's own leisure with one's own gear and music.

The reviewer can't do it for you. Another person's blind test may or may not be dispositive. But if one is willing to do a sighted evaluation in one's room, why isn't one willing to do a blind eval? As Amir has said, it isn't hard to do. After all, we keep getting bombarded by the trite phrase, "trust your ears". If one isn't willing to trust one's ears, I sure as heck would not trust his/her ears.

 

[JackD201]

True but a tad incomplete, no? Aren't they not only the best tests but the only reliable, repeatable tests?

Now you say "here's the catch", perhaps allowing room for inference that it is a bad thing. To those who deny that science applies to matters audio, i.e., faith trumps science, I suppose it is a bad thing. Yes you (or I) have to take the test. I don't see it as a catch. I see it as liberation. I see it as THE method to learn the truth, i.e., the truth for me. If I can or cannot pass a blind test in my room with my gear, I am wiser for the effort. It is the embodiment of rational thought. It is the essence of humility.

The fact of the matter is blind tests do far more than just establishing a baseline for what's good enough. For example, if one truly was interested in finding out if flipping a DAC upside down, demag-ing a CD, switching out a power cable, using cable elevators or a Clever Little Clock, etc., generates an improvement in sound quality, one could perform a blind test in one's own home at one's own leisure with one's own gear and music.

The reviewer can't do it for you. Another person's blind test may or may not be dispositive. But if one is willing to do a sighted evaluation in one's room, why isn't one willing to do a blind eval? As Amir has said, it isn't hard to do. After all, we keep getting bombarded by the trite phrase, "trust your ears". If one isn't willing to trust one's ears, I sure as heck would not trust his/her ears.

(each paragraph in the corresponding order because I can't seem to get the hang of this quoting thing)

Depends on what you mean by repeatable Ron. Change the panel and the results may change too. An unexpected human variable in one of the same panel could also have differing results. The smaller the sample the more pronounced the change in the curve is. It does not diminish the fact that DBTs are the best tests for finding what's good enough. It's the best already. For this purpose then what else matters?

I do agree that it can be liberating. In a prior post I did recommend that everybody try it. My beef is being told I shouldn't be able to hear a difference because some panel somewhere couldn't. It isn't even a matter of acuity. If I'm to take the results of a DBT done somewhere without knowing and understanding the structure of the test and the profile of the panel, doesn't it stand to reason that I would also be accepting the results on nothing but faith?

Those are pretty extreme examples Ron. Even die-hard subjectivists roll their eyes when these things are mentioned. Yet again if that clock was emitting something measurably then the next step would be to blind test it to see if it mattered one bit. Again it follows the "good enough" logic. If similarities reach a point where you can't tell the difference past 50/50, why spend more? One could always get it because it probably tells good time, and prefers the sticker to the Casio or whatever it was it was tacked on but I don't think that's what we're talking about. I just don't like being told I can't because someone else couldn't. We all agree how important acoustics are, it's an area that is hardly ever debated. Given differences in test venues alone, I don't see how someone tested in an environment different from mine could even make that assertion and make it stick. There are just too many variables assumed. That isn't good science either and could also be construed as taking a dump on the scientific method.

I don't like the trust your ears cliche either Ron, like Frantz says it could lead down the slippery slope where suddenly a cymbal sounds like it was made out of playdough instead of metal just because it sounds pleasing. I think the issue here is about being honest to yourself about what you hear. That's why panelists are trained after all. They aren't paid to be blind, they're trained to know what to listen for. Then and only then can the guess work be minimized strengthening the statistical under pinnings of both positive and negative outcomes. If you've noticed not only have I never dissuaded anybody from taking a blind test, like I said I actually recommend it. What I'm NOT recommending is that it exclude everything else.

 

[naturephoto1]

True but a tad incomplete, no? Aren't they not only the best tests but the only reliable, repeatable tests?

Now you say "here's the catch", perhaps allowing room for inference that it is a bad thing. To those who deny that science applies to matters audio, i.e., faith trumps science, I suppose it is a bad thing. Yes you (or I) have to take the test. I don't see it as a catch. I see it as liberation. I see it as THE method to learn the truth, i.e., the truth for me. If I can or cannot pass a blind test in my room with my gear, I am wiser for the effort. It is the embodiment of rational thought. It is the essence of humility.

The fact of the matter is blind tests do far more than just establishing a baseline for what's good enough. For example, if one truly was interested in finding out if flipping a DAC upside down, demag-ing a CD, switching out a power cable, using cable elevators or a Clever Little Clock, etc., generates an improvement in sound quality, one could perform a blind test in one's own home at one's own leisure with one's own gear and music.

The reviewer can't do it for you. Another person's blind test may or may not be dispositive. But if one is willing to do a sighted evaluation in one's room, why isn't one willing to do a blind eval? As Amir has said, it isn't hard to do. After all, we keep getting bombarded by the trite phrase, "trust your ears". If one isn't willing to trust one's ears, I sure as heck would not trust his/her ears.

Hi Ron,

I have 2 of the I guess original Clever Little Clocks that I won from Geoff on Audiogon. For the life of me I can't tell you if they really work. If you believe Geoff you would have to get them basically out of the house so that they don't have any affect on the sound. This could be a bit tough as a blind test.

I also now have one of the HiFi Tuning Disc Demagnetizers. I think that it works (more noticeable on some discs), but I have not and really do not have any intention of running a blind test which would again be a tough comparison. That is especially a problem since every pressed CD or digital disc supposedly sounds different. I am not sure about burned copies. Possibly you could make a comparison in a blind test that way, but there is the lag time in switching the discs themselves.

Rich

 

[amirm]

Some blind tests are incredibly difficult to set up as was just mentioned. Others are easy. I fault people for not running the easy ones. But appreciate them not taking on the difficult ones .

Someone once asked me why I didn't blind test something. I said it is a simple formula for me: how much I could learn relative to how much work it is. Earlier I asked if people did double blind tests for grocery items. Clearly no one did. Reason is that they don't really care to learn something new there and it is work to do them. Accepting such a thing should not be a sign of people being against science but rather, being human. 99% of our disagreements here are not because of our differing audio views but who we are at people and what motivates us to do something versus not....

 

[Ron Party]

Jack, great post!

Oh, BTW, except for the clock, every one of those "pretty extreme examples" come from members of this very forum who have extolled their virtue.

 

[Old Listener]

blind tests on groceries

Some blind tests are incredibly difficult to set up as was just mentioned. Others are easy. I fault people for not running the easy ones. But appreciate them not taking on the difficult ones .

I agree about the easy tests. Every audiophile should have his own BS filter to guard against hearing differences that strain credibility. Blind tests should be part of the filter.

Someone once asked me why I didn't blind test something. I said it is a simple formula for me: how much I could learn relative to how much work it is. Earlier I asked if people did double blind tests for grocery items. Clearly no one did.

A few dozen posts ago when Mark (mep) was slagging Aldi, I thought of the Trader Joe's chain which Aldi bought some years ago. Many people in the SF Bay area are big fans even though Trader Joe's sells mostly store brands (generics).

I lived in Pasadena when Trader Joe's was finding their niche in the late 60s. They started doing blind tests of wine, then food and then household items like soap. Their employees were the testers. They would write up the results in the Trader Joe's newsletter. Then they would feature the winners with prices lower than regular supermarket prices. A clever way to compete while limiting the amount of choice in their stores.

Bill

 

[amirm]

What a funny coincident. I am down in LA and we were chatting about different stores and started to talk about Trade Joe's! Didn't realize they did blind testing. I associated them with carrying higher quality things but cutting out the middleman and brand name to save money and pass that to customers. As you say, their selection is limited though so we usually go there for specific things. It is impossible to shop for groceries there as they don't carry a lot of usual items.

 

[Old Listener]

... Trade Joe's! Didn't realize they did blind testing

I don't know whether they still do blind testing now.

As you say, their selection is limited though so we usually go there for specific things. It is impossible to shop for groceries there as they don't carry a lot of usual items.

A couple of weeks ago, I was in Target buying hay fever pills and decided to try to get the groceries I needed in their very limited grocery department. I hate schepping around town going from store to store. I managed to get food for several days of meals in spite of the very limited selection. My wife visited a new Fresh Choice store near us the next week and managed to get food for a couple of meals in spite of their limited choice. Compared to those stores, Trader Joe's is different but no worse. Trader Joe's frozen quiches are our favorite "break-glass" food.

Bill