Do blind tests really prove small differences don't exist?

[Ron Party]

I provided the context Ron. You took it out when you quoted me . Let me replay it.

Myles asks me if Harman is so great at blind testing of speakers, how come MBL 101s outperform theirs. What am I supposed to tell him? I am pretty sure Harman did not blind test their speakers against the MBLs. Does this mean:

1. The Revels outperform them anyway? On what basis am I supposed to make that claim? Yes, Harman has done extensive research into speaker preferences. But how can we take this to the extreme and answer a specific question like he asked?

2. Myles is wrong and the MBLs aren't as good as Revels. How do I prove this fact?

There isn't anything to prove with this example. And again, as I previously stated, don't forget the basics. Blind testing doesn't *prove* anything.

One of the biggest limitations of our blind testing of audio is that we do so little of it relative to incredible array of equipment out there. We can say many sound the same but that still leaves so much equipment out there.

Reality is then that we rely on sighted evaluation by experts to augment our swiss cheese of knowledge here. There is no other practical way to do that.

IOW, since we can't always blind test, it is OK to rely on sighted evaluations. In my book that is a cop out. In your recent discussion with Arny and A.J., amongst others, you stated you were in their camp. I don't think so. Instead, I see you playing it extremely close to the vest.

In this discussion there seems to be an emphasis on blind testing by manufacturers. This is troublesome for at least 2 reasons.

Let's take cables as an example. The audiophile world would love to see reliable, repeatable blind testing by Nordost, Transparent, etc., showing purported proof of a difference. As my friend dizzman repeatedly states, if any of these manufacturers, for example, could point to even one such result reliable, repeatable positive finding, that would be a fantastic data point, would lead to even more sales, and quiet this on-going debate. The $1M challenge - IIRC I think that is the amount - would disappear. Absent this data, however, the audiophile world is left with "trust your ears" which implicitly means "don't trust your ears" and since there purportedly is nothing absolute in audio then for some faith trumps science.

Second, there is significant room for the end user to run blind testing. Its value definitely is not limited to manufacturers. While the end user may not be able to blind test speakers, he/she certainly can do this for other products. There can be no doubt an end user can perform blind testing for cables unless one's faith is to dismiss science. I asked you to show me a method that was more reliable and you responded with only occasional blind testing. As I stated before, bias does not go on holiday. Neither does self-delusion in thinking one is immune.

Bottom line is one either believes the perceptual world can be studied or one does not. Taste preferences, e.g. (as in your laundry list of Coke vs. Pepsi, etc.) are irrelevant. Purported *real world* practices are inherently more unreliable as a methodology for studying the perceptual world.

Keep in mind that I don't advocate things I can't measure or prove with science. That is the pre-requisite in my book. So the tice clock or whatever, is not something I am about. If I can show with objective data that a difference exists, then a lot of snake oil is taken out of the equation.

I think Arny and A.J. would beg to differ, at least in part. The part is: what is the nature of the "objective data"? Is it just math, or is it an actual blind test? We know there are measurable differences in cables. The math by itself doesn't cut it. The real world question always is: is it audible?

 

[Orb]

The problem is Ron, how would you explain-define for end users how to limit factors and variables that skew blind testing?
JA has touched a fair bit on this, so has Amir and others, including myself.
The challenge is understanding and limiting interractions that you do not want, and even being able to know when underlying interractions occur that can give wrong results.

From what I have seen but it is limited when considering what out there, most real world blind tests related to perception do not involve explicit identification tests where listener must selectively and accurately identify X against the two floating references (A and B).
I appreciate I could be very wrong here so if Kal or JA has anything to add about blind testing relating to general practices and the application of blind AB/X in the scientific-academic and engineering-manufacturing world.

The reason I raise this is simple, the discussion really needs to consider what are the actual specifics of blind testing and their scope-limitations-potential for results.
Not all blind test processes are equal especially when involving perception and behaviour (that also includes preference), for the many reasons outlined in previous posts by several posters.

Anyway here is a good paper from the Harman Group involving comparing multiple speakers, in this process they are looking for statistical significance relating to perception and preferences.
Differences in Performance and Preference of Trained Verses Untrained Listeners in Loudspeaker Tests:
http://www.aes.org/tmpFiles/elib/20110523/12206.pdf

Touching on some of what we have raised earlier is this excerpt to show just how complex and amount of variables involved, without capturing listener behaviour this would had been missed:

The rank orders of preference for loudspeakers P, B,and M were identical in both four-way and three-way tests.
However, the relative magnitude of preference between loudspeakers P, B, and M was somewhat smaller in the three-way tests.
The differences in the loudspeaker ratings were reduced between 0.51 and 1.3 preference ratings
compared to those measure in the four-way tests.
A possible explanation for this difference is that a scaling effect occurred when the number of loudspeakers increased from three to four.
The listeners may have expanded the separation and range in order to accommodate
the additional loudspeaker.
Another cause could be well-known context effect described earlier [31]–[38].
Contextual biases highlight a general principal in sensory judgment that human observers act like measuring instruments that constantly readjust themselves to the context or
expected frame of reference.

This quote is just one example and should not be seen as the only variable, which ties in with JA's perspective IMO.

Cheers
Orb

 

[Ron Party]

I see nothing in your post that invalidates blind testing, ABX being only one of many methods of blind testing. Moreover, as I asked Amir, if you've got a better methodology for studying the perceptual world, the entire scientific community is all ears. But don't come back at me with faith.

 

[amirm]

There isn't anything to prove with this example. And again, as I previously stated, don't forget the basics. Blind testing doesn't *prove* anything.

It is not me who has to remember that. I have it tattooed on my forehead . It is folks who think that one or two blind tests generates data for all people, all content, and all equipment. Sometimes this is true, but we lack the data as you say to claim that with conviction.

IOW, since we can't always blind test, it is OK to rely on sighted evaluations. In my book that is a cop out. In your recent discussion with Arny and A.J., amongst others, you stated you were in their camp. I don't think so. Instead, I see you playing it extremely close to the vest.

You can't chop down half of my statement and then run with it Ron. It is like me saying you are a great lawyer but wear cheap cloths. Take out the first and where would that leave you?

I said we use three things:

1. Measurements. We always measure. Measurement may give us too much date or too accurate of data but it is wonderful in how fast and objective it is.

2. We use trained/expert listeners. These are people are paid to be right. They don't have a job otherwise. It is like a test engineer in a company. He is not paid to say everything works right. He is paid to be critical. Yes, trained listeners can be wrong and catastrophically so at times. But overall, they are far more right than wrong.

3. Blind testing. This is to guard against the above misses for trained listeners and also to gauge the general public. To give a video example, when we developed our video codec, we opted to have higher resolution with slightly more artifacts than the other way around. General surveys showed the latter is preferred for everyday people as opposed to experts.

The above is what everyone uses. Let me say it again. That is how the real world works. You can wish that the world runs on double blind tests but it doesn't. Those tests are too slow to run, take too long, and at the end, have their limitations due to test fixtures, budget, time, etc.

If a trained speaker evaluator came and told me XYZ speaker has distorted high frequencies, I put high value on it. I will not dismiss it out of hand because he didn't run his test blind.

Now, if random Joe showed up here and said the same, I would take it for what it is worth. I would pencil it and go and test it to see if he is right or wrong.

Let's take cables as an example. The audiophile world would love to see reliable, repeatable blind testing by Nordost, Transparent, etc., showing purported proof of a difference. As my friend dizzman repeatedly states, if any of these manufacturers, for example, could point to even one such result reliable, repeatable positive finding, that would be a fantastic data point, would lead to even more sales, and quiet this on-going debate.

Isn't that the topic of this thread? The differences don't get any smaller than cables. So we need to settle the current debate before we go and ask them to run it.

BTW, have you run a cable test? It is not too hard. Find a source with dual output and run them to a pre-amp and then compare. Level matching is not necessary because if there is a level difference, the game is already over . If you get positive results, swap the cables, randomize and try again. If you are worried about a sample of one, invite others to the test.

Absent this data, however, the audiophile world is left with "trust your ears" which implicitly means "don't trust your ears" and since there purportedly is nothing absolute in audio then for some faith trumps science.

Why are you arguing with this point of view with me Ron? Does it look like I don't examine the design of a product? Its measurements? Who built it and what evaluations they did? At my company we carry only two lines of speakers: from Harman group and Paradigm. Both believe and conduct blind testing together with the other two tools I mentioned.

What I am hearing from you then is don't tell them subjective evaluation is good for anything. Well, i can't do that. International standards such as AAC audio codec were developed using tons of subjective and sighted evaluations. You name a company that designs their products 100% using blind testing and nothing else and I will change my position. Until the, I can't push the data to where it can't be pushed.

Second, there is significant room for the end user to run blind testing. Its value definitely is not limited to manufacturers. While the end user may not be able to blind test speakers, he/she certainly can do this for other products.

I love to see a thread then that objectivists show and outline all the blind tests they have run. Let's see how much they can stand the scrutiny . When I outlined a blind test to see if turning off the video circuits on an AVR or source can make a difference, these are the comments I received from the objectivist camp in another forum:

Me: "Here is a way to experiment with this idea. Does your AVR have a button to turn off the video circuits and front panel display? If so, hook up a CD/DVD player using S/PDIF coax cable. Play something quiet with lots of ambiance. Now turn up the volume good and loud (or else use headphones). Play it with the video and front panel circuits on (on both the source and AVR) and then turn them all off. Do you hear a difference?"

Poster: "This hardly looks like a proper listening test to me. If our correspondent is such an advocate of DBTs, why is he advocating such a crude sighted evaluation?"

Me: "I always think of cheap and simple exercises people can run at home to learn more about their equipment and the limits of their hearing. "

Poster:"Your cheap and simple exercise is a sales pitch in disguise. By giving you the benefit of the doubt and presuming that you simply don't know any better, I'll spare the other readers the ugliness of questions about your character."

Me: "Sales pitch for what? I asked them to test their own gear. There is nothing to buy. It is very likely that most of them don't hear any difference which is fine in my book."

Poster: "Most front panel on/off switches have discernable on and off positions. Thus the proposed experience is likely to not be the least bit blind in actuality."

Me: " The front panel switches are momentary toggle switch. You push them, they do something. You push it again, it does the reverse. There is no mechanical feel to it as it is sampled by the microprocessor and acted upon. Are you thinking of 70s hi-fi gear?

But yes, if you can cheat by feeling the button, don't do that. You don't want to cheat yourself into believing something. "

Poster: "No, its a highly flawed evaluation and one that has a predictable outcome - peole are falsly convinced of your mystical claims."

Me: "I suggested someone do blind test and you are unhappy with me?

Anyway , you should not be concerned. Most people won't hear a difference and will actually support your view. For select few, they will have an interesting realization. "

Poster: "If family and friends are involved but visible, then you have a single blind test, which is simply a flawed double blind test."

Me: "Which is miles ahead of no test which is what you are offering. And let me break the news: ALL audio tests are flawed. The good ones are simply less flawed than others. Period. Do we throw out everything they find as a result of it? I hope not. That would leave us with nothing. What we do then is to examine the test and decide how valid it is for us and what out of it can be used. "

Another poster: "Sorry, I don't see anything that makes the data gathered in such way more meaningful than random chatter seen all over the net. They are both subjective observations."

Me: "Let me understand this. You run your own randomized blind set of trials at home. You use your own ears to see if you can identify if turning off some of the circuits in your equipment makes a difference. Let' say you do and in all three instances, you find out that you prefer the circuits off. You can't possibly tell me you are not smarter on that day than you were the day before about such things .

Let me say that if you had run such a test and post here, my respect for anything you would say would go up incredibly even if it disputed the position I was taking. Indeed, my top goal in these discussions is to get people to go and learn the science and experiment. It is sharing of that data which moves our collective knowledge forward. Otherwise, it is a waste of time and words."

Poster: "You don't push toggle switches. You push push-button switches. You're not telling a consistent story....I see the bigger picture where you advise against doing proper listening tests, and have been making false claims all along about your own alleged blind tests. "

Me: "I have asked three times and I will ask again: in what way a user is harmed with blind testing of his own equipment."

Another poster: "This is not a proper test. To correct this problem, the choice presented to the test subject must be randomized, so they are comparing X with Y at each trial. But they don't know whether X is "processing on" or "processing off", and this relationship must be randomized for each trial. Then the number of trials can be chosen so the probability of picking "processing on" or "processing off" by guessing is negligibly small, not 0.5.

The problem this causes though, is the creation of a false impression that a given conclusion was reached by a properly designed experiment. I've seen people's posts where they claim to be able to distinguish two subtly different configurations in a blind test, but upon further examination one finds the probability of reaching this conclusion by guessing is 0.5, not negligibly small. So the potential harm of this approach is the spreading of misinformation under a guise of "scientific inquiry" - a common technique used by high-end audio vendors."

Me: "I am proposing textbook blind AB testing. Nothing more, nothing less. Listener is unaware whether they are listening to the system as is, or with the modification (video turned off).

What is the #1 reason we use blind testing? To remove experimenter bias. "I know video circuits do extra stuff so they must by definition worsen the sound." So I convince myself that when I hit the button, all of a sudden the sound gets better, the highs clearer, the mids engaging, the soundstage widens, and bass gets tighter .

Blind AB tests reduce the impact of such bias substantially. You don't know which state you are listening to so it is hard to try to cheat. Don't believe me? Set up an AB test and tell me how you managed to cook up the results so that it comes out the way you intended.

The choices are randomized in blind AB tests just the same. There is no hidden reference so we lose value in that, but randomization is there. I suggested multiple tries just the same.

There is no misinformation spread. The testing is done by a person for their own use. I have said this repeatedly and it keeps getting ignored. I was abundantly clear that you cannot take your results and expect to get published in AES or change the direction of these debates.

Once more, this was not put forth as the preferred scientific method a product should be evaluated. It was put forth as a method for members here, to decide for themselves if it is at all probable that turning off these circuits can make a difference. No more."

Me: "You do the blind test and time after time, no matter how many times you repeat it, the one that you like is with the circuit off. Question: which way would you leave your equipment? With the circuit off or on?

Second scenario: your car manual says that you put premium gas in it. You one day decide to put regular in it. You drive and it and it feels exactly the same. You do that 10 times in a row. Assuming I guarantee you that there is no damage to your car (and buy it from you if there were), will you pay extra for premium fuel or go regular? Crossing fingers that the proverbial car analogy doesn't come back to hunt me."

Another Poster: "Well, this is a different question. The earlier discussion centered around whether the test was "proper" (my words, maybe "valid" is a better choice), and this question relates to whether one would use the results of an invalid experiment anyway, knowing full well it was invalid. I plan on using an HT pre/pro for an audio-only system (just for bass management and possibly room correction), and I will surely turn video processing off if I have that option, just in case it might make a difference. But that is my own personal choice, and I would not try to make claims about its effect in a forum." [Car analogy not answered]

Me: "I am not asking if you would guess to get there. I asked, if you ran the experiment as suggested, would you be more inclined to follow it. You are taking the test away and then say you might do it. I didn't ask you about that.

But it is interesting that based on even less information than having run the test, you would follow that technique.

It seems to me there is such a fear that people would go and run the test, and post their outcome here. Why? How is that any different than random assertions that all modern gear sounds the same? How valid was that for Anry to say? "

Another poster: "The test is bogus, so I would ignore the results. You can't have less than no information."

Me: "You can certainly do that. Here is how I view it. All else being equal, I like to have my front panel and video circuits on. The former lets me see what the device is doing and the latter, lets me feel good that when watching say, a music video, my audio is not degraded base on existence of video.

So I perform the above test. I ran it on a number of SACD and CD players and found it to make a difference. So when using them as pure audio source, I turn off video and front panel. Some are nice in that they turn on their display when using the remote. That's nice. Others don't and I put up with it.

I also tested two high-end DACs and couldn't tell the difference between the font panel on and off. So I leave them on and can tell what they are doing as a result. This is good for example to confirm the sampling rate of the music you are listening through the PC.

So what you call less than no information, I find very useful in what I do with my hobby. I realize I can't convince you and that is cool. I write for everyone, even though I am just answering you ."

I will stop here . See how much fighting was put forth to not bother with a simple blind test at home by the objectivist camp? You want to say I don't belong in that camp? Nothing would please me more than not being associated with folks who say the above.

There can be no doubt an end user can perform blind testing for cables unless one's faith is to dismiss science.

And what if they find a difference as I did above? What was the reaction?

I asked you to show me a method that was more reliable and you responded with only occasional blind testing. As I stated before, bias does not go on holiday. Neither does self-delusion in thinking one is immune.

As I noted, measurements do take a holiday from bias. And trained listeners do that most of the time.

 

[Orb]

I see nothing in your post that invalidates blind testing, ABX being only one of many methods of blind testing. Moreover, as I asked Amir, if you've got a better methodology for studying the perceptual world, the entire scientific community is all ears. But don't come back at me with faith.

Ron can you show me where I am using faith and not talking about actual specifics?
My request to you was not flippant but actually important:

The problem is Ron, how would you explain-define for end users how to limit factors and variables that skew blind testing?
JA has touched a fair bit on this, so has Amir and others, including myself.
The challenge is understanding and limiting interractions that you do not want, and even being able to know when underlying interractions occur that can give wrong results.

Everything Amir/myself/JA/microstrip has said can be seen within that one Harman paper; it is worth reading.
We may be missing each others point-perspective on this.
Thank you
Orb

 

[Ron Party]

Ron can you show me where I am using faith and not talking about actual specifics?
My request to you was not flippant but actually important:

Everything Amir/myself/JA/microstrip has said can be seen within that one Harman paper; it is worth reading.
We may be missing each others point-perspective on this.
Thank you
Orb

I have read the paper.

No one who is being intellectually honest says blind testing is perfect. Having stated this, anyone who is rational will say blind testing is far more reliable than sighted. If one is going to dismiss blind testing while admitting that sighted testing is unreliable, then all that remains is faith, i.e., the perceptual world cannot be studied. I'm sorry, but I'm a rationalist. I don't subscribe to faith.

 

[Ron Party]

It is not me who has to remember that. I have it tattooed on my forehead . It is folks who think that one or two blind tests generates data for all people, all content, and all equipment. Sometimes this is true, but we lack the data as you say to claim that with conviction.

Huh? I'm sorry, Amir, but you should know better than to state I'm making a claim and then refute it. Strawman. Take that argument to those at AVS. It fits like a glove there.

I said we use three things:

1. Measurements. We always measure. Measurement may give us too much date or too accurate of data but it is wonderful in how fast and objective it is.

No problema.

2. We use trained/expert listeners. These are people are paid to be right. They don't have a job otherwise. It is like a test engineer in a company. He is not paid to say everything works right. He is paid to be critical. Yes, trained listeners can be wrong and catastrophically so at times. But overall, they are far more right than wrong.

3. Blind testing. This is to guard against the above misses for trained listeners and also to gauge the general public. To give a video example, when we developed our video codec, we opted to have higher resolution with slightly more artifacts than the other way around. General surveys showed the latter is preferred for everyday people as opposed to experts.

Again, no problema. Like I stated, the discussion about blind testing seems bent on a manufacturer's use of blind testing for product development. Some do, some don't. Of course those who believe in science would prefer that more of them do.

But, why can't those here at WBF who claim, e.g., that flipping a DAC upside down makes for an audible difference, perform their own blind test? One could conduct that test in their own home with their own choice of music at their own leisure for as long as they feel they need. We've had this discussion before, Amir, in the context of whether blind testing in and of itself is invalid because of some sort of test induced stress, and you answered logically and IMO correctly by stating there is no stress. Don't make me search the 3K+ posts of yours in order to refresh your recollection.

BTW, to answer your question, I have participated in blind testing of interconnects, speaker cables and power cables. I completely agree with you. "It is not too hard". Therefore the debate is settled: we indeed can ask anyone to run their own blind test when it comes to things like cables.

I didn't bother to quote or address the rest of your post/argument, all of which is, again, something you need to address with certain folks over at AVS, not me.

 

[Orb]

I have read the paper.

No one who is being intellectually honest says blind testing is perfect. Having stated this, anyone who is rational will say blind testing is far more reliable than sighted. If one is going to dismiss blind testing while admitting that sighted testing is unreliable, then all that remains is faith, i.e., the perceptual world cannot be studied. I'm sorry, but I'm a rationalist. I don't subscribe to faith.

Ah ok I understand.
Then we were misunderstanding each other, because I am not defending sighted testing and saying all blind testing is flawed, just that it is incredibly complex and difficult to do a reliable test and this is even more difficult for a typical end user as they cannot deal with the factors and variables, as explained by JA,Amir,myself,etc.
This does not mean I do not support blind tests, I do, but their focus and limitation-context needs to be understood.
However the issue is this; who decides what is deemed a valid blind test for subtle differences?
This is important as there are different blind tests, and different levels of managing and monitoring a listener's behaviour and heuristic approach.
I guess I am saying that for me I am unsure whether it is possible to do the subtle comparisons blind test that are based upon the listener absolutely identifying a product against multiple floating references that flip(AB against X as an example), for the reasons I outlined.
Furthermore the paper I linked also has further comments of its own relating to ensuring strong blind tests.
So we are not disagreeing I would say (apart from possibly my view on absolute selection against floating reference, which I would like more validation for as I mentioned much earlier involving a slightly different test).

Thanks
Orb

 

[mep]

Tim-Your mailbox is full and my head hurts from reading this thread.

Mark

 

[arnyk]

Arny felt that my jitter tests were invalid because the Miller-Dunn J-Test signal isn't dithered. I explained that the elegance of this test signal is that because both signal components are even-integer fractions of the sample rate, there is no quantizing distortion. Thus everything that you see between the signal-bins in an FFT plot of the DUT's analog output while processing this signal stems from the DUT (provided your analyzer's ADC has a greater resolution than the noisefloor of the DUT).

That's about half the argument. JA seems to like to make a big point out of *explaining* things to me, like the Dunn paper that introduced the J-test. I think this is the third time he's played this card, and the third time I had to point out that I read that paper long before I said booh! about it.

At any rate one of my hot buttons is making far-reaching conclusions based on tests that are asymmetric with real life. I can understand how a 11 KHz tone might end up in a real world recording, but that J-test signal is something else!

 

[Stereoeditor]

Arny felt that my jitter tests were invalid because the Miller-Dunn J-Test signal isn't dithered. I explained that the elegance of this test signal is that because both signal components are even-integer fractions of the sample rate, there is no quantizing distortion. Thus everything that you see between the signal-bins in an FFT plot of the DUT's analog output while processing this signal stems from the DUT (provided your analyzer's ADC has a greater resolution than the noisefloor of the DUT).

That's about half the argument. JA seems to like to make a big point out of *explaining* things to me, like the Dunn paper that introduced the J-test. I think this is the third time he's played this card, and the third time I had to point out that I read that paper long before I said booh! about it.

I don't recall you saying you had read the Julian Dunn paper. But that's not the point; at that time you were publishing jitter measurements with a dithered 11kHz tone and finding that you were getting inconsistent results yet you were criticizing my published jitter measurements on the grounds that I was not using dithered test data.

At any rate one of my hot buttons is making far-reaching conclusions based on tests that are asymmetric with real life. I can understand how a 11 KHz tone might end up in a real world recording, but that J-test signal is something else!

The undithered Dunn-Miller test signal - not 11kHz but Fs/4, ie, 11.025kHz for measuring a CD system, 12kHz for 48kHz data and so on, with an LSB-level squarewave at Fs/192 - is formally diagnostic for jitter with a system where the clock is embedded in the data. As with the chirp or MLS signal used or speaker response testing, it is irrelevant whether or not such a signal can be found in real-world recordings. What matters is that it reliably and repeatably tells you something about how the DUT behaves.

John Atkinson
Editor, Stereophile

 

[arnyk]

I don't recall you saying you had read the Julian Dunn paper. But that's not the point; at that time you were publishing jitter measurements with a dithered 11kHz tone and finding that you were getting inconsistent results yet you were criticizing my published jitter measurements on the grounds that I was not using dithered test data.

And predictably, instead of addressing the asymmetry problem, you again treat us to the following lecture:

The undithered Dunn-Miller test signal - not 11kHz but Fs/4, ie, 11.025kHz for measuring a CD system, 12kHz for 48kHz data and so on, with an LSB-level squarewave at Fs/192 - is formally diagnostic for jitter with a system where the clock is embedded in the data. As with the chirp or MLS signal used or speaker response testing, it is irrelevant whether or not such a signal can be found in real-world recordings. What matters is that it reliably and repeatably tells you something about how the DUT behaves.

While some like to build their worlds around learning *something*, I'm far more interested in learing something that is relevant to real world usage.

 

[Phelonious Ponk]

Tim-Your mailbox is full and my head hurts from reading this thread.

Mark

This made me laugh out loud. I'll go empty the box, Mark.

Tim

 

[microstrip]

Tim-Your mailbox is full and my head hurts from reading this thread.

Mark

Mark,
You are not alone - I am reaching for an aspirin!

 

[JackD201]

I'm actually enjoying how far astray the posts have gone from the title of the thread

 

[microstrip]

Trying to put is back in line

I am not an expert in sound reproduction psychoacoustics, but if by chance I find a nice short paper on this subject that is not filled with technical jargon and I can understand, I do not ignore it. One that I read some years ago discussed differential listening tests, intended to distinguish sounds. In its discussion it separated the physiological part, the associative and the memorization part and how they correlate and how each would contribute to the test results, and also the famous blind tests. One interesting point it discussed was that in sighted tests, the listener could identify much more of the different characteristics of the distinct sources as he was able to focus in listening and not in memorizing. This can be one of the reasons many people prefer to carry the sighted tests.

 

[Orb]

And predictably, instead of addressing the asymmetry problem, you again treat us to the following lecture:



While some like to build their worlds around learning *something*, I'm far more interested in learing something that is relevant to real world usage.

Arny then your issue on this subject is more with Julian Dunn because it was him who modelled and worked out how his signals-test can be compared to real world usage, and covered it enough for it to be of use to Audio Precision and Miller Audio Research.

Can you expand what by real world usage means to you as it may be possible to provide examples.

Edit:
Here is the link to the book for those who are customers of AP:
http://audioprecision.com/search/show?query=julian+dunn
Cheers
Orb

 

[arnyk]

Arny then your issue on this subject is more with Julian Dunn because it was him who modelled and worked out how his signals-test can be compared to real world usage, and covered it enough for it to be of use to Audio Precision and Miller Audio Research.

I don't take AP or MAR to be models of real world application of test results. I see that AP has a reasonable role as they enable people who produce audio products to make sure that their products are consistent with desired performance levels. But many extant peforamnce levels are orders of magnitude into overkill.

That I am being asked to explain real world usage mystified me. I think it is self evident. Real world usage is here you use audio gear to listen to music, spoken word, and drama. If an improvement in performance has meaning in terms of real world usage, then it will at least make a reliable change in the sound quality of those things.

I think that performance significantly in excess of JND levels is warranted because of the things we do in audio production. But performance that is 100 or 1000 times better than JND levels is a waste of everything done to obtain it.

 

[arnyk]

Trying to put is back in line

One that I read some years ago discussed differential listening tests, intended to distinguish sounds. In its discussion it separated the physiological part, the associative and the memorization part and how they correlate and how each would contribute to the test results, and also the famous blind tests. One interesting point it discussed was that in sighted tests, the listener could identify much more of the different characteristics of the distinct sources as he was able to focus in listening and not in memorizing.

The much-hated and misunderstood business of quick switching and listening to carefully selected short snippets pretty well eliminates as many of the memory issues as can be eliminated. Practically speaking that kind of listening is so much more sensitive and memory-independent than normal listening that it can be criticized for having too much sensitivity. IOW, they allow people to hear differences that they could rarely if ever hear in normal listening.

This can be one of the reasons many people prefer to carry the sighted tests.

No, the reason why people prefer to do sighted tests regardless of how reasonable or silly the particular test is, is because sighted tests are simply easier to do. If the differences are small, sight allows the creation of the fiction of sensitive perception without the listener actually having to hear anything.

Most sighted evaluations have many other signficiant failings such as the absence of level matching and time synchornization.

The worst thing that blind tests do to listeners is ensure that they are actually hearing a difference as opposed to randomly imagining that one is there. If it isn't about hearing things, why is it so important?

 

[Phelonious Ponk]

No, the reason why people prefer to do sighted tests regardless of how reasonable or silly the particular test is, is because sighted tests are simply easier to do.

That, and sighted tests allow you to continue to believe what you want to believe. When I first started testing blind at home, it was pretty challenging. It's humbling to discover that you can't even consistently differentiate between two components when formerly you thought one (you had paid a heck of a premium for) was clearly superior.

Tim