Comparative Listening Tests

[Al M.]

Google books has Meilgaard’s “Sensory Evaluation Techniques”, check out Chapter 6.

The easiest test mentioned is the paired comparison test, where all possible pairs have to be presented, or the same/different test, with matched and unmatched pairs. As long as the listeners are told to listen for and note differences between the two components of a pair without knowing to what kind of pair (AA, AB, BB, BA) they are listening to that specific parameter is controlled. Use one listener at a time, control SPL at listening position. There are more potential sources of bias but for a start the above would probably do. If listeners want more time to listen, give them more time.

I think that such simple tests can be done by the layman, but probably the real audiophile is not interested in knowing the objective truth. If he hears a difference and considers that this difference is worth the money, then that’s fine for him. It is most likely that there is a real physiological reaction too, one more reason to go for the fancy cable:

http://www.pnas.org/content/105/3/1050.full

Interesting link, thanks.

Actually:
1) until recently I thought the cable issue was overrated
2) I thought my interconnect was really good, for various reasons
3) initially, I was only interested in another interconnect because changes in rack/system configuration require a longer cable (1.5 m instead of 1 m)
4) the ZenWave Audio D4 interconnect is, in high-end terms, extremely cheap, so there was no 'pride of having an expensive cable' for me

So no, there was no bias towards a 'fancy cable' involved for me. Still, I found very profound improvements by the ZenWave Audio D4 over my Monster Sigma 2000 interconnect, so I bought it.

***

I do agree that careful adjustment of SPL at the listening position, especially when listening to active components, is absolutely crucial. Often just a 1 dB increase in volume can make for a perceived 'better' sound.

 

[KlausR.]

4) the ZenWave Audio D4 interconnect is, in high-end terms, extremely cheap, so there was no 'pride of having an expensive cable' for me

So no, there was no bias towards a 'fancy cable' involved for me. Still, I found very profound improvements by the ZenWave Audio D4 over my Monster Sigma 2000 interconnect, so I bought it.

Without controls, there certainly was bias that different cables sound different. This one D4 interconnect is more expensive than the 50 odd meters of IC I have in my system, so no, in my book that's not cheap at all.

do agree that careful adjustment of SPL at the listening position, especially when listening to active components, is absolutely crucial. Often just a 1 dB increase in volume can make for a perceived 'better' sound.

A colleague of mine once could obtain from a Dutch cable manufacturer some strands of gold-doped silver wire and made a DIY interconnect. When we compared that to my El Cheapo, it sounded different, but we then realized that for some reason it was louder.

Klaus

 

[microstrip]

(...)

I have no problems with others using sighted listening but I certainly don’t consider opinions as being facts.

Klaus

Surely. But experience makes us consider some opinions as more reliable than others, although as naturally we will favor opinions agreeing with ours, there is always some kind of bias.

We should never forget the words of F. Toole,

"And, what might the appropriate loudspeakers be? Two-channel stereo, as we have known it, is not a
“system” of recording and reproduction. The only “rule” is that there are two channels. At the recording end of
the chain, there are many quite different theories and practices of miking and mixing the live performance –
ranging from the purist simplicity of two coincident microphones to multi-microphone, multi-track, pan-potted
and electronically-reverberated mono. They all can be great fun, some of them even very good, but they are all
different.
At the reproduction end of things, loudspeakers have taken many forms: forward facing, bipolar
(bidirectional in phase), dipolar (bidirectional out-of-phase), omnidirectional, and a variety of multi-directional
variants. These, and various sum/difference and delay devices, have been employed in attempts to coax, from a
spatially-deprived medium, a rewarding sense of space and envelopment. In the two-channel world, therefore,
the artists could not anticipate how their performances would sound in homes. It was left to the end user to
create something pleasant. Stereo, therefore, is not an encode/decode system, but a basis for individual
experimentation."

And I still prefer informative reports on sighted listening experimentation to poorly conducted, misleading and inconclusive, blind listening tests. As always just MHO.

 

[audioguy]

I have learned over the years that I have very strong expectation bias. If a product looks or feels better, has great reviews, is highly recommended, etc. then I know that in a comparison, I "expect" it to sound better ...and it usually does.

So when possible, I have a friend assist me in "blind" comparisons. NOT double blind or ABX. Just a comparison between the product I am interested in and the one I already have. The most recent example (several years ago) was for a new SSP. I built a "box" from available components (8 XLR inputs from each of two SSP's) so that I could instantly switch from one to the other.

Setting levels identically (frequency sweeps through each one) was straight forward. All of my equipment is in a separate room so the "blind" part was easy. My friend participated in the test as well as I trust his ears. We listened to full CD's, individual songs as well as very short segments. We used our phones to communicate so that one could signal when they wanted to change sources since we were in two different rooms. And unlike others who commented about the stress associated with these kinds of tests, I had non since I really did not care which product I preferred. [Not completely accurate since I really did not want to spend $25,000 any time in the near or far future on a single new piece of equipment.]

Some high points of the tests (which lasted over several days): When we first started and before we did the blind portion, just to get a feel for what we would be listening to, we ALWAYS selected the newer (and more expensive and better looking) product. ALWAYS. And on another occasion, after we started the blind listening, when one of the two units was accidentally set to 1/2 dB louder than the other, we ALWAYS selected it. Easily audible. Once the blind portion began, then it got really tough. REALLY. We listened in just stereo for most of the trials since trying to compare a system will 7 speakers + subs would have been a bit too much.

During the blind listening, while we "thought" we knew which was which, and did slightly better than 50% when making our choice, we did not pass any of the definitions of a successful blind test. Mind you, the two units involved were a $2500 unit and a $25,000 unit.

I will NOT say the units sounded identical. But I can say, that I will no longer spend that kind of money on a product where audible differences are so subtle (to my aging ears) - or as we like to say, requires a high degree of squinting.

As a conclusion to this test, we enabled the room correction systems and then we correctly guessed 100% of the time --- and yes I purchased the more expensive unit.

I am not suggesting any one else should do this but I do so whenever possible. The first time I did this test was when I swapped 30 stock power cords with 30 very expensive power cords - after I had purchased the 30 expensive power cords. A real eye opener to say the least.

These tests can be a pain in the rear to conduct, but with the kind of $$ at stake, it is worth it TO ME.

Should anyone have nothing better to do, you can read the full description of what we did HERE

 

[Al M.]

Without controls, there certainly was bias that different cables sound different. This one D4 interconnect is more expensive than the 50 odd meters of IC I have in my system, so no, in my book that's not cheap at all.

You can believe whatever you want about my alleged bias. Just don't pretend it's the truth.

A colleague of mine once could obtain from a Dutch cable manufacturer some strands of gold-doped silver wire and made a DIY interconnect. When we compared that to my El Cheapo, it sounded different, but we then realized that for some reason it was louder.

Klaus

I checked the SPL levels. No difference between the two cables.

 

[microstrip]

Without controls, there certainly was bias that different cables sound different. This one D4 interconnect is more expensive than the 50 odd meters of IC I have in my system, so no, in my book that's not cheap at all.
(...)
A colleague of mine once could obtain from a Dutch cable manufacturer some strands of gold-doped silver wire and made a DIY interconnect. When we compared that to my El Cheapo, it sounded different, but we then realized that for some reason it was louder.

Klaus

Let us suppose that someone publishes a scientific paper reporting accepted evidence that cables can sound different. Would it change anything in your system?

 

[KlausR.]

Let us suppose that someone publishes a scientific paper reporting accepted evidence that cables can sound different. Would it change anything in your system?

Since this paper did not, for obvious reasons, examine all cables in all systems I would consider the findings to be true and valid for the experimental setup and the listeners involved.

In his paper about subjective listening tests Jon Risch said:

It must be kept in mind, that the test results, any
subjective test results, are only valid for those papticipants,
on that particular sound system in that particular room, with
the particular musical selections used. It takes a series
of tests, using different sound systems/rooms/music to
determine if a more universal outcome will result.

http://www.aes.org/e-lib/browse.cfm?elib=5538

To answer your question: No, it would not change anything in my system. The best it could possibly do is trigger my interest so that I go and try cables myself, with the best possible controls I would be able to implement as a layman.

Klaus

 

[JackD201]

For me it's really simple. If I struggle to hear any difference, I go with what's cheaper. End of story.

 

[KlausR.]

For me it's really simple. If I struggle to hear any difference, I go with what's cheaper. End of story.

For me it's even simpler: no listening, good quality cable (e.g. Belden), good quality connectors (e.g. Neutrik), soldering iron.

Klaus

 

[pkane]

I think that such simple tests can be done by the layman, but probably the real audiophile is not interested in knowing the objective truth. If he hears a difference and considers that this difference is worth the money, then that’s fine for him. It is most likely that there is a real physiological reaction too, one more reason to go for the fancy cable

Perhaps I'm the opposite of a 'real audiophile', then About 15 years ago, I spent a few thousand dollars on custom interconnects that I absolutely thought made a huge difference (and a whole lot more on speaker cables by the same manufacturer, but that's another story). More recently, after doing a significant amount of blind A/B testing, I replaced these interconnects with a $60 pair of pro-audio interconnects and I couldn't tell the difference while blind. Since then, I have occasionally swapped the cheaper cables for the custom ones to see if a long-term evaluation would reveal something different. And I must say, I always preferred the cheaper cables, so that's what stayed in my system. Am I now banned from audiophile circles?

 

[bonzo75]

For me it's really simple. If I struggle to hear any difference, I go with what's cheaper. End of story.

Same. Where we struggle is not worth knowing the difference. Go back and forth, if you hear a difference, confirm again, then confirm at a later time and in different systems.

 

[PeterA]

You can believe whatever you want about my alleged bias. Just don't pretend it's the truth.



I checked the SPL levels. No difference between the two cables.

Al, I agree, it is very difficult to asses if and how another listener is biased. Knowing how little either one of us wants to spend to upgrade our systems, I might even think we are both automatically biased against the new component on principle of cost alone. It's very hard to know for certain, but it is very easy to just claim someone is biased because of unknowns or loose or poorly described testing protocols. I also think it is difficult for someone to understand if and how he himself is biased. I know I try to convince myself that I am not when I often am.

Yes, you told us that you confirmed that the volume levels were identical. But I remember being quite surprised that one cable clearly sounded louder than the other. In fact, it was the very first difference I noticed almost instantly. The other listener said he noticed the same thing. I usually attribute it to increased clarity and sense of dynamics, but in this case, I think we disagreed about how each cable sounded in those two areas. What do you think accounts for the different perceived volume levels?

Also, did you confirm SPL levels by using test tones before we arrived or by using musical tracks and your SPL meter? I have always found it difficult to assure levels are matched when listening to regular music because of the unsteady quiet and loud passages. Steady tones are easier for me at least because the SPL meter indicates a steady number.

 

[PeterA]

I am not suggesting we stop auditioning components, stop pursuing tweaks or stop listening for significant or subtle differences. I am suggesting that perhaps we should be more realistic and circumspect -- and more skeptical -- about our expressed conclusions. We should attempt to do the best we can do, and to try to remain as intellectually honest as possible, but perhaps we should acknowledge that we may be fooling ourselves about some of our listening conclusions.

Ron, I sense that you are a bit skeptical about all of this listening evaluation stuff. You certainly seem to be questioning its reliability. You are a published reviewer and you traveled around the world auditioning, and then describing in great detail, the various super speaker systems you heard in your search for your next pair of speakers. What system of evaluation do you use and are you confident with your conclusions? How reliable is your memory and do you think it can be trusted?

I appreciate your frankness and introspection especially from the point of view of a reviewer. I don't think I have read such circumspection or skepticism about one's conclusions, especially from someone who reviews audio gear.

 

[Mike Lavigne]

Same. Where we struggle is not worth knowing the difference. Go back and forth, if you hear a difference, confirm again, then confirm at a later time and in different systems.

and sometimes there is not much difference.

but maybe later as our system matures or another variable is introduced a difference will then be uncovered. so assessing gear is not a one dimensional thing. I agree that one needs to revisit conclusions repeatedly before settling on results. or changing the context a little to see if synergy is the issue.

I'm not a believer in blind testing as a process, but accidental blind testing is ok. first rule about listening for differences is having your mind free of stress and being connected to your senses. sometimes I find I am fighting myself to hear something. when I'm in that mode; I just stop worrying about it and allow myself to relax and just enjoy. then I come back to listening for differences later when my head is clear.

my system building philosophy has always been a focus on neutrality, and allowing each recording to be itself with as little sameness as possible. I think that approach (not unique for sure) does make it easier for me to hear differences. or others to hear changes in my system, or even changes or comparative info related to gear they might bring to my system.

 

[Steve williams]

This wasn't what sold Steve on his CMS racks, was it?

He couldn't consume that much alcohol.... or was it that many stories?

I don't drink

 

[bonzo75]

and sometimes there is not much difference.

but maybe later as our system matures or another variable is introduced a difference will then be uncovered. so assessing gear is not a one dimensional thing. I agree that one needs to revisit conclusions repeatedly before settling on results. or changing the context a little to see if synergy is the issue.

I'm not a believer in blind testing as a process, but accidental blind testing is ok. first rule about listening for differences is having your mind free of stress and being connected to your senses. sometimes I find I am fighting myself to hear something. when I'm in that mode; I just stop worrying about it and allow myself to relax and just enjoy. then I come back to listening for differences later when my head is clear.

my system building philosophy has always been a focus on neutrality, and allowing each recording to be itself with as little sameness as possible. I think that approach (not unique for sure) does make it easier for me to hear differences. or others to hear changes in my system, or even changes or comparative info related to gear they might bring to my system.

Yes which is if we repeat across systems and at different points, you will find that you will also do the experiment at times when you are not stressed or not biased.

I agree as systems evolve you will hear more, but the whole point of doing it across different systems is that you will do it across systems at different levels. If you only do it in your system, the findings you had 15 years ago would differ from the ones you have today.

 

[Mike Lavigne]

Yes which is if we repeat across systems and at different points, you will find that you will also do the experiment at times when you are not stressed or not biased.

I agree as systems evolve you will hear more, but the whole point of doing it across different systems is that you will do it across systems at different levels. If you only do it in your system, the findings you had 15 years ago would differ from the ones you have today.

which obviously is the whole system maturity process/room tuning right there.

make it better --> hear more, understand more about what to change, make that change ---> now hear more still, see another problem to solve, do that ---> now compare the degree of truth you can hear compared to the starting point.

this is connected to everything.

while my approach is not to hear gear in multiple systems, I see the value in that too. but for myself, it only matters what gear does in my system. I respect that just because it does something in my system, does not mean it would give the same relative value somewhere else.

 

[bonzo75]

which obviously is the whole system maturity process/room tuning right there.

make it better --> hear more, understand more about what to change, make that change ---> now hear more still, see another problem to solve, do that ---> now compare the degree of truth you can hear compared to the starting point.

this is connected to everything.

while my approach is not to hear gear in multiple systems, I see the value in that too. but for myself, it only matters what gear does in my system. I respect that just because it does something in my system, does not mean it would give the same relative value somewhere else.

In your case that will be fine, but most systems are limited by budget and space, so if someone only relied on what they heard in their system at that time, they might not get a fair appraisal

 

[RogerD]

Since the D4 demo cable is now back on its way to DaveC (I have put in the order for my own), I re-connected my Monster interconnect. Even with the reflective side of the tube traps forward, the loss of resolution is immediately evident. It is even starker than I had anticipated. String tone is nothing like before, it sounds duller, almost all the micro-texture of massed violins is gone, and there is some hardening of tone. Also, the sound is more 'plasticky', more synthetic. It sounds more like a very good mimicry of string tone by a synthesizer. And the problem is not Redbook digital in general, or my DAC specifically -- it's the interconnect.

I think you have to be familiar with a system. There are so many variables that can effect the overall quality of the sound. I think Al M made the right choice and what difference he actually heard was real. The sound marker of resolution or clarity is the only quality that I can distinguish with any certainty. Higher resolving ability tends to be incremental in this hobby and there are possibly a few hundred levels of increments of clarity(Probably much greater). The level of clarity effects the entire reproduction of music on every level, bass,sound stage,dimensionality,tonal purity,saturation,dynamics micro and macro,ect,ect,ect

This has been a learning experience for me and the totality of that experience has spanned a lifetime. I wouldn't expect anybody to be able to come in and sit and listen and discern the differences that I hear in a short amount of time. There's just too many variables and everybody has different focuses. That being said I think Al M and Peter are well seasoned listeners...but who is more familiar with the system being heard..of course Al M is.

 

[DaveyF]

It's been my experience that there rarely are any differences in SQ with one component change that one could call ...an epiphany moment! Small changes that can add up through time and multiple equipment changes do indeed make differences.
Contrary to what so many reviewers have written, the epiphany moment just isn't that common imho.
However, a change in cabling can certainly give one a boost in SQ. It can also be a step backwards, depending on numerous variables. The greater the ability of the system to resolve, the more one can notice differences, albeit the fact that they may be small.
Isn't that what this hobby is somewhat about, always sharpening.....always sharpening!
I think Jack's post above makes the most sense...and is generally how I determine how to keep gear too.