In a post on Al M.’s thread “ZenWave Audio D4 Interconnect” Peter A., describing a comparative listening test at Al’s house, wrote:
. . . Despite your good descriptions of the differences between the D4 and Monster cables, I remain somewhat confused by the events of the recent audition. You essentially conducted an A/B/X test for me and David. Despite me being completely sure that the X cable was the same as B, I failed that test. That opened my eyes to the realities of such tests. Then, later in the evening, when we did the test again with different music, you switched the test without telling us. It was basically an A/B/C test, because you rotated the Tube Traps when we were outside of the room for the X part, changing it into C. That only confused me more.
. . .
Finally, I'd like to add a few thoughts about the A/B/X testing method. Not knowing what I was hearing made for a very intense listening session. During the A/B part, obviously we knew there were two different cables being heard, and we simply described the differences, if any, between the two. That part was fairly easy. The test changed when we heard the third X part. We did not know if you switched cables again or not. Now we were being asked to remember the sound of the previous two cables while listening to unfamiliar music and identify if the third cable was the same as the first or the second. I found this test to be fundamentally different from the first test. We were not listening for differences, but instead testing our recall ability. I thought I correctly identified it, but was clearly wrong. That knowledge of failing the test then influenced my listening for the rest of the evening and contributed to an overall very confusing listening session. I don't know much about blind testing. The fact that I failed it and that you are clearly certain which cable you prefer means either that this kind of testing is somehow flawed, . . . or that bias may enter into your decision, or something else entirely. I don't really know.
I think it is important to attempt to be intellectually honest. I want to commend Peter for being intellectually honest about a confusing situation in which he found himself with the A/B and A/B/X testing of Al’s interconnect cables.
While I have never met Peter in person, we have corresponded via e-mail and talked on the telephone extensively and frequently for several years now. I know Peter to be an
extremely thoughtful, detailed-oriented, careful and conscientious listener. Also, importantly, Peter (unlike me) regularly listens to live music. I am confident Peter's ears are better, and more accurate, than mine. So, after reading Peter’s post, I ask myself: “If Peter gets confused in a test like this, what do we, individually and collectively, even think we are doing when we compare products and listen for changes in our systems?”
I have no answers, only questions. There are a lot of opinions about the merits and problems of A/B and A/B/X testing, and about the pros and cons of short-duration A/B and A/B/X comparisons versus long duration, spend-weeks-with-a-product listening with no back-and-forth comparison.
I think Peter’s experience inclines me to view even more skeptically long-duration "comparisons." As Peter wrote, the comparison, even in the short-duration time-frame of an A/B/X test, became a test of “recall ability,” not “listening for differences.” So what realistic hope does an audiophile have trying to remember how his system
used to sound after he has been listening to a new component for weeks or even months?
Are we partially, or even completely, deluding ourselves when an audiophile visits a friend’s house to listen to music, and then returns weeks later to see if the audiophile can hear a difference wrought by some change the friend made in his system? (Let's not even think about the audiophile's differences in mood, restfulness, alcohol consumption, hunger level, stress, etc., between the two listening sessions.) After listening again weeks later, the audiophile reports hearing a significant difference in response to a minor tweak in his friend’s system (“the soundstage opened up significantly,” “the midrange glare is much less evident,” the noise floor is much lower,” "the highs are more extended," etc.). But if we can't even make reliable comparisons during the course of a single day how can we be possibly think that we can remember accurately what our friend’s system sounded like weeks ago? (Of course, critics of A/B and A/B/X tests argue there are problems and inherently confusing issues with short-term comparison tests which are solved by long-duration auditioning.)
I am not suggesting we stop auditioning components, stop pursuing tweaks or stop listening for significant or subtle differences. I am suggesting that perhaps we should be more realistic and circumspect -- and more skeptical -- about our expressed conclusions. We should attempt to do the best we can do, and to try to remain as intellectually honest as possible, but perhaps we should acknowledge that we may be fooling ourselves about some of our listening conclusions.
