The simple answer is that it isn't that relevant. The listener's emotional connection should be with the music, not with either the devices used to play that music or the format that it's carried upon.
The more complicated answer is it isn't that simple. The emotional baggage that goes into a mix tape or a CD given as a gift by a long-lost loved one has greater significance than the either the music or the medium itself. As hobbyists, there is an attendant - albeit different - emotional baggage that surrounds the equipment used in the replay chain, in a manner akin to what a fly fisher may have over their lures. They help define our position as 'audiophile' in a way that is both non-existent and incomprehensible to 'muggles'.
This is the elephant in the room with any kind of test, but especially blind or double-blind. The more forensic the audio test, the more it strives to eliminate the emotional connection that audiophiles prize. And, unless or until the hobbyist paradigm shifts so heavily away from that deeper link with the technology than mere utility, I suspect there will always be a clash between those who understand that link and those who dismiss it as irrelevant.
As this link has independently sprung up in the computer audio and headphone worlds long before the audio's old guard to get their hands on the categories, I suspect this peculiar not-quite-emotional connection with the technology has crossed the demographic fault line.
The simple answer can be acceptable if we just consider that better equipment statistically produces a better emotional connection with the listener. The issue can be complicated if you listen mainly to the type of music no one knows how it should sound. But if you listen mainly to acoustic music for which you can have good references from real non-amplified performances, the connection to the emotion is mainly caused by clues to the real think - that does not need to replicate the exact physical conditions, but requires some similitude, such as very good dynamics and timbre.
When carrying listening tests I try to look at all these clues - I have found that systems showing them are more enjoyable for me.