I liked this a lot as well! I hope you don’t mind me quoting, but the below is incredibly poignant. It highlights a huge issue of reviewing without context (room, audio journey, system synergy, musical preferences, etc.) and the swap a box, buy a box, then another mentality the magazines promote just doesn‘t work.
Sure it does. Aside from being perhaps the most subjective performance-oriented industry, this also happens to be perhaps the least scientific-oriented industry. Coincidence?
None of us will ever know a component's true performance potential as it should be impossible. That said, I suspect a swap-a-box strategy is intended to only give us an idea or a glimpse of its potential benefit / demise. I'd venture it's unrealistic to expect much more given all of the variables that make up a given playback system, including the mindset of the one doing the swap as well as our interpretations as observers / readers. Even if there was extra inventory on hand.
“As to the reviews themselves, as I suggested above, the whole process is widely misunderstood, starting with the methodological problems associated with so called ‘Reference Systems’ and the ‘one in-one out’ review technique championed by early high-end publications and still prevalent today. Because you can’t listen to a product, only a system, swapping out one product for another in single set-up only tells you which product suits that system better."
Indeed, we can only listen to a system in its entirety. But that is hardly a justifiable argument for Roy's claim that swapping a single product determines which product suits that system better. Just like every configured system, every component ought to be as unique as individual fingerprints. That goes for rooms as well. Reviewers engaging deeper really only potentially opens up more cans of worms and more questions. And it pays to remember we're already all over the map regarding most any significant topic. Not to mention that some reviewers are more savvy than others regarding certain matters and less savvy regarding other matters and we don't have a program to read which is which. Then of course, there's the reviewer's listening skills. Given all these things and more, how might introducing still more variables lead to better informed conclusions?
“What’s more, if the system is optimised around one product (say, an amplifier) it is unlikely to work at its best with another unit, unless you totally re-set the speakers to accommodate changes in low-frequency balance and the amp/speaker interface. How often does that happen? Rarely if ever – because ‘the change only one thing’ mind-set rejects the obvious, which in turn pretty much undermines or limits the usefulness of most reviews!”
—Roy, from the above interview
Many of us may have a favorite component or two but I'm somewhat unfamiliar with the concept of optimizing a system around a particular product. The concept sounds kinda' silly to me so it's probably best that it not be explained. Again, Roy's argument here should be for the birds for a number reasons.
For example. Re-setting speakers - presumably meaning repositioning / retuning (if they have dials or switches) speakers ought to be completely independent of any component. For the simple reason that repositioning / retuning a speaker has to do entirely with its interface / acoustic coupling to the associated room and not an electronic component. IOW, if after a component swap a speaker has a more musical bass and then the reviewer repositions the speaker for still more musical bass, I'll bet dollars-to-donuts that if the old component was swapped back in, it too would benefit from the speaker repositioning. But also, if a component exhibits better bass, chances are excellent that frequencies above the bass region also ought to improve because components and/or systems should lack any real ability to discriminate between frequencies. Even if only the bass region sound improved.
Another example. It's pretty clear that we're all over the map regarding vibration mgmt methods, designs, materials, knowledge, understanding, etc. If upon swapping in a new component and say the bass sounds a bit more musical, well... Since we're all over the map on numerous things, how does Roy or anybody know that when the new component was swapped in, he happened to better strategically place his Center Stage 2 pucks even a quarter inch so under the new component so as to cause a more musical impact than with the old component? Yes, a 1/4-inch reposition can make a sonic difference on the same component. And in this instance, introducing a new component and the innards including the power supply could be anywhere else than the old component. So from the outside the Center Stage 2 pucks are only a 1/4-inch out of placement but from the new component's innards the CS2 pucks could be repositioned 5 or more inches.
Any one of us could go on with numerous more examples. But the point being is, there are some-to-many black boxes in high-end audio, perhaps even some nobody has discovered yet, and we're fooling ourselves if we think there aren't or that somebody has truly mastered even one of them. And of course there are those areas for which many of us misinterpret or even poorly prioritize.
If any of these and more examples have the potential to be even remotely true, I would think the less one disrupts the apple cart, the fewer cans of worms are opened, the fewer chances for erroneous conclusions, and the more one is able to evaluate something based only on its face value alone. IOW, there's always the potential that a lot more going on beneath the surface of anything we do. Even if it's just moving an interconnect cable 2-inches this way or that way.
