If I'm being honest, I think we audiophiles do ourselves no favours by banging the "it's all about the music" drum either. There are plenty of music-lovers out there who get by just fine with lo-fi stuff, and I certainly don't think I "enjoy" music any more than they do. I know the point is that the gear and the sonics are all in service of music, but I think we should acknowledge that the sound itself is an important part of our enjoyment. Let's own up to that. It's not the same as being a gear slut, or loving the toys merely for their own sake, but it's not *all* about the music either.
While I agree that a lot of audiophile enjoyment, including mine, lies in the sound itself, better music reproduction indeed can further musical understanding. With the exception of attending live concerts of course, I have never before been able to follow the inner voicings of a symphony or of a larger jazz ensemble, or to appreciate the complexity of the polyphony of the music from an ensemble playing modern avantgarde, to the extent I am now able to achieve due to the much greater resolution of musical strands on my system over the average system or an iPod. This clearly has a great influence on depth of musical understanding. A famous composer of modern 'classical' music, Stockhausen, also has emphasized the importance of quality music reproduction.
Also, when speaking of the sound itself, more realistic sound reproduction makes you appreciate the musical event as such more. You can identify with the players more and be in the moment of the music itself. This can also bring you closer to the music, as it ceases to be a more abstract phenomenon.
I know that plenty of my non-audiophile friends despise the air of superiority that tacitly goes with the audiophile position, and I can certainly see why. It's perfectly reasonable for people to say "I enjoy my music just fine on my iPod" because they probably do. I've chosen to spend a lot of money on a system that possibly/probably brings me no more enjoyment than their iPod brings them.
I can tell you for certain that after having become an audiophile I enjoy the music more. Granted, the thrill from my old favorite rock bands has in my youth probably more or less been the same as now, or even more than now, and is on my system not much greater than when I hear the music on the radio these days, with the exception of visceral impact, of course. Yet my musical tastes have widened tremendously during my audiophile years, as has my musical understanding, see also above. But that is also due to my musical self-education over the years, which has progressed by leaps and bounds over my youth. I can tell you with confidence, however, that for example modern avantgarde music is far less enjoyable at lesser audio quality than it is on a good system or in the concert hall. Right now I am frustrated by being kept at some distance from the music of a modern string quartet piece by a young composer whose music I admire,
https://www.youtube.com/watch?v=HPdZBFIC6ig
due to the Youtube sound quality than I know would be the case had I seen the piece live (the Boston snow was in the way at the time) or listened to in great CD quality at home. I just know that under these better circumstances the musical tension and differentiation of sound would be much greater.
You can enjoy music on an iPod greatly -- up to a point. If you want to go beyond that point you have to have a good system or go to a live concert. Yes, you can "enjoy music just fine on an iPod", but "just fine" sometimes is not enough.
You can see why they wonder. The only worthwhile rebuttal is to just sit people down and let them listen. That's the real heart of this hobby, it's open to all and it requires no special skills.
Not with the intention to sound overbearing, but music appreciation, at least that of more complex music, does require special skills that can be learned.