"Audiophile" and "Music Lover" totally don't have to be mutually exclusive. But certainly, putting together a high-end system that sounds great on material other than a few "audiophile fireworks recordings" can be difficult.
I believe that if you have trouble finding recordings which sound great on your system, then it's likely you have a significant system imbalance and/or system setup problem that's working HARD against anything other than the most pleasantly recorded simple & boring "audiophile" music. Once you get the gear synergy and setup right, you'll find LOADS of stuff that sounds amazing - 70s, 80s, 90s pop and rock, orchestral, heavy metal, new music, etc. But of course, it can be cost or time prohibitive to experiment by yourself. And rooms at shows are VERY hit or miss. A great dealer can be worth their weight of gold to aid in this pursuit, but they're rare. Having local audio buddies and groups can help out.
To be fair there's certainly a significant share of badly mastered/produced music out there. But you can find plenty of gems in unexpected places. Having fine vinyl & digital sources can double your chances towards finding a great sounding copy of any given album.
When I started out with 2ch in the 2000s, I have to admit I hit on a LOT of frustrations and bad sounding systems (comprised of exceedingly expensive gear) that sounded good only with "amazing" recordings, and terrible on everything else. Perhaps the gear from that era was just particularly bad - part of me thinks this might be the case. Things started turning around for my 2ch enjoyment when:
- I discovered my love for the sound of Tannoy dual-concentric speakers - it's very important to identify speaker brands you gel with, early on
- I got into tube amps - where in fact I got my start with restored 1960's vintage tube amps like the Heathkit W5 and Eico HF-87. The vintage tube amps certainly do sound wonderful with almost any decent recording, but you can find modern amps that will also do this PLUS give you loads more detail and soundstage.
- I got into vinyl, and found that a lot of pop & rock albums which sounded terrible on CD now sounded AWESOME on vinyl
- I moved into my current apartment with solid construction quality versus the shoebox of flimsy drywall and bouncy floors (yuck!)