Real life is quite loud...distortion in playback makes things seem much louder than they really are. Horns relatively less so and thus you can listen closer to live sounds without "sounding" like it. The fatigue from listening comes more from the distortion and not just listening loud. When I recorded my ex (playing a real, honest to goodness Strad) in my apartment the pressure waves from that made my ears pulsate! It was loud but clear and amazing!
Bad noise control from the power supply/AC is typically the worst offender of making fatiguing sound. But I've found certain polyester caps can pack it on, too. Here's a good visual of why that might be. (personally I think the shots on the scope, while supposedly too small to matter, exactly represent typical cap sounds) Also I've heard too many exotic parts sound rather fatiguing with some gear (weird capacitors, wire, etc) - that goes for SS and Tubes. I've heard many painful tube amps, and many SS amps that I couldn't tolerate. In some cases a single power cord is enough to make me want to turn off a stereo.
Tests show speakers with less distortion tend to be less offensive over time, but that's kind of tentative... purchase patterns and claims don't necessarily show that. And more so low distortion speakers may really make a painful stereo shine bright like mad.