What you say here is not really true about distortion audibility and there are studies demonstrating that even very low levels of high order harmonics are audible and detrimental... This was known over 70 years ago by BBC engineers such as D.E.L Shorter who tried to quantify the impact and much more recently by Cheever.Yes but an SET is already higher distortion, so there's no real purpose in talking about that. IMO all distortion is fairly negligible. We can't really identify distortion itself until it gets pretty darn high (well past 1%). We can however engage the signature of the distortion at almost any level.
High order harmonics add “edge” to the sound and also screw with depth perception because they are perceived as loudness indicators and thus bring sound perceptually forward. I think noise is having a similar effect because the removal often results in people reporting hearing more space and depth.