I have found the same - that even very small changes in tonearm height (~20-50uM) have a noticeable audible difference! Whoever said that you cannot hear a <1 degree change in SRA was wrong (certainly with an FGS stylus profile the difference is not small!).Now as far as a remedy, there are a number of potential solutions.
1) Ignore the issue and accept that your optimization my not be ideal for every LP you play. Unfortunately, this head in the sand approach does not work well for me.
2) I generally "standardize" my playback for 180gm LPs. For 140 gm LPs, I use a 500uM Herbie's ring which is not an ideal solution but its not bad at all and better than nothing. (Its a lot easier to raise the LP height this way than to adjust it by lowering the arm. As an aside, adjusting the arm height was easy to do on the Reed 3P with its calibrated arm height dial, but the Reed 5T does not have such a handy dial so I must use an external micrometer following any arm height adjustment. As I said, the Herbie's ring is the easiest solution and works fairly well and can be rapidly employed or removed
3) For 200gm LP. I suck it up and make no arm height adjustments although I know I can improve the sonics a tiny bit if I raised the arm accordingly (20-50uM). What works best, is trying not to play 200gm LPs at all! But that is hardly a great solution!
4) The best solution. That's easy. Will someone please make a tonearm that is user adjustable remotely for VTA, and tracking force? (I don't care about anti-skate since I think pivoted tonearms should be outlawed due to their inherent tracking errors hence induced distortion,. But if you wanted to make a tonearm that also had remote adjustability for azimuth, I wouldn't complain!)
I don’t know of any arms that have remote VTA adjustment, but a number of arms make it easy to adjust VTA, some on the fly. Nagra, Schroeder, and Andre Theriault are three that come to mind.
My own POV on using tools to make these adjustments is that the Tools can add more error and do more harm that good. I am not trying to start a back and forth - if you’re happy with the sound then that’s all that matters.
Listening to various test tracks (not noise, but actual music!) and making small changes in each setting WILL get you the best results. Setting the SRA to 92 won’t do much if the tool is off by a fraction of a degree (easier than you might think) and/of if the album was not cut at 92 degrees. There is no substitute for your own ears and during actual playback.
It’s not that difficult to set everything by ear.