That might be the case if your arm lacks adjust-ability. But if its truly adjustable and you have a proper protractor then setting up by ear is luck of the draw by comparison
Just got home and I could be outside enjoying this South Georgia spring. I also spotted that little guy in the picture below back at it the same tree as I drove by just now. Want to go get more pictures. Just like audio- always want that better picture. But this is so much fun.
I guess we being audiophiles we can't help ourselves and must belabor points. So ok- I'm going to explain the mechanics of phono cartridges. Please indulge me or skip to the next post.
We have tools to measure and dial in the VTF, ATF, HTA, azimuth and antiskate in addition to test records which can aid in fine tuning antiskate and channel balance. But all of these protractors and such set up the phono cartridge using the cartridge body as the reference point. As far as the tools I use go, the HTA measurement is the only one directly referencing the stylus tip. All other measurements reference the cartridge body. So we have a tolerance stack-up from the stylus to the cantilever to the motor assembly to the cartridge body. Then additional tolerance stack-up as well as measurement repeatability using our eyes with our protractor and other measurement tools. And then as soon as the record starts moving everything changes.
The dynamic forces begin acting on the stylus the moment the record begins spinning. That changes the VTA and HTA as well as the tracking position of the stylus in the groove. For example, drop your phono cartridge onto a stationary record. Observe the side and front of the cartridge. Now start the motor. As the record begins turning notice how the cartridge squats down. You see, the VTA and HTA just changed. And looking from the front, the moment the record starts turning the stylus moves just a bit to the left relative to the cartridge body. So the tracking of the stylus just changed. How do you set all that statically to be just right when it all changes when everything is in motion? You can't. Setting up the cartridge correctly and optimally is an iterative process requiring the stuff between your ears.
Yes, some styli, just like some speakers can sound good across a range of positions but will never sound great at any position. So patience and learning are involved. That's the truth in this hobby- no shortcuts from optimizing a capacitor in an amp or speaker to fitting an acoustic panel on the wall. In the end listening is the key- the final approval.
So one trick I do when seting up VTA and HTA, I use a record that is somewhere between my thickest and thinnest record. And I slowly turn the platter (ie. record) as I adjust HTA. That at least removes some variation for me. And then a stereo test record is useful for setting the anti-skating correctly. I use a mirror and magnifier to adjust azimuth as best I can and then listen for channel balance but expensive electronic gadgets are available for fine tuning that.
One other fun fact about playing vinyl- the phono cartridge and tonearm are a spring/mass system. If the natural frequency is not between about 8-10 Hz then all that set-up work is for nought. Best case the cartridge will track poorly potentially damaging the vinyl or worst case the tonearm will skate across a record. You will eventually come across a record that will make it skip. The phono cartridge should have a specified compliance number which is one over spring rate and the tonearm spec will have an effective mass number. That's used to calculate the system natural frequency and help match a compatible cartridge to a tonearm. And you thought digital was hard.

Taken with Canon R5 MKII and RF 100-500L lens at 500mm.