I have completed most of the tests I intended to conduct.
1) The SFP modules dominate the results. Whether SFP (1Gb) or SFP+ (10Gb), Startech SFP modules sound by far the best to me, with the best colour saturation and more contrast then the others, they also consistently capture more atmosphere / ambiance and create a more 3-dimensional sound. Single mode sounds better then multi mode. Longer range sounds better then shorter. Please note that officially 40 and 80km ranges need attenuators on shorter ranges to reduce laser transmission power. I have been using 80km modules without attenuators for around 9-10 months now without issues but cannot make any guarantees about longevity.
2) Connecting the server through a dedicated switch, not shared with other devices is a very clear step up in sound quality, a FMC (Fibre Media Converter) accomplishes the same thing. This is nothing new, a lot of audiophiles are already using stacked switches.
3) Creating a separate subnet for your streaming environment can be beneficial. This is more advanced to setup. The easiest way is to use 2 routers, 1 serving your streaming network, the other your home network. When using a submask of 255.255.255.0 you can create 2 subnets by for example placing your streaming devices in 192.168.1.x and your other network devices in 192.168.2.x. You can also go more advanced by means of a managed switch where you can create separate vlans which works out at about the same. The downside is more network devices have a negative effect again. Due to complexity of setup and unpredictable results, I do not recommend going here unless you enjoy playing around with this.
4) The Netgear nighthawk pro gaming xr700 router has some "cool" features like prioritising traffic to your server, but it just does not sound good, and I have spend a full day trying all possible configurations.
5) As for 1Gb versus 10Gb versus 25Gb, ultimately I get the best results from the earlier recommended 1Gb setup, Startech SFP1000ZXST SFP Single Mode 1550nm 80km modules, with the Startech ET91000SFP2 Fibre Media Converter. My best performing 10Gb setup so far is Startech SFP10GBLRST SFP+ Single mode 1310nm 10km modules with a Planet XT-705A FMC which converts 10Gb SFP+ to 1Gb copper ethernet. Interestingly better then several 10G switches I tried. Again SFP modules dominate the outcome, the 10Gb equivalent Startech SFP modules cost around 2000 each, I did not try those and I don't plan to. There are some interesting looking QNAP 10Gb switches on presale right now, I will likely order one of those and run some more tests.
Concluding, my earlier recommendation stands. 10Gb has potential, but there's no affordable configuration available right now outperforming the recommended 1Gb setup. Looking forward to testing the upcoming uptone ether regen and qnap switches.
To add some context, do not overthink all of this, I replaced some Furutech spades and KLEI rca plugs with Bocchino connectors, they have made a far bigger difference then any of the configurations tested above.