Well, let's examine one of these embedded systems and figure out if that is the case. I believe you are a fan of Bryston BDP-2, yes? If so, I searched and could not find a high-resolution picture of its inside. But there was one at reasonable size that I could analyze and annotate. Here it is:
As shown, the board on the left is the power supply and the board on the right is the computer. Below the power supply board we see a rather large toroidal transformer leading one to think this devices uses a linear power supply. Indeed we see a rectifier and a pair of regulators/transistors on the right. But smack in the middle is a Mean Well (chinese company) 5 volt switchmode power supply! Its input is mains at 120 volts so it will be whaling all out, triggering worries people have about such devices.
From the low resolution I can't quite tell whether it is there just to power the front panel or the main CPU board. But whatever it is, it is generating noise.
The output of the power supply board is that gray cable carrying power to the CPU motherboard on the right. Note the path that is taking. It goes right over that little board that produces AES/EBU and S/PDIF digital output from the machine!!! That cable is unshielded as power cables usually are so not only could it be carrying noise from the power supply, it can also pick up noise from the computer side and radiate that out onto the digital audio output.
This is pretty poor design hygiene for an audio equipment. It is what a computer designer would do that doesn't know audio. The fix would have been dead simple: a slightly longer cable travelling in the front of the CPU board.
Another miss is where it goes on the CPU board. Notice that is almost touching the aluminum silver heatsink for the CPU! I have indicated that with a second arrow. That heat will degrade that cable over time. Is it a big deal? No. But these are the things that separate great designs from working designs.
Back to power supply, we see a coil on the right on the main CPU board. That tells me that there is another switchmode power supply on the CPU motherboard generating all the other voltages that board/CPU require. I can't be quite sure without higher resolution image or access to the board but there is high likelihood this is the case. If so, then there is no notion of a "linear power supply" here. You have two switchmode supplies creating the noise that they do which gets telegraphed to that power cable and run over the digital audio output.
The main "boot drive" for Linux is a compact flash. This is a common approach in embedded systems which assumes that the drive only gets used to boot and for infrequent access by the system. It is a rather poor choice in my opinion here. This is an audio server and that storage is used to cache/store all the activities of the system. That CF file will get far more use here than it would be in a typical embedded system. That causes two issues: slower access/response from the system and eventual demise of that Flash memory due to limited write cycles.
When we build computers we use SSDs which are designed for high-performance and much higher endurance. CF cards may not apply although you could use them just the same if you really wanted to.
At high level, there is no magic here. There is a computer and power supply, running an operating system. Its hardware design is "OK" but nothing remotely like a high-end piece of audio electronics. There should be no assumption that by mere design in this manner, it can provide superior digital audio out of its digital ports.