Nope. Not crazy. 256 kbps is even better. 320? Very few can tell the difference between 320 and CD. The analog out from the Mac is the biggest challenge, but honestly?...they're pretty quiet.
No your not crazy. The devils in the details it's the low level stuff that gets lost or has artifacts. Try using VBR when you encode just set up for a higher data rate and best quality. I use 256K VBR and it sounds very good. You get a smaller file size and you use the bits when you need them.
I wonder if it is a matter of the bit rate or of multi-tasking that keeps young people from appreciating well recorded music. You see younger people doing 5 things at once, and I ma sure none of them well.