I would argue that riding the volume up during quiet passages, or any manner of playing the material at abnormally high volume for the purpose of pushing the distortions into audibility is cooking the tests.
I agree, and I have learned to say "at normal levels" when discussing jitter and dither and the benefit of 24-bit recording. I'm pretty sure I never said "jitter is never audible in any circumstance," but if I did I meant any normal circumstance. If you have to crank the volume on soft parts such that playing other parts of the same music would blow out your speakers or your eardrums, then it's not a valid condition.
As for blind versus double blind, I think that single blind can be adequate in many cases. I've tested people blind in my own studio to my satisfaction. I knew which file was which, but the listener didn't and I'm confident my facial expressions gave nothing away. The last the I did such a test I was facing away from the listener. I also test myself blind by closing my eyes so I can't see which state the solo buttons are in.
--Ethan