That all depends on what your definition of "properly driving" is. Transformer outputs may not ideally match the impedance and will therefore change the frequency response of the speaker, but they tend to not go into heavy distortion with low impedance. They just don't deliver as much signal/power there. In the case of a more conventional amplifier with very low output impedance, it could easily run out of current driving this speaker load. When the current demands are too much, this will cause less ideal sound from amplifiers increasing distortion and/or causing clipping prematurely as the power supply sags.
On the plus side it lets audiophiles more easily use their amplifiers as tone controls.