Top end is clearly driving the profitability in the market. That is because if you make in low volume you cannot sell a lot of items (you just can't make enough) so to meet cost of living + a bit of luxury you need to sell expensive. Not many manufacturers are setup to do high volume in audio and those that are are probably making money on really high volume, low margin inexpensive stuff.
I think you are right that there are middle range buyers out there but their numbers are not so much higher than those at the top of the range. This means there is no way to sell a high volume of middle priced gear. If you can't sell any more 20K speakers in a year than 150K speakers then clearly you have a problem with making enough money at 20K speakers to have a viable business. My solution is to make my audio business a hobby business where I don't need to sell much to make it viable. A manufacturer is in another circumstance where he has to sell gear (unless he is a hobby manufactuerer just turning out a few pieces a year...but then he will be pricey). So what happens, many companies focus on one of the two ends and a lot of stuff in the middle is evaporating or moving to the high end pricewise to attract the top end buyers...Middle class hobbyists are left with buying used if they want something really good for middle money.
Several speaker brands I know make money on both their affordable and ultra-luxury segments.