To pan or not to pan...
As a reviewer, it's always a bit of a dilemma when one receives a substandard product.
If there's one weak feature (sonically) out of, say, 4-5 good ones, run the review. For some readers, the weak feature could well be unimportant to their use case, and they'd get to enjoy all the other good points. Perfectionism is not a beneficial approach.
If there are more problems, gently return the gear and spend as much time as necessary to help the manufacturer understand what's wrong with it, if they're willing to listen. Some vendors have crappy products that are selling well, so what do they care? But other companies are more open to suggestions, although few transcend the usual personality issues and are more interested in criticism than accolades.
Many of these companies are Mom and Pop operations, or perhaps just Pop, and it would be inhumane to run a scathing review, to put it mildly. It could put them out of business, but with some coaching, they could improve their present product, or get on the pathway to making something much better in the future. Not every designer has the ability and good luck to start out with a killer design right off the bat.
One well-established manufacturer recently sent me some gear, which, after weeks of working or rather struggling with it, in my opinion, simply failed to live up to any kind of reasonable expecations as to what such a device, at a rather significant price point, should do. (For those of you who know what I'm talking about, not a peep
) Almost nothing worked as advertised, and as one explored the nooks and crannies, it just went from bad to worse. And its substandard-design created problems elsewhere in my system. Great!
The vendor apparently thought that the review process meant regurgitating their standard spiel about how to use the product and what it did. While we don't necessarily reverse-engineer EPROMs and FPGAs to check on the convulution kernel code being used and all that, we do go into every single feature to make sure that it really works as advertised, and thus protect consumers from unpleasant surprises. For this product, such as approach was a disaster. There were gremlins, serious gremlins, everywhere. So back it went. The vendor was disappointed, but they took it well overall. They realized that if they want to play in the high-end arena, the product really has to deliver the goods. Better to have someone say you're not ready, and fix things up, than to stumble badly and not be able to recover.
In the end, my approach is to publish reviews of gear that can be recommended whole-heartedly. Anyone who owns the product should be really happy with it. And the reviews are structured in such as way as to set down standards against which a consumer could compare it to other products.
My opinion is that it's really hard to learn that much from unfavorable reviews. Better to encourage readers to use their own ears, and compare what they read about in a favorable review to what they're hearing in a store or with borrowed gear in their home listening room. And to start asking questions.
