Usually do not see criticism this harsh.

[DWR]

You stick to your guns Doug, maybe if more reviews were as honest as yours a perspective buyer could actually use the review to somewhat evaluate the equipment reviewed, because that is certainly not the way reviews have been done up to this point. Also the thread should have been titled Usually don't see criticism this HONEST. Thank you Doug, and keep up the good work.

Dan

 

[Jeff Fritz]

Only products likely to be outstanding and important for consumers to know about are even considered for review in the first place, and secondly, if it turns out they aren't, the review is cancelled.

Then it is not a review, but an advertorial:

http://en.wikipedia.org/wiki/Advertorial

 

[Nicholas Bedworth]

My fundamental approach is educational, and is designed to make readers able to stop relying upon what they read in reviews, and make their own competent decisions

 

[Phelonious Ponk]

This statement sums my feelings on the subject:

We should be acting as consultants to the to the reader, not to manufacturers.

I rarely read reviews any more, because so many of them fundamentally fail at the mission of critical evaluation and reporting. Evaluating a product from a well-established manufacturer that is on the market at a substantial price point, finding it is utterly substandard then returning it to the manufacturer with consultation notes instead of warning readers, is misguided at best, an abdication of responsibility at worst. And I apologize that it sounds like I'm picking on you, Nicholas. I'm really not, I think this is pretty broadly the state of audio criticism, not the exception. I would think choosing the products to review based on innovations, expectations, newsworthiness, then giving it a real, critical evaluation would be a better approach. It would certainly serve readers better. The exception might be products from start-ups and very small, low-profile companies - cases in which the review would only be of interest to readers if the product were unexpectedly excellent or some kind of breakthrough.

Tim

 

[Kal Rubinson]

I rarely read reviews any more, because so many of them fundamentally fail at the mission of critical evaluation and reporting. Evaluating a product from a well-established manufacturer that is on the market at a substantial price point, finding it is utterly substandard then returning it to the manufacturer with consultation notes instead of warning readers, is misguided at best, an abdication of responsibility at worst. And I apologize that it sounds like I'm picking on you, Nicholas. I'm really not, I think this is pretty broadly the state of audio criticism, not the exception. I would think choosing the products to review based on innovations, expectations, newsworthiness, then giving it a real, critical evaluation would be a better approach. It would certainly serve readers better. The exception might be products from start-ups and very small, low-profile companies - cases in which the review would only be of interest to readers if the product were unexpectedly excellent or some kind of breakthrough.

Tim

While we apply due consideration before agreeing to accept a product for review, Stereophile's policy is that any product accepted must be reviewed regardless of discovered faults or their source.

 

[Nicholas Bedworth]

The audio industry version of SPQR

@ Tim... I think we're on the same wavelength here on several points. Let me expand on some of them.

We are definitely trying to act as consultants to the readers, helping them get up to speed on how to listen and evaluate. Don't rely solely or even primarily upon reviews, use them as a point of reference, and find out for yourself if you agree. You're (the consumer) spending the money. You're the one who the industry needs to satisfy.

Now, giving the manufacturers feedback ultimately should benefit consumers as well. All the constituencies here are closely interrlated. And the family is very tightly-knit. We better get on.

Let me give you a recent example.

We got an initial production run unit of a particular product from a well-known (but small) company. It wouldn't synch up to certain other products, under certain circumstances, and both products enjoyed well-deserved good reputations. Uh, oh: Consumer nightmare on the horizon. it should work, but it doesn't.

To make a long story short, after getting everybody together by e-mail, talking with other purchasers of the initial run, etc., it turned out that the problem stemmed from a mistake in the "recommended design" specifications (wrong part value) from an OEM manufacturer. Ooops.

It took a couple of weeks for the designer to find someone at Gigantic International Electronics, Inc., who understood the component well enough to track down the mistake. And it would have been hard for the original designer to have corrected this problem, without feedback from reviewers, consumers and in this case, competing firms. Even if it's "someone elses" fault, both companies get "tarred" in the mind of the consumer if there's an integration problem.

Result: The Initial units were withdrawn from the field, the mistake corrected, new units go out, and things are much better for all concerned.

BTW almost all the gear that comes through here is from companies that are tiny, as mentioned earlier. Most "large" high-end audio companies are small (or very small) businesses by most criteria. Outside of forums such as this, the general public has never heard of them.

For the high-end audio industry to survive, consumers have to have a positive experience or they'll spend their time and money on something else that gives them more pleasure, rather than aggravation and frustration. Sure, an approriately negative review could be helpful, but I'm not sure the messages delivered generalize.

Thus, appropriately and realistically written positive reviews, in my opinion, are better educational tools. If the marginal products are weeded out of the review process, then it will never be necessary to be "diplomatic" about shortcomings, which in my opinion, doesn't serve consumers well.

Just present the really strong, high-performance and high-value products correctly and accurately, compare them with competing products that should be auditioned, and the consumers should be really happy.

As you say, trust your ears.

 

[das]

Hi Nicholas,

Can't agree with you on this. Sounds more to me like fear of saying a bad word. If a product's bad it's bad -- no reason to cover that up.

DS @ SoundStageNetwork.com

 

[KeithR]

I think the approach Nicholas took is a HUGE disservice to readers. As has been previously noted, it a piece of garbage is presented for review and it is not worthy of it claims, then that needs to be published. Not only will it keep an unsuspecting public from purchasing the item, it should also serve notice to other manufacturers that submitting crap has its risks. And ultimately that benefits consumers and the industry.

And I must be getting a different subscription to Stereophile. Not many negative reviews that I can remember.

Kudos to those (very, very, very few)who are willing to tell it like it is.

A few not so spectacular reviews from Sphile off the top of my head:

Ayre universal player
Magico Q5
Einstein preamp
Audio Valve conductor preamp
Bryston 7BSST monoblocks

 

[KeithR]

Hi Nicholas,

Can't agree with you on this. Sounds more to me like fear of saying a bad word. If a product's bad it's bad -- no reason to cover that up.

DS @ SoundStageNetwork.com

I agree with Doug, Nicholas. and most products up for review have already been released, so i don't buy the fact that a year later the manufacturer somehow will re-release the product that actually sounds good.

 

[jazdoc]

This thread has certainly taken a compelling and interesting turn!

As an owner of several products from boutique manufacters (Galibier, Durand Tonearms, Experience Music and Daedalus), I understand how potentially devastating even a single negative review or blog post could be. However, if you voluntarily submit a product for review, the reviewer and his/her readers should assume that this is an actual consumer product. The manufacturer should not send a beta product to a reviewer to solicit input and advice and expect a formal review, especially if the bias is to only formally review excellent products. I would posit that knowing what a reviewer doesn't like can be equally valuable to long-term readers as understanding what products that particular reviewer finds compelling.

OTOH, I do see the benefit of any manufacturer sending the products to reviewers for input and feedback. Obviously reviewers have the benefit of hearing a variety of equipment that most of us could only dream about. Reviewers hopefully have enough technical background to analyze parts and manufacturing quality in comparison to other, similar products. Certainly it benefits all of us to have manufacturers producing the best sounding (and quality) product.

I would suggest that any manufacturer submitting a product to a magazine/reviewer specificy if this is a submission for formal review or simply to solicit non-reviewed feedback. Any product submitted for formal review should be reviewed as is; good, bad or indifferent. A product submitted for non-reviewed feedback should by definition, not be reviewed. If that product is later reviewed, disclosure of the previous relationship (as well as the sonic results of any recommended changes) should be noted.

Whadda think?

 

[Phelonious Ponk]

While we apply due consideration before agreeing to accept a product for review, Stereophile's policy is that any product accepted must be reviewed regardless of discovered faults or their source.

Works for me.

Tim

 

[Phelonious Ponk]

Nicholas, I mean no offense, honestly, but we are just going to have to agree to disagree on this one. I'm out of the game for the time being, having settled into a great comfort zone with a completely integrated active system that leaves me nothing to shop for but music and, someday, perhaps a sub woofer or two. But I wandered through audiophilia for years from a mid-sized city in the Southeast, and even as the largest, most centrally located city within two states, at our peak we had three or four audiophile shops here, representing a fraction of the available options. That's not an uncommon condition, and it's becoming more common all the time. Many, many music lovers are forced to shop for playback equipment on the net. Thankfully, many online retailers offer pretty liberal return policies, allowing us to audition gear in-home. It's time consuming. It ties up cash flow until choices are made. It's a bit expensive when return shipping is involved. But it is effective. More effective, actually, than listening in store. But to that audience, the greatest service the reviewer could offer is a good ear, a well-equipped bench and a solid partner in thinning the herd before the shipping begins.

I can surf the web's audiophile boards and find fanboys for any and every product out there. From a critic, I need much more.

Tim

 

[Kal Rubinson]

This thread has certainly taken a compelling and interesting turn!

Indeed.

I would posit that knowing what a reviewer doesn't like can be equally valuable to long-term readers as understanding what products that particular reviewer finds compelling.

Very astute.

OTOH, I do see the benefit of any manufacturer sending the products to reviewers for input and feedback. ......................................

I would suggest that any manufacturer submitting a product to a magazine/reviewer specificy if this is a submission for formal review or simply to solicit non-reviewed feedback.

While the potential benefit to the manufacturer is apparent, Stereophile reviewers are specifically not permitted to do the latter. Anything accepted must be reviewed.

 

[Nicholas Bedworth]

@Tim... if you check my reviews you'll see that they are very critical and demanding of the product, and thus of value to readers trying to differentiate one from another.

Writers and magazines each have their own styles, rules and so forth. In my particular case, writing reviews of products with a lot of issues is simply not something I want to do as a person. It doesn't appeal to me. This is no criticism, BTW of your opinions, which are (as usual) well-reasoned and thoughtful. And by checking the usual responsible audio publications, a reader can get a good overview of the range of opinions out there. So it seems to me that there's plenty of room for everyone.

Most of the devices that come through here represent new product categories, BTW, looking at the current backlog.

Now, if the power here on Maui would stay up, I'd be able to get more work done.

 

[Phelonious Ponk]

@Tim... if you check my reviews you'll see that they are very critical and demanding of the product, and thus of value to readers trying to differentiate one from another.

Writers and magazines each have their own styles, rules and so forth. In my particular case, writing reviews of products with a lot of issues is simply not something I want to do as a person. It doesn't appeal to me. This is no criticism, BTW of your opinions, which are (as usual) well-reasoned and thoughtful. And by checking the usual responsible audio publications, a reader can get a good overview of the range of opinions out there. So it seems to me that there's plenty of room for everyone.

Most of the devices that come through here represent new product categories, BTW, looking at the current backlog.

Now, if the power here on Maui would stay up, I'd be able to get more work done.

Fair enough, Nicholas. I haven't read your reviews and am responding on a theoretical level. That's probably not fair enough on my behalf. Where can I find some of your work on the net?

Tim

 

[amirm]

Writers and magazines each have their own styles, rules and so forth.

I may be in a unique position, having had the product out of my group reviewed by Nick!!! Indeed, he does have his style and it is the opposite of most reviewers: he is kind and gentle in how he approaches product reviews. With most reviewers, I always felt that we were on a war path: they were trying to find dirt on our product and we were trying hard to not let them. The experience invariably was difficult and painful, even though it led to success most of the time. Granted, when you work for Microsoft, you kind of expect this but I do want to point out that it doesn't need to be so antagonistic. To wit, Nick I and built a good relationship which has spanned time and careers.

Nick is one of only two reviewers I have enjoyed having such a positive relationship.

So yes, his style is different. I for one, found it a breath of fresh air, coming from the other side. And no, he did not pass on reviewing anything we had .

 

[Nicholas Bedworth]

Here's the Sasha review... co-written with my colleague Ed Kramer of Sydney, Australia.

We thought it might be interesting to have two completely different points of view applied to the speakers being evaluated. My system is totally digital, minimalist and uses high-power, solid-state amps. His is totally analogue, with preamp and low-power tubes. Worked out well in both cases, which, actually, is rather surprising, but that's what happened. http://www.6moons.com/audioreviews/wilson/1.html

Put it this way, the 'Net has brought certainly about a democratization of audio reviewing. And guess what? The readers/consumers are in charge. In addition to 'trust your ears', 'know thy reader' is now an instantaneous reality. Within minutes of a review going up, there's e-mails and PMs asking questions, making criticisms, and getting involved. This is great, and it keeps you real, so to speak, as a reviewer.

In my recent experience, some of the incoming mail is critical of the style of the review. One person said that my writing was "painful". As some of you may have seen, my response was "wait until you meet me in person" or something like that. However, this kind of comment could come from a really knowledgeable person, or someone who is just looking for "the answer". These are probably about 10-20% of the responses, depending. The other 80-90% found at least some aspect of the review useful, and so this targeting information I take to heart, even if it makes you wince once and a while.

We're writing for the benefit of the readers, not to be some kind of modern day Petronius of the audio world. Of course, prior to being dispatched to another world by his patron, Nero, he had a lot of fun writing Satyricon.

 

[mep]

Hardly an unbiased opinion Amir. Nicholas liked your product so all was well in your world. An ezine that only reviews products they are going to give a favorable review to is not what most of us expect from an audio rag. Sending products back that don't pass muster without telling those who may be buying them is a disservice to the audiophile community. I don't think we want reviewers to be silent consultants to manufacturers. I don't want to read a "review" of a product that was silently returned to a manufacturer with tips on how to make it better and when the improvements of the reviewer are incorporated, suddenly the reviewer really likes it and gives it a rave review.

 

[mep]

In my recent experience, some of the incoming mail is critical of the style of the review. One person said that my writing was "painful". As some of you may have seen, my response was "wait until you meet me in person" or something like that. However, this kind of comment could come from a really knowledgeable person, or someone who is just looking for "the answer". These are probably about 10-20% of the responses, depending. .

That person would be me and I wasn't looking for "the answer" whatever that means.

 

[Nicholas Bedworth]

Apart from the kind words, Amir brings up a point which I hadn't brought out explicitly...

Basically most of the designers and manufacturers reviewers deal with are really, really smart and capble people who have spent their entire career focused on one single thing: their products. If you put them on the defensive, naturally they clam up, apart from possibly making a permanent enemy. If you work with them, they'll tell you all kinds of interesting things, which are important for the readers to know. They certainly know what the strengths and weakeness of their products, and their competitors products, much more profoundly that almost any reviewer could ever elicit, and there's often a very specific reason for what and why they did.

One can be critical without beating people into submission. And you'll learn a whole lot more... Sure, some products are lousy. There will be no shortage of people other than me who will review them appropriately.