How do we improve the reviewing process of High End equipment?

[MylesBAstor]

Relax, Miles. Nobody said anything about DBT. And I believe I made it clear that I understand that freelance reviewers do not have the resources to run measurements. Casual, blind listening, however, is not that difficult to accomplish, with the exception of speakers. And can you honestly tell me that you can give a more objective evaluation of a component because you know when it is and is not in the signal chain? That your evaluation would be less objective without that knowledge? And where's the "bluster?" I read reviews for entertainment value only. Others, as I pointed out, may read particular reviewers religiously, understand their positions and their points of view and actually gain useful perspective from their opinions. But as long as it is a purely subjective system, it seems to me that any real knowledge gained by the reader is going to require serious knowledge of the writer. have I got that wrong?

And are the measurements we have ineffective, or are reviewers subjective judgements often backed up by the measurements we have? If it's both, it speaks well of neither, but I suspect that's not quite what you meant to say.

P

We're obviously not reading the same thing.

I asked for proof of your bone of contention that subjective reviews have produced an outcome that is contradictory to the measurements obtained. I am asking for proof --in five published reviews -- of your assertation that . Just as you ask reviewers to do measurements to back up their subjective impressions.

 

[Jeff Fritz]

We're obviously not reading the same thing.

I asked for proof of your bone of contention that subjective reviews have produced an outcome that is contradictory to the measurements obtained. I am asking for proof --in five published reviews -- of your assertation that . Just as you ask reviewers to do measurements to back up their subjective impressions.

Myles, I could give you five from just our speaker reviews alone. If I were to go into the Stereophile archives I could find ten times that amount. Now if you care to place a wager, then I'll spend my time doing it. If not, then you'll just have to take my word for it

Oh heck, here are two, one from us and one from Stereophile, just to be fair about it:
http://stereophile.com/floorloudspeakers/zu_essence_loudspeaker/index5.html
http://www.soundstage.com/revequip/talon_khorus.htm

Do I get a coffee at CES for that?

 

[MylesBAstor]

Myles, I could give you five from just our speaker reviews alone. If I were to go into the Stereophile archives I could find ten times that amount. Now if you care to place a wager, then I'll spend my time doing it. If not, then you'll just have to take my word for it

Oh heck, here are two, one from us and one from Stereophile, just to be fair about it:
http://stereophile.com/floorloudspeakers/zu_essence_loudspeaker/index5.html
http://www.soundstage.com/revequip/talon_khorus.htm

Do I get a coffee at CES for that?

Unfortunately the Talon audio link doesn't work I heard them years ago in a reviewers system and they did nothing to rock my boot. Seemed like rock speakers. Kinda hard sounding (that could have been the amp too), overdamped and overdetailed.

As far as the Zu goes, it seems that (and Art is pretty astute reviewer) JA was saying that the speakers sounded better than the measurements indicated? I also think that Art did remark on some of the issues that turned up in the review. So is the issue here we need to see the measurments taken in Art's room or something else?

And we can get back to speakers and room interactions But among electronic components, where have you seen a discrepancy between the measurements and the reviewer's subjective review?

 

[Jeff Fritz]

Unfortunately the Talon audio link doesn't work I heard them years ago in a reviewers system and they did nothing to rock my boot. Seemed like rock speakers. Kinda hard sounding (that could have been the amp too), overdamped and overdetailed.

As far as the Zu goes, it seems that (and Art is pretty astute reviewer) JA was saying that the speakers sounded better than the measurements indicated? So is the issue here we're not measuring the right thing or need to see the measurments taken in Art's room or something else?

And we can get back to speakers and room interactions But among electronic components, where have you seen a discrepancy between the measurements and the reviewer's subjective review?

http://www.stereophile.com/tubepoweramps/704wavac/index5.html

I could go on, but I'm taking my kids to the batting cages on a lovely Saturday.

BTW, Talon link is working for me. And both reviews I linked: I was looking at the anechoic or quasi-anechoic measurements, not in-room.

And for what it's worth, I'm not casting judgement on these two reviewers. I've had the same thing happen to me, several times. I'm just using them to answer your question.

 

[MylesBAstor]

Myles, I could give you five from just our speaker reviews alone. If I were to go into the Stereophile archives I could find ten times that amount. Now if you care to place a wager, then I'll spend my time doing it. If not, then you'll just have to take my word for it

Oh heck, here are two, one from us and one from Stereophile, just to be fair about it:
http://stereophile.com/floorloudspeakers/zu_essence_loudspeaker/index5.html
http://www.soundstage.com/revequip/talon_khorus.htm


Do I get a coffee at CES for that?

Oh and BTW. We don't need an excuse for coffee Jeff

 

[FrantzM]

<snip>
Face it. The ear is still the best instrument God ever built for hearing. Can it be fooled? Of course. And so can any of our other senses. But does that make it any worse than an instrument that can only tell you what its asked and is ALWAYS subject to experimental methodology and misinterpretation? And might I add, til the new new BUZZ measurement comes out???? And why do we have new measurements? Because the old ones didn't work!!!

OTOH, there are far more examples of where measurements led to erroneous conclusions. Or my PhD advisor would have said, experiments led us down the garden path meaning if curing mice of cancer was our goal, we would have reached it 25 years ago.

Myles

That measurements come and go is not because always they didn't work... Not sufficient or inadequate or flawed is not the same thing as "did not work". A matter of degree.

Our ear is good but our brain is where the problem lies .. We hear through our brain and it is affected simultaneously by auditory stimuli but also visual , olfactory , etc ... Under the influence mp3 at 64 kb/s might sound great...
Many here talk about the review process in a way that can be constructed as worthless, except for entertainment value ... It should not be only so... Reviews can help the audiophiles. Not all of us have access to some gear. When I was in Haiti , I had to resort to reviews to buy my gears. My business travels were, that, business travels and I had to be very careful about the gear I could audition because of the time constraints. Audiophile friends, sometimes would steer me...

Now about your mention of the upper echelon... There is no doubt that at the upper level, the real one, not necessarily the higher priced models... It is a matter of taste, as we favor some part of the frequency spectrum more than others .. all of us ... we have part where we focus our attention often because of the limits of our personal auditory system or reproduction system... There again, a good, proper review, where the reviewer strives for objectivity can help and there again measurements coupled with careful and extended listening is the key.

A good room is also to me a requirement for a reviewer. Especially if he/she is dealing with speakers. I hasten to say that some aspects of the character of a given components will not necessarily be changed by the room but when a room as peaks and nulls of 30 dB within the Audio range then the real subtleties of the gear are lost, the imagined or/and the familiar starts taking over.

On the subject of moving a gear. It could be I fail to see how it can't be done...The Burmester 911 came in a road case from Germany and they worked from day one ... I would think that it would be possible to move gears around but I could be wrong... I have sen this from European magazines, especially French that there is principal reviewer but other reviewers often brings their comments, TAS also did that back in the days, don't read it anymore so ...

The last part of your post, I have some difficulty fully understanding your point .. Are you suggesting that we ought to reject measurements ? I , also am not sure that we have cured cancer from rodent, yet ... the emphasis on "far" is quite interesting .. It is an interesting indictment of the scientific method... We may need to debate it elsewhere .. however, that we can discuss it right now on this very forum in in itself a testimony of the value of the scientific, objective, measurements-based method ...

 

[MylesBAstor]

The last part of your post, I have some difficulty fully understanding your point .. Are you suggesting that we ought to reject measurements ? I , also am not sure that we have cured cancer from rodent, yet ... the emphasis on "far" is quite interesting .. It is an interesting indictment of the scientific method... We may need to debate it elsewhere .. however, that we can discuss it right now on this very forum in in itself a testimony of the value of the scientific, objective, measurements-based method ...

Not as much as we should view measurements with a healthy dose of skepticism. Measurements have lead us down some dead ends

That said, I don't think there is any top flight high end designer who doesn't use measurements in the design of their gear. I think the difference is where does one feel they have achieved their goals with the measurements and begin to voice the gear according to how the designer hears music.

 

[Phelonious Ponk]

We're obviously not reading the same thing.

I asked for proof of your bone of contention that subjective reviews have produced an outcome that is contradictory to the measurements obtained. I am asking for proof --in five published reviews -- of your assertation that . Just as you ask reviewers to do measurements to back up their subjective impressions.

Where did I make that assertion? I don't remember making it. Five examples of the subjective review producing an outcome that is contradictory to what measurements? Subjective reviews, by definition, do not contain measurements.

p

 

[vinylphilemag]

Where did I make that assertion? I don't remember making it. Five examples of the subjective review producing an outcome that is contradictory to what measurements? Subjective reviews, by definition, do not contain measurements.

p

Umm, pretty much all of Stereophile's "subjective" reviews are accompanied by a complete set of measurements, so I must respectfully disagree with your assertion at that "Subjective reviews, by definition, do not contain measurements".

 

[DaveyF]

IMHO, HP had it right many years ago when he termed the phrase 'The absolute sound'. Because, this is what I personally believe all equipment should be held up to. Unfortunately, I think in the last few years, many reviewers have been way too
forgiving of equipment and have waxed far too favorably about their favorite piece of the day. One of the things I like to demonstrate to any hobbyist who thinks that their current equipment truly sounds like 'live unamplified music' is to play my
Taylor 810 acoustic guitar and ask them what they think of how their system compares to 'The absolute sound'. I have never had any 'phile do anything except realize how far we still have to go before we can say that our systems truly recreate the
sound of live music. BTW, I have done this with 'phile friends and others that have great systems, NONE of them do anything BUT shake their head in disbelief! So, I would very much like to see the current SOTA held to a higher standard, wherein
the reviewer could along with all the platitudes include some thought on the very obvious weaknesses that all equipment still elicits. Just IMHO.

 

[Phelonious Ponk]

Umm, pretty much all of Stereophile's "subjective" reviews are accompanied by a complete set of measurements, so I must respectfully disagree with your assertion at that "Subjective reviews, by definition, do not contain measurements".

It's a disagreement of semantics, then. I wouldn't consider Stereophile's reviews subjective. Do they have a subjective element? Of course. Otherwise they would be a lab report and no fun to read. The reviews I'm talking about are simply subjective reviews. They don't even verify the manufacturers claims with measurement. They're found all over the net and in audiophile publications.

P

 

[Phelonious Ponk]

IMHO, HP had it right many years ago when he termed the phrase 'The absolute sound'. Because, this is what I personally believe all equipment should be held up to. Unfortunately, I think in the last few years, many reviewers have been way too
forgiving of equipment and have waxed far too favorably about their favorite piece of the day. One of the things I like to demonstrate to any hobbyist who thinks that their current equipment truly sounds like 'live unamplified music' is to play my
Taylor 810 acoustic guitar and ask them what they think of how their system compares to 'The absolute sound'. I have never had any 'phile do anything except realize how far we still have to go before we can say that our systems truly recreate the
sound of live music. BTW, I have done this with 'phile friends and others that have great systems, NONE of them do anything BUT shake their head in disbelief! So, I would very much like to see the current SOTA held to a higher standard, wherein
the reviewer could along with all the platitudes include some thought on the very obvious weaknesses that all equipment still elicits. Just IMHO.

Live music is a great reference, but I'm afraid it's seldom a useful one. Forget the systems of the audiophiles who are shaking their heads, the recordings don't often capture a realistic instrument sound for the systems to reproduce. And so as difficult as it is, the recording is the reference, and high fidelity to the recording is still the objective.

P

 

[vinylphilemag]

It's a disagreement of semantics, then. I wouldn't consider Stereophile's reviews subjective. Do they have a subjective element? Of course. Otherwise they would be a lab report and no fun to read. The reviews I'm talking about are simply subjective reviews. They don't even verify the manufacturers claims with measurement. They're found all over the net and in audiophile publications.

P

Gotcha. That's why I tend to prefer the term "observational", which is indeed a different beast to "subjective".

 

[FrantzM]

Gotcha. That's why I tend to prefer the term "observational", which is indeed a different beast to "subjective".

+1

I , too, prefer "observational" . Subjective has a derogatory meaning to it as in "lacking objectivity". An observation can correlate to a certain, measurable, repeatable reality. An observational review can be objective in this sense. This IMO should be the goal of any review.

 

[Elliot G.]

It would be really nice to have a set of standards for reviewing. I mean by that a methodology that we could all follow and maybe a set of musical events that we can all follow along with. In addition it would really help if all the reviewers( no Offense) would actually trade off some gear for comments on reviews rather than they all do thier own thing.
For example TAS has a new speaker of the century every month ( funny no). It seems absurd that the major reviewers never hear what the others raved over. Why? We need to open up the can of worms and release the truth!

 

[Steve williams]

Jeff Fritz here is making an admirable attempt at that E

 

[Gregadd]

It's a disagreement of semantics, then. I wouldn't consider Stereophile's reviews subjective. Do they have a subjective element? Of course. Otherwise they would be a lab report and no fun to read. The reviews I'm talking about are simply subjective reviews. They don't even verify the manufacturers claims with measurement. They're found all over the net and in audiophile publications.

P

The only two reasons I keep my subscription to Stereophile current is the industry news and the measurements.
However it appears to me that one person does the measurements in Stereophile almost always off site of the reviewer . It is not even clear whether the reviewer has even seen the measurements before it goes to press. That means the overwhelming number of reviews are qualitative not quantitative.

 

[Gregadd]

Live music is a great reference, but I'm afraid it's seldom a useful one. Forget the systems of the audiophiles who are shaking their heads, the recordings don't often capture a realistic instrument sound for the systems to reproduce. And so as difficult as it is, the recording is the reference, and high fidelity to the recording is still the objective.

P

This is a particulary troublesome argument. In theory I must accept the notion that most recordings fall short of replicating "live music" assuming that was thier goal in the first place. OTOH there exists more than an adequate number of excellent recordings to allow real music to be your reference.
To put it bluntly if live music is not our reference then what is? None of us will witness the actual recording. What you are saying then is we have no reference. We are just exercising our personal taste.
Plenty of highly respected reviewers and manufacturers use live music as a reference. As do recording engineers. recording engineer's can compare thier master tape to what they just heard.

I suggest to audiophiles, BTW this is nothing new, a system that is faithful to live music will also be faithful to any source. No audiophile should accept anything less.

 

[Kal Rubinson]

However it appears to me that one person does the measurements in Stereophile almost always off site of the reviewer . It is not even clear whether the reviewer has even seen the measurements before it goes to press. That means the overwhelming number of reviews are qualitative not quantitative.

In general, for final editing, the reviewer sees a copy proof that includes the measurements and JA's comments. However, since that occurs after the review is written, the significance is moot.