Reviewing the reviewers: Episode #1

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
This series of articles turns the tables on magazine reviewers and writers, critiquing their writing/tests as they critique something or someone else! It is only fair. No? :) Seriously, it is very hard to stay on top of so much advanced technology and mistakes do happen in how tests are performed, and writing is done. My main intent here is to use their writing as a way to explain technology and picking the topic relative to where their work falters a bit.

We start this episode with the October 2010 issue of Sound and Vision magazine and a review of three Blu-ray payers by Al Griffin. In his review of Panasonic BD player, Al starts by saying he wanted to review the Panasonic DMP-BDT350. He goes on for a full paragraph singing the praises of that player only to disappoint us by saying when he went to get one, it was out of stock! OK, I am already in grumpy mood, reading a story that he only begun but not finished :).

Moving on to the performance section, he states “all Blu-ray tests that I ran on the BD100 passed with no problem – not a surprise since this player pretty much offers the same high-quality video processing as BDT-350.” Wait a minute! You just got done telling us you couldn’t buy a BDT-350 for the review, but now we read that you knew its video processing performance? How so? By just reading its paper spec?

Al goes on to say that when playing DVDs, the player passed all tests except “for checks for 2:2 pull-down detection.” Raise your hand if you know what 2:2 pull-down is and where it is used. I see no hands up. :) Come on man. Surely the average or even avid videophile can’t be expected to know what “2:2 pull down means.

He then goes on to say, “…although that [failing 2:2 pull down] shouldn’t pose a problem with most movie discs.” Movie discs? First, in US that would be a movie shot at 30 fps where 2:2 means the frame rate is doubled to 60 Hz US standard by the encoding and then reversed out by the player. I don’t know about you but I don’t know of any movies shot at 30 fps. Europeans stuck at 50Hz, do suffer through 24 fps movies sped up to 25 fps and then converted to 50 Hz using 2:2 pull down. Lots of numbers but it nets out easily: if you live in US where this magazine is published, this test doesn’t matter to you.

In the review of the LG Blu-ray player, when speaking about LG’s DVD performance, Al critiques it by saying “… the DVD movies I watched all looked fairly noisy and soft…” OK, soft is not good although I will come back to that point later. But noisy? It might be counterintuitive to hear that noise is a figure of merit here but it is! Let me explain.

There are two types of noise in a DVD player: film grain/video noise in the original source and compression artifacts added while the video is put on the disc. In either case, that noise/artifact is stored on disc. Since we want our players to pass all the bits that exist on disc, a properly working player must put out a noisy picture if that is what on disc. The only way to take the noise out is by filtering the video. If the player is doing that, then it is likely also filtering the video material itself as it is impossible to distinguish the two in all cases. So when I hear one player has less noise than the other, I think of it as having lower fidelity, not higher.

To be clear, there is nothing in the DVD/BD player which generates noise. Yes, you can get various video artifacts if the processing is done incorrectly but all of those artifacts have proper technical terms and not described by “noise.” So the whole notion of testing for “noise” seems wrong to me.

Let me digress a bit and really drive this message home. When I go to a forum where people discuss displays, it takes all of five minutes to find a thread where someone says, “I really dislike the picture of XYZ flat plane because it is pixelated compared to other sets.” Well, as above, this is backward. What the person is actually seeing is the blocking artifacts in the original source. The more faithful the display is, the more you will see such artifacts (although admittedly, if a set’s sharpness is set too high, it will exaggerate such artifacts).

Going back to the review of the Panasonic, Al finishes his review of that player by saying, “… its Detail Clarity adjustment had a subtle and pleasing picture-sharpening effect. In fact, I liked it so much that I left Detail Clarity switched on for both DVD and Blu-ray view.” Well, I don’t know about you but I don’t want my video signal modified in the player. I want the player to pass all the pixels as is. Circuits like this boost the amplitude of high-frequency signals, which subjectively result in a picture appearing sharper but can create artifacts such as haloes around sharp edges. Had Al tested this mode through his test patterns he would have discovered its down side.

Putting that aside, might it be that he thought LG’s DVD playback was softer because he turned on the detail enhancement logic in the Panasonic player? Two bucks says he did! :) It is like him turning up the red color in the Panasonic and then concluding that LG has less red! When you are performing a shootout you better keep all conditions same. You can’t turn on a feature in one player which artificially boosts resolution and then conclude the other set is lacking in that regard.

At this point, we get to the review of Sony BDP-S770. Here he says:

“Performance on Blu-ray tests was very good for the most part. The only pattern that the S770 tripped on was the S&M [test] disc’s chroma multi-burst test, which indicates limited color-detail resolution as compared with the other two players.”

Kudos for explaining what Chroma [color] multi-burst pattern is. But how the heck do you give the “2010 Sound and Vision Certified and Recommended” award to a player that can’t resolve the full resolution of color signal on disc? People buy Blu-ray to get the best video quality. Here we have a player which can’t output what most of the other Blu-ray players do in their sleep, and it gets an excellence award? Oh, wait, he explains why:

“Despite this, the Blu-ray movies I watched on the player all looked crisp, detailed and punchy…”

Excuse me but when did he see a Blu-ray player of any brand that didn’t have all of these attributes subjectively? This is a high-definition player after all. None of this amounts to anything substantiating such an award.

To expand, unlike audio there is little room for subjective evaluation in video. Test signals speak the truth and the whole truth. The picture may have looked detailed but if he put up something with colorful patterns and compared it to another player, he would have noticed the softer picture.
He goes on to say:

“The Sony’s DVD playback was also very good: it sprinted through my full test-disc suite with not one misstep. As I mentioned before, test discs don’t always tell the full story, and I noticed that the Sony’s DVD playback looked somewhat less crisp than the Panasonic’s in direct comparison.”

Here we go again. As with LG comparison, it is entirely possible he forgot that he had left the Detail Clarity processing on in the Panasonic player and hence was seeing that difference, and not something the Sony is doing wrong.

So there you have it. As you see, it doesn’t take much to invalidate results of testing equipment. This is why I don’t do that but instead choose to critique people who do! :D
 

Gregadd

WBF Founding Member
Apr 20, 2010
10,517
1,774
1,850
Metro DC
Amir at 2010 RMAF I put the the question to Harry Pearson." Why don't they have some voluntary standards that all the reviewers would agree to follow?" He agreed it was a good idea but probably would not happen because all the reviewers were "stars." Roy Gregory talked to me afterwards and also agreed it was a good idea. We discussed the issue and thought the standards would probably have to be put together by a third party.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
You make a very good point that there is no agreed upon methodology for testing especially in audio. In video it is a bit better. For example, for the longest time people used the same Star Trek movie to test video processor fidelity (looking for jagged lines). And we have excellent test discs which everyone uses.
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
To expand, unlike audio there is little room for subjective evaluation in video.

Oh, Amir, if this were true, Samsung would sell no TVs! In a perfect world you are, of course, correct. But in the real world, people don't have test discs and like this fellow with the detail enhancement turned on the Panasonic, they like what they like and ignore the facts. The only specs most of them know are among the most meaningless:

1080p, 240hz, gazillion to one contrast ration = really good tv.

Tim
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
You are definitely right that people make decisions subjectively. I meant formal reviews must be done objectively with test discs/sources.

Funny story. Sharp rose to #1 status in Japan in display, a position which it holds still. The reason had to do with the way displays are shown in Japan. They have large department stores stuffed to the max with product. Their display floor has every TV from every manufacturer. Problem: they use harsh florescent tubes a couple of feet above the display. This washed out Plasmas of the time something silly. The LCD with its matt display and high brightness, would win hands down because its image would not wash out and you would see an array of lights reflected as you could with Panasonic plasmas. At the time they won this competition, Plasma had a huge picture quality advantage but only in a darker room.
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
You are definitely right that people make decisions subjectively. I meant formal reviews must be done objectively with test discs/sources.

Funny story. Sharp rose to #1 status in Japan in display, a position which it holds still. The reason had to do with the way displays are shown in Japan. They have large department stores stuffed to the max with product. Their display floor has every TV from every manufacturer. Problem: they use harsh florescent tubes a couple of feet above the display. This washed out Plasmas of the time something silly. The LCD with its matt display and high brightness, would win hands down because its image would not wash out and you would see an array of lights reflected as you could with Panasonic plasmas. At the time they won this competition, Plasma had a huge picture quality advantage but only in a darker room.

It's not so different today. TVs get set to their maximum output when they are put on the sales floor. This always puts plasmas at a disadvantage, because they simply aren't as bright as LCDs/LEDs, and in the showroom, bright looks like clarity, contrast and all other things good. And Samsung has taken the lead in America because they engineer their TVs for maximum brightness and saturation. At their worst, they look like candy-colors and sunburned white people, and a black shirt becomes a black shirt-shaped space devoid of detail. But boy do they pop off the showroom wall. Get one of them calibrated and they look pretty good, but the average customer in a showroom will pick the uncalibrated TV almost every time.

The other thing that hasn't changed is that plasmas are still the best, and the best LEDs...the new Sony XBR900 comes to mind...look a lot like plasmas. It's just a costlier way to get to (almost) the same place.

Tim
 

microstrip

VIP/Donor
May 30, 2010
20,806
4,698
2,790
Portugal
Recently I acquired a new 50" Panasonic plasma. The image quality is excelent, but the first time it has switched on it asked what was the environemnt: "SHOP" or "HOME"?
Happily it nerver asked it again ...
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
You are definitely right that people make decisions subjectively. I meant formal reviews must be done objectively with test discs/sources.

I'd apply that to audio electronics as well.

Tim
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
I'd apply that to audio electronics as well.

Tim
I hear you :). The competence level required is a lot higher for audio though. For video, I can show you a static picture you feed a video device, and then look for a specific thing with nothing changing. Once there, even a random person off the street can detect the problem. Not sure with audio where I have a hard time conveying what you are supposed to hearing and not sure what you hear versus what I hear.
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing