Yet another Hi-res is no good article

amirm

Banned
Apr 2, 2010
15,813
38
0
Seattle, WA
In this case the claim goes beyond just CD but that AAC lossy compression is transparent too.

The discussion is going on in AVS in the Scott/Mark thread evolving to this angle. I thought I share what I wrote there in our forum too and see how members here react:

==========

http://www.trustmeimascientist.com/...io-poll-neil-young-and-high-definition-sound/


49% of audio engineers, scientists, professional musicians, and trained listeners can't be wrong. They got the blind A/B test right so the audibility is there. ha ha. [actually it was a slightly different test than ours]


"That’s right: even among our readers, the results came out no better than a coin flip."
-Justin Colletti, audio engineer
OK, let's get down to business.

For those of you who have not read this article, the genesis of it is a prior one where they challenged people to tell the difference between 24 bit/44.1 Khz and 256 Kbits/sec AAC. They provided two sets of files each in those two instantiations: http://www.trustmeimascientist.com/...udio-poll-do-we-need-higher-definition-sound/

I expected this to be a tough comparison. After all, the follow up article's whole foundation is how people can't tell the difference between high res and lossy compressed music. That was not remotely the case! I listened to the first sample for about 5 to 10 seconds. The moment I switched to the second variation, the improvement in fidelity was substantial. Better spaciousness, cleaner high frequencies, etc. You know, the terms that our resident objectivists hate to hear. I jotted down the choice and then listened to the second set. This comparison was beyond trivial. It took just moments to tell the difference.

Of course it is abundantly easy to cheat by reading the new article where the results are revealed. You just need to take my word for it that I did not cheat. If anyone has any doubts about that, I bet you any amount of money that I can repeat this test and correctly identify the higher fidelity 24/44.1 Khz.

I like to pause here and encourage everyone to just click on the above link and take a listen. Then look up the answer in the link Zillch is providing. If you cannot tell the difference, then this whole subject is moot for you. You would be happy with anything from 256 kbps iTunes AAC downloads to CD. I would shop on basis of price then and not worry. I trust Zillch has run such a test and has landed in this bucket himself or else he should not have posted the article.

I am hoping however, that a number of you get it right as I did. And not just by guessing but hearing the precise differences that can be backed by how lossy audio compression works. If you did, then you have very good to excellent ability to hear non-linear, dynamic distortions.

Separating us into the two buckets of listeners is super important. The reason we have so many of these arguments is that non-critical listeners who can't tell the difference convince themselves that no one else can either. And hence, they argue 'till the last drop of blood is shed. To them that is the reality. To the extent some people invalidate this assumption by demonstrating differing and superior listening ability, I hope they learn from that and dial back the level of anger and angst they demonstrate in their posts.

===========

Now to the second bit. The author says something close to 50% of the people guessed right and as Zillch quoted, that means the results are no better than a coin toss. And further, that means there is no value to higher fidelity music for distribution. That conclusion of course is absurd but is a mistake often made. That is, mix someone like me who can tell the difference with others who cannot and declare that the results are no better than chance. That is totally inappropriate conclusion. What I did is independent of what others did. You can't take my results and dilute it with others.

Now, given the two samples that I passed, one can say on paper that those results are not sufficient to declare me to have guess right. Well, then test me more. Don't throw my results with countless others who we know cannot hear these differences.

To be fair, his summary results do indicate with high statistical confidence that if we sample a few hundred of their readers, as a group, they could not do better than chance. This again is what we already know. That the masses are perfectly happy with 256 kbps compressed music. This verifies that yet again which is fine. But in no way does it address the audience for high resolution audio and people taking the opposing position.

He tries to paper over this issue by assuming that his readers are recording artists (?) engineers, etc. I have no idea how he determined that or if it is true. But let's say it is. Yes, that means half the people in the business of producing music can't tell the difference between lossy music and lossless. This is sadly the truth. Anyone can set up shop and record music. They may have great ears as far as recording/mixing music. But not be sensitive to dynamic non-linear distortion. This is why Meyer and Moran bragging about including this population in their testing doesn't carry any weight. The layman assumes these people are born with the ability to hear such artifacts. But this is not part of their job and hence they are not much better situated than the typical crowd.

Of course since half the testers did vote correctly, there is a good chance that many of them could reliably tell the difference with more trials as I am confident I can. And hoping others can too in this forum.

I looked up the author's background. He seems very qualified when it comes to creating/recording music. But I see nothing in his background that would give him the training to understand the nature of lossy compression. That he says stuff like this as fact: To date, no trained listener has ever reliably picked out a properly encoded 320kbps AAC file from any higher-resolution file in a blind test. (But I’d still love to try, if someone wants to quiz me!) clearly indicates that he is playing fast and loose with this topic or else he would back this with a reference.

I will have more to say about his article :). But for now, I think this busts open Steven (Krab's) statement that even lossy compression is good enough. It isn't and I hope others run this test and see for themselves. Simply download the files. Double click on one file to play and after a few seconds, double click on the alternative. You don't need any AB program. You should be able to go back and forth and focus on the first few seconds to see if you can tell them apart. That we don't need looping, etc. means that these files do indeed sound different and easily so.

So can you please spend the few seconds it takes to do this test and report back? I especially like to hear from the vocal posters in the thread. Let's see what bucket they land in.
 

rbbert

Well-Known Member
Dec 12, 2010
3,820
239
1,000
Reno, NV
Listening through open back headphones in a noisy environment I was able to tell the difference within about 15 seconds. I can only think that in a more carefully controlled environment and listening for a longer period of time the differences would be even more apparent.
 

CGabriel

Industry Expert
Oct 31, 2013
618
92
265
WA, USA
www.shunyata.com
Sometimes I wonder why you fight this battle, Amir. There are people that have so much of their identity invested in being "right" that they succumb to a nocebo effect. There is a fundamental "conflict of perception" that overrides these arguments. One group comes from a perspective of "what is adequate" and then there is the other group that wants the "absolute best" at any cost. All but just a small percentage of the professional industry fall into the first because they must be concerned with limitations of all types including bandwidth, media types, computing power, etc.

People have different levels of ability to "hear" just as there are some people who are color blind others are tone deaf or have a lesser ability to perceive subtle timing cues. It is just more difficult to measure and tag someone with a hearing deficit that is analogous to color blindness. Then on the other hand we have people with exceptional perceptual abilities such as professional wine tasters, experts in smell and musicians with perfect pitch. Perhaps we may want to create a certification process that would include a battery of tests similar to some that you have posted about to determine who can and who can't be certified as a audio perceptionist.

In any case it is quite easy to tell the difference between a 256k file and a 44.1k RB even when listening through the computer speakers. I can't really understand why someone would argue otherwise unless their personal level of perception is exceedingly trivial.
 

Peter Breuninger

[Industry Expert] Member Sponsor
Jul 20, 2010
1,231
4
0
Sometimes I wonder why you fight this battle, Amir. There are people that have so much of their identity invested in being "right" that they succumb to a nocebo effect. There is a fundamental "conflict of perception" that overrides these arguments. One group comes from a perspective of "what is adequate" and then there is the other group that wants the "absolute best" at any cost. All but just a small percentage of the professional industry fall into the first because they must be concerned with limitations of all types including bandwidth, media types, computing power, etc.

People have different levels of ability to "hear" just as there are some people who are color blind others are tone deaf or have a lesser ability to perceive subtle timing cues. It is just more difficult to measure and tag someone with a hearing deficit that is analogous to color blindness. Then on the other hand we have people with exceptional perceptual abilities such as professional wine tasters, experts in smell and musicians with perfect pitch. Perhaps we may want to create a certification process that would include a battery of tests similar to some that you have posted about to determine who can and who can't be certified as a audio perceptionist.

In any case it is quite easy to tell the difference between a 256k file and a 44.1k RB even when listening through the computer speakers. I can't really understand why someone would argue otherwise unless their personal level of perception is exceedingly trivial.


I agree. I have thought about a reviewer and listener certification program. Anyone can claim they hear this or hear that. If we keep fueling these amateurs with no listening credentials then we are to blame. We should ignore all their claims or posts.. I won't go near AVS. It has nothing to do with our industry.
 

bonzo75

Member Sponsor
Feb 26, 2014
22,650
13,685
2,710
London
I agree. I have thought about a reviewer and listener certification program. Anyone can claim they hear this or hear that. If we keep fueling these amateurs with no listening credentials then we are to blame. We should ignore all their claims or posts.. I won't go near AVS. It has nothing to do with our industry.

Doesn't AVS have experts on Datasat and Trinnov? I thought those types would flock to AVS
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Haven't done the test above yet - not in the right environment to do so but maybe everybody would do well to do this perception test - it's interesting & personally informative
http://www.goldenears.philips.com/en/challenge.html#

I do agree that the nocebo bias is rife & far more prevalent than is admitted. Until the people that run & return null results for blind tests start to include internal controls in these tests, it will continue to remain hidden the level of this bias. I suggested this recently on a forum where another blind listening session (I refuse to elevate it by calling it a test) is being organised & was told to stop trying to interfere with the running of the session
 

esldude

New Member
I got both right using some good headphones though driven off the soundcard of a desktop computer. I have done such testing of myself and 320 kbps is more difficult. Of course with just two test files it isn't convincing either way on its own. Flipping a coin for just my choices would get the same result 1 out of 4 times on average.

The difference seemed glaringly obvious on the first track and merely obvious on the second I would say.
 

amirm

Banned
Apr 2, 2010
15,813
38
0
Seattle, WA
Sometimes I wonder why you fight this battle, Amir.
You ask a good question Caelin. I hope I have enough self-awareness to answer it correctly :). I do it for two reasons:

1. It is a hobby by itself. It is like a tennis match, seeing how the other side returns the ball.

2. I can't stand bullying and that is what these people do online. A few of them are always present and circle the wagon around anyone making subjective remarks and pound them into the ground and don't stop. They declare it all in the name of "science." So I go ahead and remind them what science really says and see how they like the pressure. They don't like it at all. They get belligerent, insulting, obnoxious, etc. all the while not realizing this is what they do day in and day out to other people.

So I hope I do some good while passing the time between tasks :).
 

wakibaki

New Member
Jan 26, 2013
16
0
0
Hi jk

I'm encouraged to see you promoting the Philips challenge. I'm one-and-a-bit sections short of goldenears. I'm 63.

It's certainly educational, I recommend everybody to try it. Taught me I could distinguish between MP3 bitrates. Lots of people on head-fi made it all the way.

You just need to take my word for it...

I never take anybody's word. You can't realistically expect people to do so. Do you? Let me introduce you to my friend Bernie Madoff.

I might gamble on, or against, somebody's word, but I'm not yet induced to do so. First I intend to test how amenable you are to reason.

Most people in these days of spamming and phishing have learned to be careful of simply taking people's word, so that I no longer appeal to people to believe or trust me, it sounds disingenuous, or downright dishonest.

This business of audio has become contentious beyond the point where trust can be presumed. Surely you recognise that? You have to provide proof, and I mean proof that will stand up to hard scrutiny better than TV magic. Otherwise you will never attract a really high standard of contributor.

I don't see what's the big issue here, somebody might have overestimated the efficacy of a particular compression. Wow.

goldenears_89.jpg
 

CGabriel

Industry Expert
Oct 31, 2013
618
92
265
WA, USA
www.shunyata.com
You ask a good question Caelin. I hope I have enough self-awareness to answer it correctly :). I do it for two reasons:

1. It is a hobby by itself. It is like a tennis match, seeing how the other side returns the ball.

2. I can't stand bullying and that is what these people do online. A few of them are always present and circle the wagon around anyone making subjective remarks and pound them into the ground and don't stop. They declare it all in the name of "science." So I go ahead and remind them what science really says and see how they like the pressure. They don't like it at all. They get belligerent, insulting, obnoxious, etc. all the while not realizing this is what they do day in and day out to other people.

So I hope I do some good while passing the time between tasks :).

Bless you. You are a true warrior.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Hey, Waki
The ABX test (so beloved of the group Amir talks about) recently returned positive results for ArnyK's set of high res Vs RB files (The infamous jangling keys audio files). I don't know if you followed the reactions to these positive results on this or PFM or AVS forums but the final reason, of many, for rejecting these results was a distrust of the testers - "the results could be gamed". Proctoring was suggested as the solution but of course the proctors themselves need to be trusted & on & on it goes. So now it's a case that the ABX test, as specified, is not to be trusted.

Seeing as honesty is the issue why not apply the same criteria to any past, unproctored results from this same test & we can all perhaps make some real progress, instead of one group continually claiming the validity of null results from such unsafe tests?
 
Last edited:

Don Hills

Well-Known Member
Jun 20, 2013
366
1
323
Wellington, New Zealand
...
Now to the second bit. The author says something close to 50% of the people guessed right and as Zillch quoted, that means the results are no better than a coin toss. And further, that means there is no value to higher fidelity music for distribution. That conclusion of course is absurd but is a mistake often made. That is, mix someone like me who can tell the difference with others who cannot and declare that the results are no better than chance. That is totally inappropriate conclusion. What I did is independent of what others did. You can't take my results and dilute it with others.
...

It's the only correct conclusion. A good statistician won't cherry pick the results. But as you say, "further testing is indicated."
 

awsmone

Well-Known Member
Apr 6, 2014
1,616
514
435
Canberra Australia
The result may indicate
1. people cannot tell the difference
2. the test is flawed and in fact there is a difference but the study failed to demonstrate it

this would depend on the number of factors

The perceived difference between the two samples
The ability of people to tell the difference
The number of people that would need to be tested to demonstrate there is a difference based on 1 and 2

If Amir is correct that there are people who can tell and they are in the minority, then having validated the percentage of the population with this ability
The ABX would need to be scaled up to observe this subgroups ability to pick the difference, and show whether there is a skew of the predicted null hypothesis as a result of a subgroup picking the correct result (the same technique used to find the Higgs Boson)

Getting away from audio, but musically related

I watched a video about the differences in key tonality in an unequalised temperament keyboard

The difference was quite obvious to me, as it was to many others

However, there were posters who said that it was a joke and there was no difference

Obviously Mozart, Bach, Beethoven and many others were deluding themselves, as far as these posters were concerned

I think if we are going to talk about ABX testing the conditions must be stated up front

I remember a story about a scandanavian radio station that was moving to compressed digital broadcasting

It tested many codec, on 14000 people

It concluded one was the best from this group

In the final trials, a sound engineer pointed out the codec had a terrible whistling sound

Immediately everyone could hear it, and the codec was canned lol
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
The test design should be directed towards answering a particular set of questions. What is the question that this audio test was directed towards? It seems to have been "Can trained listeners hear the difference?". This was run on the site "trust me I'm a scientist" with the logic/premise that "The majority of you are audio engineers, professional musicians, and ambitious hobbyists, and I figured that if anyone would be able to tell these file types apart, it would be you guys." Is this premise logical?

Is there a logical connection between the stated goal of "testing trained listeners" to actually testing an "unspecified grouping" of people who self proclaim their expertise?
 

amirm

Banned
Apr 2, 2010
15,813
38
0
Seattle, WA
There is certainly nothing scientific about the article or its testing. It does say some other interesting things though :):

The truth is that digital audio engineers have been focusing on file formats, working hard to improve them for the past decade. In 2002, I would have been crusading right alongside Neil, but today the weakest link in the average listener’s experience isn’t his high-resolution AAC file – it’s his playback system.

Listeners today often experience their music through laptop speakers and earbuds, and they hear it through D/A converters and analog signal paths that leave a lot to be desired. Up to a point, even small improvements in those areas can make a tremendous difference. The kind of difference that music fans can hear in an instant that makes them feel like we aren’t lying to them.

[...]

Because of their ability to help us overcome the placebo effect and confirmation bias, blind AB tests have the power to help us make important decisions – and to keep us from making bad ones.

When a new set of Burl converters [DACs] arrived at Strange Weather, owner Marc Goodman, engineer Daniel James Schlett and I level-matched them with the old converters and blind-tested each other repeatedly to discover if we could hear a genuine improvement. You’d have to be crazy to spend $20,000 on a set of converters without being sure it’s worth it.

Thanks to blind AB testing, we were able to discover that this new investment wasn’t snake-oil at all. If we hadn’t been able to do this, the Burl would have gone right back to the dealer, and the money would have been spent on something else that did make a noticeable difference while listening blind.


I assume that people who put this article forward didn't really read it as otherwise it is clear the author is not their friend, advocating budget audio gear.
 
Amirm, What you have said here is entirely right in my opinion. Were these people really scientists, they would be pursuing and explanation for power cords sounding different,etc.

I've also wondered how these people choose an audio system. Do they just buy the cheapest?
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing