Converter loop-back tests

[amirm]

Wasn't it at Micro's suggestion that Ethan run the test again with a different piece of music? I didn't think it was Ethan who "d[id] not like how the process is going".

Suggesting that the clips did not have good fidelity has nothing to do with Ethan refusing to disclose the public results of the test in front of us. I asked him to tell us why we are wrong and he is not telling. There is a give and take in such things. He puts forward something and we do our bit to follow through. Then we are told that we don't get to know the generational order. Why? Because he accepting that he should re-do the test? What does that have to do with anything? If this is an objective test then the two tests are independent of each other.

Now, having stated that, what if Ethan simply delays (instead of foregoes) posting the true order until the other test is performed by at least *some of us*? Would that be satisfactory?

To me? No. I no longer trust the objectivity and integrity of the process. Right now, as a matter of public record, we are told that we are "wrong." Wrong about what? Who was wrong? There were four clips in the test? How could the results be just right or wrong? Did we get all of the sequences wrong across the board? There was not one clip from one person that was correctly ranked?

Members, of course, could burn CDs.

But they are not. And stated so. Yet no concern was raised about that. That should have been reason enough to not make oneself think there is any kind of valid data point is being created. The test instead should be put forward as a fun thing to do together with no ambitions of scientific importance. We are here to enjoy a hobby together. Now look at where we are. If Ethan is after scientific results, he should run this test formally with people present, an independent proctor that can verify the accuracy of his work, and then try to get it published. Running a test in a forum instead doesn't accomplish such goals and thinking otherwise only serves to create friction and zero learning as we have seen so far.

Which forum? AVS, with such luminaries as diomania and hdnewbie? Generally speaking they are not the ones claiming to possess this kind of discriminatory hearing ability.

Why would it matter in a blind test Ron? I ask again: isn't this search for audio truth? Or is it to make people look bad? I want to learn about audio. Proving some random dude is wrong is not going to do that. I encouraged Ethan earlier to search for revealing clips. A discussion around that would be 1000 times more educational than this waste of time discussion. We would learn something of lasting value. Imagine how useful our shopping experience would be if we had revealing DAC clips. Do people really think DAC designers don't have such clips? That they stare at numbers on a scope and that is it?

Some are not. But categorically as you state? This is to state nothing of the fact that, as we both know, at least one member decided to not even take the test but, instead, *read* the difference in the clips.

I have addressed this above. Once again, I am not going to learn anything about audio or running proper listening test because XYZ did or did not take the test. I took the test as did a few others and we simply ask for Ethan to deliver on his end of the bargain. I know I would not have participated at all if this were a witch hunt to prove someone wrong. I know for sure no matter how good you are in your listening ability that you will get these things wrong. If we elevate that fear to the point of embarrassment of members, then folks would not want to take the test.

On AVS forum, a blind test was put forward. I voted. As here, I was declared to be wrong. A professional engineer who mixed movie sounds for a living was told he got it right. I showed later that the person putting forward the test had made a mistake and two of the files were identical yet he said they were different compressed clips. Countless people had taken the test and I was the only one to take exception to the results and demonstrate why. The mistake there happened when he uploaded the files and got them confused. Mistakes happen. We have to make sure that when they do they are not used as weapons or else no one will want to participate again.

I happen to be one of the few people who is not embarrassed to be wrong. I have taken so many of these where sometimes I was the hero finding differences no one else did. And zero for finding differences that there was not supposed to be any . But not everyone is in the same boat. You need to create an environment that is comfortable to many if you need them to create statistically valid results.

This concept of a generational loss test reminds me of the time here at WBF before Sean Olive and the other folks at Harman made available for public consumption that downloadable program with the dozen or so distortion tests. I remember asking the WBF membership at the time how many were going to download the program and take the test and, IIRC, I got about 3 or 4 positive replies.

I was one of the few there. Since then I have taken the same test twice at their facility and voted while many others watched. So in the context of discussing this topic with *me*, that is not a valid data point. Further, the tool instantly tells you that you voted wrong and the right answer. Yet I am sitting here with you defending the equiv of that program -- Ethan -- not disclosing the answer. How many people do you think would take that test if after 10 passes it just said, "some of the people who took the test gave the wrong answer. I will tell you sometime in the future the right answers if you take more of the test."

Strawman. Tim, I agree.

Strawman? You mean we can't discuss the science of audio and what makes a good clip and what doesn't? That asking Ethan his point of view is out of order? As I said, I am here to discuss science of audio. If folks want to censor that with statements like that, that is their prerogative. But it is not how I like to see us interact with each other.

 

[Ethan Winer]

Ethan, the people you want most to take the test are not going to be able to figure anything out from the accuracy or inaccuracy of the rankings of the first set of samples unless Amir tells them how, and they're not going to take the test anyway.

I don't know why Amir has to tell people what to listen for. This posts sums it up well:

Listening to these through Stax headphones, I found the sound different with the first pass. Forgive me the subjective verbiage, but a distinct lack of ambient fullness, oddly enough in the synth drum, as if the space of the performance retracted a bit and got harder. Each cut thereafter gets a little "hairier" with edge and compressed sounding, with less ambience.

Amir, can you promise to keep your secrets to yourself? If yes, then pubilish the results of the first round, Ethan.

I offered to tell Amir the answers by PM. But several people said they didn't like that music, so what's the point? This is why I'd like to do it again with music everyone agrees is "clean" enough to be able to notice degradation over one or more generations.

--Ethan

 

[Ethan Winer]

I asked him to tell us why we are wrong and he is not telling. There is a give and take in such things. He puts forward something and we do our bit to follow through.

Fair enough, and in the grand scheme of things revealing the first results should not compromise a subsequent test with completely different music.

Edit: To reach a wider audience and get more data, this is now a more formal article on my web site:

Converter Loop-Back Tests

If Ethan is after scientific results, he should run this test formally with people present, an independent proctor that can verify the accuracy of his work, and then try to get it published. Running a test in a forum instead doesn't accomplish such goals and thinking otherwise only serves to create friction and zero learning as we have seen so far.

I did offer to do the entire test again with anyone who wants to volunteer as a witness. But a blind test like this in a forum is absolutely valid. It's especially valid because it lets those who claim they can hear even one generation on a high-end converter prove they really can hear the degradation. And my Focusrite is a just a $250 "medium-level" converter.

I encouraged Ethan earlier to search for revealing clips.

That's why I posted a link to my proposed second clip! If I pick music based on what I think reveals degradation, people can claim I chose poorly. It's a lot of work to do ten generations! So asking people to choose in advance the material they'll be tested with makes perfect sense. You listen, and if you agree the music sounds clear enough to notice slight distortion or whatever, you say so and then we do the test. All this stone-walling just shows that some here are confident only when they don't have to actually back them up their claims.

Now, can we get on to a newer test using music that everyone agrees is acceptable?

--Ethan

 

[Ethan Winer]

Ethan, this time I think you stepped over the line in calling out specific members. This isn't a game of gotcha.

I understand, but others have called me out repeatedly. And I'm not calling people out as much as asking them to prove their claims. That's not the same as picking a fight. Here's some history from the previous thread:

I for one find it hard to believe that you could run a signal through three conversions (A/D/A) and hear no difference between the original signal before it was converted. And then they ran it through 10 conversions and still people couldn't hear any difference? It doesn't make any sense. Is there a site to go to that explains how this test was set up and how the gear was configured?

No Tim... Ethan needs to post the research so we can all read for ourselves. I've done this test with 3 A/D/A converters (AX24, PM2 and ULN8).. blindly.

Yeah, but we aren't talking about listening to generation after generation of bit perfect digital copies here. We are talking about converting analog to digital and then back again to analog and claiming there is no difference, even after 10 passes. I'm not buying it which is why I made the smart-ass comment about the test being conducted in a nursing home or a hearing aid clinic.

So far mep has totally dodged the issue by refusing to chime in on either my old SoundBlaster files or my newer Focusrite files. Bruce hasn't even posted at all. I don't see why I should be accused of instigating when I've merely asked two people who made claims to back up those claims.

Further, I never said that ten generations through either my Focusrite or SB sound cards is inaudible! All I said is that modern converters, even budget models, are pretty darn good. I also reported a story I had read (but now can't find) about a high-end manufacturer's test where nobody could hear a change even after ten passes. Both of the above people disputed that, so IMO it's not unreasonable to want to put this to the test.

--Ethan

 

[Phelonious Ponk]

I don't see any point in pretending that tests like these are scientifically valid or lead us to any statistically valid conclusions. They don't. What they could do is teach some people what they can and cannot hear, what subtle means, and bust some of the drama in the Audiophile dialogue. These differences are not dramatic, more often than not they're going to be damn hard to hear at all when we close our eyes (And they should be. This is re-production, not production), and some of the people who swear the differences are obvkous are probably finding that out for themselves, even if they never admit it publicly.

I understand Audiophile hyperbole, FWIW. Pay 5 figures for an amp, it's advantages better be BIG.

Tim

 

[Ethan Winer]

I don't see any point in pretending that tests like these are scientifically valid or lead us to any statistically valid conclusions. They don't.

I don't see why an audibility test like this isn't scientific. And it's truly blind. But okay, nothing earth shattering. I'll give you that.

What they could do is teach some people what they can and cannot hear, what subtle means, and bust some of the drama in the Audiophile dialogue.

There ya go. What I don't understand is why people prefer to cling to beliefs rather than learn the truth about their own hearing and perception. Is it not useful to learn that a $2 wire sounds just as good as a wire costing $2,000? Don't people want to know that they can get the exact same sound from a $200 amplifier as from a $15,000 amplifier? I sure want to know that!

Likewise, if it turns out someone really can hear one wire from another, or CD demagnetizer etc, I want to know that too. As I've said to Michael Fremer many times, as soon as he proves to me in person that he can hear a demagnetized CD, I'll not only change my opinion, I'll post all over the forums that I was wrong.

--Ethan

 

[Phelonious Ponk]

What I don't understand is why people prefer to cling to beliefs rather than learn the truth about their own hearing and perception. Is it not useful to learn that a $2 wire sounds just as good as a wire costing $2,000?

That's not hard to understand at all. It's useful if you haven't bought wire yet. It's reassuring if you bought the $2 wire. If you bought the $2k wire and told a few thousand people on the internet that the difference between it and the $1500 wire it replaced was dramatic, extraordinary, the difference between mere reproduction and real music, well, being unable do distinguish it from lamp cord in a blind listening test is just damned humiliating.
Tim

 

[Gregadd]

I guess what I am taking away from this test, (and I used high quality headphones, two types and an emu something or other super low distortion sound card (for rightmark audio testing) is that generational differences are pretty darn small as far as the bit level presented at and so there you go. But I do believe there is a hearing threshold and its getting worse every year for all of us.

I think Ethan just has to finally come to the conclusion that those who hear differences and all that will never stand out inthe sunlight under scientific review (OK, Carver challenge the one exception and we all know how that turned out) because they just can't acknowledge that they have a hearing threshold, their identity and prestige is all wrapped up in their magic hearing. I don't know why but there you have it IMO.

The problem is that those who "pass" run the risk of being called "statistically insignificant", a "lucky coin" or even worse being asked to continually do it agian.
Michael Fremer participation in the amplifier test is a prime example. He claims to have visited Sean Olives facilitys, if I remember correctly.

Of course we all have hearing thresholds . I thought this test was about gauging what that threshold is. On the question of magic hearing you might want to know that Sean Olive referred to his trained listeners as "golden ears."

As far as Ethan being forced to accept something, despite his protestations, I never met anyone with stronger opinions or beliefs. He has already emphatically stated his beleif on this subject.

Pleas tell me what is scientiifc about this test?

 

[Ron Party]

Well, since yesterday I see Ethan has disclosed the order so, Amir, most of your points, regardless of whether or not I agree with them, have been rendered moot. WRT the strawman argument, it still applies. You did not address it. But you did build several more strawmen.

 

[Gregadd]

Then tell me the minimum length you require and I'll make an excerpt that long so you'll be able to participate without any reluctance.



Why would you not listen using the highest quality playback available? Do you not have a way to play Wave files through your good system? You do know how to burn a CD, yes?

--Ethan

Actually I have been busy practicing "tieing my shoes." I wonder why you would even be intereted in my opinion.

I aksed the questions not of you but to see what other members were doing in thier evaluations.

I was at the library when I encountered your post. They make earbuds available for free and do not allow for CD burning.
On numerrous occaisions I have indicated my requirements for evaluating equipment. I do not participate in
D-double dog dares since my playgorund days.
I listend to your clips. Then I pulled this video form U-tube. The sound quality difference is significant.

In answer to your question when doing evaluations I listen to about ten CD of this quality. I usyually include a "ringer." A poorly recorded CD just to insure the device is not editorializing everything.

I am realy not quite that interested in a perfect copy of imperfection.

Please be advisedt hat I have followed yuor debates around the other audio forums. I am pretty familiar with how they play out.

 

[JasonI]

In general, the scientific method is applicable in a test even if the results aren't statistically significant. This test is uncontrolled and even if there were enough participants to generate a significant result, there is no way to make this more than an interesting exercise for the individual participants and perhaps a few outsiders that enjoy following fun forum topics. Anyway, I may join in if I can find the time. I have really enjoyed and been enlightened by the the Harman training program.

 

[MylesBAstor]

Well, since yesterday I see Ethan has disclosed the order so, Amir, most of your points, regardless of whether or not I agree with them, have been rendered moot. WRT the strawman argument, it still applies. You did not address it. But you did build several more strawmen.

Sure you're not referring to Etan Ron?

 

[amirm]

Fair enough, and in the grand scheme of things revealing the first results should not compromise a subsequent test with completely different music. Here's the order:

Original: focusrite_b.wav
1 generation: focusrite_d.wav
5 generations: focusrite_a.wav
10 generations: focusrite_c.wav

Great. Let's review a prior set of votes:

In generational order : b (original),d,a then c (being the worst).

4 out of 4 right. Hmmm . Did that influence you at all Ethan in not wanting to give the answers? How do you think he accomplished that? Pure luck? Great ears? Him outsmarting you?

That's why I posted a link to my proposed second clip! If I pick music based on what I think reveals degradation, people can claim I chose poorly. It's a lot of work to do ten generations! So asking people to choose in advance the material they'll be tested with makes perfect sense. You listen, and if you agree the music sounds clear enough to notice slight distortion or whatever, you say so and then we do the test. All this stone-walling just shows that some here are confident only when they don't have to actually back them up their claims.

Well, the only stone-walling occurred in us getting a simple set of answers.

 

[Ron Party]

Focusing on someone who only read the results? Hmmm. This whole thread has been trashed by all who've participated.

If there is any legitimate interest in this test, a new thread should be started. Amir, given your concern over Ethan's involvement, and given your capabilities, perhaps you could start the thread? Maybe even perform the generation test with a few different tracks that would more readily as well as less readily (if at all) be detectable.

 

[jkeny]

Focusing on someone who only read the results? Hmmm. This whole thread has been trashed by all who've participated.

If there is any legitimate interest in this test, a new thread should be started. Amir, given your concern over Ethan's involvement, and given your capabilities, perhaps you could start the thread? Maybe even perform the generation test with a few different tracks that would more readily as well as less readily (if at all) be detectable.

Yes you all should be ashamed of your less than honest approach to a thread that was a real attempt at flushing out the truth! I mean come on people the truth is all the Mr Winer is interested in & waking up the public to the hype that surrounds audiophiles. Your attempts at thwarting this valiant effort really just further reinforces how self-deluded you all are.

 

[MylesBAstor]

Focusing on someone who only read the results? Hmmm. This whole thread has been trashed by all who've participated.

If there is any legitimate interest in this test, a new thread should be started. Amir, given your concern over Ethan's involvement, and given your capabilities, perhaps you could start the thread? Maybe even perform the generation test with a few different tracks that would more readily as well as less readily (if at all) be detectable.

Hardly Ron. His reputation precedes him.

It's a contest between Etan and Romy the Cat as to who's been banned from more websites.

 

[Ron Party]

The truth, whatever it may be, knows no author now, does it? No one person owns the truth. The truth does not depend upon how many websites one has seen fit for whatever reason to post, nor does it depend upon from how many websites one has been banned.

Whatever one's predisposition may be towards others, I would hope that one has the honest intellectual curiosity to, at a minimum, know for oneself whether one can discern that which one may claim or, like me, whether one can discern a difference one way or the other.

Me personally? I'm not particularly interested in whether others can or cannot hear the difference. OTOH I'm deeply interested in whether I can. Having never taken the kind of test being discussed in this thread, I personally have no idea whatsoever if I could or could not detect any difference.

A few have posted for public consumption their results. I applaud them. They have the courage to do so and IMO have the security of knowing it is not the end of the world - OK, I know that is dramatic but go with me - to be shown in a public light to be wrong.

Is one more interested in discussing the person or the topic? Is one more interested in learning for him or herself if one possesses the discriminatory hearing capability, or is one more interested in discussing the person who makes a claim, one way or the other, that there is an audible difference in the generations of the track. We know we have members here who fall on both sides of the fence with respect to whether or not there is an audible difference.

So I offered up what I thought might be a more palatable solution to the membership here. That some, but rest assured not all, do not trust Ethan to sponsor this test, had me raise what seemed at least to me to be an obvious solution. Let's have Amir sponsor the test. Does the membership here distrust Amir?

 

[jkeny]

So, Ron, what has changed the "gotcha" motivation of this thread (even if administered by someone else) since you last called it?

"Now, having stated that, Ethan, this time I think you stepped over the line in calling out specific members. This isn't a game of gotcha."

 

[MylesBAstor]

The truth, whatever it may be, knows no author now, does it? No one person owns the truth. The truth does not depend upon how many websites one has seen fit for whatever reason to post, nor does it depend upon from how many websites one has been banned.

Whatever one's predisposition may be towards others, I would hope that one has the honest intellectual curiosity to, at a minimum, know for oneself whether one can discern that which one may claim or, like me, whether one can discern a difference one way or the other.

Me personally? I'm not particularly interested in whether others can or cannot hear the difference. OTOH I'm deeply interested in whether I can. Having never taken the kind of test being discussed in this thread, I personally have no idea whatsoever if I could or could not detect any difference.

A few have posted for public consumption their results. I applaud them. They have the courage to do so and IMO have the security of knowing it is not the end of the world - OK, I know that is dramatic but go with me - to be shown in a public light to be wrong.

Is one more interested in discussing the person or the topic? Is one more interested in learning for him or herself if one possesses the discriminatory hearing capability, or is one more interested in discussing the person who makes a claim, one way or the other, that there is an audible difference in the generations of the track. We know we have members here who fall on both sides of the fence with respect to whether or not there is an audible difference.

So I offered up what I thought might be a more palatable solution to the membership here. That some, but rest assured not all, do not trust Ethan to sponsor this test, had me raise what seemed at least to me to be an obvious solution. Let's have Amir sponsor the test. Does the membership here distrust Amir?

Ron, you're deluding yourself. This is the internet and these people have an agenda, hidden or not.

And you're going to make a decision based on what you hear on your computer speakers. Now that's a reliable test. If it sounded good on computer speakers, I'd run the other way.

 

[FrantzM]

Hardly Ron. His reputation precedes him.

It's a contest between Etan and Romy the Cat as to who's been banned from more websites.

Hi

This is amazing how much we, audiophiles can be intolerant. Our zeal to repeal those who do not prostrate to the altar of High End Audio Orthodoxy seems to be non pareil. While I do have my own set of problems with Romy The Cat, Ethan contributions are to me real. As long as he doesn't get into ad hominem attacks his points should be taken at their value. If they are logical fallacies or unprovable assertions by all means shot them. If they are only contrary to our point of views however is it wise to automatically reject ideas because of their author/source and /or because they challenge our belief system? I for one don't agree with many of his points but I would not reject them all especially when I am not able to refute them on purely logical and scientific grounds.

I don't expect this to change at once. It is after all primarily an audiophile forum but I wish more openness in discussions here, bad faith is getting prevalent on both sides. I have seen people dancing around issues rather than admit they can't prove it or can't sustain their point of views. I , like, many did learn a lot in the beginning in this forum. Right now we are simply trowing darts at each other not building up more knowledge. Our system can evolve when we understand more about what make a good system and IMHO there are sets of attributes of a good system and it doesn't seem to be that much different between audiophiles. The weight attached to the parameters may vary but there is more consensus than we like to think. True knowledge and understanding of these parameters, increase the enjoyment we derive from our gears, that, come only from honest and frank discussions. Not the automatic rejections of ideas or people who challenge us.