Converter loop-back tests

[Ethan Winer]

Nobody in the rec.audio.pro newsgroup could recall the details of that loop-back test using high-end converters, so I did my own today with my mid-level Focusrite Scarlett ($250). I did ten sequential passes, with the last pass being a 10th generation copy. I used short excerpts from two different types of music - a classical string section and a pop tune. The four files are Original, Pass 1, Pass 5, and Pass 10, and each is about 5 MB:

Edit: To reach a wider audience and get more data, this is now a more formal article on my web site:

Converter Loop-Back Tests

If each generation sounds more grainy or "hairy" as was reported earlier with my SoundBlaster files, maybe some of you will be able to guess which file is which with this newer converter. However, I trimmed the file lengths slightly to be sure that will not give you a clue.

I'll be interested to see how many people can discern the various generations from each other and from the original. Especially Bruce. But please folks: if you don't state your guesses before I post the answers in a few days, do not post later that you could tell which is which!

--Ethan

 

[opus111]

In generational order : b (original),d,a then c (being the worst).

 

[Phelonious Ponk]

The pressure and stress are far too much for me. I'm suffering performance anxiety. Just tell my which one is which, then I'll relax and I'll tell you how they sound different.

Tim

 

[Phelonious Ponk]

Seriously...what should I listen for (it's ok if you tell me after the game is over)? I confess I couldn't stand to listen to the pop track more than once, so I concentrated on the strings. I listened to the high frequency information in the violins to see if one sounded more hissy or sibilant than the other. Then I listened for wooliness -- excess bloom and boom in the low end. Anything I'd say at this point would be a WAG.

Tim

 

[Phelonious Ponk]

I went back and tried to hear what Tom and Opus heard. I just compared b and c. B sounded better to me on the first run through, but after that I couldn't even talk myself into hearing a difference. Whatever is there is uber-subtle. And I'm listening on cans.

Tim

 

[amirm]

Here is the generational order.:

1. d
2. a
3. c
4. b (many generations down)

Unless of course he has cheated and cooked the test in which case, shame on you Ethan.

 

[Ethan Winer]

I'm really glad to see people unafraid of posting their choices. Please keep 'em coming, especially those who believe they can hear even one generation through the finest converters. I've seen tests like this in the past at Gearslutz where the people with the strongest opinions about converter quality refuse to post their guesses. Hopefully, we'll get many more replies.

Unless of course he has cheated and cooked the test in which case, shame on you Ethan.

Why would I cheat? As always, I'm glad to make the entire SONAR (DAW software) project available if people want to verify. Further, anyone near me in western Connecticut who'd like to see the files and setup in person is most welcome to visit. The project is all set up, so you can even witness the copying done all over again in person.

--Ethan

 

[amirm]

Why would I cheat?

'Cause it is common for people to do that . I have told this story before but once again when I worked at Microsoft we acquired Pacific Microsonics -- the inventor of HDCD. After we acquired them, one of the test engineers said he thought the thing really didn't do much. I.e. embedding 20 bits in 16 bits had no value. I told him he was dead wrong and that and I had tested them and the 20 bit version was better. So right there he gives me a headphone and says to turn around and he will AB them for me. He plays the clips and I am confident which is which and I tell him that. The son of a gun turns around says that he kept playing the same identical file! I turn back and have him repeat and of course now they both sounded the same . Fortunately we did not acquire them for HDCD. They had built this neat speaker correction technology for computer speakers which we were interested in.

Back to this topic, I have been through online tests like this where people played the same trick with the same goal: not testing what is said but to show the weakness in human subjective evaluation. My reasoning for saying that was not this though. I will explain later. It is not what you think . Definitely was not calling your ethics into question but rather, how careful you were in creating this test. Again, more when everyone is done and your reveal the results.

 

[Ethan Winer]

He plays the clips and I am confident which is which and I tell him that. The son of a gun turns around says that he kept playing the same identical file! I turn back and have him repeat and of course now they both sounded the same .

This is why I offered not only the entire SONAR project, and not only to visit me in person to hear the files, but even to watch me do the entire test again. It took me a few hours to set up (shoulda been 15 minutes, don't ask ), but now it's all set so it's easy to do again in about half an hour.

Definitely was not calling your ethics into question but rather, how careful you were in creating this test.

I've publicly screwed up tests like this in the past, so believe me, I was extremely careful to do it correctly this time! I even aligned a few sets of files and nulled them to prove the files really are copies. Had they nulled completely the copies would be suspect. But they do not null completely, so I'm fully confident I did the test correctly. Each file really is different from the previous one.

BTW, what makes tests like this possible to screw up is the fancy (read: overly complicated) mixing software that's bundled with so many sound cards. When you can route channels to each other via software, that can override the physical loop-back from analog output to input. But I'm quite certain I got it right. Heck, just seeing different sets of guesses shows me that this modern sound card is at least decent. If it sucked as bad as some suggest, everyone who posted so far would have given the same list because it'd be so obvious. The beauty of a test like this is it's done entirely in software having very low distortion, and people can listen on their own earphones or speakers.

I'm still waiting for more people to post. Pretty please? The more people that chime in, the more statistically significant the results will be.

--Ethan

 

[Ethan Winer]

^^^ Thanks so much for your thoughtful contribution. I'd be glad to do the test again with different music, though I imagine you'll find a reason to not post your choices no matter what music was used.

--Ehan

 

[mep]

^^^ Thanks so much for your thoughtful contribution. I'd be glad to do the test again with different music, though I imagine you'll find a reason to not post your choices no matter what music was used.

--Ehan

Ethan-Did you change the spelling of your name? And I thought it was a thoughtful contribution because micro shared his feelings with you about how your music selections affected him.

 

[audioarcher]

I can not stop your imagination, but can add two suggestions. Use selections at less some minutes long - otherwise we (at less I ) can not dig in the music and the comparisons will become tiresome . Also choose a recording having a natural quality with real acoustics. If the early generation sounds poor the differences will surely be minimal. But I also understand it is your test and you are free to choose the methodology.

I agree with using better material. Preferably recorded with real instruments in a real space not a dead studio with reverb added back in.

I only have the capability to play these through my netbooks mono speaker. At first I thought b sounded better than a. After I listened to all of them many times it is too hard to pick an order. C seems to be the best but I would not bet on it. Perhaps if I could play it back on something better I could be of more help.

 

[Phelonious Ponk]

Micro, FWIW, I thought the second track was pretty awful too. I ended up just using the first track.

Tim

 

[Ethan Winer]

Micro, FWIW, I thought the second track was pretty awful too. I ended up just using the first track.

Okay, fair enough guys, I'll do it again with a better recording. I think I have a nice cello concerto that a friend recorded in Russia with a real orchestra, that is not available commercially. But I can't make the clips too long or they'll total too many MB.

--Ethan

 

[Ethan Winer]

Before I go to the trouble to make ten copy generations, please folks let me know if you agree that this recording is high enough quality for the test:

Edit: To reach a wider audience and get more data, this is now a more formal article on my web site:

Converter Loop-Back Tests

This short excerpt is the gentle ending of the Victor Herbert cello concerto, 2nd movement, performed by the Moscow Symphony Orchestra. The file is just over 7 MB.

--Ethan

 

[amirm]

Before redoing the tests, are you going to reveal the results of this one?

 

[Ethan Winer]

Before redoing the tests, are you going to reveal the results of this one?

No, because not enough people chimed in. So why even bother? But if it matters, I'll tell you that you did not choose correctly. If you email or PM me, and promise not to share, I'll gladly tell you the correct order.

--Ethan

 

[amirm]

No, because not enough people chimed in. So why even bother? But if it matters, I'll tell you that you did not choose correctly. If you email or PM me, and promise not to share, I'll gladly tell you the correct order.

--Ethan

Ha? You are declaring my results as being wrong but refuse to give the answers so that we can discuss them? I have no use for private version of the results. It matters not whether I am wrong or right on some random test. What may matter is what we talk about regarding the test. The learning is in that. You keeping the results secret will not allow for examination of the test and its results. This is beside the fact there is no data point of interest here anyway. Who here has ever asked about generational loss in digital? Or what happens there when you play the files on a computer which is what people are doing?

Anyway, please forget that I asked. It is clear that you have no interest in discussing the science but only want some numbers from this forum to go and publicize and for that you need more stats. Seeing how hardly anyone here cares, I suggest trying another forum. I would close your thread except that I don't take moderator action when I am involved in such discussions. I leave it to my colleagues to make such a decision.

 

[Ethan Winer]

You completely misunderstand. The main point is with only two people guessing, it's not nearly enough to determine statistical significance. Plus, I want to be sure I don't taint the next test by leaking what might be a way for people to identify the degradation with each generation.

If people are not interested in doing this, that only tells me that deep down they know they won't be able to hear the degradation. Including (especially) those who insist they can hear one generation let alone five or even ten. Indeed, this test will put a lot of these "arguments" to bed permanently. Or maybe some people don't want that either? Do people really prefer to not know what is easily known?

So, is this new music excerpt acceptable to folks or not?

--Ethan

 

[mojave]

I guess C-original, D, B, A-10th. They are very close it is just the way the music feels rather than any real difference that I can point to.

I listened before I read that you were going to do new music. I'm not exactly sure how other music will help. The process should degrade any music the same.