Comparative Listening Tests

[stehno]

Oh but I did check your link ( http://enenews.com/ ), I thanked you for it. I started a thread some while back, with dramatic videos. ...

One post had this link: https://www.infowars.com/fukushima-radiation-detected-on-u-s-west-coast/

Another thread; Documentary Heaven (Food for your brain) had this post:
http://www.whatsbestforum.com/showt...or-your-Brain)&p=440811&viewfull=1#post440811

Oh, sorry. Yup.

IMO, you wanna discuss the host of conspiracy theories (actually conspiracies that arent' theories)? None of it really matters much anymore.

IMO this dwarfs anything and everything else that we may entertain or imagine. Cuz there ain't no best case scenario and there ain't no technology to remedy the problem.

 

[KlausR.]

Perhaps you and Klause can identify all sorts of flaws with this test, and it may not be rigorous enough for your standards, yet it must be somewhat legitimate for you to use my result in test #2 to prove your point that I need to sober up and admit how dumb I am and that others should read this thread and conclude the same thing.

By no means I’m an expert in these matters, but having a look at the paper by Zielinski, Rumsey & Bech on biases in audio listening tests there might have been quite a few flaws. Judge yourself, the paper is here:

http://www.acourate.com/Download/BiasesInModernAudioQualityListeningTests.pdf

Here is some more reading stuff:

http://usir.salford.ac.uk/34338/1/F...How to Design and Conduct Listening Tests.pdf

https://tech.ebu.ch/docs/techreview/trev_283-kozamernik.pdf

https://helda.helsinki.fi/bitstream/handle/10138/155061/thedesig.pdf?sequence=1

The fact that you know that you are listening to two different cables albeit without knowing which is which makes the test a sighted one. Listener bias is a strong factor and I can understand that it’s not easy to accept that when one has not taken this into account all one’s life.

 

[jkeny]

He misread my post but that doesn't matter. What matters is that this thread is created by Ron, quoting Peter word for word, and then expressing strong concern regarding method of evaluation by subjectivists:



If Peter thinks Ron has mis-stated the events that led to such strong concerns on his part, he should address Ron, not me.

Wow, where have I heard that before - "I'm just repeating what someone else said - take it up with them"(take it up with Judge Napolitano on Fox News in this case)

I am simply agreeing with Ron that such tests easily produce faulty results.

So based on Ron's misunderstanding of the listening session you still want to agree with his concerns from this this mistake? Now that you know Ron was wrong are you going to be intellectually honest??

Since you are volunteering to help, do you understand Ron's concerns and do you agree with them?

Do I agree with Ron's mistaken concerns? No - it just shows how home organised ABX testing is a sciencey ruse propagated by people who like to portray themselves as objective. Again some intellectual honesty needed from you Amir.

 

[PeterA]

By no means I’m an expert in these matters, but having a look at the paper by Zielinski, Rumsey & Bech on biases in audio listening tests there might have been quite a few flaws. Judge yourself, the paper is here:

http://www.acourate.com/Download/BiasesInModernAudioQualityListeningTests.pdf

Here is some more reading stuff:

http://usir.salford.ac.uk/34338/1/F...How to Design and Conduct Listening Tests.pdf

https://tech.ebu.ch/docs/techreview/trev_283-kozamernik.pdf

https://helda.helsinki.fi/bitstream/handle/10138/155061/thedesig.pdf?sequence=1

The fact that you know that you are listening to two different cables albeit without knowing which is which makes the test a sighted one. Listener bias is a strong factor and I can understand that it’s not easy to accept that when one has not taken this into account all one’s life.

Klaus, I'm sure the series of tests that Al presented that evening were full of flaws. Thank you for explaining the notion that knowledge of the items under test, even though unseen, makes it a sighted test. I did not know that. I think that I did learn a few things, and that is always good. Not knowing that Al had rotated the Tube Traps prior to the last A/B/X test to identify which cable was which convinced me that I could hear a difference but I did not know that a new variable had been introduced, so I what I heard confused me. Does this introduction of a new variable unknown to the test subject qualify as "unsighted" in this instance?

The whole exercise was to assist Al in his cable decision by offering additional listening impressions, biased, unbiased, sighted or unsighted, from a friend who knows his system pretty well. It was not a rigorous investigation, but a casual effort to learn and confirm impressions, flawed as it was. I think it helped Al, and I did learn something. It involved minimal effort and was accomplished with what he had on hand and available to him. It may not have been ideal, but it got him a bit further along in his quest to improve his system. In that sense, it is all good, as the younger generation is fond of saying.

 

[amirm]

Wow, where have I heard that before - "I'm just repeating what someone else said - take it up with them"(take it up with Judge Napolitano on Fox News in this case) So based on Ron's misunderstanding of the listening session you still want to agree with his concerns from this this mistake? Now that you know Ron was wrong are you going to be intellectually honest??

Do I agree with Ron's mistaken concerns? No - it just shows how home organised ABX testing is a sciencey ruse propagated by people who like to portray themselves as objective. Again some intellectual honesty needed from you Amir.

Ron did not misunderstand anything.

Now run the simple test of playing the same file over and over again that I suggested and report back if you hear it identically. That would tell me a lot about your honesty.

 

[amirm]

Do I agree with Ron's mistaken concerns? No - it just shows how home organised ABX testing is a sciencey ruse propagated by people who like to portray themselves as objective. Again some intellectual honesty needed from you Amir.

And another thing. This is not an ABX test. It is an AB test. There is no reference. And there are only two (known) alternatives.

 

[amirm]

That is not quite true, amirm. Al gave me a four tests:

I didn't say you were given one test. I said one test was conducted where Al knew the answer with certainty and you did not. Remember we were talking about one taking an exam where your instructor knows objectively what the right answer is. This was the case with Al. He had you give one set of impressions sighted, then he gave you a test where the knowledge of which cable was which was hidden from you. You did not manage to give the same response as you did in sighted test.

Unfortunately for you all , we get to rely on that negative outcome and ignore any (single) positive outcome. For any positive outcome it needs to rise above someone guessing. Getting one answer right out of one trial means the probability that you knew the answer is just 50%. If you ran the test again, then it would rise up to 75% if you got the right answer again. Standard practice requires 95% confidence. To get there you need 7 or 8 trials (from memory).

I can't tell you how many times I have run a blind ABX test where I got three or so answers right in a row but then could not get them right from there on with the result being that I simply guessed right the first few times. It is not that hard to guess which way a coin lands three times in a row. But seven or eight times, gets really hard.

You and Al did great here experimenting. Would be great if you get together and repeat in this manner. First listen sighted, with your own material or whatever you think is revealing to you. Do as much as this as you like. It is actually encouraged in testing protocols (it is called training). Then the formal part of the test starts and all you want to do is identify which cable is which. Don't worry about characterizing them. If you can't tell them apart reliably, it doesn't matter what you think of their sound. Do this 7 or 8 times. If you get them right every time, then you are done. Unfortunately if you miss one or two times, then you have to keep going. I have written an article on this here: http://audiosciencereview.com/forum/index.php?threads/statistics-of-abx-testing.170/

Anyway, let me tell you much credit also for participating in this type of testing. Much appreciated.

 

[SoundAndMotion]

Hi Peter,

Since the evening of testing you describe is the basis of the OP, I want to understand a little better.

Al gave me a four tests:

1. Identify differences if any and preference if any between two cables, A and B.
I completed this test and was able to identify a difference, a preference, and able to describe differences.

I wouldn’t really call this a test. It is a task or test preparation.

2. Listen for a third time to X and recall whether it is A or B.
I failed to correctly identify X as A or B.

This is either a test or one trial of a test (see below). Alone, it doesn’t say much to me.

3. Using different musical cuts, listen to both cables again in random order and identify differences and preferences between A and B.
I completed this test and was able to identify a difference, a preference, and able to describe the differences. I was also able to pick which cable matched the cables in test #1 in the correct order, not knowing the identify of the cables. This contradicts result of test #2.

I’m a little unclear, you listened twice with music you wanted: once to unknown cable “X” and once to unknown cable “Y” and then you correctly identified which was A and which was B (from step 1 above)? Or were there more comparisons? Was it agreed upon that X and Y were different, or was it left open that they might be the same?

4. Listen for a third time to X with the new musical cuts and recall whether it is A or B.
This was difficult as I thought that the third sample did not match either A or B, yet I thought those were the only two possibilities, so I gave up saying I was confused. Al then revealed that he actually changed the parameters by rotating the Tube Traps while I was out of the room between #3 and #4, in effect creating a C variable. So we concluded that this was a flawed or false test because an additional variable was introduced without me knowing it.

Agree.

You are incorrect to describe the event as one test. You are also incorrectly describing how I failed and not acknowledging that I did complete and pass two of the three given tests.

It looks to me like you did 2 tests: 1 identification task with 1 trial, and 1 matching task with 1 trial. In perceptual experiments (psychophysical, psychoacoustic), it’s unusual to say “passed” or “failed”. You certainly had 1 correct and 1 incorrect. You are right that the one correct and the one incorrect are both part of the picture. I won’t deliver a lecture on psychometric functions here (although if there’s interest I can give a little guidance), but there are 3 reasons you could have answered the first one wrong: you are near your threshold for “the effect”, lapse rate (knew the answer; gave it wrong), or you guessed (wrong). Many believe it can only be the last option, incorrectly. It may well be, but there is not enough info to confidently draw that conclusion. Many others believe that the fact you were doing tests blocks your ability to distinguish. This does not have to be true, but it’s worth discussion.

I have a couple more comments, but I have to run now.

 

[jkeny]

And another thing. This is not an ABX test. It is an AB test. There is no reference. And there are only two (known) alternatives.

Oh dear, dear - you really are so deep into this attempt to win every point that you can't even read properly
PeterA said this:
"2. Listen for a third time to X and recall whether it is A or B.
I failed to correctly identify X as A or B."

YES that was an ABX test - what deflection or denial do you now wish to put forth or will you just leave for another 2 weeks?

 

[amirm]

Well I now seen for myself how they "sanitize" threads over at ASR. Mike made another mockery of Amir, and they already started deleting his posts. Mike says give it a few days and go back and good chance his posts will be edited, along with other posts to completely change how things went in Amir's favor.

http://audiosciencereview.com/forum/index.php?threads/close-in-jitter.1621/page-5#post-40611

Mike, I just looked and the only post was deleted was this:

"Amir you need to learn when it's time to throw in the towel. You can't bring a cap gun to a showdown and expect to win. Especially if your opponent is packing a 44 magnum. You already lost. And it's okay to accept it. It's part of being an adult."


You need to grow up as an adult and talk about audio science on ASR Forum. This type of personal commentary has no place there. Put forward data and we can discuss it. So far other than posting some graphs and measurements from others and papers which you don't show any understanding of either, is not it.

 

[SoundAndMotion]

The fact that you know that you are listening to two different cables albeit without knowing which is which makes the test a sighted one.

Hi Klaus,
I'm sorry. This is wrong.

 

[jkeny]

Ron did not misunderstand anything.

Now run the simple test of playing the same file over and over again that I suggested and report back if you hear it identically. That would tell me a lot about your honesty.

Why don't you prove your own honesty first & answer what I & others have asked you many times & which you sneaked off leaving such questions unanswered:
- Do you not understand USB audio to the extent that you would claim Swenson said that USB packets were transmitted?
- Do you not understand FFTs & think that oversampling is a part of FFT as you claimed?

 

[SoundAndMotion]

And another thing. This is not an ABX test. It is an AB test. There is no reference. And there are only two (known) alternatives.

Hi Amir,

I'm sorry. Steps 1 and 2 as described by Peter together make and ABX test.

 

[KlausR.]

Klaus, I'm sure the series of tests that Al presented that evening were full of flaws. Thank you for explaining the notion that knowledge of the items under test, even though unseen, makes it a sighted test.

On second thought I was not detailed enough: when A and B are part of an ABX you are supposed know that A and B are different, that's part of the test, so in your case that would be ok. When A and B are part of a paired comparison, you should not know that they are different.

A variant of ABX is the triangle test, where the only thing you know is that two of the three samples are the same and the third is different: you present all possible combinations ABB, BAA, AAB, BBA, ABA, BAB.
Another variant would be the duo-trio where you present a known reference and two samples one of which matches the reference.

The introduction of a new variable such as rotating the Tube Traps would invalidate the test altogether.

Klaus

 

[amirm]

Oh dear, dear - you really are so deep into this attempt to win every point that you can't even read properly
PeterA said this:
"2. Listen for a third time to X and recall whether it is A or B.
I failed to correctly identify X as A or B."

YES that was an ABX test - what deflection or denial do you now wish to put forth?

That is what he said. That doesn't make it an ABX test.

An ABX test would start with a known reference. The listener would be told that is the reference. That is A. An alternative sample is presented as B. Then a third sample, X, is presented and the listener compares that to A or B to see if he can tell which one it is. X is randomized and test repeated multiple times until the desired number of trials is reached.

There is no reference in this test. No presentation was made for Peter to keep hearing A against B and comparing it to X. No randomization or repeat of the trial was performed.

If you wanted to make an ABX test, you would start with a generic cable as the reference. Then you introduce only one of the two cables into the equation as B/X. You run that and see what the outcome is. Then you repeat the test with the other cable. You don't test two variations against each other. That is AB testing.

 

[SoundAndMotion]

That is what he said. That doesn't make it an ABX test.

An ABX test would start with a known reference. The listener would be told that is the reference. That is A. An alternative sample is presented as B. Then a third sample, X, is presented and the listener compares that to A or B to see if he can tell which one it is. X is randomized and test repeated multiple times until the desired number of trials is reached.

There is no reference in this test. No presentation was made for Peter to keep hearing A against B and comparing it to X. No randomization or repeat of the trial was performed.

If you wanted to make an ABX test, you would start with a generic cable as the reference. Then you introduce only one of the two cables into the equation as B/X. You run that and see what the outcome is. Then you repeat the test with the other cable. You don't test two variations against each other. That is AB testing.

Amir,
You have mixed up ABX with ABC/hr (aka BS.1116). Peter and Al did an ABX.

 

[KlausR.]

Originally Posted by KlausR.
"The fact that you know that you are listening to two different cables albeit without knowing which is which makes the test a sighted one."

Hi Klaus,
I'm sorry. This is wrong.

Difference paired comparison A vs B: you present all possible combinations AA, AB, BA, BA and ask whether they are same or different. Obviously you should be left in the dark about what the respective pair includes.

 

[Steve williams]

Mike, I just looked and the only post was deleted was this:

"Amir you need to learn when it's time to throw in the towel. You can't bring a cap gun to a showdown and expect to win. Especially if your opponent is packing a 44 magnum. You already lost. And it's okay to accept it. It's part of being an adult."


You need to grow up as an adult and talk about audio science on ASR Forum. This type of personal commentary has no place there. Put forward data and we can discuss it. So far other than posting some graphs and measurements from others and papers which you don't show any understanding of either, is not it.

Sorry but Silly Goose is NOT Mike

 

[PeterA]

Ron did not misunderstand anything.

Now run the simple test of playing the same file over and over again that I suggested and report back if you hear it identically. That would tell me a lot about your honesty.

Amir, I don't understand this. I can think of at least these several possible outcomes:

1. After repeated listening, the file sounds the same to him, he reports that it sounds the same to him. He is correct and he is honest.
2. After repeated listening, the file sounds the same to him, he reports that it sounds the same to him. He is incorrect and he is honest.
3. After repeated listening, the file sounds the same to him, he reports that it sounds different to him. He is correct and he is dishonest.
4. After repeated listening, the file sounds the same to him, he reports that it sounds different to him. He is incorrect and he is dishonest.
5. After repeated listening, the file sounds different to him, he reports that it sounds different to him. He is correct and he is honest.
6. After repeated listening, the file sounds different to him, he reports that it sounds different to him. He is incorrect and he is honest.
7. After repeated listening, the file sounds different to him, he reports that it sounds the same to him. He is correct and he is dishonest.
8. After repeated listening, the file sounds different to him, he reports that it sounds the same to him. He is incorrect and he is dishonest.

One might assume that because the file is the same, that it should always sound the same. Trained listeners like yourself, knowing that the file is the same ahead of time, may repeatedly hear it as the same, either because of their training or their bias caused by knowing that it is the same file. But an untrained listener may start to notice different things because his mind may wander, loose focus and concentration, and then may either hear it as the same, because he has been told it is the same, or as different because of his wandering sense of perception. And to complicate matters, he may report it honestly or dishonestly, because he is human. I don't see how you could know with certitude which is which and arrive at a conclusion about the listeners honesty?

The fact that you write that his reporting would tell you a lot about his honesty implies to me that you are inherently biased because you presume to know which of the possible outcomes I listed is the correct one.

 

[amirm]

Amir,
You have mixed up ABX with ABC/hr (aka BS.1116). Peter and Al did an ABX.

No, ABC/hr allows quality rating which is actually close to what Peter was doing. He definitely was not doing an ABX test which only identifies if the two samples are different.