Why do some Objectivists fear Psychoacousitics?

[FrantzM]

Hmmm, I think you are confused between acoustics & psychoacoustics. I seem to have to repeat myself. I already said that JNDs need to be revisited as they have been based on simplistic test signals - signals that ignore what we now know about psychoacoustics.


OK, lets clear this up as I believe what you said on another forum to be dismissive of Amir's results:

John

Interestingly enough you keep on coming back to Amir results .. Which were under blind conditions ...

 

[Whatmore]

But I don't insist on sighted tests to "prove" anything but many objectivists insist on DBTs.
Even when a valid, rigorous DBT is run, (& not many are), I find it only revealing of some elements in the soundfield
I find that long term listening to be more revealing of the other elements - the ones I consider more important - things like the illusion of realism of the sound. This "realism" illusion won't be so evident if the distortions, frequency/amplitude anomalies, are present - the ones DBTs are directed towards.
So, to my mind, long term listening are more revealing than quick A/B style DBTs & less prone to error

What I'm suggesting is the inclusion of internal controls in DBTs - it's not my idea, it's recommended in the published standards for conducting DBTs. I'm just highlighting why it's needed & the lack of such controls in almost all BTs I've seen.
If you or Tomelex's wish to argue that these controls are not necessary then do so.
Don't distract by talking about another test - defend the test you support or admit that it's flawed & do something about it.

Is this objectivity of objctivists in action?

How valid are long term listening tests? Can you provide some rigour around the incidence of false positives?
If you can't, then long term listening tests are not valid.

If you disagree, please tell me why?

 

[jkeny]

You're just being silly now, Whatmore & behaving like a troll

 

[amirm]

Let's not get personal please.

 

[Whatmore]

You're just being silly now, Whatmore & behaving like a troll

I'll ignore the insult.
I'm interested in why you don't demand the same rigour around sighted tests, be they short or long term, as you do around blind tests.
I'm possibly misunderstanding you though, I have assumed you mean long term *sighted* tests are better.
How about long term blind tests?

 

[jkeny]

I'll ignore the insult.
I'm interested in why you don't demand the same rigour around sighted tests, be they short or long term, as you do around blind tests.
I'm possibly misunderstanding you though, I have assumed you mean long term *sighted* tests are better.
How about long term blind tests?

It wasn't meant as an insult, just an observation that posting the exact same post more than once is not engaging in dialogue, it's more like trolling (posting to get a reaction, etc.)

Yes, I'm making the distinction between quick A/B listening tests Vs long term relaxed listening.

For a start, we don't have the experience, money or dedication to run rigorous blind tests as research scientists do - so the tests I'm talking about are the usual ones run on forums.

These quick A/B listening (usually blind) tests are so fraught with difficulties, with all the psychological factors at play, that it is very unreliable. Failure to recognise this is just blind belief, as far as I'm concerned.
A suggestion to improve this & a huge step forward, in my book, would be to include internal controls in these tests. It would give us all a way of evaluating the reliability of the results.

I find that natural, relaxed listening over a period of time has less factors at play & any that are at play tend to be reduced in effect due to the use of many different listening sessions, with different music, in different times of the day, with different music, etc. So if we are tired on one listening session, it only has an influence in our listening for that session - same with moods, our hearing, etc. As I've said already I find that anything uncovered in quick A/B differences that is of importance in our long term enjoyment of the device, will be evident in long term listening.

Now the whole notion of knowledge/sightedness is way over-emphasised, in my view - I'm sure it has some small influence but nothing like the influence that is attributed to it (based on the number of posts on forums). That was one of my questions about the Harmon test - the exaggerated high scores of the sighted results are the result of the psychological bias that the Harmon employees brought to the test. It is incorrect to assume that all normal people (who don't have a pre-conditioned bias) will be so influenced by sight/knowledge of the device. Or to put it another way, if there is some initial attraction of the visuals of a device, these will lessen over the time of longer term listening. The same applies to an initial sound that attracts (for the wrong reasons) - over time this is usually found to be tiring over long term listening.

I recognise that, based on the look/reputation of a device, we have a pre-conditioning about what it will sound like - but we are not slaves to this. So we expect a large, expensive looking speaker will have a large, enveloping, well balanced sound but I have never experienced being fooled by this - in fact, if the speaker doesn't deliver on my expectations (which are usually greater than reality) the speaker is diminished, in my initial view.

A guy who was looking to get a new DAC organised a sighted listening of some DACs during the summer, a DCS stack was introduced which we all expected to hear great things from - it looked expensive, it felt expensive & we knew it's reputation. The disappointment was palpable when we heard it - it quite obviously didn't match the other DACs we had heard. As it turned out, some days later, the guy discovered that we had not connected the two dCs boxes optimally. Point being, nobody, was swayed by the dCS because of it's visuals & reputation & price. Was the sound that much different to the other DACs - no, it was just lifeless & uninvolving.

BTW, even during this relaxed, sighted listening session, I was bored after listening to the same or an alternative track after about 5 or 6 trials- I found my mind wandering, my focus drifting i.e not really listening. The guy had more motivation & kept going but he also knew that he needed to listen to all the DACs over the following week to form an opinion on which he liked. He also changed his preference during that week

 

[Whatmore]

It wasn't meant as an insult, just an observation that posting the exact same post more than once is not engaging in dialogue, it's more like trolling (posting to get a reaction, etc.)

Again you accuse me of trolling. Please stop

I pose the same question over and over again because I'm interested in your answer. Why will you not answer it

I'll try a different way:

You require that blind tests need

..... to include internal controls in these tests.

But your evidence that long term testing is better is given simply as

I find that natural, relaxed listening over a period of time has less factors at play & any that are at play tend to be reduced

Do you see that this is a double standard?

You insist on hard evidence that blind tests do not produce false negatives and yet you claim

Now the whole notion of knowledge/sightedness is way over-emphasised, in my view - I'm sure it has some small influence but nothing like the influence that is attributed to it

Do you see that you are applying a double standard if you provide no more evidence than, "in my view" ?


So again, why will you not answer (with proper evidence) the question as to why sighted testing or sighted long term listening is better than blind testing or blind long term listening?

 

[jkeny]

Again you accuse me of trolling. Please stop

I didn't accuse you again - just explained my reason for initially saying you were trolling[/quote]

I pose the same question over and over again because I'm interested in your answer. Why will you not answer it

I'll try a different way:

You require that blind tests need


But your evidence that long term testing is better is given simply as


Do you see that this is a double standard?

You insist on hard evidence that blind tests do not produce false negatives and yet you claim

Do you see that you are applying a double standard if you provide no more evidence than, "in my view" ?


So again, why will you not answer (with proper evidence) the question as to why sighted testing or sighted long term listening is better than blind testing or blind long term listening?

I said "I find" long term listening more valuable & gave my reasons.
I'm not using it as a cudgel to beat people up with (as is often the case with objectivists & DBTs) - it's my personal experience. I also gather people's impressions of their audio devices if they have lived with them for a while & us this to derive a shortlist of what is probably worth investigating.

There's no double standard - I find quick A/B tests unreliable for all the reasons I've given & suggest using controls to improve their reliability. And BTW, this is part of the standards set for running blind tests - using internal controls - it's not "my requirement", I think it would be a very good idea for everybody concerned, though.

So far you have not argued against this specifically on the basis of why you think it is not a good idea for blind tests, instead you have tried to pitch sighted Vs blind testing.
Either you accept that internal controls will improve the reliability of blind tests or you don't? I don't know, I haven't heard your logic about this matter?

 

[Whatmore]

A duckin' and a weavin'....

Of course better controls will improve the reliability of *any* test. So what controls do you think could improve the reliability of sighted long term tests ?

 

[jkeny]

A duckin' and a weavin'....

Of course better controls will improve the reliability of *any* test.

Yes, that's one of the usual stock answers I hear when this is brought up i.e. "we have no interest in improving blind tests - it gives us the results we want, oops I meant the correct results" And onward we go - no controls, unreliable blind test results & those citing them clinging on with blind belief

So what controls do you think could improve the reliability of sighted long term tests ?

None, as far as I can see but if you have any suggestions I'm willing to listen - I welcome any opportunity to make my comparisons more accurate.

 

[esldude]

I said "I find" long term listening more valuable & gave my reasons.
I'm not using it as a cudgel to beat people up with (as is often the case with objectivists & DBTs) - it's my personal experience. I also gather people's impressions of their audio devices if they have lived with them for a while & us this to derive a shortlist of what is probably worth investigating.

There's no double standard - I find quick A/B tests unreliable for all the reasons I've given & suggest using controls to improve their reliability. And BTW, this is part of the standards set for running blind tests - using internal controls - it's not "my requirement", I think it would be a very good idea for everybody concerned, though.

So far you have not argued against this specifically on the basis of why you think it is not a good idea for blind tests, instead you have tried to pitch sighted Vs blind testing.
Either you accept that internal controls will improve the reliability of blind tests or you don't? I don't know, I haven't heard your logic about this matter?

http://www.nousaine.com/pdfs/Flying Blind.pdf

You might have seen this before. It isn't academic research quality testing. But as Whatmore complains, I too find your automatic value to long term listening being trustworthy suspect and lacking in evidence. The above article could even be said to have a positive control of sorts. Distortion levels known to be reliably detected blind in short term tests.

Given long term listening, in this case weeks, people failed to detect 4% distortion vs clean. Some of those very same people then did a blind test using short intervals of a few seconds which they got at or near 100% correct. Not a perfect test, but more than nothing. Some data that long term doesn't equal short term perceptual limits. And your data for the obverse idea?

 

[Whatmore]

I said

Of course better controls will improve the reliability of *any* test. So what controls do you think could improve the reliability of sighted long term tests ?

you said

Yes, that's one of the usual stock answers I hear when this is brought up i.e. "we have no interest in improving blind tests - it gives us the results we want, oops I meant the correct results" And onward we go - no controls, unreliable blind test results & those citing them clinging on with blind belief .

did you read my answer? I agreed that better controls would be an improvement.
How do you twist that to read "we have no interest in improving blind tests...."

 

[jkeny]

I said



you said



did you read my answer? I agreed that better controls would be an improvement.
How do you twist that to read "we have no interest in improving blind tests...."

Based on the fact that I have come across this reply before & yet have never seen these controls used in forum run blind tests. To me it reads as a generalised dismissive agreement to the principle rather than actually engaging with it.

So, apologies if I misread your reply, let's then talk about internal controls in blind tests. Do you agree that blind tests need improvement in order to be reliable? In what ways do you see the controls would improve the reliability of these tests?

 

[jkeny]

http://www.nousaine.com/pdfs/Flying Blind.pdf

You might have seen this before. It isn't academic research quality testing. But as Whatmore complains, I too find your automatic value to long term listening being trustworthy suspect and lacking in evidence. The above article could even be said to have a positive control of sorts. Distortion levels known to be reliably detected blind in short term tests.

Given long term listening, in this case weeks, people failed to detect 4% distortion vs clean. Some of those very same people then did a blind test using short intervals of a few seconds which they got at or near 100% correct. Not a perfect test, but more than nothing. Some data that long term doesn't equal short term perceptual limits. And your data for the obverse idea?

I did see that report before

I have no evidence for you that will satisfy you or probably anybody else other than this is what my experience has led me to conclude - that sighted long term listening is the lesser of two evils. Yes it's prone to mistakes but I would contend less so than the typical forum run blind test & it is fundamentally different in that it pays as much attention to the whole audio performance as it does to the very fine grained differences examined in A/B testing.

I also would direct you to the fact that audio product development do not use blind tests to continually check their progress & yet they progress to better sounding products.

I would also point to the fact that in an ABX blind test a sighted period is used to identify areas in the audio sample that will be focussed on in the blind trials in order to differentiate between the two tracks.

 

[Whatmore]

I did see that report before



I also would direct you to the fact that audio product development do not use blind tests to continually check their progress & [I]yet they progress to better sounding products.[/I]

what evidence do you have of this?
To be clear, I mean evidence of the same standard that you require of blind testing

 

[esldude]

I did see that report before

I have no evidence for you that will satisfy you or probably anybody else other than this is what my experience has led me to conclude - that sighted long term listening is the lesser of two evils. Yes it's prone to mistakes but I would contend less so than the typical forum run blind test & it is fundamentally different in that it pays as much attention to the whole audio performance as it does to the very fine grained differences examined in A/B testing.

So after all this, after trying to hold blind testing's feet to fire, you have nothing. You have your experience. You harp on various deficiencies in blind testing that may be important as if it invalidates much of such testing done. Offering long term listening as the superior alternative, and all you have in support of it is your contention this is so from your subjective unmonitored unconfirmed experience. I expected more.

I also would direct you to the fact that audio product development do not use blind tests to continually check their progress & yet they progress to better sounding products.

Harman is using it for loudspeakers. There have been pro audio companies that use it for improving amplifiers. Harman is doing the tests to figure out how to use it for better headphones.

Very weak on your part.

 

[jkeny]

what evidence do you have of this?
To be clear, I mean evidence of the same standard that you require of blind testing

Well, do you think audio hasn't improved continually over many years?

I guess you don't want to actually investigate internal controls & their usage in blind tests, then?

 

[Robh3606]

I also would direct you to the fact that audio product development do not use blind tests to continually check their progress & yet they progress to better sounding products.

Hello Jkeny

Companies under the Harman umbrella such as JBL, Revel and Infinity certainly do. Others may not but that's besides the point. Takes quite commitment and financial investment in R+D that may be beyond smaller companies. They actually have more than one facility.

Rob

 

[jkeny]

So after all this, after trying to hold blind testing's feet to fire, you have nothing. You have your experience. You harp on various deficiencies in blind testing that may be important as if it invalidates much of such testing done. Offering long term listening as the superior alternative, and all you have in support of it is your contention this is so from your subjective unmonitored unconfirmed experience. I expected more.

I'll repeat again - typical blind tests we see on forum are unreliable. The use of internal controls would aid greatly in qualifying those of some value.
Whether long term sighted listening is more or not reliable than the above, matters not to the above. Your expectations change nothing in what I said

Harman is using it for loudspeakers. There have been pro audio companies that use it for improving amplifiers. Harman is doing the tests to figure out how to use it for better headphones.

Very weak on your part.

As I said & will repeat, manufacturers (even Harmon) don't continually use bind testing for every development stage in their product evolution - it's infeasible & uneconomical to boot & yet they improve their products using sighted listening.

 

[jkeny]

Hello Jkeny

Companies under the Harman umbrella such as JBL, Revel and Infinity certainly do. Others may not but that's besides the point. Takes quite commitment and financial investment in R+D that may be beyond smaller companies. They actually have more than one facility.

Rob

Rob, as I said to esldude "manufacturers (even Harmon) don't continually use bind testing for every development stage in their product evolution - it's not feasible & uneconomical to boot & yet they improve their products using sighted listening."