Conclusive "Proof" that higher resolution audio sounds different

[Tony Lauck]

Since there continue to be some discussion regarding levels, I would like to describe the procedure that I used to verify that levels were not a problem for various sample rate conversion tests that I ran some years ago. Some tests were purely objective (examination and numerical processing of digital sample data) and some purely subjective (listening tests).

1. No sample ever hit the maximum (0 dBfs). IMO clipping a digital signal should be a capital crime. In general, I leave at least -0.5 dB if I have control over gain, much more with live recordings where the signal level may be unpredictable. If one is concentrating on the performance of the media (digital format) then one should use high level signals in this range, otherwise one is unnecessarily throwing away resolution. If one is concentrating on performance of a particular DAC then experiments should be done at a variety of levels, so as to understand the behavior of various sections of the DAC under near-overload conditions.

2. Unity gain when processing mid range test tones (e.g. 440 Hz, 1 kHz). Here I verified that the peak levels (dBfs) were the same within 0.01 dB. Since I worked with 24 bit files (or larger precision), gain changes could be used if necessary to achieve an accurate gain at one frequency. However, sample rate conversions inevitably involve some use of filters; there may be gain issues in the middle of the audio band due to effects characterized as "pass-band" ripple. I generally use test tones that peak at -12 dBfs to minimize the risk of "ear bleed". Looking at peak levels rather than RMS levels on these test tones is more reliable, because if one samples more than a single complete cycle of a periodic waveform the positive and negative peaks will be precise, whereas time averages (e.g. RMS) will depend on the phase at the sample end points.

Before conducting any time consuming listening tests, it is also desirable to vet an SRC to verify that it behaves in a linear fashion, i.e. any errors are determined by the noise floor of PCM word length and not numerical errors due to accumulation of round-off error or grosser errors. If the SRC isn't linear then it probably makes no sense to talk about "gain". See the Infinite Wave SRC web site for examples of non-linear SRC software. One example is the SRC that originally came with Sony SoundForge. Some years ago there was a lot of garbage DSP software on the market and one should not assume any software is acceptable unless it has been vetted.

If one is interested in the performance of various digital formats one must accept that these can never be completely determined by testing since one is comparing complete record-playback chains. If one wants to compare formats one needs to carefully vet the entire record playback chains to ensure that they do not contribute effects that may result in false negatives or false positives. This requires technical skill, including understanding how the equipment being used in the tests actually works, and assumes a reasonably high level of understanding of mathematics, physics, and electrical engineering, something beyond what one would learn in college level courses. All the necessary material is available for free and/or purchase on the Internet, with the real cost being in patience and time.

 

[arnyk]

Very impressive sir. I especially like the fact that you are "understating your formal credentials" by quite a bit.

Can you, with all due respect, tell me how that qualifies you to be a more informed, astute listener within the context of judging the quality of reproduced music?

PS: Science experiments don't count.

As far as being a more informed, astute listener goes, all of the alphabet soup has no direct bearing.

What my formal education did for me was teach me that the most important thing is knowing how to pursue and learn relevant new knowledge aggressively. Virtually nothing that is relevant to the current discussion was even a twinkle in all but very few people's eyes when I left the university for financial reasons in 1973. ABX was the primary tool for me learning how to be an effective listener.

 

[arnyk]

Totally agree! And one of the points of this thread is to drive some rigour into such testing.

The last 40 or so years of the development of blind listening tests for audio has had exactly that as its primary goal.

This rigour is being resisted at every step of the way,

Of course the first form of resistance has been 30+ years of refusal by advocates of sighted evaluations who have come up with all sorts of fantastic reasons why blind tests are inherently faulty. Virtually every high end publication and high end audiophile has come up with their own stories, one more fantastic and imaginative then the next.

I would presume that it would not be necessary to provide a bibliography of these bogus articles, since there have been so many of them. Any serious audiophile has no doubt seen dozens of them.

One of the means of resistance has been demands that blind tests be absolutely blameless. Any fault whether trivial or significant has been blown up and seized as justification to ignore the results of test results that don't make people feel good.

Here is an example:

"I see no attempt at improving the test with pos & neg controls."

Ignores the fact that years of effort have already been put into improving the tests with positive and negative controls. When given links to tests where this has been carried to a high level, there is no response.


Another form of resistance has been numerous erroneous technical analysis claiming the existence of non-existent faults in the preparation of the test files.

Ironically, some of this this was in the context of an attempt to add better positive and negative controls.

Another form of resistance has been personal attacks such as this one:

"This rigour is being resisted at every step of the way, with the excuse that such tests are good enough because they expose the sighted Vs blind debate (which is really what all these amateur tests are really about - to shame/goad/"prove" the sighted listener as wrong in their perception. "

and this:


"It may be that they are wrong but these amateur tests (& this is one of the better ones) show nothing other than the test designers bias.

and this one:

"I have had a lifetime of experience dealing with people who are first rate. I have enjoyed working with these people despite many of them having a "difficult" personality. On the other hand, I never was able to reach a level of personal maturity where I could tolerate second raters who were difficult to get along with, particularly those who were more interested in winning an argument than uncovering knowledge."
Exactly where I stand too - sometimes I wish I could overlook hypocrisy but I have a built in reflex to it - probably comes from my upbringing in catholic Ireland? [/QUOTE]

There is no evidence that the person raising this complaint is even 454th rate in this context. But they feel that they are judges of people who have achieved some very worthwhile things in this area of endeavor.

I also have a lifetime of being personal friends with people who are highly respected in the field of audio, three of whom are AES Fellows. We sometimes chat about these things and have a good laugh at all of the people with an audio interest who are obviously caught up in a high level of denial and resort to the strategies listed above.

 

[arnyk]

Since there continue to be some discussion regarding levels, I would like to describe the procedure that I used to verify that levels were not a problem for various sample rate conversion tests that I ran some years ago. Some tests were purely objective (examination and numerical processing of digital sample data) and some purely subjective (listening tests).

1. No sample ever hit the maximum (0 dBfs). IMO clipping a digital signal should be a capital crime. In general, I leave at least -0.5 dB if I have control over gain, much more with live recordings where the signal level may be unpredictable. If one is concentrating on the performance of the media (digital format) then one should use high level signals in this range, otherwise one is unnecessarily throwing away resolution. If one is concentrating on performance of a particular DAC then experiments should be done at a variety of levels, so as to understand the behavior of various sections of the DAC under near-overload conditions.

2. Unity gain when processing mid range test tones (e.g. 440 Hz, 1 kHz). Here I verified that the peak levels (dBfs) were the same within 0.01 dB. Since I worked with 24 bit files (or larger precision), gain changes could be used if necessary to achieve an accurate gain at one frequency. However, sample rate conversions inevitably involve some use of filters; there may be gain issues in the middle of the audio band due to effects characterized as "pass-band" ripple. I generally use test tones that peak at -12 dBfs to minimize the risk of "ear bleed". Looking at peak levels rather than RMS levels on these test tones is more reliable, because if one samples more than a single complete cycle of a periodic waveform the positive and negative peaks will be precise, whereas time averages (e.g. RMS) will depend on the phase at the sample end points.

Before conducting any time consuming listening tests, it is also desirable to vet an SRC to verify that it behaves in a linear fashion, i.e. any errors are determined by the noise floor of PCM word length and not numerical errors due to accumulation of round-off error or grosser errors. If the SRC isn't linear then it probably makes no sense to talk about "gain". See the Infinite Wave SRC web site for examples of non-linear SRC software. One example is the SRC that originally came with Sony SoundForge. Some years ago there was a lot of garbage DSP software on the market and one should not assume any software is acceptable unless it has been vetted.

If one is interested in the performance of various digital formats one must accept that these can never be completely determined by testing since one is comparing complete record-playback chains. If one wants to compare formats one needs to carefully vet the entire record playback chains to ensure that they do not contribute effects that may result in false negatives or false positives. This requires technical skill, including understanding how the equipment being used in the tests actually works, and assumes a reasonably high level of understanding of mathematics, physics, and electrical engineering, something beyond what one would learn in college level courses. All the necessary material is available for free and/or purchase on the Internet, with the real cost being in patience and time.

Rest assured Tony that the samples I contributed to these tests were qualified by the means described above and then some.

By the way it is easy to compare the performance of various digital formats by testing by simply performing and controlling the entire process, from acoustic source to distributed samples, as I have.

 

[Phelonious Ponk]

Hey when the smart kids figure out what it is that folks are hearing here, would you send word to the alley, behind the school, where we're busy teaching your sisters how to make out? Thanks.

Tim

 

[jkeny]

The last 40 or so years of the development of blind listening tests for audio has had exactly that as its primary goal.

Nobody denying that - just a pity that half-arsed tests are the norm on audio forums.

.....
Here is an example:

"I see no attempt at improving the test with pos & neg controls."

Ignores the fact that years of effort have already been put into improving the tests with positive and negative controls. When given links to tests where this has been carried to a high level, there is no response.

Pity positive & negative controls were NOT used in your test. Pity positive & negative controls were NOT used in Meyer & Moran's test. Both of these tests you have cited continuously in the past while ignoring these basic flaws. Are you trying to defend them now? I've no problem with any tests that include such controls as it's a basic requirement & first step to provide some semblance of reliability. I know these controls are used in professionally designed tests.

Another form of resistance has been numerous erroneous technical analysis claiming the existence of non-existent faults in the preparation of the test files.

You mean the equivalent of your unproven claims of IMD or gaming or cheating with your test when positive results were returned? Never saw you raise any suggestion of these issues when negative results were being reported?

...Ironically, some of this this was in the context of an attempt to add better positive and negative controls.

Ah, you're referring to your skewed testing for IMD? Another example of test designer's bias.

 

[arnyk]

Nobody denying that - just a pity that half-arsed tests are the norm on audio forums.

Yes, all those casual sighted evaluations.

And look what happens if someone has the temerity to try to reverse that by adding a little much needed rigor.

Pity positive & negative controls were NOT used in your test.

Actually, there were both positive and negative controls in my tests. I've explained this here already, but denial and defensive nit-picking appear to such strong temptations for some.

If I show just one negative or positive control I falsify the claim above, because it is stated in such a dogmatic way. But I will do more than that. I will show examples of both positive and negative controls.

A positive control was added in the form of the series of IM tests where there were elements that had to be heard (the -30 dBFS 4 KHz tone and the clicks) to obtain a positive verification of the test setup.

A negative control was added to the IM tests in the form of the segments where the ultrasonic probes were required to sound the same as the same segments when downsampled to obtain a positive verification of the test setup.

Other positive and negative controls include the basic nature of the ABX test that accepts both positive results (example A sounds like X) and negative results (example A does not sound like X) as representing accurate perception.

Ah, you're referring to your skewed testing for IMD?

There was no skewing which I proved by correcting errors in certain false interpretations of FFT test results. One false interpretation underestimated the actual ultrasonic content of the keys jangling sample by inappropriate choice of the time scale, and another false interpretation was due to erroneous assessments of the relative levels of sine waves and music.

There is a big problem around here of people who use common tools like FFT analysis but they are apparently unaware of certain common pitfalls.

 

[Tony Lauck]

Rest assured Tony that the samples I contributed to these tests were qualified by the means described above and then some.

By the way it is easy to compare the performance of various digital formats by testing by simply performing and controlling the entire process, from acoustic source to distributed samples, as I have.

Rest assured that I have done this. I have spend many hours trying various sample rate converters and various parameter settings to these sample rate converters, dither algorithms, etc., and listening to the results of these transformations and their musical significance. I have also examined the waveforms output from these conversions, done difference tests and done various spectrum plots. I have done enough blind testing to know that these tests are difficult and that it is often quite hard to notice differences that later prove to be obvious. I know how to make sample rate conversions that will obscure the differences between hi-res and CD format along any particular sonic dimension, e.g. tonality, imaging and sound stage. I have also discovered how conversion parameters sonically interact and that while it is often easy to optimize one aspect of a sonic conversion, it is hard, if not impossible, to optimize all three. I have also played with dither algorithms and reached similar conclusions. Here things were often more insidious, because some noise shaping processes that sounded better ended up inducing more listening fatigue on a long term basis. (This was sensitive to the particular original source material, with results being particularly bad when recordings that had already been noise shaped had to be processed and were noise shaped a second time.)

One other point I would like to make. ABX double blind tests are not "objective". They are subjective tests that produce "objective" data. They do not get to the heart of the problem. By their very nature these tests are quite effective at eliminating false positives (when done correctly, e.g. not with the PC ABX gear). However, by their very nature they are unable to effectively deal with the question of false negatives, both for statistical reasons and for reasons of psychology. There are parts of human perception that impact more than the conscious brain state and these parts are not accessible by multiple guess testing, except possibly as a slight change in probability of a correct guess. Obtaining statistical significance from slight biases in guess probabilities is very costly if not impractical because of the huge sample sizes required and considerations of listener fatigue.

Your key samples were perfectly reasonable for test purposes, by the way. It's the suggested test tools that I had a problem with, namely the impact of click switching by the PC ABX software, which modulates the switching waveform with the signal and produces aliases of the ultrasonic frequencies in the audible range.

In the case of the corrected music files that are the subject of this thread, I haven't formed an opinion as to their suitability. I thought I could hear differences on initial listen all the way through for the first two files, characterizing one as more "forward" than the other, but not forming an opinion as to which was the original. My initial impression was that I preferred what turned out to be the 44/16 file, but this impression was coupled with an uneasy feeling that I could well be wrong.

Note that it is possible to prefer reduced resolution files over full resolution files. This sometimes even happens when comparing 44/16 to 320 kbps MP3. I don't claim to understand what is going on, but I have encountered this a number of times as have others. I have also observed that using perceptual coding based noise reduction, such as with iZotope RX, can make resulting audio sound cleaner, not just reduce obvious noise. Unfortunately, I have also seen the down side of over processing with this tool, wherein one discovers in short term listening that the sound has gotten obviously better, only to discover after a few minutes of continued listening that a great sense of unnaturality and artificiality pervades the recorded sound. These experiences has done a lot to convince me personally that "hearing" sonic differences is not something that can be reduced to answers in a multiple guess test.

As to anyone planning additional SRC comparison tests. Simply avoid using Sonic Solutions SRCs or any others that do not provide correct gain. There is no point in a 0.1 dB reduction. This reduction is unnecessary and serves no useful purpose, but it does complicate any SRC experiments. Recording engineers have no basis in shaving headroom so close as 0.1 dB unless they have already sold out to the loudness wars. While I'm knocking Sonic Solutions, let me put in a comment about iZotope's otherwise excellent "64 bit" SRC. It has subsample delays as a result of the filter lengths used. This prevents a good null even when linear phase filter parameters have been selected. I agree with the designer's comment that this has no "sonic significance" (private email) but it is a pain in the ass when it comes to using this converter as a test/measurement tool in experiments such as these.

 

[jkeny]

Yes, all those casual sighted evaluations.

Right & half-arsed audio forum blind tests are the equivalent anecdotal evidence - no better - glad we agree on that.

And look what happens if someone has the temerity to try to reverse that by adding a little much needed rigor.

Actually, there were both positive and negative controls in my tests. I've explained this here already, but denial and defensive nit-picking appear to such strong temptations for some.

If I show just one negative or positive control I falsify the claim above, because it is stated in such a dogmatic way. But I will do more than that. I will show examples of both positive and negative controls.

A positive control was added in the form of the series of IM tests where there were elements that had to be heard (the -30 dBFS 4 KHz tone and the clicks) to obtain a positive verification of the test setup.

A negative control was added to the IM tests in the form of the segments where the ultrasonic probes were required to sound the same as the same segments when downsampled to obtain a positive verification of the test setup.

Other positive and negative controls include the basic nature of the ABX test that accepts both positive results (example A sounds like X) and negative results (example A does not sound like X) as representing accurate perception.

Yes, I applaud you for trying to introduce some controls into the test after 14(?) years. Exactly at the time when positive results began being reported. So, again we are in agreement - the test was half-baked to begin with. It's very typical of audio forum tests (but slightly better as I already said)

There was no skewing which I proved by correcting errors in certain false interpretations of FFT test results. One false interpretation underestimated the actual ultrasonic content of the keys jangling sample by inappropriate choice of the time scale, and another false interpretation was due to erroneous assessments of the relative levels of sine waves and music.

There is a big problem around here of people who use common tools like FFT analysis but they are apparently unaware of certain common pitfalls.

There is a big problem with someone including test tones in the same audio file at such a difference in level to the jangling keys signal that when playing back at the normal listening level used in the test will then cause the test tones to clip most playback devices.

So, not only was your test skewed but your controls also.

It seems Amir didn't report any IMD when he ran the test tones & hence it appears this was not the reason for his positive results. But I guess you still have cheating & proctoring as two other fall-back reasons to deny the results?

 

[arnyk]

Rest assured that I have done this. I have spend many hours trying various sample rate converters and various parameter settings to these sample rate converters, dither algorithms, etc., and listening to the results of these transformations and their musical significance. I have also examined the waveforms output from these conversions, done difference tests and done various spectrum plots. I have done enough blind testing to know that these tests are difficult and that it is often quite hard to notice differences that later prove to be obvious. I know how to make sample rate conversions that will obscure the differences between hi-res and CD format along any particular sonic dimension, e.g. tonality, imaging and sound stage. I have also discovered how conversion parameters sonically interact and that while it is often easy to optimize one aspect of a sonic conversion, it is hard, if not impossible, to optimize all three. I have also played with dither algorithms and reached similar conclusions. Here things were often more insidious, because some noise shaping processes that sounded better ended up inducing more listening fatigue on a low term basis. (This was sensitive to the particular original source material, with results being particularly bad when recordings that had already been noise shaped had to be processed and were noise shaped a second time.)

Got DBT results supporting the things claimed above?

One other point I would like to make. ABX double blind tests are not "objective". They are subjective tests that produce "objective" data.

You have my permission to argue with yourself. I've never said that DBTs are objective.

They do not get to the heart of the problem.

I didn't know that there was just one problem! ;-)

By their very nature these tests are quite effective at eliminating false positives (when done correctly, e.g. not with the PC ABX gear).

PCABX software is of course not gear so you've kinda lost me already. However PCABX and other software DBT listening test software is generally recognized and widely used in industry and academic environments.

However, by their very nature they are unable to effectively deal with the question of false negatives, both for statistical reasons and for reasons of psychology. There are parts of human perception that impact more than the conscious brain state and these parts are not accessible by multiple guess testing, except possibly as a slight change in probability of a correct guess.

Can you cite authoritative support for that rather exceptional claim?

Obtaining statistical significance from slight biases in guess probabilities is very costly if not impractical because of the huge sample sizes required and considerations of listener fatigue.

Again you have my permission to argue with yourself.

Your key samples were perfectly reasonable for test purposes, by the way. It's the suggested test tools that I had a problem with, namely the impact of click switching by the PC ABX software, which modulates the switching waveform with the signal and produces aliases of the ultrasonic frequencies in the audible range.

Please clarify. "Click switching" is not a term that I've found used by anybody else. Producing aliases by means of switching between A, B, and X is one of those things I'd have to see proven with actual tests. Also the tool that has been used for most of these tests is FOOBAR2000, not PCABX.

In the case of the corrected music files that are the subject of this thread, I haven't formed an opinion as to their suitability. I thought I could hear differences on initial listen all the way through for the first two files, characterizing one as more "forward" than the other, but not forming an opinion as to which was the original. My initial impression was that I preferred what turned out to be the 44/16 file, but this impression was coupled with an uneasy feeling that I could well be wrong.

Sounds like the results of sighted evaluations.

Note that it is possible to prefer reduced resolution files over full resolution files.

I thought we were talking about ABX tests which are most definitely well known to not be preference tests.

 

[arnyk]

There is a big problem with someone including test tones in the same audio file at such a difference in level to the jangling keys signal that when playing back at the normal listening level used in the test will then cause the test tones to clip most playback devices.

I guess you didn't get the memo that has been sent out several times by now - those claims of big level differences were falsified as being artifacts of unskilled use of FFT analysis tools. The peak levels of the jangling keys signal was within a dB or 2 of the peak levels of the test tones.

 

[arnyk]

Yes, I applaud you for trying to introduce some controls into the test after 14(?) years

Many of the controls I mentioned, both positive and negative have been standard features of ABX tests since their inception back in the middle 1970s. That is more like 30 or 40 years ago.

 

[Tony Lauck]

I have no interest in double blind confirmation of the work I did some years back. I did this for my own benefit to better understand how to use the tools at hand to make recordings and did not have time to waste on protocols whose only value would be to convince skeptics. I do not have any interest in convincing skeptics, since my experience is that the vast majority of skeptics have closed their minds and will simply discard any results that don't fit their pre-existing mind set. (These recent threads confirm this viewpoint amply.) My peers in this work (actually my superiors) are experienced mastering engineers who have vastly more experience doing actual conversions and other post-processing of recorded audio. These people make their living making correct and rapid judgements of sonic quality and their work serves as the best proof of their competence. It is not possible to produce consistent recordings of high quality without the ability to hear well and understand the significance of what one hears.

As to subconscious perception, do your own research. There are FMRI tests showing brain activity and there are other examples of sub threshold perception affecting probabilities of guesses. This is rather obvious, and totally consistent with signal detection theory. You might also note that there are various schools of psychology beyond the rat school pioneered by B. F. Skinner and his ilk. I suggest you do some reading about Gestalt psychology. If you understand the applicability of the foreground / background concept to audio then perhaps you will get a glimmer of the complexity of human auditory perception, or more correctly, the complexity of the human mind.

PCABX software is definitely gear as much as any other "in the box" digital signal processing, such as VST plugins, audio editing suites such as iZotope RX, etc.. It happens to be software and it happens to be free, but neither of these properties prevents it from being "gear".

I suggest you stop arguing and start reading and thinking about what I have already posted about audible clicks in rapid switching. I assume you are sufficiently knowledgeable to understand the mathematics and the practical concepts involved here. If you are not then your inability to understand these concepts and their practical implications disqualifies you as a potential "authority". For the non-technical reader no mathematics is needed, just a little playing around with the ABX test tools and paying attention to what one hears, particularly the sound of the start up transients. Since these recent threads began and I began using PC ABX I have yet to find a pair of files where I could not find click artifacts that completely unblinded the ABX test, rendering the test software useless. I am using a Mytek Stereo 192-DSD DAC that directly drives a pair of Focal twin - 6 BE near field monitors and a Sub 6 subwoofer. This system has been measured to be reasonably flat from below 30 Hz to above 20 kHz and is spec'd to be flat to 40 kHz and deliver peak sound levels at my listening position of 118 dB (both channels together). I have at least 6 dB undistorted head room from the listening settings that I used for all of these listening tests. It could be that other people's systems obscure what I can obviously hear. There is no point publishing any of my ABX tests, because every single time I ran them I quickly found ways to get perfect accuracy. However, I make no claims that I heard differences between the files this way. Instead I make a simple guess based on experience that what I heard was the result of the start-up transients. I await the arrival of decent test software, at which point I am open to forming a different potential explanation for what I heard. In the meantime, I have disqualified the test "gear" as unsuited for purpose. I am unable to perform any ABX tests that might be requested.

As I understand this thread, the original test was not an ABX test, but rather a blind system identification test. In that regard, my initial listening to the files was in the spirit of the thread's creator. It was blind, as I did not know when I listened which file was which. It was only after initially forming my opinion (which inescapably included characterizing what I heard and forming a preference and preliminary prediction) that I unblinded the two files by using my audio editor. As I have explained before, blinding is a necessary tool to convince other people that a person has "heard" something. It can be a useful ego-deflating tool to convince an inexperienced listener that one does not always "hear" what one thinks one has, but it is not a particularly useful tool for a recording engineer working with sound files for a purpose, namely producing good recordings. The reliance on randomization and large number of trials to gain statistical power are appropriate in research activities that are well funded, but simply impractical when one is interested in effecient production of high quality recordings. A recording engineer must have developed the skills to quickly and reliably hear and decide what action to take to produce a preferred result. In a broader context blind testing is just one tool of many in a large tool bag. For those not brainwashed by dogma, blind testing is not a religious icon or sacred cow.

 

[amirm]

And look what happens if someone has the temerity to try to reverse that by adding a little much needed rigor.

Appearance of rigor is far worse than lack of rigor. The later can be recognized. The former will fool all but the people who do that for a living.

Countless forum readers have been fooled by tests such as Meyer and Moran because a) they were ABX double blind tests and b) "peer reviewed" by JAES. Both gave strong appearance of rigor but unfortunately they did not remotely practice such.

Thanks to positive outcome in these tests, for the first time, real rigor is being talked about in the form of controls, employment of trained listeners, not having listeners who have obviously damaged hearing (e.g. like yourself), on and on. This is what I think is the biggest benefit of these results.

I wonder if you feel any responsibility for the situation prior to these tests Arny.

 

[jkeny]

Many of the controls I mentioned, both positive and negative have been standard features of ABX tests since their inception back in the middle 1970s. That is more like 30 or 40 years ago.

Indeed but you didn't bother with any controls until 14 years after you created the test. And then only because positive results were reported. The motivation for now introducing your new controls is very obvious. If you were interested in some semblance of a true, competent test, controls would have been included from the start.

 

[Orb]

I guess you didn't get the memo that has been sent out several times by now - those claims of big level differences were falsified as being artifacts of unskilled use of FFT analysis tools. The peak levels of the jangling keys signal was within a dB or 2 of the peak levels of the test tones.

Although as pointed out the keys is not entirely comparable to those constant 0dbfs tones.
But then also Tony Lauck mentions the level of those keys would not cause any issues unless something is not right with the hardware (or unless played excessively loud to cause clipping/near to clipping related distortion, which tbh would break the abx framework anyway).
Are you still contending those that passed the test were hearing IMD or accept that it is a non-issue unless hardware pushed to/beyond limits?
Thanks
Orb

 

[Orb]

....
Your key samples were perfectly reasonable for test purposes, by the way. It's the suggested test tools that I had a problem with, namely the impact of click switching by the PC ABX software, which modulates the switching waveform with the signal and produces aliases of the ultrasonic frequencies in the audible range.

....

Tony,
just to add this does depend upon the listeners methodology-approach to listening correct?
For this type of test (due to nature of what listening for) one does not need to sequentially switch between ABX with real-time stream; several of us use the approach of identifying-isolate an event (anchor) where the anomaly is perceived and then narrow the critical listening window-segment down to a very short span (2 seconds at most usually less with the anomaly within it) and play the short segment that then stopped before switching (listening methodology for that as well), so there is no simultaneous streaming/switching between sound sources.
Seems that is part (not all of it) of how a few of us listen who had some background training (mine very early 90s relating to transmission-broadcast), including Amir.
Oversimplifying as a fair bit of this has already been discussed much earlier, including those who did not hear differences and then did using this technique.
I appreciate there are other times-specific ABX tests where it is desirable to sequentially switch real-time.

Thanks
Orb

 

[arnyk]

Appearance of rigor is far worse than lack of rigor. The later can be recognized. The former will fool all but the people who do that for a living.

Countless forum readers have been fooled by tests such as Meyer and Moran because a) they were ABX double blind tests and b) "peer reviewed" by JAES. Both gave strong appearance of rigor but unfortunately they did not remotely practice such.

IMO the worst thing that Meyer and Moran did is get blinded by hype of the segment of the recording industry that was attempting to profiteer by retreading existing recordings. They were not alone as the whole high end audio industry was just as blind.

I wasn't consulted, either formally or informally on any phase of the Meyer Moran tests or the article so I absolutely refuse to take any responsibility for it. I also have been very clear when asked, as I am now that the current value of that work related to the ongoing controversy about high resolution, high sample rate audio is pretty limited.

Thanks to positive outcome in these tests, for the first time, real rigor is being talked about in the form of controls, employment of trained listeners, not having listeners who have obviously damaged hearing (e.g. like yourself), on and on. This is what I think is the biggest benefit of these results.

The global statistical significance of the current tests appears to be very highly limited. Just not enough people involved, not enough people reporting postive results from DBTs, and of course, no external controls on the self-administered ABX tests.

I wonder if you feel any responsibility for the situation prior to these tests Arny.

I don't take any responsibility for things I had nothing to do with (e.g. Meyer and Moran), and I won't take any responsibility for or support any attempts to inflate the global importance of minuscule, questionable, and fragmentary results.

 

[arnyk]

Although as pointed out the keys is not entirely comparable to those constant 0dbfs tones.

There are and were no constant 0 dB FS tones in my test files. They were -1 dB FS which is consistent with the stated practice of many others, including some WBF posters such as Mr. Lauck.

I have long been an advocate of limiting peaks to -1 dB FS in digital recordings because too many converters, both ADC and DACs that are otherwise OK get flaky above that. In fact most of the flakiness comes above -0.5 dB FS and more often above -0.1 dB FS at which point this is just all a tiny misunderstanding.

 

[Orb]

There are and were no constant 0 dB FS tones in my test files. They were -1 dB FS which is consistent with the stated practice of many others, including some WBF posters such as Mr. Lauck.

I have long been an advocate of limiting peaks to -1 dB FS in digital recordings because too many converters, both ADC and DACs that are otherwise OK get flaky above that. In fact most of the flakiness comes above -0.5 dB FS and more often above -0.1 dB FS at which point this is just all a tiny misunderstanding.

So why do you insist the IM ultrasonic tone measurements JA did reflect the keys proving IM is a possibility
In reality it was taking the existing 19+20khz 0dbfs IM stress-duress test and pushing it to ultrasonic FR range.
As I mentioned and so has JA, the really good hardware still handles even this severe ultrasonic test, the two products that had difficulty with 0dbfs (causing -50db IMD from signal in audioband) had no problems when it was backed off from 0dbfs and closer to what the jangling keys were.

So we can agree then IMD is not an issue and not heard by anyone doing this test unless their hardware was faulty (although this has been proved not to be the case with them doing further tests).

BTW several times you have insisted JAs measurements shows IMD being a possibility, which is opposite JA's conclusion,mine, and several others who also used other equipment/did further testing.
Edit:
Just to clarify I do appreciate such test signal is not just 0dbfs but also includes using ones further away from full scale reference, point being here though to stress test and further test if something was found (which was done with the two products).
That said much equipment reference their spec to 0dbfs for IM.
Thanks
Orb