Null testing why isn’t it done more in audio reviews or would the results be too unpalatable ?2

[awsmone]

Ever since Winer did his null test on a couple of audio cables I have wondered about this in audio reviewing
Why isn’t it done more often ?

It seems such a conclusive way of telling if there are differences eg for PC interconnects etc

Double blind trials are so difficult to do, this seems or more practical way of testing equipment

I suppose though there could be some unhappy companies if their results are no better than lamp cord from home depot

As this is used a lot in the recording industry why not in the user end of the market?

Please I don’t want a flame war, just genuinely curious?

I found an interesting article that compared MQA of Emmylou Harris with the 44.1 version and found no difference to -70db which begs the question what you are actually getting ?

Of course complete null such as with DAW is more conclusive

s

 

[Vienna]

 

[Folsom]

Ethan didn't do frequency sweeps. I can't take it very seriously. Also, a singular load that isn't like equipment isn't useful either.

Also if you can't completely do away with every difference, then how do you know that low level noise isn't changing things?

You can ask the question you're asking, but I'll ask how would a null test be useful at all? As far as I can see it's just a big thorn that objectionists like to apply. Where I agree with Ethan is that it's all measurable, but whether we know how to or not is a very different thing.

 

[awsmone]

Ethan didn't do frequency sweeps. I can't take it very seriously. Also, a singular load that isn't like equipment isn't useful either.

Agreed but I am not specifically talking about Ethan’s experiment null test are used in pro audio as my video shows

Also if you can't completely do away with every difference, then how do you know that low level noise isn't changing things?

Agreed but what level of difference is significant, and complete null is conclusive

You can ask the question you're asking, but I'll ask how would a null test be useful at all? As far as I can see it's just a big thorn that objectionists like to apply. Where I agree with Ethan is that it's all measurable, but whether we know how to or not is a very different thing.

That’s an important question

Let me give you a practical example

We have a digital Converter

It’s claimed a certain PC improves the sound quality

We record the output with cord x
We record the output with cord y

We invert phase of one output

And null them

If the null is zero there is no difference between the two pc

If not I agree we have to discuss how low we consider is relevant

 

[ddk]

Ever since Winer did his null test on a couple of audio cables I have wondered about this in audio reviewing
Why isn’t it done more often ?

It seems such a conclusive way of telling if there are differences eg for PC interconnects etc

Double blind trials are so difficult to do, this seems or more practical way of testing equipment

I suppose though there could be some unhappy companies if their results are no better than lamp cord from home depot

As this is used a lot in the recording industry why not in the user end of the market?

Please I don’t want a flame war, just genuinely curious?

I found an interesting article that compared MQA of Emmylou Harris with the 44.1 version and found no difference to -70db which begs the question what you are actually getting ?

Of course complete null such as with DAW is more conclusive

s

He should have null tested that shirt before getting in front of a camera !

We use the same concept when we have asymmetrical mirrored designs but I don't see the application in context of power cords Andrew. At best you might see a difference in the sine way but what does it mean in terms of sound quality in case of power cords where there's no real explanation for the differences in sound quality because no one knows how or what to measure, so what's a null test going to tell anyone? Even for equipment you measure two poor sounding components and there's a difference in the sine waves at certain frequencies, what does that prove? Do you trust a digitized signal and/or a digital oscilloscope like the one in DAW, I know several high end manufacturers who do to use digital test equipment only analog ones. I don't see the point or value in this context of reviewing but maybe measuring differences of channels.

david

 

[microstrip]

The null test in an extremely useful test for specific purposes, the main question in audio is how do we correlate the residuals with subjective sound quality.

Modern audio analyzers, using ADCs with high resolution and low noise associated with high computational power have displaced the need for null tests most of the time - their dynamic range is outstanding.

BTW, IMHO the different sound of cables is a system property - carrying null tests in the audio band in an isolated cable is of little interest.

 

[Folsom]

BTW, IMHO the different sound of cables is a system property - carrying null tests in the audio band in an isolated cable is of little interest.

Yes, you basically would have to measure at the speaker for every test. But for a few reasons that'll probably be inconclusive.

 

[DaveC]

If the null is zero there is no difference between the two pc

That's starting with a flawed assumption and the entire premise is unreasonable... unless you are coming from a place where you think cables can't make a difference, and think measurements totally define what we hear, but IMO that's ridiculous and so is Ethan Winer.

 

[ddk]

BTW, IMHO the different sound of cables is a system property - carrying null tests in the audio band in an isolated cable is of little interest.

I disagree! IME you can place some of the better known cables with strong qualities like Transparent, Nordost, Shunyata, Purist, etc. in any system and their signature sound will be apparent in any system vs a more do nothing cable which will remain neutral in any system. This isn't about system synergy nor am I arguing personal preference just that signature qualities remain the same and audible in any system with decent resolution.

david

 

[awsmone]

Yes, you basically would have to measure at the speaker for every test. But for a few reasons that'll probably be inconclusive.

Yes I saw a measurement of the transfer function of speaker cables attached to real speakers which clearly showed differences in frequency response

 

[awsmone]

I ca

That's starting with a flawed assumption and the entire premise is unreasonable... unless you are coming from a place where you think cables can't make a difference, and think measurements totally define what we hear, but IMO that's ridiculous and so is Ethan Winer.

i cannot see your reasoning ?
I am not siding with Ethan Winer, but few people have actually refuted what he did, though I agree you need to do it on real loads

I believe interconnects and pc do make a difference but am trying to understand null test results

If I perform a null test using two PCs and they actually null, there is no difference in principle

Whether there are limitations of the way the test is done is obvious in the testing methodology

But if your saying in principle that null tests are invalid when they do actually null

You would need to explain it

As Mark Twain said it’s easy to fool people , but it’s harder to convince them they have been fooled

 

[awsmone]

He should have null tested that shirt before getting in front of a camera !

We use the same concept when we have asymmetrical mirrored designs but I don't see the application in context of power cords Andrew. At best you might see a difference in the sine way but what does it mean in terms of sound quality in case of power cords where there's no real explanation for the differences in sound quality because no one knows how or what to measure, so what's a null test going to tell anyone? Even for equipment you measure two poor sounding components and there's a difference in the sine waves at certain frequencies, what does that prove? Do you trust a digitized signal and/or a digital oscilloscope like the one in DAW, I know several high end manufacturers who do to use digital test equipment only analog ones. I don't see the point or value in this context of reviewing but maybe measuring differences of channels.

david

Thanks David

I agree that where there is a residual you cannot interpret this in some quality way

I do believe you can explain effects of PCs on power supplies and this can be seen as distortions in the sine waves and current delivery of the power supply

These changes are real and measurable

In science we assume when we compare things a null result not the other way around so am surprised at the resistance to there may be no difference in switching things around

I heard an interview with Richard Fryer of Spectral who is a psychologist by training and very interested in how we perceive audio sound

He made a telling point when he said humans have the ability to hear audio anomalies in a haystack

He gave the example that loudspeakers are the most high distortion part of the reproduction chain , yet if we change the preamplifier which are low distortion devices we immediately here the difference , which demonstrates that low level differences are easily perceived

Conductors can easily hear a flat or false note/late note in a symphony orchestra playing together

This probably is a survival advantage if someone is talking to you and a twig snaps behind you a way off you immediately spin around

But it probably a “burden” to the audiophile

 

[ddk]

Thanks David

I agree that where there is a residual you cannot interpret this in some quality way

I do believe you can explain effects of PCs on power supplies and this can be seen as distortions in the sine waves and current delivery of the power supply

These changes are real and measurable

In science we assume when we compare things a null result not the other way around so am surprised at the resistance to there may be no difference in switching things around

I heard an interview with Richard Fryer of Spectral who is a psychologist by training and very interested in how we perceive audio sound

He made a telling point when he said humans have the ability to hear audio anomalies in a haystack

He gave the example that loudspeakers are the most high distortion part of the reproduction chain , yet if we change the preamplifier which are low distortion devices we immediately here the difference , which demonstrates that low level differences are easily perceived

Conductors can easily hear a flat or false note/late note in a symphony orchestra playing together

This probably is a survival advantage if someone is talking to you and a twig snaps behind you a way off you immediately spin around

But it probably a “burden” to the audiophile

It's not a burden we become audiophiles and audio fixated by choice in pursuit of some ideal. It's burdensome when one doesn't know clearly what they're chasing. Worse still is following the words of some "reviewer" without knowing that person and following the path to a goal not set by another person's words.

I'm not against testing of any kind where applicable just don't see null testing of any value to audiophiles in this context. I happen to agree with Dave C here.

david

 

[BlueFox]

ddk. Thanks for putting sentences together into paragraphs and making a readable post versus the nonsensical multiple 1 sentence paragraph posts. Those posts hurt my eyes and are hard to read.

 

[Atmasphere]

There are two topics here: Why Ethan Should Always Be Ignored
and
Why a Null Test Won't Always Tell You What You Want.

With regards to the first- the cables need to be installed in a system and then the response of the system measured in the room. Then the equipment changed so its not a cheap Pioneer amp like Ethan likes to use and the test run again. I am sure the results are measurable but it would take some doing and Ethan (IME/IMO) isn't about doing the really hard work. He seems to like going for the low hanging fruit. I challenged him once about power cords and discovered that (oddly) he had never measured their effects, simply took the position that they can't make a difference. OTOH I had some measurements and knew that they respond to Ohm's Law like everything else (and thus can affect the power, distortion and output impedance of a power amplifier).

With regards to the second- The problem here is that if you want to compare an amplifier to the original signal or the like you do have some very real problems that have to do with what is practical and how the ear perceives sound. The practical bit is that no circuit can be built that has zero distortion- they all have some. The issue with the ear/brain system has to do with the fact that there is a thing called the 'masking principle' that makes designing an amplifier quite tricky. If you design it to have the best null characteristic, it will be very low distortion. But it will also sound bright and likely harsher than real life. The reason for this is the ear uses higher ordered harmonics to perceive sound pressure (and thus is extremely sensitive to the higher orders). Any amp that has really low distortion will have mostly very slight amounts of distortion- but almost entirely higher orders. It turns out that you need a bit of 2nd and 3rd to mask the higher orders- and that means you won't get a good null. But you'll have a more neutral sounding design.

This is why tube amps sound smoother than solid state, despite the fact that solid state amps have less higher ordered harmonic distortion content (at least in terms of amplitude). The lower orders that exist in a tube amp mask the higher orders, which otherwise are perceived as brightness and harshness (since the ear converts all forms of distortion into some sort of tonality).

The problem here is that higher ordered harmonics are quite audible in small amounts. This isn't news- we've known this since the 1930s... but our test and measurement systems ignore this fact entirely. As a result we have spec sheets that don't tell you how the circuit sounds- but of course that's how the industry wants it- spec sheets are a good real life example of the Emperor's New Clothes.

What is needed is a weighting system where lower orders are assigned a base value- '1' perhaps, while higher orders like the 7th or 13th get a '500' or '1000'. We have the ability to do this, but lack the will. So conversations like this one will be filling the internet for some time to come...

 

[DaveC]

There are two topics here: Why Ethan Should Always Be Ignored
and
Why a Null Test Won't Always Tell You What You Want.

With regards to the first- the cables need to be installed in a system and then the response of the system measured in the room. Then the equipment changed so its not a cheap Pioneer amp like Ethan likes to use and the test run again. I am sure the results are measurable but it would take some doing and Ethan (IME/IMO) isn't about doing the really hard work. He seems to like going for the low hanging fruit. I challenged him once about power cords and discovered that (oddly) he had never measured their effects, simply took the position that they can't make a difference. OTOH I had some measurements and knew that they respond to Ohm's Law like everything else (and thus can affect the power, distortion and output impedance of a power amplifier).

With regards to the second- The problem here is that if you want to compare an amplifier to the original signal or the like you do have some very real problems that have to do with what is practical and how the ear perceives sound. The practical bit is that no circuit can be built that has zero distortion- they all have some. The issue with the ear/brain system has to do with the fact that there is a thing called the 'masking principle' that makes designing an amplifier quite tricky. If you design it to have the best null characteristic, it will be very low distortion. But it will also sound bright and likely harsher than real life. The reason for this is the ear uses higher ordered harmonics to perceive sound pressure (and thus is extremely sensitive to the higher orders). Any amp that has really low distortion will have mostly very slight amounts of distortion- but almost entirely higher orders. It turns out that you need a bit of 2nd and 3rd to mask the higher orders- and that means you won't get a good null. But you'll have a more neutral sounding design.

This is why tube amps sound smoother than solid state, despite the fact that solid state amps have less higher ordered harmonic distortion content (at least in terms of amplitude). The lower orders that exist in a tube amp mask the higher orders, which otherwise are perceived as brightness and harshness (since the ear converts all forms of distortion into some sort of tonality).

The problem here is that higher ordered harmonics are quite audible in small amounts. This isn't news- we've known this since the 1930s... but our test and measurement systems ignore this fact entirely. As a result we have spec sheets that don't tell you how the circuit sounds- but of course that's how the industry wants it- spec sheets are a good real life example of the Emperor's New Clothes.

What is needed is a weighting system where lower orders are assigned a base value- '1' perhaps, while higher orders like the 7th or 13th get a '500' or '1000'. We have the ability to do this, but lack the will. So conversations like this one will be filling the internet for some time to come...

There have been some weighted harmonic distortion schemes, ex here of HI-2 weighted distortion:

https://www.klippel.de/fileadmin/kl...AN_07_Weighted_Harmonic_Distortion_(HI-2).pdf

 

[DaveC]

I ca


i cannot see your reasoning ?
I am not siding with Ethan Winer, but few people have actually refuted what he did, though I agree you need to do it on real loads

I believe interconnects and pc do make a difference but am trying to understand null test results

If I perform a null test using two PCs and they actually null, there is no difference in principle

Whether there are limitations of the way the test is done is obvious in the testing methodology

But if your saying in principle that null tests are invalid when they do actually null

You would need to explain it

As Mark Twain said it’s easy to fool people , but it’s harder to convince them they have been fooled

What are you measuring and nulling? Frequency vs time on an o-scope? I don't think that gives you a complete picture. When we can't even correlate what we hear to what we measure how can this possibly work out?

 

[Duke LeJeune]

The problem here is that higher ordered harmonics are quite audible in small amounts. This isn't news- we've known this since the 1930s... but our test and measurement systems ignore this fact entirely.

I totally agree. The audio industry is often very good at solving the wrong problems.

In 2003 Earl Geddes and Lydia Lee published a series of papers on distortion perception in which they proposed a metric specifically weighted to correlate well with subjective perception. They call it the "Gedlee Metric". Unfortunately it hasn't really caught on with the industry in general, but acceptance seems to be slowly growing. Anyway here is an article written by Keith Howard about it, which is a considerably easier read than the actual papers:

http://www.gedlee.com/Papers/THD_.pdf

 

[microstrip]

(...) In 2003 Earl Geddes and Lydia Lee published a series of papers on distortion perception in which they proposed a metric specifically weighted to correlate well with subjective perception. They call it the "Gedlee Metric". Unfortunately it hasn't really caught on with the industry in general, but acceptance seems to be slowly growing. (...)

What evidence can we have that acceptance of use of any specific metric weighted to correlate with subjective distortion seems to be slowly growing? As far as I could see around in measurements carried in a few reviews of what we consider top amplifiers their is no systematic at all in their distortion spectra.

 

[Duke LeJeune]

What evidence can we have that acceptance of use of any specific metric weighted to correlate with subjective distortion seems to be slowly growing?

Last year a company whose name escapes me released a measurement system that gives results in the Gedlee Metric. I interpreted that as "slowly growing acceptance".