Do measurements matter or is that we don't measure enough?

[amirm]

Read this article first and then let's discuss! http://www.benchmarkmedia.com/discuss/sites/default/files/Headphone-Amplifier-Performance-Part-1.pdf

-----

"Recording Engineers and Audiophiles often distrust audio measurements and specifications. It is not uncommon to hear claims that a product measures poorly but sounds good. Occasionally we also hear claims that a product measures well but sounds bad. This whitepaper documents significant differences between three headphone amplifiers that have nearly identical published specifications. Each is equipped with a high-quality low-jitter digital to analog converter, and each can accept 96 kHz 24-bit digital audio. All three products have a high-quality headphone output with very low output impedance, high drive capability, low distortion, and a flat frequency response. The published specifications strongly suggest that it should be impossible to hear any difference between these products. Nevertheless, in our informal listening tests, we had the distinct impression that there were significant sonic differences between these products when listening to music.

We replicated the manufacturers' tests and found that the published specifications were accurate and correct. All three manufacturers followed standard measurement procedures and provided accurate specifications....."

 

[fas42]

Exactly as you asked, we don't measure enough: it's perfectly stated in the document's summary:

We verified that the manufacturer's specifications were accurate, but we have shown that these published specifications are not sufficient to tell the whole story. In this case, the published specifications were not a good representation of typical operating conditions. Under typical operating conditions, unit 2 and unit 3 suffered from elevated noise levels. These same two units also suffered from distortion problems when driving the difficult load presented by a pair of headphones. In contrast, the Benchmark DAC1 HDR was able to maintain its performance over a wide range of operating conditions. All three products sounded very good at normal listening levels, but differences were noticeable. Our measurements, taken under typical listening conditions, confirmed that audible differences should exist. The published specifications did not lie, the manufacturers did not lie, and our ears did not lie. The truth lies here: Specifications must be measured under typical operating conditions if they are to be useful in predicting audible differences.

Frank

 

[microstrip]

Amir,

I have read this paper some days ago, as it was also referred in another forum. There is nothing new in it. Expecting that the "Headphone Output at Maximum Gain, -3 dB FS Digital Input, No Load" distortion spectrum is a significant measurement is ridiculous. Anyone having studied basic electronics and the 741 operational amplifier basics will immediately laugh at such proposition.

The paper can be summarized in a single sentence - headphones have a loading effect on amplifiers and are a dynamic load. Who would have expected it to be otherwise ?

BTW, as far as know , Benchmark units do need this type of papers to promote themselves - they are of excellent quality and do not need enhancing comparisons with poorly designed units.

 

[fas42]

The paper can be summarized in a single sentence - headphones have a loading effect on amplifiers and are a dynamic load. Who would have expected it to be otherwise ?

And when was the last time you saw spec's of a high end amplifier's distortion measured this way??

Frank

 

[Phelonious Ponk]

I'd love to know what the other two DAC/amps were . I'd also love to see these tests run against some high-end DAC/headphone separates.

Tim

 

[amirm]

I'd love to know what the other two DAC/amps were . I'd also love to see these tests run against some high-end DAC/headphone separates.

Tim

I did some googling given the MSRP. I hit on one unit that was a tube amp. I could not find any measurements on it. The specs provided were all at specific headphone type impedance so that didn't jive with their assertion of advertised THD being with no load.

BTW, the no load measurement actually has a load of 100 K. It is at the end of the report and is likely the input load of Audio Precision. So there is a load but not a realistic one.

I should also add that I did not mean this to be a headphone analysis paper. But rather, the notion that given a single measurement, THD, so many variations of the results can be extracted. I especially liked the shape of THD with the noise removed, showing far more distortion at low frequencies for two devices.

 

[JackD201]

Personally I believe either we don't measure enough or we are having problems translating what's measured to what is heard......a correlation issue. No doubt there are people walking the earth that have dedicated their lives to this and CAN correlate what's measured and what's heard. The question is, should this type of knowledge and skill be mandatory for an end user.

That's what recording engineers and audiophiles are. To demand that a recording engineer and audiophile possess this type of skill set is like demanding that a restaurant patron know how to cook as well or better as the chef and his staff or a rabbit know how to farm carrots.

To me knowing more about how my gear performs and how they are made enhance appreciation. It's a plus but to enjoy the primary activity which is to listen to the music, it surely isn't mandatory knowledge. If I were a designer/DIY builder/worker/manufacturer however, which is not the focus of this article, things would be very different.

 

[Phelonious Ponk]

Audiophiles don't need this information because their final decisions are ultimately subjective, and if they prefer the difference they hear in a system, does it matter if that difference is a distortion of the input signal? They like what they like. Recording engineers, or at least the good ones, want their signal chains to be as neutral as possible so they know exactly what's getting to the monitors. More information is better.

By the way, this just reinforces, by comparison, what has been reported on the Benchmark after pretty extensive testing by a few independent reviewers: From digital input to headphone or line level output, the thing is about as neutral as an electronic component can be, and the noise floor is way down there. Audiophiles who don't like the sound of the Benchmark don't like the sound of the recordings. It's really that simple, and that's ok. Change it. Be happy.

Good to know that kind of performance can be reached, by a relatively small company, for <$2k.

Tim

 

[microstrip]

Audiophiles don't need this information because their final decisions are ultimately subjective, and if they prefer the difference they hear in a system, does it matter if that difference is a distortion of the input signal? They like what they like. Recording engineers, or at least the good ones, want their signal chains to be as neutral as possible so they know exactly what's getting to the monitors. More information is better.

By the way, this just reinforces, by comparison, what has been reported on the Benchmark after pretty extensive testing by a few independent reviewers: From digital input to headphone or line level output, the thing is about as neutral as an electronic component can be, and the noise floor is way down there. Audiophiles who don't like the sound of the Benchmark don't like the sound of the recordings. It's really that simple, and that's ok. Change it. Be happy.

Good to know that kind of performance can be reached, by a relatively small company, for <$2k.

Tim

Tim.

And those who never listened to Benchmark? How do you class them?

But curiously you still refer "More information is better" . It is just the first thing any audiophile will tell you - there are systems that can present more information than the Benchmark!

 

[JackD201]

Recording engineers, or at least the good ones, want their signal chains to be as neutral as possible so they know exactly what's getting to the monitors. More information is better.

I would beg to differ on this one Tim. The good ones have their bag of goodies in the form of favorite mics and outboard gear. They do enjoy a wide latitude in terms of artistic input and their electronics are their instruments. Just like Audiophiles there are long running debates on the exact same themes and then some. While active vs. passive may not be debated, which active definitely is.

 

[Mark Seaton]

Read this article first and then let's discuss! http://www.benchmarkmedia.com/discuss/sites/default/files/Headphone-Amplifier-Performance-Part-1.pdf

-----

"Recording Engineers and Audiophiles often distrust audio measurements and specifications. It is not uncommon to hear claims that a product measures poorly but sounds good. Occasionally we also hear claims that a product measures well but sounds bad. This whitepaper documents significant differences between three headphone amplifiers that have nearly identical published specifications. Each is equipped with a high-quality low-jitter digital to analog converter, and each can accept 96 kHz 24-bit digital audio. All three products have a high-quality headphone output with very low output impedance, high drive capability, low distortion, and a flat frequency response. The published specifications strongly suggest that it should be impossible to hear any difference between these products. Nevertheless, in our informal listening tests, we had the distinct impression that there were significant sonic differences between these products when listening to music.

We replicated the manufacturers' tests and found that the published specifications were accurate and correct. All three manufacturers followed standard measurement procedures and provided accurate specifications....."

I did some googling given the MSRP. I hit on one unit that was a tube amp. I could not find any measurements on it. The specs provided were all at specific headphone type impedance so that didn't jive with their assertion of advertised THD being with no load.

BTW, the no load measurement actually has a load of 100 K. It is at the end of the report and is likely the input load of Audio Precision. So there is a load but not a realistic one.

I should also add that I did not mean this to be a headphone analysis paper. But rather, the notion that given a single measurement, THD, so many variations of the results can be extracted. I especially liked the shape of THD with the noise removed, showing far more distortion at low frequencies for two devices.

As usual, the problem is not measuring enough, but not measuring what directly matters in terms of audibility.

In the quote above is the statement: "All three products have a high-quality headphone output with very low output impedance, high drive capability, low distortion, and a flat frequency response."

Unfortunately I'd say only output impedance and flat frequency response directly tell you anything interesting with respect to the perceived sound. The other measurements merely suggest a possibility of low distortion and clean sound. Without further measurements, we don't know. More to the point, good THD measurements at best don't preclude good sound from a device on their own, but they don't guarantee it. It's been proven over and over that the order and spectrum of the distortion is much more important than the total level. Similarly, the change in behavior at differing levels and under different loads is much more indicative of audible differences. Some of the measurements presented under differing loads and levels give a glimpse at such differences.

Making the corollary to loudspeakers and subwoofers, I am just as interested in how the frequency response and distortion changes with level as I am with the absolute smoothness and distortion. Combine this info with a 3 dimensional view of the frequency response and you can get a good idea of an expected sound character. Of course that is far from a complete picture, and in most cases it's important to give a quick listen to hear if something was overlooked and not obvious in the particular measurements taken thus far.

Unfortunately you can only devise a reasonable set of measurements to characterize a device if you assume many things are non-issues, which is not always the case. Accounting for all possible issues or variations and to never miss them in measurements alone make for a super extensive set of measurements.

 

[Phelonious Ponk]

I would beg to differ on this one Tim. The good ones have their bag of goodies in the form of favorite mics and outboard gear. They do enjoy a wide latitude in terms of artistic input and their electronics are their instruments. Just like Audiophiles there are long running debates on the exact same themes and then some. While active vs. passive may not be debated, which active definitely is.

We don't disagree, Jack, I overstated. Of course they shape the sound with microphone choices, effects, eq. All the more reason good engineers look for transparency in DACs, amps and pres -- so they know what they're hearing is the mic, not a deliberately colored component.

Tim

 

[Phelonious Ponk]

As usual, the problem is not measuring enough, but not measuring what directly matters in terms of audibility.

In the quote above is the statement: "All three products have a high-quality headphone output with very low output impedance, high drive capability, low distortion, and a flat frequency response."

Unfortunately I'd say only output impedance and flat frequency response directly tell you anything interesting with respect to the perceived sound. The other measurements merely suggest a possibility of low distortion and clean sound. Without further measurements, we don't know. More to the point, good THD measurements at best don't preclude good sound from a device on their own, but they don't guarantee it. It's been proven over and over that the order and spectrum of the distortion is much more important than the total level. Similarly, the change in behavior at differing levels and under different loads is much more indicative of audible differences. Some of the measurements presented under differing loads and levels give a glimpse at such differences.

Making the corollary to loudspeakers and subwoofers, I am just as interested in how the frequency response and distortion changes with level as I am with the absolute smoothness and distortion. Combine this info with a 3 dimensional view of the frequency response and you can get a good idea of an expected sound character. Of course that is far from a complete picture, and in most cases it's important to give a quick listen to hear if something was overlooked and not obvious in the particular measurements taken thus far.

Unfortunately you can only devise a reasonable set of measurements to characterize a device if you assume many things are non-issues, which is not always the case. Accounting for all possible issues or variations and to never miss them in measurements alone make for a super extensive set of measurements.

I'm not at all sure what point you're trying to make, Mark. Of course you could always measure something else. And unless you've imagined everything that could possibly be measured, and measured it, and without going to that theoretical extreme, I suppose you can leap to the conclusion that it was assumed that everything that wasn't measured was a non-issue. That's a broad assumption, but you can go there. In this case, though, if you do, you've missed the point. They started with 3 DAC/headamps that sounded subjectively different but measured essentially identical, given normal manufacturers' measurements. They made exactly the same assumption that you did - that the measurements were insufficient and only suggested the possibility of lower distortion and cleaner sound. They measured more -- less theoretically and more realistically -- and they discovered good objective reasons why the components sound subjectively different.

Did they measure every possible parameter and uncover every possible reason for the difference in sound? Nope. But the pushed past the highly selective kinds of measurements manufacturers too often use to make their products appear to be better than they are, and they uncovered a couple of good reasons why gear that seems to measure the same, can sound different. And while it is in their interest to do so as long as their product comes out on top, they didn't even tell us what they competitive units were.

Good for them on all counts.

Tim

 

[microstrip]

(…) They started with 3 DAC/headamps that sounded subjectively different but measured essentially identical, given normal manufacturers' measurements.

No, Tim, they did not. No load (or 100kohm load, due to the input impedance of the AP, as explained by Amir) is not a normal manufacturer measurement for amplifiers, unless the manufacturer did not study basic electronics – not the case, surely.

They made exactly the same assumption that you did - that the measurements were insufficient and only suggested the possibility of lower distortion and cleaner sound. They measured more -- less theoretically and more realistically -- and they discovered good objective reasons why the components sound subjectively different.

I would love to know what is measuring less theoretically and more realistically in audio. It seems a too poor excuse for bad measuring practices.

Did they measure every possible parameter and uncover every possible reason for the difference in sound? Nope. But the pushed past the highly selective kinds of measurements manufacturers too often use to make their products appear to be better than they are, and they uncovered a couple of good reasons why gear that seems to measure the same, can sound different. And while it is in their interest to do so as long as their product comes out on top, they didn't even tell us what they competitive units were. Good for them on all counts.

Their subjective listening opinions, that have not been validated by any means, are not strong enough to withstand public challenges. They show a few common reasons why the other products seem technically flawed. Sometimes it is better staying incognito then fighting in other people arena.

Happily what is good for them on all counts is the quality of their products, not their technical propaganda tactics…

 

[fas42]

I would love to know what is measuring less theoretically and more realistically in audio. It seems a too poor excuse for bad measuring practices.

"Theoretically" means:

1) Extremely convenient for the manufacturer, that is, takes minimal time, is part of the production line
2) Shows equipment in best possible light
3) Is in an electrical environment 100 miles from what exists where it will actually be used

"Realistically" means:

1) Taking measurements when equipment is dead cold, hasn't been used for weeks; then when warmed up, after being nicely "run in" over some period; and finally, at the end of a day when it's been hammered hard, pushed continuously at close to its limits of operation. Then comparing those 3 sets of figures ...
2) Using ancillary gear that are the worst combinations, for example, attaching loudspeakers that have completely different load characteristics, for amplifier testing. Also, do the testing in an electrically extremely noisy enviroment, say, a couple of computers working flat out next to the component, plugged into the same outlet, as well as a fluorescent light or two, and some really nasty light dimmers.
3) Do tests where absolutely everything electrically mimics the typical domestic listening environment, where the component is actually operating as part of a realistic, typical audio system.

Frank

 

[Phelonious Ponk]

I completely agree with Frank's theoretical. For the "realistic," I'll settle for measured driving real loads across a real range. A typical case scenario, not a worst case scenario. The Benchmark test here seems pretty real to me.

Tim

 

[amirm]

What do we think of worst case measurements? Useful, or useless?

 

[fas42]

Worst case scenarios would be extremely useful, because they would give excellent guidance as to how "robust" the equipment's performance is when confronted with generally difficult circumstances; environment or source material. Tom's worry about THD of tape, say at 0dB is not really relevant because no real music contains tones at peak levels for anything more than transient moments. But an analysis of the reproduction of multiple simultaneous tones could be very eye opening, I agree, and with today's DSP processing it wouldn't be hard to find out exactly what the equipment was doing wrong ...

Frank

 

[microstrip]

What do we think of worst case measurements? Useful, or useless?

Amir,

Although in an audio context Worst case measurements is just a subjective statement, and our interpretation of it can be ambiguous, these type of measurements can be useful only if you can correlate something audible with them. I think they should be very useful at the development stage, but if presented to consumers they can be misleading, and most probably you will end up paying for something you really do not need and choosing equipment based on non relevant specifications.

 

[fas42]

So, Frank, at just what level (db)does distortion increase above 2%, and at what frequencies? I have had some experience with R2R, and tape, and perhaps you might retract what you just said or find measurments to support it. Reel to Reel distorts much worse than say a proper pre-amp and talk about tone controls...all measurements are relevant, whether worst case tape or any other component.

Tom

Tom, I don't want to start an argument for sake of it, I'm just attempting to point to areas where real efforts need to be made to understand what more diligent measurements have to be used to understand what's causing sound quality to not come up to scratch. And worrying about peak level distortion is not one of them. Vinyl, tube amps, speakers all have terrible distortion figures at the limits of their range of capabilities, but it doesn't get in the way of these mechanisms producing highly musical playback. And that's because the vast majority of the music being listened to spends hardly any time at these levels, as I tried to indicate. Unless you listen to modern, ultra compressed material at high volumes; but last I heard this is not an area where too many audiophiles spend much time ..

So, yes, tape does 2% distortion at the MOL for 1kHz say, but drop down to where 99.5% of the music is, and the distortion is very benign: there has to be some reason for many people raving about R2R playback ...

Measurements are of less relevance if they only relate to something that happens now and again, like a tiny crackle or pop every so often when playing vinyl: the ear/brain can handle this, can take it in its stride.

Frank