Creating a Audio Dartboard

[MylesBAstor]

I missed this before .

I am not sure any two audiophiles ever agree with anything. One thing I hope we can agree on though is the respect for people who spend a lifetime trying to understand how we hear audio. Sean came from the famed Canadian NRC. No one done large scale characterization of what it is we like about audio reproduction. Sean's writing is usually authoritative and allows one to learn a ton about sound reproduction. Whether one agrees with everything he has to say is not necessary to learn what he knows about audio. I firmly believe NRC and the researchers who led that work and now work at companies such as Harman and Paradigm do bring to us knowledge in addition to opinion. And that knowledge has surely advanced the art of audio design, turning it into at least 50% science. The idea of a guy sitting in the room simply tweaking a speaker without such science does not make sense to me. What happens when he gets 10 years older and grows somewhat deaf?

Two things we can agree on. We can never agree on anything!

Second, I think that high-end audio gear, in contrast to the mass marketed Sony, Pioneer, etc, represents a blend of science and art. As we know from the initial digital and solid-state debacle, we can only measure for what we know. Intuitively, I'd like to believe that we can explain things with science. But as I said earlier, reductionism doesn't always works because of unpredictable synergies. But somewhere, the ear still has to be final arbitrar.

 

[amirm]

For consumers, the ear can be the final arbiter. It is your money and can spend it the way you like . But if you do this stuff or a living, then that won't be the case -- at least not all the time. People design audio gear for wide audience. One's ear may not be representative of what the larger set of consumers like. This is why NRC research was so groundbreaking. It quantified what large collection of consumers like. This is why when I explained audio compression, I put equal weight on both factors.

BTW, the reason subjective testing gets used a lot in designing audio gear is not because it is better per-se but because it is much, much faster and cheaper. I can listen to something in two seconds and decide it is good or bad. Trying to do the same thing with a population study in blind manner will set you back orders of magnitude more work. Therefore, we rely on "expert listeners" which we hope are most of the time are hearing the right things to steer us in the proper direction.

Here is another related point: there is a big difference between setting up DBT to prove someone wrong vs trying to learn something. When we want to learn something, we don't give up when results are inconclusive for example. We change the experiment until we get positive confirmation that a difference is detected and it is good or bad.

 

[Gregadd]

No offense intended

For consumers, the ear can be the final arbiter. If the ear is not the final arbiter for the designer then what is? I assume at least for the high-end we are trying to recreate the sound of the live event. You can do anything you want to get there but that is the final test. If you fail at that nothing else matters It is your money and can spend it the way you like . But if you do this stuff or a living, then that won't be the case -- at least not all the time. People design audio gear for wide audience. One's ear may not be representative of what the larger set of consumers like. This is why NRC research was so groundbreaking. It quantified what large collection of consumers like. This is why when I explained audio compression, I put equal weight on both factors.Who cares what the mass market likes. Hell the mass market thinks Briney Spears is a musician. The mass market likes boom and sizzle. The mass market likes compressed music. Again the standard is the live event. Anything else is subjective

BTW, the reason subjective testing gets used a lot in designing audio gear is not because it is better per-se but because it is much, much faster and cheaper. No not at all the the reason we use listening(subjective tests)because that is what it comes down to. When we sit down to listen do we get the illusion of the live event. You can try to get there by any means necessary. In the final event that's what it all about.I can listen to something in two seconds and decide it is good or bad. Maybe you can . I have spent thirty years trying to figure that out and I am still not sure. Harry Pearson said he needed six months to get the true measure of equipment. Trying to do the same thing with a population study in blind manner will set you back orders of magnitude more work. Therefore, we rely on "expert listeners" which we hope are most of the time are hearing the right things to steer us in the proper direction.No we rely on expert listeners solely because they undertook the job. A job that so called objectivists and scientist either were unwilling to undertake or deemed unnecessary because measurements told us all we needed to know. By your own admission blind tests are not being done even by its own proponents. I never heard anyone sold called subjectivist say there method is to hard.

Here is another related point: there is a big difference between setting up DBT to prove someone wrong vs trying to learn something. Agreed When we want to learn something, we don't give up when results are inconclusive for example. We change the experiment until we get positive confirmation that a difference is detected and it is good or bad.

We also consider that the test may me be flawed and consider other methodologies. Not a dogmatic adherence to ABX/DBT.

 

[amirm]

The ear makes mistakes. It is also a subjective instrument that cannot convey data to another person who may also be involved in a design. It is a great tool to be sure, but for design of the equipment, it has a lot of flaws as with anything else.

BTW, I made no reference to "mass market." The number of people buying gear is >> than the person who designed it. Unless it can be proven that we all hear exactly like the designer, his subjective experience cannot be definitive. This is why it is important to gather data from more than one person.

As to replicating live event, that is the goal. But none of us know what that is. The live event was recorded using means unknown to us. We think we know what it is supposed to sound like. But like the kid said in movie Matrix, "Maybe they couldn't figure out what to make chicken taste like, which is why chicken tastes like everything." We have plenty of measurements that tell us how well a piece of equipment is working, some of which is far more accurate and revealing than one's ear.

Here is Amir's rule of audio: stick to either religion and you will have a hard time defending yourself in all situations. That is why I stay smack in the middle. In this thread, I am defending why objective data is important. In other places where people dismiss all subjectivity, I show them how we can't measure everything or understand what we measure. Of course, that gets me arguments from both sides . So be it. I know there is much less chance I will be caught with my pants down .

 

[Gregadd]

Living on the edge with your pants down is just so much more fun. You should try it sometime.

 

[amirm]

Living on the edge with your pants down is just so much more fun. You should try it sometime.

If someone paid me to design A/V technology that way, I would. But no one does since having pants on is a requirement to work at most companies .

 

[Gregadd]

If someone paid me to design A/V technology that way, I would. But no one does since having pants on is a requirement to work at most companies .

I gave it a shot. I just think you miss the point in so may ways. I'm just thankful that so many recognized the shortcomings of your philosophy and went out on their own. I guess thiere has to be both a yin and yang to everything. I am reminded of a saxophone player. His father had trained, The interviewer remarked that the guy never made much money and was relatively unknown. He replied his father had taught him to master his instrument not to be famous or make money. He went on to tell the interviewer, You found me didn't you.

 

[FrantzM]

Two things we can agree on. We can never agree on anything!

Second, I think that high-end audio gear, in contrast to the mass marketed Sony, Pioneer, etc, represents a blend of science and art. As we know from the initial digital and solid-state debacle, we can only measure for what we know. Intuitively, I'd like to believe that we can explain things with science. But as I said earlier, reductionism doesn't always works because of unpredictable synergies. But somewhere, the ear still has to be final arbitrar.

Hi
I don't think there is another hobby where people bend and weave around basic issues as in High End Audio.
This said:
Do we admit that the ear can be fooled? If Yes then let us move forward if NO, then let us stop right there.
Do we admit that our senses are unreliable? If Yes let us continue , else no reason to even read anything else from this point ....
Do we admit that how and what we hear is thoroughly explainable by science? If yes, let's move forward else let's stop..
If you've made it that far, it seems sensible that the gear that we love and their performance are technological pieces, that are applications of known science. That some manifestations escape our present understanding does not require us to invoke anything outside science to explain it. Let’s be clear Audio designers are not magicians, nor Artist they are Technologists … "Art" in that context is the ability to balance constraints so as to present results which despite the difficulties and constraints are deemed plausible by the intended audience; NOT to PRODUCE art.
Audiophiles have a conflicting relationship with science. If Science seems to go in the direction of their beliefs then Science is well seen and is cited to explain how great the gear if it goes toward infirming their views then it is because the gear is “Art” thus can’t be explained by science. Scientific notions are invoked to simply prove appoint regardless of context thus the notion of “synergy” favored by many audiophiles. If an experience infirm a claim then it is because the methodology is flawed. It is on this account interesting how audiophiles, myself included never questioned judging a gear while seeing , it knowing its reputation its supposed performance, e tc… This methodology is rarely if ever questioned. We turn argument on its head… We use arguments such : Blind Testing doesn’t work because Audio memory is too short. So when we substitute a gear do we reliably remember how the previous one sounded ? I know the answer to that.. I will most likely bow out …
Just one thing to repeat myself: Audio Designers are not Artists nor are Engineers in the application of their craft. Audio components are Technological and their performance thoroughly explainable by science.
Frantz

 

[amirm]

I gave it a shot. I just think you miss the point in so may ways. I'm just thankful that so many recognized the shortcomings of your philosophy and went out on their own.

I am not sure why you are personalizing this. I am not trying to convince you of my way but what the industry at large uses. MPEG is an international organization designing audio//video compression such as MP3, AAC and MPEG-4. Video codecs used in Blu-ray were chosen based on DBT and subjective testing just the same. The methodology of mixed subjective and objective testing is used by standards groups extensively.

I also was very clear that as end customers, you are free to choose whatever method suits you. I am not trying to suggest, nor is it practical, that you perform DBT for every piece of equipment you buy. If by "everyone" you mean people go and design equipment without any sort of objective testing using instruments and blind testing in addition to their ear, then I disagree. I know a ton of people in the industry and with some exceptions they all use the various techniques to achieve excellence.

 

[Gregadd]

Wow where do you guys get this stuff?
DBT does not deal with the ear being tricked. It deals with sight bias. That is the listeners supposed desire for a brand or a device to have an effect on the perceived sound.
Lets' condense what Admir said
That DBT is necessary to reach a valid scientific conclusion. Therefor the conclusions of the subjectivists are invalid. But he does not do DBT because its too hard and expensive. Logically then if my results are invalid because of a lack of DBT then so are yours.
That his standard is not live music but is based on what someone will pay him for and what a large number of people like. By definition that is subjective and probably biased. Finally then living with a piece of equipment in your home for a protracted period creates substantially more acoustic memory than rapid AB switching. Moreover one can bring the equipment home and listen as long as the dealer will allow. Moscode gives you thirty days as does Sanders.

 

[apanton1952]

Fascinating.

On the subject of live, lets consider a $100 Yamaha classical guitar. Not earth shattering in terms of brand cachet or even technical specification (at that price anyway). However, it is guaranteed that though there may be similarities in the sounds between a number of those instruments, no two will be precisely alike. This due to a myriad of reasons including environment, humidity, clothing, pick/finger nail variances etc and then finally the ear. I may prefer instrument a, another b and so it goes on. However, the musical piece is a decider too. And it goes on and on and on.

But this I can tell you, no system on earth can reproduce the sound of any of those guitars perfectly. I have heard a monster ML system come close, but at the price rather go and buy a guitar and learn to play, then join an orchestra. The goal must be live sound (agreed) and the final decider is the ear and even better if you can compare it directly to the particular instrument. This of course raises the bar all the way back to the recording methodology and technology and opens another can of worms..

But you can always detect the switch from live to recorded and back again, at least on a classical guitar - piano may be even more of a challenge.

So perhaps what is required is a catalogue of standard "sounds" referenced to a particular known instrument, player and environment. This may provide a solid basis for at least comparing components/systems on the basis of a deviation in the first instance from the live instrument as assessed by music "experts" and then defining the variances again for the particular components/systems. Of course, there would have to be a reference system as well, something most experts probably never would agree on!

Hope I'm making sense here...

 

[amirm]

You are definitely right on the guitar. It is one of the most challenging sounds to reproduce. The transients come with vengeance, revealing a lot of issues especially in digital systems. A quiet solo guitar piece can be exceptionally informative in finding DAC problems for example.

 

[MylesBAstor]

Every instrument is a challenge to record and for audio equipment to reproduce ranging from the little clavichord to the organ to the drums to the violin......

 

[Gregadd]

Microphone technique is also critical. The ML/CLS is is very transparent. A close microphone placement reveals the sound of the pick scraping across the strings. Pull the microphone back and all you get is the rich overtones. Listen to a close microphone placement on the saxophone and you can actually hear the feint sound of the valves clicking. Same experience with piano. A microphone placed directly over the strings allows you to hear the hammer strike the strings. When the hammer strikes it prevents part of the string from vibrating. You get the overtones when the hammer is lifted. A different technique where you try to mimic what the audience is hearing might miss some of that.

May artifacts of sound are captured with a solo performance but are drowned out by other instruments. That's why the recording is so critical. Is the equipment under evaluation actually revealing and transparent or was it just a close microphone recording?

 

[amirm]

Every instrument is a challenge to record and for audio equipment to reproduce ranging from the little clavichord to the organ to the drums to the violin......

Not in all cases. Violin for example is far more harmonic and lacks the transients of guitar. Try compressing violin and guitar side by side and you will see that the latter loses far more fidelity than the former and its artifacts far easier to identify.

Drums have transients and can lose their impact if not treated well but since their frequency is lower and the ear is less sensitive in that region, its artifacts are somewhat harder to detect although once you know it, you can also hear them. Harpsichord is also a tough instrument for reasons that are difficult to explain but it is a good test clip and is used by MPEG for compression testing. So if clarichord is similar to it, then we are in sync there .

Note that all of my comments are in regards to digital systems reproducing such signals and whether it is hard to find artifacts in them.

 

[MylesBAstor]

Not in all cases. Violin for example is far more harmonic and lacks the transients of guitar. Try compressing violin and guitar side by side and you will see that the latter loses far more fidelity than the former and its artifacts far easier to identify.

Drums have transients and can lose their impact if not treated well but since their frequency is lower and the ear is less sensitive in that region, its artifacts are somewhat harder to detect although once you know it, you can also hear them. Harpsichord is also a tough instrument for reasons that are difficult to explain but it is a good test clip and is used by MPEG for compression testing. So if clarichord is similar to it, then we are in sync there .

Note that all of my comments are in regards to digital systems reproducing such signals and whether it is hard to find artifacts in them.

Transients are one thing but I was thinking more about the directionality and overtones of the violins complicating matters.

 

[MylesBAstor]

For consumers, the ear can be the final arbiter. It is your money and can spend it the way you like . But if you do this stuff or a living, then that won't be the case -- at least not all the time. People design audio gear for wide audience. One's ear may not be representative of what the larger set of consumers like. This is why NRC research was so groundbreaking. It quantified what large collection of consumers like. This is why when I explained audio compression, I put equal weight on both factors.

BTW, the reason subjective testing gets used a lot in designing audio gear is not because it is better per-se but because it is much, much faster and cheaper. I can listen to something in two seconds and decide it is good or bad. Trying to do the same thing with a population study in blind manner will set you back orders of magnitude more work. Therefore, we rely on "expert listeners" which we hope are most of the time are hearing the right things to steer us in the proper direction.

Here is another related point: there is a big difference between setting up DBT to prove someone wrong vs trying to learn something. When we want to learn something, we don't give up when results are inconclusive for example. We change the experiment until we get positive confirmation that a difference is detected and it is good or bad.

Actually Amir I have to disagree here. Do you think high-end designers go strictly by measurements. Measurement only take them so far; every designer I know listens and selects the active and passive parts like caps, resistors, switches, connectors before deciding on the unit's specs. All of these parts certainly have a big impact upon the sound of the unit and in actuality represents the designers "voicing" of each unit.

 

[amirm]

No I don't believe they go strictly by measurements. I think even mass market consumer manufacturers perform subjective testing in addition to measurements. I actually have nothing to disagree with regarding what you have said.

Good designers will use all techniques available to them:

1. Start with measurements since they are quick, precise and usually repeatable. Good companies create their own tests on top of industry standards.

2. Add subjective evaluation not only by the designer but other expert listeners. Like to see at least 2-3 people here in addition to the designer.

3. Blind evaluations especially as compared to competition. Again, good companies create constructive evaluation plans and test fixtures to advance the art here. The Harman speaker sled which switches speakers in under 4 second as to make sure the person doesn't forget the sound of one gear when switched to another, is an example of such innovation.

Where we might differ is that you may not consider #3 an essential component and I do. The only justifiable reason to not do it is because one can't afford it. Not that no data can be gained from it. I have been part of, and conducted many blind tests which had very interesting data and good learnings.

 

[MylesBAstor]

No I don't believe they go strictly by measurements. I think even mass market consumer manufacturers perform subjective testing in addition to measurements. I actually have nothing to disagree with regarding what you have said.

Good designers will use all techniques available to them:

1. Start with measurements since they are quick, precise and usually repeatable. Good companies create their own tests on top of industry standards.

2. Add subjective evaluation not only by the designer but other expert listeners. Like to see at least 2-3 people here in addition to the designer.

3. Blind evaluations especially as compared to competition. Again, good companies create constructive evaluation plans and test fixtures to advance the art here. The Harman speaker sled which switches speakers in under 4 second as to make sure the person doesn't forget the sound of one gear when switched to another, is an example of such innovation.

Where we might differ is that you may not consider #3 an essential component and I do. The only justifiable reason to not do it is because one can't afford it. Not that no data can be gained from it. I have been part of, and conducted many blind tests which had very interesting data and good learnings.

Ok we agree more than we disagree But what I would ask about pt. 3 is what is the correlation/relationship between blind listening tests and long term listening test results and consumer satisfaction?

 

[amirm]

what is the correlation/relationship between blind listening tests and long term listening test results and consumer satisfaction?

A precursor to that question is what is gained from extended listening? Answer is familiarity. Most of the time, we don't know what we are listening to.

Proper double-blind listening of music will include time and availability of material in advance to allow the user to become familiar with them. There should be no time limit to that process allowing the person to become firmly trained. In standards groups, the same clips are used over and over again so that this training is not necessary (that creates its own set of problems unrelated to topic at hand).

So, a round about way to say that well executed tests that are designed for learning rather than tricking someone to fail, does include provisions as you mention although obviously not to the level of allowing months to go by before the tests are done.

When I setup blind testing at home, of course I have all the time in the world available to do things to my satisfaction. If I want to spend a month doing it, I can.

Of course, no test is perfect. Blind tests can fall victim to many problems which could invalidate part or all of their results. Nothing is more difficult than creating proper audio test protocols. I usually have no problem finding half a dozen flaws in tests published by experts in respected circles such as AES. This is hard stuff to do right. Worse yet, they are too expensive to repeat once problems are found.