Dr. Sean Olive: New Evidence That Gen Y'ers Prefer Accurate Sound Reproduction

[Ron Party]

I also write a blog on the science of audio called Audio Musings. The article I just wrote today summarizes some recent listening experiments that show evidence that sound quality is not lost on Generation Y. For me, this is very encouraging as there is hope that the future of sound quality in sound reproduction will continue on, in spite of what the media is reporting these days.

I look forward to future interesting discussions.

Cheers
Sean Olive

I have several questions for you about this study, as I'm sure others here at WBF do, but for now I'll start with questions regarding program content:

If I correctly understand the slide presentation - a big assumption on my part - I see that three of the programs included female vocals. How did you select the programs? Is there something about these programs that made you believe these were amongst the best suited for testing the Gen Y preferences?

Related to this, I often read that one of the concerns with blind testing is familiarity with program content. Was this a concern in any way with the Gen Y study? I ask because my teenage son, who I believe to be like many of today's Gen Y'ers, listens to a lot of hip hop but rarely listens to the kinds of music used in the study.

 

[tonmeister2008]

I have several questions for you about this study, as I'm sure others here at WBF do, but for now I'll start with questions regarding program content:

If I correctly understand the slide presentation - a big assumption on my part - I see that three of the programs included female vocals. How did you select the programs? Is there something about these programs that made you believe these were amongst the best suited for testing the Gen Y preferences?

Related to this, I often read that one of the concerns with blind testing is familiarity with program content. Was this a concern in any way with the Gen Y study? I ask because my teenage son, who I believe to be like many of today's Gen Y'ers, listens to a lot of hip hop but rarely listens to the kinds of music used in the study.

Hi Ron,

The main criteria I used for selecting the music programs were that they be relatively well recorded and reveal audible differences between the MP3 and lossless formats. The program selection was done mostly on a trial and error basis (over a weekend) using my limited knowledge of how perceptual audio codecs work and what sorts of signals tend to trip them up: program material with dynamic range, containing high frequencies and lots of transients,where simultaneous and temporal masking is least effective, thus challenging the performance of MP3 codecs at lower bit rates. [ see note below].

Female vocals, percussive instruments like drums, massed strings, applause all meet this criteria, and were included in my selections. Some of the most critical programs used in previous audio codec tests are on the European Broadcast Union Test CD, and include solo instruments like triangles, glockenspiel,etc. I decided those were too far removed from what most people listen to

My theory (and I have no proof yet) is that much of the poorly recorded dynamically compressed material these days may be less effective at revealing Mp3 artifacts, and that is why kids may tolerate listening to MP3 - besides the convenience/storage benefits. Clearly I need to include some of this program in future tests to improve the ecological validity of the tests.

Besides the program material, the other variables that are known to influence audibility of perceptual audio codecs and may influence listener's music format preference are: the quality of the headphones/loudspeakers, the masking effect of room reverberation, the use of 2-channel-7 channel up-mixers, the training/selection of listeners, and the listening test methodology itself. These are all variables that I hope to examine and test more thoroughly in the near future.


[Note: A more sophisticated process for screening programs might have been to use a software program based on the PEAQ (Perceptual Evaluation of Audio Quality) model that compares and analyzes the MP3 encoded file versus the original lossless file using a perceptual model, and then calculates a Mean Opinion Score based on the predicted audible degradation. I would still need to do the listening tests since am interested in "preference" not audible degradation (remember that Berger reported that his students preferred the audible degradations of MP3).]


Cheers
Sean Olive
Audio Musings

 

[Ron Party]

In the accompanying slide presentation I see a photo of a yellow school bus but otherwise do not see any information regarding the identity of the high school. Is there any information you are comfortable in sharing regarding the high school, e.g., is it a public school, is it a school which has an emphasis in music?

Related to this, how did it come to pass that these particular 18 kids were the ones who would serve as test subjects? Did they just volunteer? Was this *field trip* part of some sort of academic curriculum, e.g., a music theory or science class?

Finally, I note that there were 5 female and 13 male students. I know that generally speaking us older folk can't hear much above 15 kHz, if not 13kHz, and my wife tells me her hearing is better than mine, so given this context did you find any noteworthy differences in preference between the girls and boys, either in the MP3 vs. CD or speaker testing?

 

[Ron Party]

The main criteria I used for selecting the music programs were that they be relatively well recorded and reveal audible differences between the MP3 and lossless formats. The program selection was done mostly on a trial and error basis (over a weekend) using my limited knowledge of how perceptual audio codecs work and what sorts of signals tend to trip them up: program material with dynamic range, containing high frequencies and lots of transients,where simultaneous and temporal masking is least effective, thus challenging the performance of MP3 codecs at lower bit rates. [ see note below].

Female vocals, percussive instruments like drums, massed strings, applause all meet this criteria, and were included in my selections. Some of the most critical programs used in previous audio codec tests are on the European Broadcast Union Test CD, and include solo instruments like triangles, glockenspiel,etc. I decided those were too far removed from what most people listen to

<snip>

[Note: A more sophisticated process for screening programs might have been to use a software program based on the PEAQ (Perceptual Evaluation of Audio Quality) model that compares and analyzes the MP3 encoded file versus the original lossless file using a perceptual model, and then calculates a Mean Opinion Score based on the predicted audible degradation. I would still need to do the listening tests since am interested in "preference" not audible degradation (remember that Berger reported that his students preferred the audible degradations of MP3).]

Amir, does this bring back memories of your days with Microsoft?

 

[Nicholas Bedworth]

Sean,

Glad to see you're participating on the forum...

Nick

 

[tonmeister2008]

The students all attended a LAUSD Music/Performing Arts Magnate located in Los Angeles. The teacher contacted me regarding a class field trip to Harman, and I asked if they wanted to participate in two listening tests, which they volunteered to do, and seemed to be quite motivated to participate. We intend to recruit another sample of students who have no musical training to see if there are any effects related to musical training,etc. Keep in mind that Jonathan Berger's study at Stanford University used music students, and they tended to prefer MP3 @ 128 kbps over lossless formats. So musical training may not be a factor in preference for MP3 versus CD music format. It certainly has never been a factor in any loudspeaker tests we have done over the last 25+ years. In fact, there is evidence from other researchers that musicians are often among the worst listeners in sound quality tests, at least until they've received some training and experience (perhaps they are too focused on musical details and not paying attention to the sound quality differences).

Regarding Gender:
The most consistent listener in the group was a female who preferred CD over MP3 format in 12 out of 12 trials. I haven't done any analysis yet on gender but I don't think gender was a factor in these tests. Gender has also never been shown to be a factor in loudspeaker tests over the past 20 years. Hearing loss has been shown to be a factor in terms of listener performance, and historically females have been shown to have better hearing than men based on their lower absolute hearing thresholds. The reason for this is believed to be occupational and recreational-related: men tend to have noisier jobs and hobbies that result in greater noise-induced hearing loss.

Cheers
Sean Olive
Audio Musings

 

[MylesBAstor]

The reason for this is believed to be occupational and recreational-related: men tend to have noisier jobs and hobbies that result in greater noise-induced hearing loss.

Cheers
Sean Olive
Audio Musings

The same occupational hazards result was shown for men vis a vis sensory nerve endings in the skin. Men had jobs like using jackhammers, etc.

 

[Ron Party]

Have you reported your findings to the test subjects? And if so, what kind of responses, if any, did you get?

 

[Gregadd]

Gender me be a factor to their approach to the test. When I tutored math in college men took advice after they screwed up. Women listened up front. Women also tended to take the the test more seriously and follow directions more closely. This is anecdotal of course.


The problem is when you begin to talk about listener preferences vs fidelity to live music. If you goal is maximum sales to the general public then listener preference may be your standard. My standard as is that of most audiophiles is fidelity to live music. I think that requires a higher standard than listener preference and a deeper explanation by the listener why he/she made a particular choice.

Now the standard response we never know what the original recording sounded like. I consider that a red herring. We could make our own recording and have the musician there to play it for us.

 

[amirm]

Amir, does this bring back memories of your days with Microsoft?

It does indeed! When I heard Sean used people clapping as one of the samples, I knew he would get a good hit ratio of people detecting the compressed clip. It is amazing how difficult it is to encode such a clip! Transients are also good clips.

 

[amirm]

[Note: A more sophisticated process for screening programs might have been to use a software program based on the PEAQ (Perceptual Evaluation of Audio Quality) model that compares and analyzes the MP3 encoded file versus the original lossless file using a perceptual model, and then calculates a Mean Opinion Score based on the predicted audible degradation. I would still need to do the listening tests since am interested in "preference" not audible degradation (remember that Berger reported that his students preferred the audible degradations of MP3).]

We never found such objective analysis tools to be of value in these applications. But rather, we used to detect gross distortion in automated tests. Each one of us had our favorite suite of "codec killers" we used in addition to classic MPEG clips.

 

[tonmeister2008]

Have you reported your findings to the test subjects? And if so, what kind of responses, if any, did you get?

I sent the results to the teacher to distribute to the students. Haven't heard any feedback yet so I will probably follow up soon. I also need some clarification on their music training since that subject wasn't addressed in the pre-survey.

 

[tonmeister2008]

Interesting. I don't have much experience with these tools except we briefly had a demo version of OPERA from Opticom which we used for measuring loudspeaker distortions. Of course, that was not its intended application, but there was some correlation between audibility and the calculated MOVs.

Years ago, someone from Universal Music used our listening labs to evaluate different CODECS for possible use for a new optical based mini-disc that required lossy compression: the media never came to market.

On most rock music samples I couldn't hear many differences. Then he switched to the EBU Disc torture tracks and the audible differences among the codecs were like night and day. Most of them sounded broken with these program tracks with the exception of one or two.

 

[tonmeister2008]

Gender me be a factor to their approach to the test. When I tutored math in college men took advice after they screwed up. Women listened up front. Women also tended to take the the test more seriously and follow directions more closely. This is anecdotal of course.


The problem is when you begin to talk about listener preferences vs fidelity to live music. If you goal is maximum sales to the general public then listener preference may be your standard. My standard as is that of most audiophiles is fidelity to live music. I think that requires a higher standard than listener preference and a deeper explanation by the listener why he/she made a particular choice.

Now the standard response we never know what the original recording sounded like. I consider that a red herring. We could make our own recording and have the musician there to play it for us.

Of course, the majority recordings made today (notwithstanding live performances), are studio creations where the "live" music event never happened. So what is the standard or reference for judging the fidelity of the reproduction then? The best we can do is try to replicate in our homes the sound that the artist/producer heard in the studio. This is difficult to do right now since there are no meaningful playback standards in either the recording or the playback chain: we are caught in the endless "Circle of Confusion" that Floyd Toole refers to. This is the "Red Herring" you are talking about.

The solution is for the professional and consumer audio industry to develop some meaningful standards that define the performance of the loudspeakers and their interaction with the room acoustics. The science already exists to do this. On the consumer side, we are already working towards this goal within CEA and CEDIA. On the professional side, we need to educate the audio professionals/artists and convince them to adopt a standardized/calibrated playback chain so we know better what the artist was hearing and what they intended us to hear.

I've argued with many people about the scientific pitfalls/fallacies of the live versus reproduced listening test. Edison, who was the ultimate audio salesman/promoter used these live versus reproduced parlor tricks in the early 1900s to convince people that his phonograph recordings sounded identical to a trained opera singer who would stand behind a curtain and sing along with the recording. People were apparently fooled which one was live and which one was recorded, which tells me the methodology is not very sensitive, and is inherently biased due to too many uncontrolled nuisance variables.

I agree with you regarding the need to have an explanation for why people prefer a particular audio component. We get that information routinely from our listening panel who are trained to quantify in precise terms the different sound quality attributes among the stimuli in terms of spectral, spatial, dynamics/distortion aspects. With untrained listeners, it's challenging to get useful information because they don't have the skills and common terms. We did get comments from the high school students as to why they preferred CD over MP3, and their comments generally confirm what a trained listener would report.

Finally, based on 30+ years of loudspeaker testing, I would argue that preference and fidelity in sound reproduction may be the same thing. The fact that the most accurate and linear loudspeakers consistently win the double-blind tests is proof of that. Many years ago we switched from fidelity ratings to preference ratings and the same loudspeakers still won the tests.

 

[Ron Party]

How was it that you decided to use MP3 @ 128?

By the way, some of those photos at the beginning of the slide presentation are priceless!

 

[Ron Party]

Finally, based on 30+ years of loudspeaker testing, I would argue that preference and fidelity in sound reproduction may be the same thing. The fact that the most accurate and linear loudspeakers consistently win the double-blind tests is proof of that. Many years ago we switched from fidelity ratings to preference ratings and the same loudspeakers still won the tests.

This would be a great subject for a new thread.

 

[Gregadd]

You know Sean let me take the gloves off. I think what you really do is not advance the state of the art. What you actually do is try to see how low you can go and have the general public still buy it. Fancy terms like "lossless data versus compressed data " don't mislead the careful examiner. The fact of the matter is we can know what a trumpet sounds like. Different quality trumpets may have different sounds. A recorded trumpet should still sound like a trumpet when it is played back. If we really want to know how it sounds we can have the trumpeter stand there and play the trumpet while listen to whatever media your are trying to sell to the public or evaluate for your employer. Then you can compare live vs whatever media you desire. My standard is not whether crap A sounds better then crap B.

I think what you mean by "lossless data" is the original untreated recording something like if you recorded directly to a Master Tape. Compressed data means an inferior recording where elements of the music were loss to facilitate what I consider an inferior storage medium. That medium was chosen more for its commercial viability than its musical fidelity. That's nothing new and is what drove most of us to the high end anyway.
Finally I assure you that that "preference and fidelity" by definition are not the same thing. I find at really amazing that that those who are willing to dismiss as a subjectivist is willing to make decisions on listener preference. Especially when IMO the are choosing from inferior options.
No offense intended.

 

[The Smokester]

I know I can tell the difference between 128k MP3 and 320k or lossless and that my preference is for the higher sampling rates. When I first bought my iPod Classic I converted my flac library to mp3 to get it on the player. I was very disappointed in the iPod's performance and posted about it on Head-Fi where I was lambasted thoroughly. I then discovered that the setting on the format converter was inadvertently set to 128k and had to retract my post.

Some of what is written in this thread reminds me of what becomes ones favorite recordings: Namely, ones that your system plays well. This is a type of self-selection. Is it not a possibility that MP3 encoding complements the sounds that the latest generation wants, and is accustomed, to hearing?

 

[amirm]

You know Sean let me take the gloves off.

I hope you mean that figuratively . We want to have a respected discussion, not one that is personal. Let's stay on the topic rather the person and their motives. BTW, the fact that someone works for a company for a living, doesn't automatically invalidate what they have to say unless we can challenge the information on its merits.

I think what you really do is not advance the state of the art. What you actually do is try to see how low you can go and have the general public still buy it.

In what way using expert listeners as Sean said they use to evaluate audio systems is for "general public" to buy something? The expert listeners I have worked with are able to outdo most audiophile in how they are able to discern flaws in audio systems. They go through special training which allows them to hear things that some of the best audiophiles often miss.

Fancy terms like "lossless data versus compressed data " don't mislead the careful examiner.

??? Lossless compression means using a compression format for ripping the content where the source can be reproduced perfectly such as using FLAC or WMA Lossless. [Lossy] compressed means format such as MP3 which lose fidelity in the process of compressing audio. In what way using those terms is misleading? He is using standard terminology which is not only clearly understood in the industry but also by enthusiasts just the same.

The fact of the matter is we can know what a trumpet sounds like. Different quality trumpets may have different sounds.

I don't know how these two sentences are consistent with each other.

A recorded trumpet should still sound like a trumpet when it is played back. If we really want to know how it sounds we can have the trumpeter stand there and play the trumpet while listen to whatever media your are trying to sell to the public or evaluate for your employer. Then you can compare live vs whatever media you desire.

How do you do that with music you buy from some place? That is the point Sean is making. He said that most recordings we listen to are not only the result of what the live instrument sounds like by itself, but how it sounded in the recording *room* which added many other characteristic. It was that sound that was heard, QE'ed and adjusted by the recording engineer and based on feedback from the talent, that was put on the media. And unfortunately, there is no way for us to ever know what they really heard as no metadata was captured to tell us that. And trying to match the sound of a live trumpet would do you no good at all because that is also arbitrary and at any rate, is not what the two people above attempted to record.

Is your goal in audio reproduction to create your own sound or that the talent intended for you to hear?

I think what you mean by "lossless data" is the original untreated recording something like if you recorded directly to a Master Tape.

That is not what I understood him to mean. He used the term in two contexts:

1. Lossless as opposite of lossy compression. I.e. WMA Lossless/Flac as opposed to MP3

2. Compressed dynamics where the average loudness level is increased as in typical pop music releases today.

Finally I assure you that that "preference and fidelity" by definition are not the same thing. I find at really amazing that that those who are willing to dismiss as a subjectivist is willing to make decisions on listener preference. Especially when IMO the are choosing from inferior options.
No offense intended.

So are you saying that accuracy is less important than personal preference for a sound? 'cause that is what I am understanding Sean to mean by his terminology. Fidelity = trueness to the source. Preference = liking something whether it is or isn't the same as the original.

Also, Sean didn't dismiss anyone. He said that when they look at the scores for fidelity vs preference they seem to track together. That is, people tend to like more accurate sound not less. If there is data to the contrary and is the view of audiophiles that less accurate sound is better, I am sure both he and I would love to see that data .

 

[amirm]

Some of what is written in this thread reminds me of what becomes ones favorite recordings: Namely, ones that your system plays well. This is a type of self-selection. Is it not a possibility that MP3 encoding complements the sounds that the latest generation wants, and is accustomed, to hearing?

Interesting point . I have only experienced isolated examples of this in large scale listening tests. In one example, we had encoded a bunch of material in two codecs: one that had wide response close to CD and the other, half the CD (i.e. its highest frequency was 11 Khz). The population study was our own employees in the group. And there, we had one female voter who voted against everyone in favor of the version with half the response. So after the test I went and asked her why she preferred that one. Her answer was that it was more mellow and softer sound which of course, was true since it had little high frequency.

In video testing, codecs which filter the source and hence the noise with it, often rate higher, even in some cases higher than the original! Of course source resolution is also lost in the process of filtering but people tend to not notice that as much as the "improvement" they think they are seeing in the form of less noisy picture.

In case of audio test above, we dismissed that single vote and went with the majority. In case of video, it was much harder to figure out if one has to use that trade off. So our decision was to use similar techniques at lower bit rates (think youtube video) but at high definition data rates, we took off all filtering.

Beyond these corner points though, we have extensive experience after years of compressed audio being in the field that other than filtering, there is little that makes the sound better. I have done tens of thousands of such comparisons, including many instances of using my family members and I have never found them to like compression artifacts. Their feedback goes from hearing no difference to not liking the artifacts.