Old vs. new violins; the perils of blind testing?

marty

Well-Known Member
Apr 20, 2010
3,025
4,171
2,520
United States
Fabulous reporting. Thanks. So many surprises here, not the least of which is that its in PNAS (usually devoted to publishing advances in the rigorous biological sciences). But better yet, if the paradigm here isn't a model for high end audio, I'd be shocked. There is no question in my mind, that in the hands of a blinded listener, "older and very expensive" audio gear can be equalled if not exceeded by their more contemporary and inexpensive counterparts. However, you'll have to excuse me now while I purchase some magic flooby dust for $500 that promises to make my 6 figure rig sound even better....
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
The perils of bad experimental procedure :D

Do you believe that any musician (or anybody for that matter) who has to wear welder's goggles, with scent wafting up their nose while playing in an unfamiliar hotel room will have their powers of discrimination affected? It's a while since I read that report but if I remember it correctly the typical mistake & major mistake with all such blind tests was made - no controls were used to check if this environment did actually affect peoples discrimination abilities. The experimental setup seems to me to be conducive to guaranteeing a null result i.e no differences would be found. With the absence of any hidden controls, the test is flawed & I'm surprised that such a flawed experimental procedure was not picked up in peer review before publishing - let's not lose our powers of reason because a paper is published by PNAS?

I'm not saying that there is or is not an audible difference between violins but this experiment provides no help in answering that question - only further confusion. So I agree with the OP -"The perils of blind testing" - avoiding a rigorous experiment

Of course I could be wrong.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
There has been some criticism on the experimental design including the hotel room.
They repeated the experiment, this time in a concert hall.


http://www.pnas.org/content/early/2...ract?sid=5a3bf425-8e2c-4e71-874b-37790fb8bfb6

Don't have time to read it but when I said "environment" I didn't mean just the Hotel room - I meant the whole experimental design "environment"
In the new test did they address the fundamental flaw of the experiment - internal controls to evaluate how the test procedure affects the participant's ability to discriminate known differences & to what level? A validation & calibration of the test procedure/setup/participants, if you will.

Otherwise the test & it's results are meaningless!!
 

rbbert

Well-Known Member
Dec 12, 2010
3,820
239
1,000
Reno, NV
Hard to say, only the abstract is free online AFAIK, but this experiment was specifically designed to address at least some of the shortcomings of the welder's goggles / hotel room study you mentioned. Still, a big part of the reason I posted it was just as you say; it's very difficult to design and perform a truly blind but sensitive test in the behavioral sciences. As Victor says, though, just the fact that it's in PNAS suggests that some of the common shortcomings have perhaps been avoided or minimized.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Hard to say, only the abstract is free online AFAIK, but this experiment was specifically designed to address at least some of the shortcomings of the welder's goggles / hotel room study you mentioned. Still, a big part of the reason I posted it was just as you say; it's very difficult to design and perform a truly blind but sensitive test in the behavioral sciences. As Victor says, though, just the fact that it's in PNAS suggests that some of the common shortcomings have perhaps been avoided or minimized.

Sorry, I must have missed where Vincent said this but in any case I don't accept that is the case - having read the PNAS paper the flaws are blindingly obvious & as I said before I'm surprised that this paper was passed for publication. It's not as if there aren't published & recognised procedures for testing small audible difference that have been mentioned here before.
 

rbbert

Well-Known Member
Dec 12, 2010
3,820
239
1,000
Reno, NV
Is there something different in the paper than the abstract? According to the abstract, the study wasn't designed to detect differences (although it did), it was designed to determine preferences.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Is there something different in the paper than the abstract? According to the abstract, the study wasn't designed to detect differences (although it did), it was designed to determine preferences.
But Preferences can only be determined by & implies (is predicated on) differences, no?

Edit: I didn't see the Youtube video - Vincent must have put this up later
Yes the second test seems much more comprehensive compared to the PNAS paper but I still don't see any evidence of hidden controls within the test.
It would be interesting to see the new paper & if the seemingly preferred "new violin" showed some spectral characteristics that accounted for it being preferred?
 
Last edited:

MylesBAstor

Well-Known Member
Apr 20, 2010
11,236
81
1,725
New York City

rbbert

Well-Known Member
Dec 12, 2010
3,820
239
1,000
Reno, NV
Huh? Preferences can only be determined by & implies (is predicated on) differences, no?
...
In that case I totally miss your point, because differences were detected and preferences expressed. Whatever you might think of the study in other ways, it certainly did show differences.
 

rbbert

Well-Known Member
Dec 12, 2010
3,820
239
1,000
Reno, NV

MylesBAstor

Well-Known Member
Apr 20, 2010
11,236
81
1,725
New York City
Better take it up with the PNAS editors :D:p


You should know that there are articles that get published in every journal that raise people's eyebrows. You and I know it's all about who gets the piece for review. They probably had no idea who to send this article to for review.
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
Denial is not a river in Africa. Welcome back, John.

Tim
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
In that case I totally miss your point, because differences were detected and preferences expressed. Whatever you might think of the study in other ways, it certainly did show differences.
What I mean is that there is no consistency in differences/preferences as evidenced by this quote
How consistent were the subjects? Of the 15 who chose new
violins more often than old ones in part 1, 7 later chose old violins
to take home. Against this,?ve subjects who chose old violins more
often in part 1 later chose new violins to take home (SI Text). By
this measure, just 9 of 21 were consistent

No consistent preferences means that guesswork is probably at play or at least requires this possibility to be investigated for some rigour
Instead this confused logic was proffered to explain the above
—although this ?nding
seems unsurprising given the way preferences shifted as time was
spent with individual instruments (SI Text). What was consistent
through parts 1 and 2 was a preference for new violins and a
speci?c dislike for O1


Again, I refer to the idea of using controls in such tests & also a test sample that has some statistical significance.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Denial is not a river in Africa. Welcome back, John.

Tim

Thanks, Tim - it looks lik ethis place needed a bit of livening up :D
I see you haven't changed :)
Let's restart from where we left off - are you really trying to say I'm in denial based on this first paper? I presume you haven't read the second paper?
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
You should know that there are articles that get published in every journal that raise people's eyebrows. You and I know it's all about who gets the piece for review. They probably had no idea who to send this article to for review.
Agreed, & as I said before let's not lose our critical senses just because a paper appears in a scientific journal - it just becomes a belief system but with "science" (i.e. not good science) substituted for god
 

microstrip

VIP/Donor
May 30, 2010
20,806
4,698
2,790
Portugal
From Wikipedia:

Criticism

Earl Carlyss, a member of the Juilliard String Quartet, was critical of the study, saying it used totally inappropriate methods of evaluating the quality of the instruments. He said that what makes the older violins better is how they sound to an audience in a concert hall and that it is irrelevant whether a violinist prefers a certain violin in a hotel room. He felt the test was as valid as comparing a Ford and a Ferrari in a parking lot.[9]

Samuel Zygmuntowicz, a noted violin maker, however, said the study is "highly credible" and that it "puts cold water on some old myths and should certainly be good news to young musicians who yearn for violins that they will never afford."

Some commenters (including participants in the study) have criticized that the old violins may not have been recently adjusted or may have had older strings, while the new violins were adjusted by the study organizers.


As I am not a violin maker, I will not spend my time reading it. ;)
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Micro, the second study seems to attempt to address these criticisms & I would like to read the full paper but anybody using the first paper as some sort of evidence/proof that blind testing is the great myth-buster is really badly mistaken & rather proves why almost all blind tests return null results - they are not rigorous in dealing with the all the variables, thinking that eliminating sighted bias is enough.
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing