View Single Post
  #16   Report Post  
Steven Sullivan
 
Posts: n/a
Default Comments about Blind Testing

S888Wheel wrote:
Which does not make the conclusions drawn any more tenable...all it means
is that both reviewer and consumer tend to evaluate the sound of a
component in a manner that is unlikely to be particularly
reliable or accurate. But it does make them feel good.


Which is the bottom line in any hobby.



In butterfly collecting, one wants to know the
*actual* facts of the creature's taxonomy, life cycle,
geographical distribution, etc....not a collection of
comforting communal beliefs. One may *want* to believe
that a Luna moth is a rare species, when one has one in
a collection -- they are quite pretty, after all --
or that it's life cycle is peculiar, but in fact it's not
and it isn't. Lepidopterists accept these facts with
no complaint or cavilling about the small-mindedness
of entomologists. They don't indulge in special pleading
that 'maybe science just doesn't know' or 'science
can't measure everything'.

In most hobbies, claims of difference are based on
incontrovertibly visual evidence.


Indeed. But these 'subjective reviews' you talk about do *not* confine
themselves purely to statemenst such as, "I liked this one", "I didn't like
that one"; "This one made me feel good", or the like. They instead often
*do* involve comparisons (often to the reviewer's 'reference system')
and they *do* make *specific claims* about one 'throwing a better soundstage'

or 'sounding less harsh' or 'clarifying the bass'. Under thy typical
review conditions there is a distinct possibility that these
impressions are NOT the result of
actual sonic differences, but are *entirely* psychological in origin -- a
likelihood that science tells us cannot be confidently dismissed without more
controlled comparison.


All true and all a possible reaction that any given consumer may have to such a
new piece of equipment. Just because the reviewer is more detailed in
describing their impressions of a given piece of equipment doesn't make their
impression any less real or any less like the sort of impressions that any
consumer may also have.


And it doesn't make these 'impressions' any closer to *true*. In the
case where in fact there is *NO DIFFERENCE*, isn't it absurd to talk of the
impressions being 'real'? They are only 'real' in the trivial sense that
a false statement remains 'really' a statement.

Your line of reasoning ends up having to assert that the audiophile hobby
isn't particularly concerned with what's true.

Reviewers pretend this isn't the case.


The disclaimer says enough IMO. It is clear that the editor of Stereophile does
not hold the subjective opinions of the reviewers as definitive. I haven't seen
any reviewer specifically claim that thier subjective impressions are
universally infalable.


What disclaimer? Have you seen any reviewer claim that their impressions
could in fact be entirely spurious? Have you seen any admit that,
absent controls for bias, level matching, etc, the likelihood that impressions
about cable or amp or CD 'sound' are *entirely* spurious, cannot be dismissed?

Yes, the community of belief is a strong component of the hobby. But is the
belief
warranted by the facts? At what point does the hobby become more
'faith based' than fact-based?


At the point where people draw their conclusions based on what they are told
rather than what they experience. You cannot blame a magazine that tells it's
readers to not use subjective reviews as definitive claims of performance and
to audition equipment for themselves before purchasing equipment for some
audiophiles acting on faith.


But 'audition for yourselves' is not a sufficiently detailed piece of
advice, Scott, as regards *verifying audible difference*,
any more than 'trust your ears' is. It's little more
than slogan or mantra, unless accompanied by information about factors
that can produce false impressions during those auditions.

Do these disclaimers -- where are they printed , btw, in the current issues
of the usual audio journals? -- further recommend that the personal auditions
be
carried out in a scientifically reliable manner? Repeating the error of the
reviewer yourself isn't going to get you closer to reliable.


They do not suggest how people audition equipment for themslves. Just that they
do so.


Exactly. Which makes these disclaimers no more useful as truth-finding
advice about the 'sound' , than the reviews themselves. The same
essential flaws in the methodology apply.

I believe these disclaimers can be found in the introduction to every
recomended components list. It is your opinion that a reviewer or a consumer is
in error if they do not use scientifically valid methods for testing thier
subjective impressions. But then *all* of the advocates of DBTs on RAO clearly
chose the one component that they believe matters most in the same unreliable
manner. The bottom line how a hobbyist feels about his or her participation in
the hobby.


For the hundredth time...

People choose components for lots of reasons, not all of which involve the
sound: looks, budget, features. Visual differences are easy to verify.
Sonic ones often aren't. But journals don't confine themselves only
to reviewing the looks and features , do they?

DBT advocates differe from others only in their willingnes to
*admit* that without DBT, their impression of the 'sound' of
cables and amps stands a strong chance of being entirely wrong.
Or else they differ in their willingness to actually perform such
comparisons.

It's also rather schizophrenic of journals to display test measurement
results, but then perform their reviews sighted. Why make a nod
to some science, but ignore other science? Clearly the journals want
to present *some* sort of semblance of technical accuracy.



I'll never understand why you think telling me that *lots* of journals
have poor standards of objective proof for their claims, is any sort of
argument *for* sighted comparison.


I agree that it is unlikely you will ever understand. I suggest you give
careful thought as to why skeptics spend their time on bigfoot UFOs and
holistic medicine and not hobbies like audio and photography.


The answer is that far more of the public at large cares about Bigfoot
and UFOs and holistic medicine, than about the folklore of audiophiles.
And what controversies in photography are analogous to 'cable sound' or
'green pens', Scott? Are there treatments or devices that claim to
induce *visible* differences in photographic results, that require
a blind comparison for verification?


What sorts of products are you thinking of?


Toys for grownups.


And because they're 'toys', it's acceptable for the hobby to be pervaded by
child-like, magical thinking about how those toys work? How odd.
Usually children grow out of such beliefs about their toys.






--

-S.

"They've got God on their side. All we've got is science and reason."
-- Dawn Hulsey, Talent Director