studio
contact
rss

Hi-Fi Versus the Armchair Audio Pundits

Bookmark and Share

by Josh Ray on April 17 '06

Sys-Images Guardian Pix Site Furniture 2003 04 02 Badscience128

Skeptic-for-hire Ben Goldacre across the pond at the Guardian sets his myth-busting cross-hairs at high-end audio, aiming to prove our little industry is filled with bad science. Here's his bone:

But the most striking parallel is the widespread notion in the hi-fi community that blinded trials - where you ask listeners to identify a cable without knowing if it's cheap or expensive - are somehow intrinsically flawed.

He then goes on to hammer Stereophile editor John Atkinson for a blind listening test he held between a SS amp and a tube amp with inconclusive results. Atkinson hits back in a Stereophile article where he gives his defense of said test and why Goldacre is a boob:

Sighted listening has its own pitfalls, of course, and no one has said otherwise. But Mr. Goldacre appears to be making the naïve assumption that the mere fact that a test is blind inherently—his word was intrinsically—confers legitimacy on the test and its results. That assumption, I suggest, is "bad science"—even voodoo.

Aktinson gives a good argument, though I'm somewhat surprised he didn't mention (as far as I can tell) his measurements of Stereophile. Aktinson is Mr. Measurements, about as scientific as one can get. Regardless, audio is SUB-FUCKING-JECTIVE! Excuse the vulgarity, but, seriously, we're not testing Cialis here (or are we??), but audio equipment. Like sports cars, yeah, it hits 200mph or 20Hz, but that doesn't mean it's a great car or speaker. A NOS-happy Honda can hit 200mph, big whoop. As we know, a plantain-pulp single driver speaker will never sound remotely close to a tower with 20 diamond drivers, but so what? It's all preference.

Anyway, Goldacre throws down the gauntlet to the hi-fi community. His challenge involves power cords, easily the stupidest part of hi-fi. If I've said it once, I've said it a million times: we should never, ever, ever talk about power cords. Power cords are the hemorrhoidal creams of our industry -- they're needed and they work, but people blanch when we describe our experiences with them. In any case, here's his challenge. Does this make sense? Any takers?

The manufacturer of my expensive power cable has assured me that it will have an impact on the sound of an expensive CD player, or a cheap one, and that it will affect optical outputs as much as normal phono outputs. So I propose, as a pilot study, to place, on badscience.net, two files, clearly labelled (Hah! He spelled "labeled" wrong! Idiott! -ED), one recorded from the CD player powered by the expensive cable, one powered with a normal kettle lead. There will also be a third file, and you can vote on which is which. You can do what you like to identify them: burn it to a CD, listen with headphones, examine the raw data, whatever. But the catch is this: it's a palaver, and I'm not going to bother, if the hi-fi buffs are just going to bleat about how unfair the test is. So this is your chance to participate in the design of an experiment. Post your thoughts on badscience.net. I want everybody to be happy.

Comments

When I moved up into more high-end audio I was contemptuous of ninnys who agonized over things like power cords fer chrissake...much to my chagrin, I have now heard the difference. I try to keep it sane, my PC only cost $180, but try to explain even that relatively paltry sum to a non-audiophile and you will get "the look"...you guys know what I mean.

Posted by: Steve Carlson | April 17, 2006 1:10 PM

When I moved up into more high-end audio I was contemptuous of ninnys who agonized over things like power cords fer chrissake...much to my chagrin, I have now heard the difference, even though I was predisposed NOT to. I try to keep it sane, my PC only cost $180, but try to explain even that relatively paltry sum to a non-audiophile and you will get "the look"...you guys know what I mean.

Posted by: Steve Carlson | April 17, 2006 1:11 PM

Looks like DBTs in the high-end realm will keep being the issue of controversy for a long, long time. I can recall a thread on the Stereophile.com forums about this very subject, which turned to be the most popular... but for all the wrong reasons. I tend to resolve these issues in a rather simple way: If the price of two components being reviewed/compared one to each other is something at least remotely close to a budget range I can afford (which is something closer to Busch Light than to 1970s Dom Perignon), then I pay attention. Otherwise, it would be just putting salt in the wound, and why would I ever want to do that? While it is tempting to go and join the "audiophile cables are BS" crowd, I can attest this is not always so. The Zu Mobius upgrade cable I bought for my Sennheiser HD600s definitely retrieves more detail and sounds more natural than the stock cable that came with them, on an a/b comparison I did not so long ago. But your mileage may vary - you may even think otherwise. After all there is no subsitute to find out on your own which system/cables/accesories sound better to you, milspec measurements be damned.

Posted by: beto | April 17, 2006 4:30 PM

Plus: More Goldacre fun about this subject (and subsequent reader retorts) here and here.

Posted by: beto | April 17, 2006 4:41 PM

Beto, on this A/B test you've made with stock cable for your Sennheiser versus the upgrade one, did you use both brand new? Or just the upgrade cable was new? Because if the stock was somewhat old, corrosion could have affected it. And was the test blind or sighted?

I don't mean to flame things up, I just want to make a point here. Tests, whether DBT or not, can be carried flawy. But that does not make excuses to assert that DBT is rubbish as many frequently do. DBT is a valid tool either for testing Cialis or testing audio equipment. As long as the appropriate procedure is be used, it is an important tool for assessing our imperfect preference system. If the procedures are not appropriate, let us correct them, not abolish the tool.

Posted by: Bruno Z de Moraes | April 17, 2006 7:52 PM

Bruno: From what I remember, I did the a/b testing after giving the Zu a good dose of break-in time (100+ hours), since I wager the stock cable must have been used at least that much. (I bought the HD600s used). I definitely ignore how old the stock cable is. As it is pretty evident, there are a lot of variables that can affect the result of a DBT comparison. Only thing I'm damn sure about is that at the time of comparison the Zu cable's performance was noticeably superior to that of the stock cable so in that respect it was worth my money and put the case to rest. Test was not blind, yet I tried my best to be brutally honest not to get carried away by the more-expensive-equals-better mentality. I should give that test a rerun soon.

Posted by: beto | April 17, 2006 9:07 PM

Cables absolutely make a difference, I don't believe anyone in the scientific community would argue about that. In truth, the majority of cable superiority comes down to shielding. As you know, a huge length of cable becomes presto-antenna. Teflon and Kapton and air dialectics as well as geometries are made to prevent your cables from picking up local talk radio. Additionally, I've done a number of cable demos with true switching. Manley makes the Skipjack for instant comparisons and there are a number of products out there that allow for multiple runs. Recently, I was at an in-store cable demo for an active power cable. Active power cables have a secondary charge running in the sleeve to provide this and that benefit. The cable was stupidly expensive and the differences were not worth the cash, imo, but were definitely there. So, yeah, bombarding anything that carries a signal with electrons will affect the sound in some way or other.

Posted by: Josh Ray [SonicFlare] | April 17, 2006 9:23 PM

I believe "labelled" is a British spelling, since Guardian, the last time I checked is from Britain it would make sense.

Posted by: foma | April 17, 2006 9:39 PM

We can talk for days about the listening experiences, we have for audio equipment. What the differences are and such, but the Stereophile editor is wrong. Just plain wrong. Anytime someone tests anything in a non-test environment, it will affect the results of the test. Why am I saying this? Because the man has the audacity to say that a blind listening test for audio, is different than a blind taste test. He says one of the reasons it’s different is, because audio quality is a mental construct. Hello everything we perceive in life is a mental construct. Numbers themselves are abstract representations of what we call numbers. You cannot hold 1 or 2 in your hand. So when we taste wine, our taste buds interact with the wine (btw not just the wine but what ever you ate in the past hour or so, the saliva in your mouth, and a myriad of other bacteria that exist in your mouth) and your mind puts in a category of Merlot (or Melotess, as the Sterophile author would describe it.) Merlot is something you mind made a category for when it tasted this wine. When you hear audio equipment with your ears you would have a preconceived notion of where to categorize the sound heard (i.e. noise or music). If you believe "then the test itself becomes an interfering variable" then you cannot evaluate anything. Because you will always take into account you are testing something, and therefore you are listening defiantly than normally would. Just as the taster would taste differently than they normally would. This is because while you are tasting, or listening, you are thinking of things. You are using your brain to go beyond the simple experience of just tasting, and hearing, to evaluate the process. When you do that you are able to write down more descriptive words that describe your experience. This is not a flaw of experimentation of something subjective. This is the reality of it. Every time you evaluate something subjective you are looking for things that are not (at hand) or you can grasp corporeally. Instead you are searching your brain for a way to convey to someone else what you are experiencing. This is the nature of Subjective Evaluations. This is why audiophiles have such a hard time out of the gate convincing people there is a real reason to buy $100,000 in audio equipment. Unless the audio evaluator can convince everyone in a tactile way the price difference, people will not come in droves to buy the products. Hearing is just as complicated as seeing, tasting, touching, and smelling. It is not more complex as the Stereophile author would have you believe. Bottom line it does not interfere with the process. That is the process. This is the main problem with most scientists or anyone in an advanced field. They sometimes forget to take a step back, and remember what they are doing. They turn simple things into complex organisms, and forget about the original motivation of discovery.

Posted by: Gambit | April 18, 2006 12:14 PM

Regardless, audio is SUB-FUCKING-JECTIVE! Well, it certainly CAN be. And if audiophiles actually kept it all subjective then there would simply be nothing to argue. But many audiophiles aren't content to remain in the subjective realm. Instead they go on to make OBJECTIVE claims (such as claims of actual audibility) and then become indignant and often downright insulting when their objective claims are questioned and/or challenged. Can't have it both ways. Can't make objective claims and expect them to be sacrosanct as subjective claims are. Objective claims can and should be questioned and challenged and ultimately the onus is on the one making the claim to substantiate it. Objectively. Oh, and by the way, whatever Mr. Goldacre may be, he's not an idiot for spelling "labeled" wrong. "Labelled" is the accepted spelling in the UK where Mr. Goldacre hails from and where his Guardian article was published. What was it someone once said? English and Americans are two peoples separated by a common language? :) se

Posted by: Steve Eddy | April 18, 2006 1:43 PM

Audio is entirely subjective, that's why it's great and why, really, there's absolutely zero consensus on great equipment. In fact, I can't name any two reviewers with similar speakers, amps, etc. It seems like the authorities in this industry all rock different equipment. Interesting, no? As for blind tests, I don't believe I've ever read a blind automotive review. If there is one industry we should model hi-fi after, it's automotive. Not only is a blind driving test ridiculous for the obvious reasons, it's ridiculous for the reasons I talk about above: cars are subjective. Which is better, Porsche or Corvette? There is no "better," only taste. Ferrari or Rolls Royce? It's a silly question, just as silly as comparing Wilson to, say, a horn speaker or planar. Interestingly, Revel came out with their unique blind listening tests to evaluate their speakers. The idea was to get together a bunch of people with golden ears (discern certain frequencies with pinpoint accuracy) to test their speakers. But then most people realized there is no logic in listening tests. So a person can hear 19kHz off axis, what does it mean? Quantifiable results do not equal great sound, just as 0-60 in 4.0 does not make a great car. As for Englishisms, yes, "labelled" is theoretically correct. At the very least, unlike underground audio publications, the Guardian has an editorial staff to check that kind of stuff. Personally, I never pass on a chance to throw sand across the pond. And I believe it was Oscar Wilde who made the language comment. Either him or that Wilde wanna-be, Shaw.

Posted by: Josh Ray [SonicFlare] | April 18, 2006 2:03 PM

Josh, why do you so determinedly state that the hi-fi industry should be modeled after the automotive? Why couldn't it be more like, let's say, the wine industry? I agree with you when you say choosing between a Ferrari and a Rolls-Royce is a matter of taste. But so is choosing between this or that loudspeaker, or this or that wine. DBT does not intend to discredit that. The difference is that the DBT can be used as a valuable tool to select wine, perfume or audio equipment, but certainly not cars. And you're apparently mixing up preference assessment and parameter measurement. Measuring whether how fast a car accelerates or how low a woofer goes has nothing to do with the pleasure one feel by driving a car or listening to a loudspeaker. But DBT can be used to assess what loudspeaker sound would make you feel better, which one being tested is chosen by your preference system without taking the brand effect, the visual appeal, the price tag or other non-primordial factors into account. (By non-primordial I mean audio experiencing, not, for instance, the ever-present pride-of-ownership when acquiring "jewelry".) Two final things. First, if audio is entirely subjective as you say, what kind of tools the designers use to come up with their products? Trial and error? Or scientific method? Second, quoting Audio Critic's editor Peter Aczel, "audio consumers give credence to the most ignorant exudations of gonzo audio journalists and loudmouthed dealers while tending to regard with suspicion and skepticism a superbly accredited and commercially disinterested authority" like professors and members of Audio Engineering Society and other highly recognised boards.

Posted by: Bruno Z de Moraes | April 18, 2006 8:46 PM

Bruno, Wine is a great model. I use autos because they hit the same target market I believe audio should go after: males 18-35. And sports cars have an inherent "cool factor" similar to what I believe audio can have. If audio is to grow, we must embrace the iPodders in all their trendy glory, not staid aficionados. But, hey, that's just my opinion. In any case, the correct analogy relative to wine would be doing a DBT between a pinot noir and, say, a pinot grigio. A reviewer would say, "yup, that's a pinot grigio" and the DBT would be meaningless. As for audio equipment, if the reviewer owns Wilsons and you own Magnepans, then the review is inconclusive. In fact, most every review is inconclusive unless you own identical equipment. So you do a DBT between one massive SS amp and another massive SS amp. Unless you own the exact same speakers, components and cables, then there is no conclusion that means anything. There are simply too many variables to make an audio DBT meaningful. If it were one system versus another system, then that I could see doing a DBT. Otherwise, it's Pinot Noir versus Gris. And that's why about 50% of reviews end with "...deserves an audition." Audio is not like wine where you get home, pop the cork and experience excellent bass and treble. Lord-knows how XYZ component will sound in your vastly different system unless you demo it. That's why most audio reviews have become diary entries about their systems, not analytical comparisons. As for how designers come up with their products, some EQ the suckers as flat as possible, others do it all by ear. One big-time tube amp designer, when asked how the amps sound, said, "how do you want them to sound? I make a few tweaks and it can sound any way you like."

Posted by: Josh Ray [SonicFlare] | April 18, 2006 10:12 PM

you are searching your brain for a way to convey to someone else what you are experiencing. And therein lies another part of this process that will always be controversial and inherently flawed. It has been proven over and over again that most people can probably pick out the last waiter that served them a meal from a line-up. But the same people, when asked to describe the same person, and THEN pick them out of a line-up, fail to do so. As the above quote so eloquently points out, we are all victims of how we relate information to each other. Although our communication is very advanced by any standards, we still cannot effectively put into words what we are hearing, or subjectively thinking. And if that's the case, then blind testing is moot, unless you are doing it solely for yourself. There is no way you should ever read blind test, or any test results for anything and take that on faith, because of what we just discussed. Good posts all. . .

Posted by: B. Riley | April 19, 2006 2:49 PM