Can You Reliably Distinguish Between Electronics Without an ABX Blind Test?

Welwynnick

Distinguished Member
Joined
Mar 16, 2005
Messages
7,445
Reaction score
1,059
Points
1,128
Location
Welwyn, Herts
I started out in HiFi at university when everything was vinyl and stereo. Compact disc was the bogeyman, and there were no mobile phones or internet. I joined an audio society, and we read magazines, listened to each other’s systems, and occasionally organised visits to dealers and manufacturers. I even did a final year project on loudspeakers. But we really loved listening to music, and played records at every opportunity. A good record would transport me out of the room, and I became quite addicted.

Before my first proper job, I worked at a good hifi shop. Obviously I had a great time and learned all sorts of interesting things, but eventually grew up, bought a house, a DVD player, and an expensive AVR. Listening to music slowly and sadly faded into the background. I didn’t realise how much the AVR had poisoned my music listening until I replaced it. What went wrong? It had universally great specs, great reviews and great measurements, but it was rubbish. How do you find that out?

The internet provides all the information you could possibly want. If you want to find a particular opinion about something, you can usually find someone who has written what you want to hear. So if you want to read that amplifier A sounds totally better / totally indistinguishable / tragically worse than amplifier B, then you can find it. It’s one of many obsessive hobbies out there, and you’ll find some convincing arguments.

However, if you want to find independent ABX DBT results for HiFi, good luck. It’s too time-consuming and expensive, and nobody wants the information badly enough, so nobody does it. So how do you find out what sounds best? For a while I did it the hard way, and begged, bought and borrowed all the equipment I could lay my hands on. I did countless auditions and comparisons (and some AVF shoot-outs), and found out what I wanted to know, and told everybody about it. I subsequently became the AVF retained amplifier reviewer for a while.

September and October - Blu-ray player shootout
Procesor Shootout: Arcam AV9, Lexicon MC12, Proceed AVP2, Upgrade Company Onkyo SC885
Power Amp Comparison


Many years of blind-testing with many friends had shown that comparisons become strangely difficult when you can’t see what you’re listening to. Sceptics everywhere insisted that sighted tests counted for nothing and only ABX DBTs were reliable. What I did was level-matched, but it was all sighted, so how could I be sure that I wasn’t deceiving myself without ABX testing (HiFi doesn’t pay the mortgage)?

I think I found the answer by accident, and I’ve had the proof all along.

Ten days ago this week I borrowed an Onkyo SC886 from Mad Mr H, and was able to hear the difference between player and receiver decoding of DTS MA soundtracks. It was at a time when I was getting to grips with the causes and effects of jitter, and how it was affected by the digital audio replay architecture. Because that was changed by the decoding, I reasoned that it must change the sound quality, notwithstanding the contemptible “bits-are-bits” flat-earthers. Many people said outputting compressed audio from the player sounded, whereas others insisted they must be imagining it, because bits were bits. I was dying to hear the difference for myself, so I could prove myself right and feed my intellectual and golden eared ego.

However, that all came crashing down when I could barely hear any difference, and anything I could hear was probably unreliable, impossible to repeat, and really didn’t matter at all. Perhaps it was everyone else that had the golden ears, and not me. It was very embarrassing, but I nonetheless owned up straight away and wrote:

Price list for new Onkyo PR-SC886
I think what I found with amplifier decoding is easiest to summarise first. This is what I was really keen to hear for myself, never having had the opportunity before. Every time I made the comparison, I fancy that LPCM sounded better, but the difference was so small that I doubt I could tell the difference reliably. LPCM seemed to have a bit more sparkle and transparency, which is the opposite of what I was expecting of course. There were much bigger differences elsewhere, and if my player or amp constrained me to using one or the other, it really wouldn't be a factor.
I unwittingly used a musical scene from the Happy Feet blu-ray, which actually had an LPCM soundtrack, so there was no decoding to be done anywhere. Therefore when I thought I switched between native and LPCM output, it was LPCM both ways. That was a very disappointing day. It was a sighted test, and the expectation bias could hardly have been greater, but I still declared no reliable audible difference. The next day I realised my mistake and repeated the test with a Dolby-HD soundtrack. What a difference, amplifier decoding was far better, as I hoped it would be, and I set the record straight. It was a dumb mistake, but I got the result I wanted.

Which doesn't matter in this context.

Here’s the point. In spite of considerable expectation bias, I couldn’t reliably distinguish between two things that I thought were different, but were in fact the same. I can’t speak for other people, but in the context of recent discussions about whether electronics sound audibly different, I think this provides proof that I’m not susceptible to expectation bias in sighted tests. I’ve done that on a few other occasions as well. If two things sound the same, then I say they’re the same.

Logically then, I don’t need ABX DBT’s to confidently distinguish between amplifiers.

How other people can achieve the same thing is another matter entirely


Regards,



Nick
 
Can I reliably distinguish between two amplifiers without an ABX blind test?

Yes, one of them is black and the other one is silver :) Joking aside, I feel more comfortable holding onto the belief that I'm not sure.

I suspect it largely depends on what you mean by 'amplifiers'. I felt quite sure that I could distinguish between my first AV receiver (Yamaha) and my old stereo integrated amplifier (Cyrus) when playing stereo music through the same speakers. But that doesn't really seem to answer the question to me, as there are more factors than just amplification going on.

I feel I can reliably distinguish differences between playing sources though my Townshend Allegri passive 'pre-amp' (autotransformer volume control) versus playing them through the analogue inputs of my Audiolab AV processor, both fed to the same power amp. I appear to share the view of many others, which is that I think if there are differences between amplifiers then the pre-amplification is much more important than the power amplification. But again, there may be somewhat more going on than just 'amplification' in the Audiolab, which I presume has a digital volume control.

I've only had two different pure power amps, and they were more of a sideways move than an upgrade, from a Cyrus to an Audiolab. When I made the change, things did sound subtly different to me in terms of how they drove my speakers, more so on some music (e.g. guitar based rock) than other music (e.g. acoustic and electronic). I suspect that if the difference was real then it was at least partly due to differences in the magnitude of gain and sensitivity. I'm not entirely confident I haven't imagined the difference. It seemed too much hassle to do all the plugging and unplugging for side by side tests (I bought the Audiolab to have a cosmetic match with my processor) and by the time I would have done all the unplugging and plugging I wouldn't have trusted my memory of what I'd just heard anway.

What I can say for sure is that I'm still using a Cyrus mono block to power my centre speaker, and to me it sounds seamless with my Audiolab powering the front left and right.

I really don't want to sound critical, and I really value your contributions here, but I'm struggling to distinguish between the point of this thread and the point of the other one on the same subject.
 
Last edited:
I really don't want to sound critical, and I really value your contributions here, but I'm struggling to distinguish between the point of this thread and the point of the other one on the same subject.
Yes, it's more about hifi electronic equipment in general, rather then about amplifiers specifically. I've changed the title.

My point is the ABX tests aren't the only way to reliably distinguish the differences in sound quality between hifi equipment that measures very well.

What's needed is a proper scientific approach. An ABX DBT is one way, but not the only way. You can also use a control, as I described.

I agree with your comments about amplifiers. I think that power amplifiers have the least impact on the overall system performance (especially with easy to drive speakers). With audio, I think the problems are mostly caused at the small signal stages, and where any D to A conversion is performed.

Nick
 
Last edited:
"notwithstanding the contemptible “bits-are-bits” flat-earthers."..Wow, that hurt!!.
I also try to keep something of an open mind,but it is difficult, because I am constrained by a 50 years indoctrination and training in science and engineering. So I have this unreasonable belief in the power of measurement. If two things sound different, they are different and there is a measurement which on paper show the difference..It may need to be a full graph of power versus frequency, or power versus impedence, or transient function response. This unreasonable belief has keot me nearly sane for decades.
Bits are bits.. the relative positions of the holes on my 1987 CDs, have kept their positions quite well so that they are still readable after 40 years, and all the information recorded then is still as as available now. If played via any of my few working CD players,or even blu ray players the bits are reassembled into words and bytes and get to my various tablets,tvs music centres... I am ready to believe that the bit pattern getting to these devices is identical to that impressed some 4 decades ago.
Then the magic happens. The digital words are converted into audio .. analogue voltage levels sent to amplifiers and headphones. An entire industry gets built on the errors which is supposed to happen at this stage. Two major culprits are currently identified.. jitter and filtering. . There were others... missing codes, digital pre processing, digital volume control reducing resolution ,poor analog amplifier
Now are jitter and filtering real problems?.
Jitter is real, it can and is measured. It is a constant problem with communications systems and there are standard methods..the fisheye diagram illustrating the problem graphically. ..Without correction, digital terrestrial TV would not work. Now the link between a CD player and a DAC is one of the most benign environments, imaginable,, so sorting out Jitter is not a difficult problem..
Filtering ... Well yes. All filters create phase shifts, and artifacts so moving the cutoff frequency away from the audio band is just good sense.
But back to the kernel of your post..
The philosopher who said you cannot step into the same stream twice ,has it right. You cannot hear the same passage of music twice.
The medical world is highly conversant with placebo effects etc and have as good as possible protocols in place. They also know that giving a control group cynide and the others asprin will show the aspirin a good light. (A bit extreme ,i know)
 
Dannneill, it’s a pleasure debating with you. My reference to the bits-are-bits brigade was ten years ago (and you only joined last year). I mentioned it to underline the great importance of being able to hear the difference in system configurations, so that I could prove that I understood what made systems perform, and how that piled the pressure on the expectation bias. According to the objectivists, that should have given me a false positive result, but it didn’t.

I also have a lengthy indoctrination in science and engineering, perhaps without quite the depth or duration of yours. My degree was applied physics and electronics, and I’ve worked in EMC, EW, RADAR, telecoms and satcoms ever since. I definitely sit on the science side of the fence, but I still don’t think that objective measurements are able to predict what we hear. I hear DACs with ruler-flat frequency response that sound bright, and others that sound dull. I hear amplifiers with low distortion that sound muddled, and some with high distortion that sound clean.

To my mind, the correlation between jitter and sound quality is perhaps the best. People often say that you can’t characterise jitter with a single figure (notwithstanding the fact they do that with power and distortion), but in general sources with high jitter sound poor, and those with low jitter do indeed sound good. I have a cheap LG blu-ray player that puts out as good a picture as my dreadnought Denon, but LPCM audio sound quality is dire.

So respectfully, I tend to disagree that objective tests can tell us exactly what we hear. I think it’s about time we were able to do that, but sadly I have little to contribute. For what it’s worth, I think we should stop looking at the way that audio signals degrade as we approach clipping, and concentrate instead on how they degrade as signals approach and sink beneath the noise floor.

If you want to know what something sounds like, you have to listen to it.

Best regards,

Nick
 
Of course 'electronics' sound different
(unless we're talking about power amplifiers that is).

Here's a quote from J Gordon Holt, founder of Stereophile magazine.

As far as the real world is concerned, high end audio lost its credibility during the 1980's, when it flatley refused to submit to the kind of basic honesty controls (double blind testing for example) that had legitimized every other serious scientific endeavor since Pascal. This refusal is a source of endless derisive amusement among rational people and of perpetual embarrasment for me.
 
HiFiChoice used to do blind tests of lots of equipment during the '80s. I remember them well, and treated those little books like the audio bible. It didn't last though, because not enough people wanted good information. I wish it could come back, but it just ain't gonna happen.

Nick
 
We have a sequence of electronics that convert a physical recording (record grooves, CD pits, hard drive sectors, etc.) to sound waves that we hear and our brain then converts what we hear to our perception of the sound.
If I change an electronic component it either causes a change in the output or it doesn't.
If it causes a change then I can either measure the change or I can't.
If I can measure the change then I can either physically detect the change with my ear or I can't.
If I can physically hear it then I can either mentally perceive it or I can't.

My hearing is affected by genetics and age.
My perception is influenced by training (learning to recognise differences) and prejudice (expecting to hear something due to brand / previous information).

An ABX test helps to ensure you are detecting a meaningful difference consistently, in this case using your hearing and training (if any).
A blind test helps to eliminate the prejudice/bias.

Where the change in the output is large and detectable by all then the ABX becomes less necessary.
Where the observer is honest then the blind test can be unnecessary for conscious bias.
Which just leaves us with unconscious bias...

Countering an implicit stereotype is difficult. Again if the observer is honest they could prime themselves by thinking about the two things they are comparing and try and negate their stereotypes (if known). Brand X has a long history for quality... but this might be their first bad product. Brand Y is new and from country Z which has produced poor quality in the past... but this might be their first good product.

Even if the observer can convincingly prime themselves then other unconscious biases can come into play.
The McGurk effect always reminds me how entangled our sensory input can be when forming a perception. If you haven't seen it before try this fun video:

Seeing and knowing what you are listening to can heavily influence your decision even if you are a fundamentally honest person...

So what is needed to distinguish between components?
In my limited experience:
Volume levelling is the first most important thing to do when switching between your test samples. (Metering pink noise or similar)
Recognising large differences can then be done without ABX Blind.
The next most important is a quick way to switch between samples so that you have a recent impression for the first sample when listening to the second. Having a remote (or friend) that can quickly switch between components is important.
Recognising smaller differences is where things get tricky.
I would argue a blind test (even if just AB) becomes essential even if you think you have no prejudice at work.
ABX then becomes essential as differences are on the edge of detectability.

Again in my limited experience speaker differences can give rise to reasonably large differences that many people can easily agree on. In general Record Players are less different. Amplifiers even less so and Digital sources less again. Cables and power supplies... well erm. :)
I treat reviews of the audio performance of Amplifiers and CD players and network streamers with a pinch of salt. Mainly with these things I'm looking for transparency, as well as non-audio qualities such as build quality / features/ ease of use / aesthetic.

In conclusion, my answer to the question posed in the title is 'depends' :)
(on how big the difference is)
I hope that helps ;-)
 

The latest video from AVForums

Is 4K Blu-ray Worth It?
Subscribe to our YouTube channel
Back
Top Bottom