One reason AB or ABX etc ,testing does not work

Posted by: jonni on 28 February 2003

There are some who insist that blind AB ABX testing proves that there are no real differences between many hifi components, and that those who cliam to hear differences are fools.Many blind tests give a 50/50 result the same as pure chance.
But there are many reasons why tests like these dont work.The prime one is that they are testing the wrong thing.What happens is that they are not testing whether there are audible differences between componentes but they are testing YOUR ability to do dicern these differences under test conditions and accuratly predicting what component is which.
I will use the example of vision which help demonstrate a little about how the mind works.
Most of us will have at some time done a spot the difference quizz in a comic or maganzine.
You are presented two very similar pictures with slight differences .When you look at the picture since your eye can see them both all the differences there are technically in your field of vision.But can you disern them and pick them out in an instant?No this requires examination of the picture over time in order for you to work out exactly what you are seeing and for the brain to regonize those differences.
Seeing is not the same as regoniztion.
The advantage you have with these pictures is that they are static.
Now take music.Music is constantly changing, so that the sounds you hear are not static like a picture.In a quick AB test you now have to identify a moving target and look for the differences.This is a skill, a skill that needs to be learnt, and yes hearing does work like seeing in this aspect.It take time to regonise differenes and doing it with music under test conditions is not an easy task.
It can take moths or years to learn the "sound" of your system and recognize and be able to describe how it sounds.The thing is , is that when you spot those differences and they are not plesant the system may now irritate you.This may not have been apparent in a quick AB test.
So it does not matter whether a test is blind or not , time is what is needed in order to disern differences between components and to predict which coponent is which.It is a skill that you learn and you can not just take volenteers off the street and expect them to spot these differences in quick blind tests.
Posted on: 28 February 2003 by Mekon
You don't need objective measures, although you could use a measure of autonomic arousal, such as electrodermal activity if you you really wanted to. I don't know how you would differentiate between negative and positive emotions though, given they measure the same afaik. The important neasure is self reported enjoyment of the experience of listening, IMO.

It's very easy to put together a decent double-blind test, controlling for the variety of confounding factors that seem to spoil most of these tests, and with a high level of validity. I've offered to do it for giggles before, but nobody is really interested.
Posted on: 28 February 2003 by jayd
You've touched on one of the key differences between the visual and the auditory arts - the temporal dimension. It's possible to get an overall impression of a work of art such as a painting or sculpture in an instant. Certainly there are details that won't be conveyed in that instant, but overall... you see the entire work.

A piece of music is inherently different. A symphony takes time to be completely revealed. Our impression of the work as a whole depends in part on how well we remember the entire course of the work.

Audio A/B comparisons compare memories and impressions. The success of the comparison depends less on the true differences in detail between components A and B than on our ability to evaluate, store, recall, and compare those details.

Maybe. Maybe not. Just talkin'.

jay
Posted on: 28 February 2003 by ejl
Mekon,

I for one would like to hear about a nicely operationalized test like you've suggested.

jonni,

quote:
The thing is , is that when you spot those differences and they are not plesant the system may now irritate you.This may not have been apparent in a quick AB test.


This *might* be true; I for one am certainly willing to grant for the sake of argument that there could be subtle differences than only appear over time. But even granting this, the inference to your conclusion that there is something wrong with AB testing in hifi doesn't follow.

The reason for this is that many of us are looking for value for money in buying expensive gear. I personally do NOT consider tiny differences that I only will notice over a period of months to justify the often considerable outlay that upgrades require. If I can't hear the difference within a few minutes in an AB session, it's not a difference I'm willing to pay for. And although it's conceivable that differences that are not discernable in a blind A/B test would somehow significantly influence my enjoyment, it would quite frankly take a fair bit of hard empirical data to persuade me of this (the opposite conclusion that people are claiming to hear subtle differences in order to rationalize/justify their expensive investments currently seems to me at least equiplausible).

So even granting your claim about subtle differences, blind AB tests have a legitimate role in hifi purchases for many, myself (and Nick S, apparently; see his "gutted" thread) included. Indeed, I find it a bit odd that some people seem willing to defend the outlay of thousands of pounds/dollars for "improvements" that they themselves admit are not perceptible except in some almost unconscious wa