[BlueWizard]

While I don't discredit A/B/X testing, I can't speak to the validity of any A/B/X test unless I am there and know the details.

Someone might be brilliant at understanding scientific method, but terrible at understanding the nature of audio and acoustics. Equally, someone who conducts the test might thoroughly understand the audio part, but not quite get the nature of scientific method.

In drug trials, which we would assume would be cut and dried, there is still a huge psychological aspect that skews the tests. So, psychology can't be dismissed.

With audio, you might as well be trying to do an A/B/X test for love. Just the mere fact that people are involved in an active test causes distortion to their perception.

Next is the element of control. If you are simply march en masse into a room, sat down on folding chairs, and asked 'which sounds better (or different) A, B, or C', you have induced a huge psychological block to getting usable data.

Also, as I mentioned before, if the test is under the control of the people gathering that data, they have induced another psychological factor that blocks reliable data. If you give control to the listener, while still maintaining the blind aspect of the test, and you give them time to make their decision, you are far more likely to get reliable data.

Next is a matter of environment and acoustics. If you haven't, in your scientific method, provided an environment with acoustics that allow for the possibility of a difference to be heard, then the data means nothing.

So, if such a test were devised, it would have to be devised in a way that enhanced the likelihood of getting worthwhile data. It would have to be devised by people with a thorough understanding of audio and acoustics, as well as a working understanding of the psychology of audio and of general psychology, and a thorough understanding of scientific method. And that all these areas are applied in the construction of the test.

Next, as I've already mentioned, are the test subjects. With other randomized blind tests you do indeed want a truly random selection of the population. But is that really what we want here? I don't think so, IF you intend to get reasonable data reasonably fast.

I think this is especially true if this is a 'which sounds better' test, rather than a 'can you hear a difference' test. But even in a can you hear a difference test, it hinges on the ability of the listener to be sufficiently discriminating to hear a difference when one exists.

Keep in mind to the unconditioned non-discriminating ear, bad stereo sounds good, and conversely, good stereo sounds bad.

Would it be consistent with scientific method to pre-screen listeners to make sure they at least had some basic ability to discern small differences in music sounds? Would that enhance the data or compromise it? I think students of scientific method would have differing opinion on which was more valid.

Further, is there value in not only pre-screening, but also pre-conditioning listeners? Perhaps 5 or 10 minutes in a dimly lite room listening to white noise might enhance their ability to discriminate.

Another aspect is the ancillary equipment. If we are trying to determine if two amps sound different, then we must have speakers that are capable of demonstrating that difference.

The reason I'm skeptical of A/B/X tests that claim you can't hear a difference, is because I have no information verifying that the data was gathered in a way that made the data relevant and useful.

The illusion of scientific method is not scientific method.

Also, keep in mind that I'm convinced that all audio testing is flawed, just flawed in different ways. All provide some useful information, but not universally and completely useful information.

But then, that's just my opinion.

Steve/bluewizard