Clever Hans Redux. Sort of.

Clever Hans Redux. Sort of.

A favorite historical example of how humans can fool themselves and, as a result others, is Clever Hans.

Clever Hans was a horse that could do mathematical calculations. Well, not really. It was his owner that did the calculation and the horse was responding to his owners involuntary cues to get the correct answer. It is a classic example of the observer-expectancy effect

a form of reactivity in which a researcher's cognitive bias causes them to unconsciously influence the participants of an experiment.

The effect is very important to consider when testing implausible/impossible pseudo-medicines, especially those that have a strong hands on component.

Humans are also prone to the Clever Hans effect, although unlike horses we are better able to rationalize away the effect.

There is a form of pseudo-medical diagnoses best demonstrated by a type of applied kinesiology where a patient has their muscle strength tested before and while they hold a vial of a substance to diagnose allergies and deficiency. It is form of pure bunkum often used by chiropractors.

Homeopaths have their own version where a diagnosis is made by exposing a patient to a vial of a homeopathic remedy and measuring their skin conductance. This technique was evaluated in Physiological effects of homeopathic medicines in closed phials - a critical evaluation..

Not surprisingly unblinded evaluations were much better (85%) than blind evaluations (65%); 

The data suggested that under blind conditions the verum and placebo phials were not distinguished as well as they were in open trials.

As is so often the case with pseudo-medicines, when the expected outcome is known in advance, the experimenter gets the results she wants.

But they took it a step farther to see how in the unblinded study the conductivity was altered to give a positive result.

It turns out the tester was causing the result

Since the tester holds the electrode in his hand the difference in pressure must be attributed to the tester. Changes in the tester's muscle tone must be responsible for the results of the blind tests. However, the tester is not aware of the difference in reaction.

This was because

The comparison of pressure and conductivity in the open trials showed that the pressure was applied differently by the tester in the trials with verum phials compared to placebo phials. In the situation that verum was tested the tester gradually increased pressure for 2–4 s before reaching maximum pressure, whereas in testing placebo phials the pressure was almost immediately at maximum value.

Not quite Clever Hans. But close. The issue of subconscious bias altering the results of a trials is usually difficult to tease out.   

In the acupuncture literature there is often a slight difference between sham and real acupuncture. I wonder how much inadvertent information is passed on to patients about which kind of acupuncture they are receiving, biasing the results in favor of 'real' acupuncture.

I also appreciate the understated snark of the conclusion

With respect to the serious errors which are made when conducting a blind test, the question can be raised whether these errors also occur in medical diagnostics practice. Based on the outcome of the blind and open experimental tests, such error is expected to occur. Electro-acupuncture practitioners think that in medical practice there are fewer errors because of an increased tester–patient interaction. As yet these claims are not justified.

​New report recommends that FDA make food health c...
Points of Interest 12/26/2015

Related Posts