Report from TAM 7 (Part 5)

Closed eye psychic

Yes, I know TAM 7 was a month and a half ago. I told you I was busy. Finally, here is my last report. This is what fun I had on Sunday, the last official day of the conference. (You can see the earlier reports here: Part 1, Part 2, Part 3, and Part 4)

From 9 until noon were the paper presentations. Some of them were pretty good. But that was a long time ago, and I don’t remember much about them.

I managed to corner James Randi at one point that morning and asked him a question that has been bugging me for a while. I mentioned to him that there were two types of psychics: Those who know they are frauds and those who don’t.

Randi replied that he calls them “open eye” and “closed eye” psychics (I think that’s the term he used.).

I’ve noticed that a lot of the so-called psychics at the new age fairs and similar places seem to genuinely believe that they have paranormal abilities. What I wanted Randi’s opinion on was whether any of the big-name faux-psychics thought they were real. Randi said that he was convinced that every one of the famous frauds knew they were frauds. He said they can’t get that successful without active deception and manipulation.

That goes along with my perceptions of the business. Every time I see one of these charlatans on TV, it’s clear to me that they’re using all of the conventional tricks.

The self-deluded psychics I’ve seen locally often bumble around with their predictions. Their marks fall for the scam, because they want to believe it. Confirmation bias and selective memory make most people who are stupid enough to part with $10 for a reading convince themselves that it was money well spent.

However, that accidental success through ineptitude doesn’t stand up to scrutiny. If you’re on TV, you need to have a higher success rate. Anybody who has reached that level of financial success in that business almost certainly is actively manipulating the marks by way of cold and (probably) warm reading techniques, various magic tricks, and other methods of deception. It also helps when they have full control over how the show is edited.

I Need a Psychic, Stat!

After lunch, self-proclaimed dowser Connie Sonne was reaching for the brass ring, and we got to watch. She was going to be tested for the JREF Million Dollar Challenge. This was just a preliminary test, but if she passed it, she could then go on and take the final test and earn the million.

Like any scientific endeavor, testing psychics relies on the arcane art of statistics to measure the likelihood that what you see reflects reality.

The JREF has tested hundreds of alleged psychics. They have a million dollars on the line. If they aren’t careful, the odds are that somebody would pass the test by dumb luck and claim the million dollars! That’s the nature of statistics. You can never be 100% sure. There’s always a chance you’re wrong. When setting up a statistical test, you need to determine your tolerance for getting a wrong answer. A false positive in this situation is an expensive error, so the test has to be designed to make it very difficult to pass by luck alone. We’ll come back to this matter in a minute. I’m not convinced the test was set up properly.

The Million Dollar Challenge

The JREF works with the alleged psychic to come up with a test that both sides consider fair. In this case, Connie Sonne claimed to be a dowser. The test that she agreed to was to use her pendulum to find a randomly-selected card.

Prior to the test, they had somebody take a deck of cards and sort them by suit. The jacks through kings were discarded. For each suit, they took the ten remaining cards (ace through ten) and placed each card in a separate envelope and sealed it. Then they shuffled the ten envelopes and stuffed them into a larger envelope.

On the day of the test, with 600 of us watching in the audience, Connie sat down on one side of a table and Banachek sat on the other side. He took the ten envelopes from one of the suits and spread them in front of Connie in a horseshoe pattern. He then rolled a ten-sided die and announced the result. That would be the card that Connie had to find. Connie would then dangle her pendulum for a few minutes and finally announce which envelope contained the card. Banachek marked her guess on the card and set it aside. They repeated this process twice more.

The JREF had determined statistically (remember, the threshold has to be high to rule out dumb luck) that Connie would need to get all three cards correct in order to pass this challenge and move on to the next (final?) round. Connie agreed to this in advance.

So one by one, Banachek opened the envelopes and showed us which card was inside.

Of Connie’s three guesses, she got none correct.

Connie’s Reaction

She was initially gracious in her defeat. Now she’s claiming it was rigged. Sorry, Connie, nobody cheated.

The problem with people who have convinced themselves that they’re psychic is that there is nothing you can do to convince them otherwise. It’s like any firmly-held irrational belief. Evidence and logic never enter the picture.

Was the Test Set Up Properly?

The odds of finding the right card three times in a row purely by chance are 1000:1. That is reasonable insurance against giving away the money by accident.

But if Connie truly were psychic, was this the right test? I’m not sure it was. I wasn’t in on the design of the test, so I don’t know if my concerns are justified. I’ll just mention them here for the record.

I am confident that this test fairly showed that Connie has no strong paranormal ability. If she had, she would have been able to get at least one of the three guesses correct. My concern is that if she has a weak paranormal ability, this test could not have detected it. More broadly, if the JREF uses this approach frequently, they run the risk of missing somebody who does have paranormal ability (assuming such a thing exists).

The primary responsibility for coming up with a good test rests with the applicant. The applicant has to clearly state, in quantifiable terms, what her paranormal ability is. She just can’t say “I’m psychic” or even “I read minds”. What, precisely, can you read? How often are you correct?

Let’s say that I claim to have the psychic ability to tell you the suit of any card chosen at random. If psychic powers really exist, they’re probably weak. It’s unreasonable to expect me to get ten in a row correct. It may even be unreasonable to expect me to get three in a row correct.

Let’s pretend that I really am psychic, but it’s a weak force. It’s just strong enough that I can guess the suit of the card one third of the time.

First of all, it would be my responsibility to know what my approximate success rate is and communicate that to the JREF when I make my application. Assuming I did this, I’m hoping the JREF would design the test accordingly.

What would be a good test in this case? It would obviously require more than three or even ten guesses. If you did 100 cards, that probably still wouldn’t be enough, because the difference between random chance (25) and the psychic power (33) is only 8, which could easily be swallowed up by normal random variation. This is where the statistics come in. There are formulas for determining how many trials you would need to run in order to be confident that the effect you’ve detected is probably real. I used to know how to calculate this sort of thing. I suspect you’d have to run somewhere between 500 and 1000 cards to be sure.

The test I just described is a far cry from what Connie did. I’m hoping that Connie told them that she is 100% accurate, because the test they ran is only appropriate for such a claim.

Connie Sonne

Connie Sonne taking the Million Dollar Challenge
(Image from Scott Hurst’s Flickr stream.)

10 Responses to “Report from TAM 7 (Part 5)”

  1. Alcari Says:

    Well, that’s exactly why both the JREF and the “psychic” set up the tests together.

    If she didn’t agree to the test, there was ample opportunity to say so. But, she did agree.

  2. Brian Says:

    I’m just jealous that you got to meet James Randi. His book, “Faith Healers”, was one of the first books I read years ago that helped me to realize I’m an atheist. May he continue to vex frauds and charlatans for many more years.

  3. thoughtcounts Z Says:

    I’ve never heard of a psychic or anyone else claiming paranormal abilities saying, “I can do it X% of the time.” Though it would be refreshing to hear! Figuring out such a percentage would require assessing success/failure after each reading (or whatever), rather than just asserting correctness, and would probably also require that the recipient of the reading (or whatever) not be so eager to believe anything.

  4. YogaforCynics Says:

    I love it when someone asks my sign. I’ll say “I’m not into that astrology stuff.” They’ll say “come on, just tell me your sign.” I’ll say “scorpio.” They’ll say “obviously!” Then I’ll say “actually, I’m a Libra.” They’ll say “okay, I knew you were either a Scorpio or a Libra.” “Actually, I’m a Sagittarius.” “Okay, that explains it…” “Really I’m a Leo…” And so on. No matter how many times I fool them, they still keep acting like my sign proves something…

  5. RBH Says:

    What you’re suggesting is that the particular test set for Sonne had insufficient statistical power to detect a weak psi ability. As you no doubt know, there are two sorts of errors one can make in testing a hypothesis. One can incorrectly reject the null hypothesis (no psi) and accept the alternative (psi works), or one can incorrectly fail to reject the null (no psi) and fail to accept the alternative (psi works). The latter is the problem you’re suggesting and is associated with the “power” of a statistical test to correctly reject the ‘no psi’ null.

    A difficulty is that the more statistically powerful a test/experiment, the more vulnerable it is to biases and artifacts. Suppose, for example, that there is a tiny bias in the randomization routine that determines to order of cards in a card guessing experiment. If one runs a large enough number of trials that bias could produce statistically significant results in the total absence of psi abilities.

    Excessive statistical power also amplifies the likelihood that poor experimental design will lead to falsely accepting an alternative hypothesis. One of the errors in a good deal of psi research couples massive numbers of runs with an optional stopping rule such that the experimenter stops the run when statistically significant results occur. But, of course, by the nature of the stats, one is guaranteed to find statistically significant outcomes in a purely random process of unconstrained length. So (among other things) the number of trials must be fixed in advance.

    Decades ago I had an exchange of papers in the Journal of the American Society for Psychical Research in which I pointed out that sort of error in Dean Radin’s “research” on humans sensing the contents of computer memories. Psi researchers are still making the same errors.

  6. Another Steve Says:

    A million bucks and odds of 1000:1? Those are better odds than you will get from any state lottery. I hereby claim to have psychotic powers. Someone test me!

  7. Ron Britton Says:

    RBH:

    Thanks for the great comment. In the first draft of this article, I had gone into great depth explaining the Type I and Type II errors. I chopped all that out, because it was boring even me.

    I didn’t really think about the small biases showing up in bigger tests, but you’re right. If your test is sensitive enough to pick up small real effects, it will also pick up small biases and skew your results. The test the JREF ran on Connie lacks that sort of power and vulnerability. If the die they used to pick the number were biased or if their technique for shuffling the cards weren’t truly random, those effects are unlikely to skew their results, because they only have an N of 3.

    You’re right about needing to define your statistical tests in advance. It’s so easy on the computer to take your data and keep looking at it in different ways until you see something. “Aha! I knew there was an effect in there somewhere!”

  8. Parrotlover77 Says:

    Another Steve – I was thinking the same thing. Most lotteries are on the order of 10s to 100s of MILLIONS to 1. 1000:1 lottos (pick-3 type) usually only net you like a hundred bucks, if you are lucky.

    On a related note, we did a Fantasy Football draft order pick out of the hat yesterday for our work leagues. The two league commissioners both picked themselves out of the hat last, which in our leagues means you draft first. In addition, the brother of one of the league commissioners was picked second to last. So, essentially we have what can appear as some sort of cheating or collusion, but I was a witness and helped with the pick, so I know it wasn’t.

    Coincidences can and do happen all the time! Bring on the psychic challenge! ;-)

  9. melior Says:

    Naaah, if Connie had merely “weak” card-guessing skills, say one-in-three, we would know it by her massive collection of World Series of Poker Bracelets jangling and bangling on her wrists.

  10. El Guapo Says:

    Actually, getting one of the three correct is not impressive. The probability of getting at least one correct in three tries (assuming that the cards were shuffled and replaced after each try) is 27%.

    Me, I’m a one out of three quality psychic. Want to offer say, $100?