What We Have Here Is A Failure To Replicate, Notes on Bem

Scientific research is difficult to do well, and people are flawed and biased. As Carl Sagan noted, science is not just an ideal abstraction, but is very much a human endeavor, and as such is messy and imperfect. Nature itself is random and quirky and doesn�t always cooperate with our desires to penetrate its secrets.

The power of science as a tool for understanding the world comes largely from the fact that it is self corrective � it doesn�t always get it right the first time, but it has the potential to fix any mistakes and get it right eventually. Part of this self-corrective power comes from the humble replication � scientists repeating the work of other scientists to see if they get the same results. If a certain result represents a real effect in the world, it should generally replicate, no matter who is doing the experiment. If the result represents some methodological error, or was just a chance result, then it will not consistently show up in replication.

This fact has particularly plagued parapsychology research, whose research paradigms have not historically survived replication. Skeptics interpret this fact quite simply � psi effects are not real, they are the product of research error. Psi proponents have fancier explanations, which sound a lot like special pleading: skeptical replicators inhibit psi phenomena, or perhaps there is a �decline effect� inherent to psi phenomena itself. It may be the nature of psi that it is elusive in the laboratory, or repeatedly looking for it makes it go away. These are not convincing arguments.

The latest psi research to fall victim to a failure to replicate is Daryl Bem�s �feeling the future� research. A recent article in PLoSone reports the result of three independently carried out precise replications of Bem�s 9th study. All three report completely negative results. This paper presents a good opportunity to delve deeply into the issues surrounding replication and research error.


The Experiment

In early 2011 Bem published a series of 9 studies in The Journal of Personality and Social Psychology, a prestigious psychology journal. All of the studies followed a similar format, reversing the usually direction of standard psychology experiments to determine if future events can affect past performance.

In the 9th study, for example, subjects were given a list of words in sequence on a computer screen. They were then asked to recall as many of the words as possible. Following that they were given two practice sessions with half of the word chosen by the computer at random. The results were then analyzed to see if practicing the words improved the subject�s recall for those words in the past. Bem found that they did, with the largest effect size of any of the 9 studies.

Needles to say, these results were met with widespread skepticism. There are a number of ways to assess an experiment to determine if the results are reliable. You can examine the methods and the data themselves to see if there are any mistakes. You can also replicate the experiment to see if you get the same results.

Researcher Degrees of Freedom

As we see, there were many problems with the methods and statistical analysis of Bem�s original paper. Bem argues that each problem was small and by itself would not have changed the results. This argument, however, misses a critical point, made very clear in another recent paper.

Simmons et al published a paper demonstrating how easy it is to achieve false positive results by exploiting (consciously or unconsciously) so-called �researcher degrees of freedom.� In the abstract they write:

�In this article, we accomplish two things. First, we show that despite empirical psychologists� nominal endorsement of a low rate of false-positive findings (! .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis.�

In my opinion this is a seminal paper that deserves wide distribution and discussion among skeptics, scientists, and the public. The paper discusses the fact that researchers make many decisions when designing and executing a study, and analyzing and reporting the data. Each individual decision may have only a small effect on the final outcome. Each decision may be made perfectly innocently, and can be reasonably justified.

However, the cumulative effect of these decisions (degrees of freedom) could be to systematically bias the results of a study toward the positive. The power of this effect is potentially huge, and likely results in a significant bias towards positive studies in the published literature.

But even worse, this effect can also be invisible. As the authors point out � each individual decision can seem quite reasonable by itself. The final published paper may not reflect the fact that the researchers, for example, looked at three different statistical methods of analysis before choosing the one that gave the best results.

The authors lay out some fixes for this problem, such as researchers disclosing their methods prior to collecting data (and no peeking). But another check on this kind of bias in research is replication.




The Power of Replication

The degrees of freedom issue is one big reason that replicating studies, especially precise replications, is so important. A precise replication should have no degrees of freedom, because all the choices were already made by the original study. If the effect being researched is real, then the results should still come out positive. If they were the result of exploiting the degrees of freedom, then they should vanish.

There are also the other recognized benefits of replication. The most obvious is that any unrecognized quirky aspects of study execution or researcher biases should average out over multiple replications. For this reason it is critical for replications to be truly independent.

Another often missed reason why replications are important is simply to look at a fresh set of data. It is possible for a researcher, for example, to notice a trend in data that generates a hypothesis. That trend may have been entirely due to random clustering, however. If the data in which the trend was initially observed is used in a study, then the original random clustering can be carried forward, creating the false impression that they hypothesis is confirmed.

Replication involves gathering an entirely new data set, so any prior random patterns would not carry forward. Only if there is a real effect should the new data reflect the same pattern.




Replicating #9

To get back to the recent replication of Bem�s research - Ritchie, Wiseman, and French each replicated, as precisely as possible, Bem�s protocol for his 9th study. They point out that sometimes even trivial deviations from the original protocol are used to dismiss negative replication results. They point out any changes they did make (all minor), such as changing some words to their British equivalents. They used the exact same software for the study, provided by Bem (to his credit, he did encourage and facilitate replications of his research).

In my assessment, after reading their methods, they seem like fair and accurate replications. There results were entirely negative. A separate fourth replication by Eric Robinson was also negative.

The weight of the evidence (even without considering plausibility) is heavily on the side of concluding that �feeling the future� is not a real phenomenon. This has important implications that go beyond this one study and this one topic.

This means that studies that look clean on paper, and that can be reasonably defended from criticism, can still be completely wrong. I cannot explain exactly why Bem�s results were positive when they were probably false positive (given the negative replications), but I can give reasons to be skeptical of the results. Invisible (or sometimes visible) researcher degrees of freedom are enough to generate false positive results.


And especially for Zombie


The Straw Man vs Skepticism

This is exactly why skeptics do not take research results seriously (especially when they appear to demonstrate the impossible) until they have been independently replicated sufficiently to show a consistent pattern of positive results with rigorous methods.

Prominently displayed at the top of the Society for Pyschical Research�s website is this quote:

"I shall not commit the fashionable stupidity of regarding everything I cannot explain as a fraud." - C.G.Jung

Clearly that quote reflects the prevailing attitude among psi researchers of external skepticism of their claims and research. Every skeptic who has voiced their opinion has likely been met with accusations of being dismissive and closed-minded.

But this is a straw man. Skeptics are open to new discoveries, even paradigm-changing revolutionary ideas. Often I am asked specifically � what would it take to make me accept psi claims. I have given a very specific answer. It would take research simultaneously displaying the following characteristics:

1 � Solid methodology (proper blinding, fresh data set, clearly defined end points, etc.)

2 � Statistically significant results

3 � Absolute magnitude of the effect size that is greater than noise level (a sufficient signal to noise ratio)

4 � Consistent results with independent replication.

Most importantly, it would need to display all four of these characteristics simultaneously. Psi research cannot do that, and that is why I remain skeptical. These are the same criteria that I apply to any claim in science.



In addition, I do think that prior probability should play a role � not in accepting or rejecting any claim a priori, but in setting the threshold for the amount and quality of evidence that will be convincing. This is reasonable � it would take more evidence to convince me that someone hit Bigfoot with their car than that they hit a deer with their car. There is a word for someone who accepts the former claim with a low threshold of evidence.

You can convince me that psi phenomena are real, but it would take evidence that is at least as solid as the evidence that implies that such phenomena are probably not possible.

It is also important to recognize that the evidence for psi is so weak and of a nature that it is reasonable to conclude it is not real even without considering plausibility. But it is probably not a coincidence that we consistently see either poor quality or negative research in areas that do have very low plausibility.

Science may be difficult and complex, but we have gained much hard won knowledge and experience in how to apply science rigorously to arrive at reliable conclusions. All skeptics want is to apply that knowledge fairly and consistently to all claims. When we apply rigorous science to psi claims the best conclusion we can come to at this time is that they are very implausible and very probably not true.

Thanks Duck.

Stuff replication,i have good reason to believe that ghosts exist and in some point in time you are going to find out.

Not without good scientific protocol we're not.

*see 200 years of failure in parapsychology* for more.

Post deleted.Not suitable.

This post has been edited by lifesmate on Mar 19 2012, 07:19 PM

try and be more open minded....what makes you think that experiments like this work, even if it has been repeated multiple times with negative results and in regular science it would be considered a fail.

when talking about something like this, think about whether you would accept it if the sceptics said something like that about paranormal evidence-like if someone took a famous ghost photo and loads of sceptics were able to make a fake one just as good, or refused to accept your point because ghosts dont exist.

You can't convince a believer of anything; for their belief is not based on evidence, it's based on a deep seated need to believe.

Carl Sagan

Carl Sagan

And this, IMHO, is where science is missing out, in many cases.

Carl Sagan.

Carl Sagan.

But do you understand what he's saying?

DUCK KNOWS BETTER.. THERE IS NOTHING MUCH HERE OR AFTER HERE BUT HE WILL NOT TAKE ON THE EXPERTS THAT I WILL FREFER TO HIM.what do you say duck? are you up for it?

Experts in what? If its dress making or hair dressing you may have won already.

All this replication talk is brilliant in most respects but I wonder at how long or how many times some experiments were worked on before they got the replication they were looking for?

Isn't this what experiments are about - for changing this or that until we get that replication?

What we need to realise is that we don't have the answers to everything. Not now, not ever.


Saying nothing has been found out in 200 years of paranormal experiments is wrong. There are evidences and research is going on all around the world on a regular basis. People dedicate their lives to the subject. If it takes a certain kind of person and certain requirements to get abundant results, then so be it. We have to realise not everyone will be able to do it. You do need to have a certain amount of openess towards the phenomena, it seems.

However, of course we can't expect everyone to go along with that and it's a good thing we are all different in out thoughts. We need factual people to help us reason and keep our feet firmly on the ground. And we need imaginative people to accept their might be more to life than basic facts, and who will keep exploring against the odds.

So the answer is, keep testing, keep refining the test to make it better and more valid. But, always stay within the parameters of a good scientific experiment and replication by external groups is a corner stone.

I think you're confusing experimentation (trying to get a given outcome from a set of experimental parameters) and peer review (trying to get the same outcome from an independent, but duplicate, set of experimental parameters). The first makes the claim and the latter seeks to corroborate the claim.

In peer review you would not adjust any of the parameters - you would duplicate them entirely. They either produce the same outcome, in which case the claim would have been confirmed, or they don't, in which case it will be rejected by the scientific community.

All this report is saying is that Bem's claims could not be replicated - making them invalid.

There is plenty of evidence - but all of it is extremely poor - either anecdotal or unreplicateable. If we realise that 'not everyone will be able to do it' is not even the standard of scientific rigour - it only asks that it can be replicated by someone independent, it makes no claim for 100% replication by all people trying - then we realise just how unconvincing the past 200 years has been in terms of paranormal enlightenment. That not a single claim has been successfully peer reviewed by independent parties shows how weak these claims are. It is therefore absolutely true to say that nothing 'paranormal' of any merit has been discovered (or even observed) in the last 200 years by the mainstream scientific community.

Science is completely agnostic to all types of claims so doesn't require openness to existence or non-openness to existence. In fact, the default position of any claim is that it does not exist - which goes for all non-paranormal claim too. The results should be self-evident and not subject to human emotion (open-mindedness/closed-mindedness) i.e. if the evidence points to existence proven then, regardless of human bias, the results will speak for themselves - just as long as those results are replicated independently. The belief position of individuals undertaking the replications is wholly irrelevant.

This is how science works and how we have, as a species, been able to exploit scientific enlightenment. Just as it should be.

I still maintain that even believers would be upset if the police started to use ghosts or the afterlife as an answer to a murder.

Some posts have been deleted and/or edited.

Any offensive posts will be removed.Please do not respond to such posts in like but let a mod or admin know and it will be dealt with.

Thanks.
Admin.

I wonder who that could be

No failure to replicate there, that's for sure!