::  Posts  ::  RSS  ::  ◂◂RSS  ::  Contact

Publication Bias

April 17th, 2012
science  [html]
If I correctly predicted the overall direction of the stock market (or a specific stock) five days in a row, that would be pretty good evidence that I've got my finger on the market's pulse. Getting that right with random guessing would only happen about once every 32 (2 to the 5th) times, so isn't this p=0.031? Wouldn't you let me manage your money, winking at my "past performance is no guarantee of future results" disclaimer?

Let's say I start out with 32 people. I pick a stock. I tell half of them it will go up, the other half it will go down. Tomorrow 16 people will have seen me be right, 16 will have seen me be wrong. I give up on the second half and pick a second stock for day two. Of the 16 who have never seen me be wrong I tell 8 it will go up and 8 it will go down. I repeat this until day six when I have one person who has seen me be right the past five days.

This is an illegal con, but something similar happens in studies. I claim "Foobaritol lowers blood pressure in tall people", but maybe I ran studies on several different drugs, only publishing the one with the strongest results. How do you know I didn't run the numbers on several ways different ways of dividing up my study participants (age, gender, location, race, height, ...) and am able to make my claim about tall people only because I sneakily neglected to make a Bonferroni correction?

What if we required preregistration as a condition for IRB approval?

Comment via: google plus, facebook

Recent posts on blogs I like:

Absolute scale corrupts absolutely

The Internet has gotten too big. Growing up, I, like many computery people of my generation, was an idealist. I believed that better, faster communication would be an unmitigated improvement to society. "World peace through better communication,"…

via apenwarr August 19, 2019

Traces

At naptime Anna listens to recordings of novels recorded by Jeff’s grandmother. It is the main way she will know Winnie, as it is the main way I have ever known Winnie. Some of the recordings are missing parts, and Suzie often fills in the first few sente…

via The whole sky August 18, 2019

One Night Ultimate Werewolf

I really like One Night Ultimate Werewolf (ONUW), and recommend trying it out. The Daybreak expansion is also good (but I’d skip the later games in the series). App For those who’ve played a lot and wish the night took 20 seconds rather than a few minutes…

via The sideways view August 18, 2019

more     (via openring)

More Posts:


  ::  Posts  ::  RSS  ::  ◂◂RSS  ::  Contact