• Posts
  • RSS
  • ◂◂RSS
  • Contact

  • Publication Bias

    April 17th, 2012
    science  [html]
    If I correctly predicted the overall direction of the stock market (or a specific stock) five days in a row, that would be pretty good evidence that I've got my finger on the market's pulse. Getting that right with random guessing would only happen about once every 32 (2 to the 5th) times, so isn't this p=0.031? Wouldn't you let me manage your money, winking at my "past performance is no guarantee of future results" disclaimer?

    Let's say I start out with 32 people. I pick a stock. I tell half of them it will go up, the other half it will go down. Tomorrow 16 people will have seen me be right, 16 will have seen me be wrong. I give up on the second half and pick a second stock for day two. Of the 16 who have never seen me be wrong I tell 8 it will go up and 8 it will go down. I repeat this until day six when I have one person who has seen me be right the past five days.

    This is an illegal con, but something similar happens in studies. I claim "Foobaritol lowers blood pressure in tall people", but maybe I ran studies on several different drugs, only publishing the one with the strongest results. How do you know I didn't run the numbers on several ways different ways of dividing up my study participants (age, gender, location, race, height, ...) and am able to make my claim about tall people only because I sneakily neglected to make a Bonferroni correction?

    What if we required preregistration as a condition for IRB approval?

    Comment via: google plus, facebook

    Recent posts on blogs I like:

    Who Should Bear the Risk in Infrastructure Projects?

    The answer to the question is the public sector, always. It’s okay to have private-sector involvement in construction, but the risk must be borne by the public sector, or else the private sector will just want more money to compensate for the extra risk. …

    via Pedestrian Observations November 30, 2020

    Fireside Friday, November 27, 2020

    Hey folks! Fireside this week. A bit of a change-up in terms of the coming attractions. I had planned to start “Textiles, How Did They Make It?” next, but I want to do a bit more reading on some of the initial stages of textile production (that is, the pr…

    via A Collection of Unmitigated Pedantry November 27, 2020

    Thoughts you mightn't have thunk about remote meetings

    Welcome to this week's edition of "building a startup in 2020," in which all your meetings are suddenly remote, and you probably weren't prepared for it. I know I wasn't. We started a "fully remote" company back in 2019, but …

    via apenwarr November 23, 2020

    more     (via openring)


  • Posts
  • RSS
  • ◂◂RSS
  • Contact