{"items": [{"author": "Julian", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202816046461380", "anchor": "fb-202816046461380", "service": "fb", "text": "Since we all encounter dozens - hundreds, even - of such claims, we should expect one in twenty to be false. I'm impressed by the fact that so many people think 0.05 is a tiny number???", "timestamp": "1321543315"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202817543127897", "anchor": "fb-202817543127897", "service": "fb", "text": "@Julian: except much more than 1 in 20 are false.", "timestamp": "1321543546"}, {"author": "Alex", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202820249794293", "anchor": "fb-202820249794293", "service": "fb", "text": "It seems to be that a reasonable attitude toward scientific studies is the same attitude people like Nate Silver take toward political polls: one poll by itself doesn't tell you too much, and instead we should look at the trendline from many polls to inform our understanding of the political landscape. I think you may be advocating for something similar: don't take a single study at face value, regardless of its claimed significance. Unfortunately, the mainstream media is structurally inclined to rumor-monger by leaping at shocking results the first time they hear about them.", "timestamp": "1321543923"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202821436460841", "anchor": "fb-202821436460841", "service": "fb", "text": "@Alex: if I saw a medical study claiming p<0.001 I would probably take it seriously.", "timestamp": "1321544064"}, {"author": "Alex", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202823963127255", "anchor": "fb-202823963127255", "service": "fb", "text": "I actually almost typed the polling corollary to that. If a poll has a huge sample size and does a lot of other things right, you can probably trust it more than you normally would a single poll.", "timestamp": "1321544439"}, {"author": "Joshua", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202894319786886", "anchor": "fb-202894319786886", "service": "fb", "text": "@Jeff If a new medication is 50% more effective than an older one, a study showing p<0.001 would be prohibitively expensive to run. (I haven't actually figured out the sample size required.) Given our current state of knowledge, p<0.001 is impractical for some fields. I think that efforts to counter the biases (cherry-picking positive results, not publishing negative resuts, etc.) and educating the public what results from a single unreplicated study mean would be more effective than demanding that a single study virtually eliminates chance.", "timestamp": "1321554529"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202905506452434", "anchor": "fb-202905506452434", "service": "fb", "text": "@Joshua: I wasn't trying to claim that single studies should be designed to yield extremely high confidences. I was quibbling with Alex who had said \"don't take a single study at face value, regardless of its claimed significance\".
If you're interested in sample sizes, the main thing that matters in addition to the effectiveness ratio is the absolute effectiveness. So if the old drug cures 1/3 of patients and the new one 1/2 vs 1/30 and 1/20, both of these are \"50% more effective\" but they need very different numbers of people. The first one needs ~800 people for p<0.001 and ~400 for p<0.05. The second one needs ~8000 for p < 0.001 and ~4000 for p<0.05. Very roughly, you need twice as many people to report at 0.001 than 0.05 (because 0.05 squared is 0.0025), and you need ten times as many people for something ten times less effective.
Looking at the numbers, doubling the number of people to go from 0.05 to 0.001 is not actually prohibitive after all. Though I might be doing this wrong; stats person?", "timestamp": "1321555997"}, {"author": "Joshua", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202915179784800", "anchor": "fb-202915179784800", "service": "fb", "text": "@Jeff Got it. I missed the context of your comment. Thanks for the quick info on sample sizes. Of course p only measures the errors from chance. Confounding variables, poor study design, bias, and nonrandom samples can be large causes of errors.", "timestamp": "1321557290"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=202916876451297", "anchor": "fb-202916876451297", "service": "fb", "text": "@Joshua: definitely.", "timestamp": "1321557543"}, {"author": "Todd", "source_link": "https://plus.google.com/112947709146257842066", "anchor": "gp-1321584453861", "service": "gp", "text": "Perhaps I over-adjust for this. I tend to put basically no stock in any single study.", "timestamp": 1321584453}, {"author": "George", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=203135009762817", "anchor": "fb-203135009762817", "service": "fb", "text": "Jeff I am disappoint! Please read http://en.wikipedia.org/wiki/P-value#Misunderstandings
(Another reason to hate frequentist p-values: everyone gets them wrong.)
The p-value is NOT the probability a result is due to chance! When computing the p-value, one ASSUMES the result is \"due to chance\" so there is no way the p-value can be a probability of this.
You have hit one of my pet peeves. >_<", "timestamp": "1321589189"}, {"author": "George", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=203136919762626", "anchor": "fb-203136919762626", "service": "fb", "text": "\"The p-value is not the probability that a replicating experiment would not yield the same conclusion.\" I'm sorry to keep posting the same point, but I couldn't read any farther into your post and when I tried again I was derailed by an inescapable compulsion to post the same correction. I probably have some sort of disease. People seem to naturally want something that frequentist statistics can't give them and so they constantly misinterpret p-values (even people who should know better do this more often than I can stand). As a Bayesian, p-values seem absurd things to compute because they are conditional probabilities conditioned on something I don't believe.", "timestamp": "1321589587"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=203262933083358", "anchor": "fb-203262933083358", "service": "fb", "text": "@George: In cases where our prior probability is 50%, do these claims about p-value become true?", "timestamp": "1321622119"}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1321622343822", "service": "gp", "text": "George points out (on facebook) that I'm misinterpreting p-values here. They're not the inverse probability of replication or the probability that results are due to chance, because they ignore prior probability.", "timestamp": 1321622343}, {"author": "George", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=203609709715347", "anchor": "fb-203609709715347", "service": "fb", "text": "No. Prior of what?", "timestamp": "1321674893"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=203741706368814", "anchor": "fb-203741706368814", "service": "fb", "text": "@George: Say my prior for \"most people are over 5.3ft tall\" is 50%, and I run an experiment where I go around measuring people, finding that most people are indeed over 5.3ft, and coming up with a p-value of 0.03. Is 3% now my best estimate for how likely it is that most people are not actually over 5.3ft tall?", "timestamp": "1321707830"}, {"author": "Frederic", "source_link": "https://plus.google.com/118156077148469167305", "anchor": "gp-1321746356814", "service": "gp", "text": "There are adjustments to make that eliminate your \"interpretation bias\" problem (the simplest one is the Bonferroni correction). You need to account for the number of hypotheses that you're testing when calculating your p-value; the more hypotheses, the more evidence you need to attain a certain p-value. Of coure, unscrupulous or inept experimenters can ignore this, but it's not some sort of intractable problem from the setup end.", "timestamp": 1321746356}, {"author": "George", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=204049319671386", "anchor": "fb-204049319671386", "service": "fb", "text": "No. It is of course possible to concoct a set of hypothesis along with probabilistic assumptions and appropriate numerical values that can create the absurd coincidence you seek, but if your post hinged on the particular misunderstanding of p-values in the few sentences I read, you should revise it. Furthermore, frequentist probability CAN'T assign a probability to hypotheses being true. If you want to use a framework where that is possible, you shouldn't be computing p-values. The frequentist p-value is the probability of the data *assuming* the null hypothesis is true.", "timestamp": "1321757163"}, {"author": "Bronwyn", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=204067803002871", "anchor": "fb-204067803002871", "service": "fb", "text": "Jeff, 3% is the probability that you would have gotten the measurements you got if most people were not over 5.3ft tall (if the null hypothesis were true). You can then reason that you probably observed the null hypothesis being false rather than the rare case of outlying data with a true null). As I think you realize, prior probabilities and p-values don't fit into the same statistical framework .
With regards to the Bayesian/frequentist comments above: I am firmly in the \"lets all get along\" camp of the frequentist/bayesian debate. There's nothing wrong with p-values (or Bayes factors or cross validation or posterior predictive checks...) as long as you understand what each technique does and does not tell you. If you have good scientific reasons to include prior information or want certain types of computational simplicity, Bayesian methods may be appropriate. But that doesn't mean you can't evaluate your model using frequentist techniques if you wish.
With regard to the earlier sample size estimation discussion: If you do wish to calculate required sample size estimates for a particular p-value, you have to take into account the (expected) variance of your population as well as the (expected) difference in the means between your two groups. You also have to consider the power that you want your study to have (the probability you will actually reject the null when you should). The chart in the wikipedia page for this topic seems applicable to the original medical study example though it only works out the numbers for p=0.05 (http://en.wikipedia.org/wiki/Sample_size_determination).
On the original point: I think it's important to understand the factors involved when deciding whether to believe a study. p-values are useful but they aren't the be-all end-all of scientific research, even though they are too often presented that way. Interpreting studies requires more active thought on the part of the reader (and information on the part of the writer) than one number can provide.", "timestamp": "1321760923"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/202795936463391?comment_id=204219372987714", "anchor": "fb-204219372987714", "service": "fb", "text": "@George: \"if your post hinged on the particular misunderstanding of p-values in the few sentences I read, you should revise it.\"
I don't think it did, though I should fix it anyway.", "timestamp": "1321797054"}]}