{"items": [{"author": "Jay", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812", "anchor": "fb-771960517812", "service": "fb", "text": "The big thing that sticks out to me is there was a massive under response rate, which could easily have created this effect.", "timestamp": "1455944887"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771960587672", "anchor": "fb-771960517812_771960587672", "service": "fb", "text": "→ Under-response, in that the follow-up survey only reached a small fraction of people in the two groups?", "timestamp": "1455944984"}, {"author": "Jay", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771961515812", "anchor": "fb-771960517812_771961515812", "service": "fb", "text": "→ More that only a small fraction of the ones reached responded. For example, someone who is vegetarian might be more likely to respond to a survey asking about eating habits.", "timestamp": "1455945097"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771961870102", "anchor": "fb-771960517812_771961870102", "service": "fb", "text": "→ @Jay: I had initially thought it was important to make sure that the study didn't look like it was about vegetarianism, but they ran a Mechanical Turk pilot study and concluded that this wasn't needed: https://impactresearch.hackpad.com/Do-We-Need-to-Disguise...", "timestamp": "1455945508"}, {"author": "Jay", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771962119602", "anchor": "fb-771960517812_771962119602", "service": "fb", "text": "→ So, that address the question of if people put different answers if they know it's about vegetarianism. That doesn't test if people would be more or less willing to take a survey that was about vegetarianism.
But my actual complaint didn't make sense anyway, because that would bring higher than expected vegetarians in both the control and experimental equally, so it wouldn't make a difference. Objection withdrawn.", "timestamp": "1455945757"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771962289262", "anchor": "fb-771960517812_771962289262", "service": "fb", "text": "→ I think your objection still holds somewhat. Imagine current vegetarians are way more likely to take the followup survey. This would understate the difference between the two groups, since you're pulling in an accidentally homogenized group.", "timestamp": "1455945902"}, {"author": "Jay", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771962578682", "anchor": "fb-771960517812_771962578682", "service": "fb", "text": "→ Not sure I'm interpreting what you're saying correctly, you're saying that if there's a real difference between control and experimental, but vegetarians took the survey at a higher rate then the difference between the groups would be less?
If that's what you're saying, then I don't think that's correct because the difference in percentage points would be the same in both cases, and that's what this test uses.", "timestamp": "1455946255"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771964155522", "anchor": "fb-771960517812_771964155522", "service": "fb", "text": "→ I'm pretty sleepy, so I might not be thinking about this right, but say there are 100k people each in control and experimental, and that 5% in control are vegetarians and 8% in experimental are vegetarians. If you sample 1% randomly, you'll get 1k people in each group, of which 50 are vegetarian in control and 80 are vegetarian in experimental. We correctly conclude that 5% of the control are vegetarian, that 8% of the experimental are vegetarian, and that we increased the rate of vegetarianism by 3%.
Now say instead vegetarians are 10x more likely to respond to the survey, so we sample 1% of non-vegetarians and 10% of vegetarians. Now our control sample is 1450 people, with 500 vegetarians, and our experimental sample is 1750 people, with 800 vegetarians. We incorrectly conclude that 34% of the control sample are vegetarians, that 46% of the experimental sample are vegetarians, and that we increased the rate of vegetarianism by 11%.", "timestamp": "1455947190"}, {"author": "Jay", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771964759312", "anchor": "fb-771960517812_771964759312", "service": "fb", "text": "→ Oh yep, that seems right to me too!", "timestamp": "1455947513"}, {"author": "Jay", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=771964978872", "anchor": "fb-771960517812_771964978872", "service": "fb", "text": "→ One way to test that is if that's happening is if control is significantly higher than we would expect. That's gonna be hard to measure though, since people who use facebook are probably vegetarian at higher than average rates to begin with.", "timestamp": "1455947694"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771960517812&reply_comment_id=772189264402", "anchor": "fb-771960517812_772189264402", "service": "fb", "text": "→ Jay - For what it is worth, there was nothing in the actual invitation to take the survey that indicated it would be about food or vegetarianism, so the drop-off would have to be when they see the actual questions.", "timestamp": "1456096033"}, {"author": "Dan", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792", "anchor": "fb-771961525792", "service": "fb", "text": "I'm confused. Do you have the data for how many vegetarians were in the groups before they saw the video?", "timestamp": "1455945104"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792&reply_comment_id=771961780282", "anchor": "fb-771961525792_771961780282", "service": "fb", "text": "→ People clicking on the ads were randomly assigned to control or experimental. Control people saw something about tropical diseases, experimental people saw a video against eating meat. Several months later they followed up with both groups. Since only the experimental group saw the video, and there aren't any other differences between the groups, and difference in the survey results (like how many vegetarians there are) is due to seeing the video or not.", "timestamp": "1455945302"}, {"author": "Dan", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792&reply_comment_id=771962593652", "anchor": "fb-771961525792_771962593652", "service": "fb", "text": "→ There are literally always initial differences between groups. The way you account for these differences is by measuring the initial values for your groups before the intervention. This is especially important when you're looking at small changes in a group following an intervention.
It is true that the larger your sample size, the smaller these differences SHOULD be, but a 2% could totally have been initially present.
...Unless I'm totally misunderstanding you and there is data saying that the groups were initially identical-ish in vegetarian membership.", "timestamp": "1455946270"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792&reply_comment_id=771963711412", "anchor": "fb-771961525792_771963711412", "service": "fb", "text": "→ @Dan: It sounds like you're saying you would like to have seen each person take the same survey twice, so we could see differences? There are two problems: bringing people in for a survey on the internet doesn't work well, so needing to catch the same fish twice would massively reduce your sample size, and people might respond differently to a survey if they've already seen it earlier.
But not needing to survey people twice is one of the benefits of a randomized controlled trial. These two groups are only different in composition in that some of them got \"heads\" when they hit the server and some got \"tails\". It's not like they went to Chicago to recruit people for the experimental group and LA to recruit people for the control.", "timestamp": "1455946650"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792&reply_comment_id=771963726382", "anchor": "fb-771961525792_771963726382", "service": "fb", "text": "→ Another way to say it is, 2% could totally have been present initially, but given the sample sizes involved that's less than a 10% chance.", "timestamp": "1455946690"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792&reply_comment_id=772189394142", "anchor": "fb-771961525792_772189394142", "service": "fb", "text": "→ Dan - the groups were randomly assigned from the same population of people who clicked on the ad. Why would we expect a difference between the two groups, if not for chance?", "timestamp": "1456096094"}, {"author": "Dan", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961525792&reply_comment_id=772198086722", "anchor": "fb-771961525792_772198086722", "service": "fb", "text": "→ Oops yeah ok. My resident statistician tells me as long as the groups are sufficiently large, the statistical tests will account for the initial differences.
So if the results had been significant, the statistical testing would have accounted for the initial differences.", "timestamp": "1456101784"}, {"author": "Alexander", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092", "anchor": "fb-771961875092", "service": "fb", "text": "I agree that this is a natural way to analyze the data, and that I was also surprised this wasn't the approach taken. But I'm not sure where your p value is coming from - when I run a linear regression on this data in R, I get a point estimate of .01985 with a standard error of .01235, which gets a p value of .108. Also, I initially categorized people who ate fish as vegetarian, and the results were further from significance then.", "timestamp": "1455945510"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092&reply_comment_id=771962124592", "anchor": "fb-771961875092_771962124592", "service": "fb", "text": "→ If you're doing a linear regression I think we're not taking the same approach. My p-value just comes from taking:
control: 864 samples, 55 vegetarian
experimental: 934 samples, 78 vegetarian
and putting that into a calculator like https://vwo.com/ab-split-test-significance-calculator/", "timestamp": "1455945757"}, {"author": "Alexander", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092&reply_comment_id=771962493852", "anchor": "fb-771961875092_771962493852", "service": "fb", "text": "→ A regression should get the right result. Are you sure the calculator you're using is using a two-tailed test of significance?", "timestamp": "1455946157"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092&reply_comment_id=771965188452", "anchor": "fb-771961875092_771965188452", "service": "fb", "text": "→ Alexander: I'm not sure that the calculator uses two-tailed, and your .108 is very close to twice my 0.053 which suggests that the calculator was one-tailed. I agree that two-tailed is the right thing to use here.
Here's one where you can see the math behind it, though I'm not sure enough what's going on to tell if this is one or two tailed: https://docs.google.com/.../1q7x9aGGsM20SlCVKq54q.../edit...", "timestamp": "1455947873"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092&reply_comment_id=771994579552", "anchor": "fb-771961875092_771994579552", "service": "fb", "text": "→ Sorry, you're right. The test I was using must have been one-tailed. I redid it computationally and also got 0.108.
Post updated.", "timestamp": "1455979820"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092&reply_comment_id=772189439052", "anchor": "fb-771961875092_772189439052", "service": "fb", "text": "→ Alexander - this wasn't the approach taken because MFA believes that the main impact of their campaigns is in meat reduction (not just meat elimination) and therefore merely looking at vegetarian vs. non-vegetarian understates any potential effect.
...So it's ironic that Jeff found a stronger effect that way.", "timestamp": "1456096166"}, {"author": "Krystal", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771961875092&reply_comment_id=772474552682", "anchor": "fb-771961875092_772474552682", "service": "fb", "text": "→ Hi Alexander - I posted this comment below but wanted to post again here because it's also relevant to this discussion. I redid Jeff's analysis with a cleaned/trimmed dataset and obtained slightly different results. I put my (long) comment in this Google Doc. https://docs.google.com/.../1BYt2X_rqi.../edit...", "timestamp": "1456259328"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602", "anchor": "fb-771966111602", "service": "fb", "text": "This is a negative result, and the Facebook summary should reflect that. It is not reasonable to say that something had an effect size \"way higher than I would have expected\" when the effect was not significant (p=0.053 with no Bonferri correction) and someone else's methodology found an (also non-significant) effect in the opposite direction.", "timestamp": "1455948907"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=771966670482", "anchor": "fb-771966111602_771966670482", "service": "fb", "text": "→ \"This is a negative result, and the Facebook summary should reflect that.\"
I put the p-value right in the summary.
\"with no Bonferroni correction\"
I only did one test, and I committed to doing only that test in advance, so I don't see why I would do a correction?", "timestamp": "1455949403"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=771968052712", "anchor": "fb-771966111602_771968052712", "service": "fb", "text": "→ Earlier today, someone mentioned the Mercy for Animals study and said that it had found increased meat consumption, an effect in the opposite of the expected direction. I did a double take, because I thought I had read about that study and that it had found an effect in the expected direction, ie, decreasing meat consumption. Investigating now, this summary is where that false belief came from.
Yes, the p-value is there. But I skimmed it quickly, and everything about the summary other than the p-value speaks as though the result were significant. In particular, the sentence \"This is way higher than I would have expected\" is wrong unless you were expecting the ads to decrease vegetarianism.
While you only performed one analysis, the original authors of the study performed another, and the selection between these analyses creates one bit of potential filtering. I'm not sure whether that calls for Bonferri correction, but it's sort of moot since none of the results are significant anyways.", "timestamp": "1455950970"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=771968132552", "anchor": "fb-771966111602_771968132552", "service": "fb", "text": "→ Usually I would just roll my eyes at this sort of thing, since most of the press gets these things consistently wrong. It's just... I really didn't expect you-in-particular to make that mistake. You were meticulous and careful with methodology, right up until the moment when it came time to write the headline, and then you threw away your analysis' conclusion and wrote something else.", "timestamp": "1455951135"}, {"author": "Jacy", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=771970532742", "anchor": "fb-771966111602_771970532742", "service": "fb", "text": "→ ", "timestamp": "1455953861"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=771986176392", "anchor": "fb-771966111602_771986176392", "service": "fb", "text": "→ @Jim: If there had been one more vegetarian in the experimental group then my analysis would have found a p-value of slightly less than 5 instead of slightly more than 5. If that had happened, would you have the opposite view here, and consider the results \"significant\"?
The 5% cutoff is arbitrary, and instead of a binary approach of \"if it's under p=0.05 then update on it, otherwise ignore it\" we should smoothly update more strongly for lower p-values and less strongly for higher ones.", "timestamp": "1455972464"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=772002309062", "anchor": "fb-771966111602_772002309062", "service": "fb", "text": "→ You misunderstand. I am not against reporting results that have p>0.05. I am against this sentence:
> \"This is way higher than I would have expected.\"
The p-value sets an upper bound on how much you could have been surprised in that direction, and the magnitude of the surprise falls unambiguously short of what would warrant using an intensifier in this sentence.", "timestamp": "1455984797"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=772006700262", "anchor": "fb-771966111602_772006700262", "service": "fb", "text": "→ Maybe you two just use qualifiers differently, and there's no substantive disagreement? If I took a draw from a standard normal and got 1.6 (the corresponds to Jeff's original .053), I'd probably say that 1.6 is \"way higher\" than I expected. Less sure I'd say that about 1.2, but maybe.
(Sorry, standard normal isn't a good example - I'd rather use something people have more real-life intuition about.)", "timestamp": "1455988747"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=772032393772", "anchor": "fb-771966111602_772032393772", "service": "fb", "text": "→ An example with more straightforward intuitions: roll 1d20. p~=0.05 is a natural 20, p~=0.1 is a 19+.", "timestamp": "1456001810"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966111602&reply_comment_id=772032668222", "anchor": "fb-771966111602_772032668222", "service": "fb", "text": "→ Yeah, 19 or 20 are way higher than I'd expect.", "timestamp": "1456001968"}, {"author": "Howie", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882", "anchor": "fb-771966969882", "service": "fb", "text": "Isn't your measure of vegetarians likely to be an overestimate? Some people who report not eating meat for those two days likely either forgot or are not vegetarians.
Also see all of Andrew Gelman's work showing that significant but under powered results are often due to chance", "timestamp": "1455949752"}, {"author": "Jacy", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=771967134552", "anchor": "fb-771966969882_771967134552", "service": "fb", "text": "→ ", "timestamp": "1455949839"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772233490772", "anchor": "fb-771966969882_772233490772", "service": "fb", "text": "→ Howie Jacy Do you have a concrete citation from Gelman I can look at? I'd be interested to follow up but I wasn't able to find a paper that showed this via Googling.", "timestamp": "1456127019"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772260985672", "anchor": "fb-771966969882_772260985672", "service": "fb", "text": "→ \"Isn't your measure of vegetarians likely to be an overestimate? Some people who report not eating meat for those two days likely either forgot or are not vegetarians.\"
Yes, you're right, my measure should count basically all vegetarians plus some meat eaters. I think few American meat eaters have eaten no meat in the last two days, though?
\"Also see all of Andrew Gelman's work showing that significant but under powered results are often due to chance.\"
Is that talking about things like publication and selection bias, which I don't think apply here, or something else?
Simple frequentist interpretation of the pvalue says this is 10% likely to be due to chance, but if you had a strong prior that the ads were less convincing then you could think the 2% is extremely likely to be chance.", "timestamp": "1456150052"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772261369902", "anchor": "fb-771966969882_772261369902", "service": "fb", "text": "→ \"Simple frequentist interpretation of the pvalue says this is 10% likely to be due to chance ... \"
Just to nitpick: That's not a frequentist interpretation. (Frequentist interpretation is that it's 10% likely to occur by chance if there's no difference, but says nothing about the probability it did occur by chance.)", "timestamp": "1456150294"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772261629382", "anchor": "fb-771966969882_772261629382", "service": "fb", "text": "→ David Chudzicki Yeah, it's annoying we get P(data | no difference) when we want P(no difference | data)...", "timestamp": "1456150561"}, {"author": "Howie", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772267512592", "anchor": "fb-771966969882_772267512592", "service": "fb", "text": "→ Right - and when power is low there's a very big difference between the two. Here's an example of a low power study where a statistically significant result has only a 76% chance of being the right sign. Also, if power is low, we only get significance when the estimate is larger than the true effect size. http://andrewgelman.com/.../17/power-06-looks-like-get-used/", "timestamp": "1456153986"}, {"author": "Howie", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772267841932", "anchor": "fb-771966969882_772267841932", "service": "fb", "text": "→ Peter I was thinking of this piece by Gelman. http://www.stat.columbia.edu/.../publi.../PPS551642_REV2.pdf
This piece is also relevant and good:
http://www.nicebread.de/whats-the-probability-that-a.../", "timestamp": "1456154220"}, {"author": "Howie", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772268031552", "anchor": "fb-771966969882_772268031552", "service": "fb", "text": "→ Also this:
http://andrewgelman.com/.../p-values-and-statistical.../", "timestamp": "1456154429"}, {"author": "Howie", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966969882&reply_comment_id=772268320972", "anchor": "fb-771966969882_772268320972", "service": "fb", "text": "→ Jeff Kaufman: I'd guess it's true that \"few American meat eaters have eaten no meat in the last two days.\"
However, given that the base rate of vegetarianism is pretty low, I'd guess that if we look at a group of people who have not eaten meat in the last two days, a meaningful number of them are meat eaters (especially given recent trends towards meat reduction). This is even more true once we account for measurement error in the survey. Some of the people you're identifying are meat eaters who forgot about the one time they ate meat over the last two days. This shouldn't cause much bias (because that's ~equally likely to happen in the control and experimental group) but it does make it important to note that you're not actually detecting an effect on vegetarianism.", "timestamp": "1456154726"}, {"author": "Jacy", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966984852", "anchor": "fb-771966984852", "service": "fb", "text": "", "timestamp": "1455949759"}, {"author": "Jacy", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771966984852&reply_comment_id=771967394032", "anchor": "fb-771966984852_771967394032", "service": "fb", "text": "→ ", "timestamp": "1455950372"}, {"author": "Ron", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771967982852", "anchor": "fb-771967982852", "service": "fb", "text": "Control group?", "timestamp": "1455950862"}, {"author": "Nix", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771967982852&reply_comment_id=771968107602", "anchor": "fb-771967982852_771968107602", "service": "fb", "text": "→ Not sure what your asking. There was one. See the post for more details.", "timestamp": "1455951071"}, {"author": "Daniel", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771967982852&reply_comment_id=771968242332", "anchor": "fb-771967982852_771968242332", "service": "fb", "text": "→ Was shown END7\u2019s video about reducing neglected tropical diseases", "timestamp": "1455951277"}, {"author": "Alyssa", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968092632", "anchor": "fb-771968092632", "service": "fb", "text": "Claire Zabel Scott Alexander", "timestamp": "1455951047"}, {"author": "Daniel", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262", "anchor": "fb-771968277262", "service": "fb", "text": "I note that the study was conducted on people who clicked an ad with text \"Why does Ariana Grande leave meat off her plate? Watch the shocking video to see why.\"
That's both control and experimental, but we must be cautious about extrapolating to the general population.", "timestamp": "1455951432"}, {"author": "Jacy", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=771970672462", "anchor": "fb-771968277262_771970672462", "service": "fb", "text": "→ ", "timestamp": "1455954008"}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772034035482", "anchor": "fb-771968277262_772034035482", "service": "fb", "text": "→ If that was really the ad text, it almost certainly biased study participants in favor of vegetarianism. Lots of meat eaters don't give a flying fuck why Ariana Grand, whoever that is, makes what we consider incomprehensibly stupid dietary choices, and consider articles with headlines such as that one to be noting but lies or propaganda or both.", "timestamp": "1456002876"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772040477572", "anchor": "fb-771968277262_772040477572", "service": "fb", "text": "→ @Kiran: Both the experimental and control group saw that ad text when they came in for the first time. For the follow up, though, the two groups saw an ad offering a survey by \"nonprofit research\".", "timestamp": "1456007934"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772040542442", "anchor": "fb-771968277262_772040542442", "service": "fb", "text": "→ (The target audience for the ads was also teenage girls.)", "timestamp": "1456007986"}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772042433652", "anchor": "fb-771968277262_772042433652", "service": "fb", "text": "→ Ah, okay.", "timestamp": "1456009628"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772043371772", "anchor": "fb-771968277262_772043371772", "service": "fb", "text": "→ Uh, Kiran, I get that you disagree, but \"incomprehensibly stupid\"...?", "timestamp": "1456010297"}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772043566382", "anchor": "fb-771968277262_772043566382", "service": "fb", "text": "→ Those words were chosen to indicate how strongly meatatarians would be turned off by such an ad.", "timestamp": "1456010402"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772043775962", "anchor": "fb-771968277262_772043775962", "service": "fb", "text": "→ @Kiran: From the perspective of the people funding these ads, that's not a problem. They're targeting teenage girls with an ad they think the vegetarian-leaning ones will find appealing, and showing them a video they think will convince them to stop eating meat. This study was a randomized controlled trial to figure out what fraction of the people who do see the video end up giving up meat.", "timestamp": "1456010739"}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772053656162", "anchor": "fb-771968277262_772053656162", "service": "fb", "text": "→ Ah, so they were trying to influence teenage girls, not at all a random population. I bet that's the best target audience. I guess I was misled by your use of \"people\" into believing that they were targeting a broad spectrum.", "timestamp": "1456015333"}, {"author": "Linchuan", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772183346262", "anchor": "fb-771968277262_772183346262", "service": "fb", "text": "→ Kiran you seem to be very aggressively agreeing with the study implementers.", "timestamp": "1456091971"}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772183979992", "anchor": "fb-771968277262_772183979992", "service": "fb", "text": "→ Agreeing? I have no opinion on whether targeting teenage girls would actually *work*, but if they think teenage girls are the group most likely to be persuaded to adopt new diets, then we do agree about that.", "timestamp": "1456092345"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771968277262&reply_comment_id=772189598732", "anchor": "fb-771968277262_772189598732", "service": "fb", "text": "→ Since we're testing the ROI on a cost-per-click campaign, we only really care about those people who actually click.", "timestamp": "1456096291"}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1455972684575", "service": "gp", "text": "@Julia\n\u200b The study didn't let us see this. It used cookie based retargeting, which limits the period you can follow up with to his long cookies stick around in practice. So this was looking at about 4 months out, and longer ones address probably not practical without a different follow up method.", "timestamp": 1455972684}, {"author": "Gregory", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=771996999702", "anchor": "fb-771996999702", "service": "fb", "text": "I agree with the approach of limiting tests rather than analyzing everything and correcting for multiple testing. Bonferroni is known to be very conservative, and thus applying it will lose a lot of signal. If (eyeballing Jacy's histograms, assuming I understand them correctly) the serving data is that highly skewed, nonparametric methods should probably be used over T-tests, although in fairness T-tests are (I think) fairly robust to skew.
Interpretation looks challenging. If I read you right, vegetarian fraction is sensitive to non-response. It also seems a bit incongruous with the index analysis, showing (non-significant) increases in meat consumption. Maybe there's polarization, or maybe it is safest to talk about a flat result. Looking at minimum detectable difference might give an envelope on the upper bond on the efficacy of this sort of advertising.", "timestamp": "1455980646"}, {"author": "Brian", "source_link": "https://plus.google.com/114156500057804356924", "anchor": "gp-1455984031765", "service": "gp", "text": "I'd guess that, if this effect is real, it's because there are some people who were already thinking about trying it.", "timestamp": 1455984031}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=772006795072", "anchor": "fb-772006795072", "service": "fb", "text": "I confirm this is the analysis Jeff precommitted to by email. It's also the analysis you'd expect from what he wrote 3 years ago (http://www.jefftk.com/p/vegetarian-survey-proposal).
I'm glad there aren't a bunch of other people popping out of the woodwork claiming he precommitted to a different analysis.", "timestamp": "1455988917"}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=772033142272", "anchor": "fb-772033142272", "service": "fb", "text": "I ate no meat on the two days preceding the day of your post. (Plenty of eggs, cheese, cream, and butter, though.) It amuses me that you would consider this important.", "timestamp": "1456002468"}, {"author": "Jacy", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=772033142272&reply_comment_id=772043406702", "anchor": "fb-772033142272_772043406702", "service": "fb", "text": "→ ", "timestamp": "1456010306"}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1456061501045", "service": "gp", "text": "@Brian\n\u00a0I do think it's likely that this was people who were already thinking about trying it, but because this was a randomized controlled trial we can see that there was still a substantial effect of seeing the anti-meat\n
\nvideo.", "timestamp": 1456061501}, {"author": "Brian", "source_link": "https://plus.google.com/114156500057804356924", "anchor": "gp-1456070400599", "service": "gp", "text": "Yes, I agree. But if the advertising works mostly with fence-sitters, then it implies it would stop working when you run out of fence-sitters. Maybe. Unless there are other effects, like fence sitters who decide to do it influencing their friends.", "timestamp": 1456070400}, {"author": "Marcus", "source_link": "https://plus.google.com/115811589251174483775", "anchor": "gp-1456154758207", "service": "gp", "text": "You probably don't have a large enough sample for this to give a meaningful signal, but I would be curious if there's a geographic trend in who ended up going vegetarian. In particular, I would suspect the video to be less effective in large, liberal cities like Boston, because we're already awash in vegetarian propaganda.\n
\nSomeone in say, Des Moines, might be curious or interested in vegetarianism, but would be less likely to have multiple vegan friends or see advertisements on public transit urging \u00a0them to \"go veg\".", "timestamp": 1456154758}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1456163981870", "service": "gp", "text": "@Marcus\n\u00a0The raw data they released includes IPs which we could geocode, so it would at least be possible to check if the trend is in the direction you're expecting.", "timestamp": 1456163981}, {"author": "Krystal", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=772474163462", "anchor": "fb-772474163462", "service": "fb", "text": "Thanks for this post and analysis Jeff. I wanted to share some additional analysis that I did, but my comment ended up being long so I put it in a Google Doc. Anyone is free to comment. I pose one question that I'd like more feedback on, and that's how we can best interpret the p-values alongside the power of the study. Look forward to comments. https://docs.google.com/.../1BYt2X_rqi.../edit...", "timestamp": "1456259026"}, {"author": "Tom", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=806467560422", "anchor": "fb-806467560422", "service": "fb", "text": "Hi Peter, would you be able to comment on why MFA did not pay for more ads in order to get more responses, and thus higher power? Was it a funding constraint?
The methodology google doc says \"the underlying ad spend we need to make to recruit 6K participants at expected response rates is at the upper threshold of what MFA\u2019s current ad network can handle without loosening it\u2019s targeting (and expected cost-effectiveness)\" - but I don't understand what this means. :)", "timestamp": "1471775975"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/771959135582?comment_id=806467560422&reply_comment_id=806596382262", "anchor": "fb-806467560422_806596382262", "service": "fb", "text": "→ While money was a constraint and we used all the funding we thought we could find with reasonable effort (and did feel funding limited), another concern we had was saturation of FB. We can only run so many ads at any particular time -- in order to run more ads, we would have to either run the study for a longer period of time or relax our demographic targeting. In retrospect, we should have raised more money and run the study for longer, but we didn't know that at the time!
You can see http://effective-altruism.com/.../more_thoughts_and.../ for a lot more detail on the study if you haven't already.", "timestamp": "1471828399"}]}