{"items": [{"author": "Rebecca", "source_link": "https://plus.google.com/108076092316272654620", "anchor": "gp-1367931730489", "service": "gp", "text": "Based on my own internet behavior, I would guess your rates for people taking the survey would be very very low. An ad asking me to take a survey in exchange for a chance to win $25 sounds like textbook spam to me and I wouldn't click on it. With a smaller prize, it is not worth clicking on the ad and with a bigger prize, it just sounds that much more like spam.\u00a0\n
\n
\nMaybe you could solve that problem by working backwards - partnering with an established market research group that works through Facebook, if such a thing exists. Use their cookies to show ads to their members, and then have them put the questions you want into a survey that those people take later. Disclaimer: I know nothing about the technical feasibility of any of this.\u00a0", "timestamp": 1367931730}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1367932118228", "service": "gp", "text": "@Becky\n\u00a0\"An ad asking me to take a survey in exchange for a chance to win $25 sounds like textbook spam to me and I wouldn't click on it.\"\n
\n
\nBut you probably also wouldn't click on the initial ad on facebook, right? \u00a0These are ads aimed at people who do click on ads, so the response rate might not be that bad.\n
\n
\nI'm guessing we get 10% of the people who clicked on one ad to click on another, but maybe that's still too high?", "timestamp": 1367932118}, {"author": "Marcus", "source_link": "https://plus.google.com/115811589251174483775", "anchor": "gp-1367933571737", "service": "gp", "text": "Personally I'm more likely to take surveys that appeal to a sense of obligation or helping out, though I also never click on FB ads, so I'm not really the target audience.", "timestamp": 1367933571}, {"author": "Marcus", "source_link": "https://plus.google.com/115811589251174483775", "anchor": "gp-1367934551501", "service": "gp", "text": "There's research that suggest giving people monetary incentives undermines altruistic incentives.\u00a0 ie.\u00a0 If someone is convinced that something repugnant is the right thing to do, then you offer them money to do it, they become more reluctant.\n
\n
\nSee:\n
\nhttp://www.nytimes.com/2007/07/02/opinion/02schwartz.html\n
\nfor example.", "timestamp": 1367934551}, {"author": "Chris", "source_link": "https://plus.google.com/112938759017605010116", "anchor": "gp-1367934994365", "service": "gp", "text": "Thanks for investigating this!\n
\n
\nThis is probably just a personal story rather than actionable data, but in my experience the path that people follow to become vegetarian can be very indirect and drawn out. \u00a0I read Peter Singer's essays and found them logically sound and persuasive a few years before I decided to go ahead and become vegetarian -- even then, it took me about a year to transition into being a consistent vegetarian, and then two years after that before becoming vegan (which I'd been ethically convinced to become from the start). \u00a0It can take time to learn how to make large diet and life changes, especially if you're an incompetent cook.\n
\n
\nBut if you asked me what the causal factor was in my becoming vegan, I'm pretty sure it was the Peter Singer essays that I read five years beforehand. \u00a0I think becoming good friends with a few vegetarians was probably the tipping point that made me feel that I could do it.\n
\n
\nI'm not going to try and make a point like \"even if the people report that they still eat (some?) meat a few months later, that doesn't mean that the ad failed\"; I do think measuring the direct effect soon after exposure is still useful. \u00a0But it might be good to try to find out more about how veg*ism happens, in order to know what and when to measure: \u00a0is it a slow reduction in eating animal products for most people, or an immediate feeling of disgust and rejection of them?", "timestamp": 1367934994}, {"author": "Ben", "source_link": "https://plus.google.com/105971154852908006603", "anchor": "gp-1367938380253", "service": "gp", "text": "Investigating this seems like a really good idea and I hope you go through with it. I'd be pretty likely to contribute to a kickstarter, if that happens.\n
\n
\nI wonder if the Givewell Labs people would be willing to advise you about survey design.", "timestamp": 1367938380}, {"author": "David Chudzicki", "source_link": "https://plus.google.com/106120852580068301475", "anchor": "gp-1367965945780", "service": "gp", "text": "I'd contribute (either informally or through a Kickstarter), b/c I think the knowledge will be useful for the animal rights movement (and also because I'm interested).", "timestamp": 1367965945}, {"author": "Simon", "source_link": "https://plus.google.com/116669225133133799072", "anchor": "gp-1368176199807", "service": "gp", "text": "Hi Jeff,\n
\n
\nFirst, your proposal sounds very smart.\n
\n
\nAlso, just so I understand the study setup. Half of ad clickers will see the video, half will see something else (unrelated?), and the allocation will be made as randomly as possible. Thanks to cookies, we can later target all clickers with ads for the follow-up survey, and we will know who were in the intervention and who were in the control group. Correct?\n
\n
\nBelow are some thoughts, nothing new to you probably, but just to cover some aspects.\n
\n
\n1.\tIf there are one thousand clicks on survey ads, do we have a sense of what % will complete the survey?\n
\n
\n2.\tWhen to do follow-ups: Ideally, we would make several spread out in time over several years. Say 1 month, 1 year, 3 years. Does that work technically? E.g., how long do the cookies last, will too many participants get new computers so we lose track of them?\n
\n
\n3.\tSurvey questions: Since the survey won\u2019t be connected to the video, can\u2019t it be a diet survey without posing leading questions? E.g. asking (illustration suggestion): do you eat a) potato, b) rice, c) chicken, d) egg, e) fish? When deciding what animal foods to ask about, it makes sense to focus on those with most impact on suffering, so afaik that means chicken, egg, and fish. If I could only ask about one, I\u2019d probably ask about \u2018chicken\u2019 rather than \u2018meat\u2019.\n
\n
\n4.\tSpeculating: It would be great to get data on what people choose to consume rather than what they say they consume. Say instead of asking them \u201cdo you eat,\u201d we offer \u201cchicken\u201d vs \u201cveggie burger\u201d (oversimplified example) and record the choice. I don\u2019t have any good ideas for how to do that but if anyone else does, please let us know.\n
\n
\n5.\tThe cookie will be tied to the computer, right? So someone might use the computer to watch the video and someone else to answer the survey. Maybe this is not a substantial problem since most in the target group use their own computers, but it needs to be considered. E.g. can one see whether a public computer was used to view the video and filter those replies out?\n
\n
\n6.\tIf people need to allow/enable the cookies, it should lead to another step of participant loss, and we would need to account for that in the choice of number of initial ad clicks we need.\n
\n
\n7.\tIs it considered ethical to do this kind of research? I mean targeting and surveying people without them ever knowing what it\u2019s about. Doesn\u2019t seem like a problem to me but something to consider.\n
\n
\n8.\tFyi, I\u2019m also interested in funding and possibly raising funds among friends for this kind of research. Say $3-20k, depending on when the money is needed (just to give you an idea of what sums I\u2019m talking about).\n
\n
\nTake care :)", "timestamp": 1368176199}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1368188347017", "service": "gp", "text": "@Simon\n\u00a0\"Half of ad clickers will see the video, half will see something else (unrelated?), and the allocation will be made as randomly as possible. Thanks to cookies, we can later target all clickers with ads for the follow-up survey, and we will know who were in the intervention and who were in the control group. Correct?\"\n
\n
\nYes.\n
\n
\n\"If there are one thousand clicks on survey ads, do we have a sense of what % will complete the survey?\"\n
\n
\nI don't know, but the smaller and simpler the survey the better a response we should get. \u00a0We could test out the survey with a quick pilot to learn what response rate we could get.\n
\n
\n\"how long do the cookies last\"\n
\n
\nIt varies. \u00a0Some people use the same browser for months and never clear cookies, others clear cookies regularly. \u00a0I thought 30 days might be a good compromise. \u00a0This is probably something that people who work with remarketing cookies know, but quickly reading some blogs I don't see. \u00a0I suspect the fraction of people still using the same browser and computer after three years would be extremely small.\n
\n
\n\"can\u2019t it be a diet survey without posing leading questions?\"\n
\n
\nI'm not sure. \u00a0I don't want people to think they're supposed to answer a certain way, and I think in a diet survey you're more likely to get what people think they should be eating as opposed to what they actually eat. \u00a0The uses of the 2011 survey I've seen get a fraction of people who are vegetarian and from there get an estimate of the cost of a vegetarian-year, so it seems like \"are you a vegetarian\" would be ideal. \u00a0Except that I'm remembering a survey I read in TIME about 10 years ago where 25% of the people who claimed to be vegetarian also claimed to have eaten meat in the last 24 hours, so I think a lot of people don't know what it means. \u00a0Hence \"do you eat meat?\".\n
\n
\n\"someone might use the computer to watch the video and someone else to answer the survey\"\n
\n
\nYes, this is possible, and we don't have any way of knowing when it happens. \u00a0I think it should be rare, but it's hard to say.\n
\n
\n\"If people need to allow/enable the cookies\"\n
\n
\nOn almost all computers you can set cookies automatically.\n
\n
\n\"Is it considered ethical to do this kind of research?\"\n
\n
\nI don't see anything unethical about it, except maybe the showing the veg ads in the first place.\n
\n
\n\"I\u2019m also interested in funding and possibly raising funds among friends for this kind of research\"\n
\n
\nGreat! \u00a0I'll definitely let you know when we've got something that's ready for testing.", "timestamp": 1368188347}, {"author": "Simon", "source_link": "https://plus.google.com/116669225133133799072", "anchor": "gp-1368286010896", "service": "gp", "text": "@Jeff Kaufman\n\u00a0\n
\nThanks for your replies, they make sense :)\n
\n
\nSome other things that you've most likely thought of:\n
\n
\n1.\tIt would be good to use two different videos and a control group so that one can perhaps learn which video is better. Especially if increasing the sample is low cost, which it seems to be since Nick said you can use ads that will run anyway.\n
\n
\n2.\u00a0In my view,\u00a0the study won't be able to produce strong evidence for or against the impact of the ads because, among other things, the follow-up is so quick and since it will disregard people that eat less but still eat some. It still sounds worthwhile though, I just mention the limitation so that one goes into it with the right (modest) expectations.", "timestamp": 1368286010}, {"author": "Simon", "source_link": "https://plus.google.com/116669225133133799072", "anchor": "gp-1368351000810", "service": "gp", "text": "Hi again,\n
\n
\nIt would be better to also have data on meat consumption before the intervention, to know whether the intervention and control groups differed in meat consumption already before watching the video (which could make the results misleading). One possibility is to do the survey in three steps:\u00a0\n
\n1.\tFirst run ads only for the before-intervention-survey\n
\n2.\tThen target only those completing the survey with ads for the video\n
\n3.\tFinally, target only those who clicked the video ads with ads for the after-intervention-survey\n
\n
\nThe before and after surveys can (should?) probably be the same. Maybe this idea isn\u2019t feasible since we would need too many participants in the first survey round to get a reasonable sample size in the end. Another drawback is that my method adds a selection layer so that only people who completed the survey get the video ads, which makes it harder to generalize the results. But still, it would be good to have before and after data, and maybe one can get it with a better study design than my three step idea.", "timestamp": 1368351000}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1368375171928", "service": "gp", "text": "@Simon\n\u00a0\"to know whether the intervention and control groups differed in meat consumption already before watching the video\"\n
\n
\nWhy would this happen? \u00a0We'd be randomly assigning people to the groups, so with enough people they should have very similar initial meat consumption patterns.", "timestamp": 1368375171}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1368375377548", "service": "gp", "text": "@Simon\n\u00a0\"learn which video is better\"\n
\n
\nI think optimizing for facebook likes or \"no-more-meat\" comments probably gets you to the best video faster and cheaper.\n
\n
\n\"the study won't be able to produce strong evidence for or against the impact of the ads because, among other things, the follow-up is so quick\"\n
\n
\nWhat fraction of people who go veg in part because of the video would you estimate do it in the first 30 days after?\n
\n
\n\"disregard people that eat less but still eat some\"\n
\n
\nBut it also overcounts people who ate little and now eat none.", "timestamp": 1368375377}, {"author": "Simon", "source_link": "https://plus.google.com/116669225133133799072", "anchor": "gp-1368473025396", "service": "gp", "text": "\"Optimizing for facebook likes or \"no-more-meat\" comments probably gets you to the best video faster and cheaper.\" Good point.\n
\n
\n\"What fraction of people who go veg in part because of the video would you estimate do it in the first 30 days after?\" No idea really, 10%? There should be people for which this is the first exposure to animal suffering info , and it might spark something but they may not go veg for years. On the other hand, you have people for which the video is the tipping point. Then we have all the cases in-between these extremes. In my limited experience, going veg often takes a good while (one or a couple of years?) even after watching a video about factory farming, and in that perspective the 30-day period is a fairly narrow window to measure people becoming vegetarians.\n
\n
\n\"But it also overcounts people who ate little and now eat none.\" Yea, I agree. Having data on degrees of changed consumption would give a better \u00a0picture of the impact though. But there are pros and cons to trying to measure the degree (cons like maybe posing leading questions.)\n
\n
\nRe the need for baseline data, my impression is that it's considered good practice in RCTs to collect it and verify that the intervention and control groups are similar enough (could be different by chance or failed randomization). It also makes sense that the effect size estimate should be more precise with baseline data as long as the intervention and control groups are at least a little different due to chance. I could be wrong though and I don't know in which situations it's redundant. \u00a0I just base my impression on reading things like the quotes below and if I ran the study, I'd need to dig into it more. \u00a0\u00a0\n
\n
\nLecture notes on RCTs:\n
\n\"An overview of the RCT design....\n
\nThe amount of baseline data that needs to be collected does not have to be excessive, because the randomization process should result in identical groups. However, it is always necessary to collect sufficient baseline information regarding demographics and clinical characteristics to prove this point.\" \nhttp://learn.chm.msu.edu/epi/Coursepack/EPI546_Lecture_7_course_notes.pdf\n
\n
\nThe CONSORT Statement recommends reporting baseline data:\u00a0\n
\n\"CONSORT, which stands for Consolidated Standards of Reporting Trials, encompasses various initiatives developed by the CONSORT Group to alleviate the problems arising from inadequate reporting of randomized controlled trials (RCTs). The main product of CONSORT is the CONSORT Statement,\" \nhttp://www.consort-statement.org/home/\n
\n
\n\"Investigators and editors developed and revised the CONSORT (CONsolidated Standards of Reporting Trials) Statement to help authors improve reporting of two-parallel design RCTs by using a checklist.... The checklist includes the 25 items selected because empirical evidence indicates that not reporting the information is associated with biased estimates of treatment effect, or because the information is essential to judge the reliability or relevance of the findings. \" \nhttp://www.consort-statement.org/consort-statement/overview0/#checklist\n
\n
\n\" 15 - Baseline data.... Although proper random assignment prevents selection bias, it does not guarantee that the groups are equivalent at baseline. Any differences in baseline characteristics are, however, the result of chance rather than bias.(32) The study groups should be compared at baseline for important demographic and clinical characteristics so that readers can assess how similar they were. Baseline data are especially valuable for outcomes that can also be measured at the start of the trial.\" \nhttp://www.consort-statement.org/consort-statement/13-19---results/item15_baseline-data/\n
\n
\nTwo statisticians write:\n
\n\"Testing before the intervention\n
\nSo far we have looked at an \u2018after only\u2019 design for experiments:\n
\n\u2022 set up the two groups for the experiment\n
\n\u2022 carry out your experiment\n
\n\u2022 measure whatever it is at the end.\n
\nThere is a more powerful design, still under the randomised trials umbrella, which can be schematised as:\n
\n\u2022 set up the two groups for the experiment\n
\n\u2022 measure whatever it is at the beginning\n
\n\u2022 carry out your experiment\n
\n\u2022 measure whatever it is at the end.\n
\nThis is a more powerful design than the simple \u2018after only\u2019 design, and often a substantially more powerful one.... Baseline data also has a role in correcting for any imbalance in outcomes between the groups of the trial that might have resulted at randomisation. Such imbalance could have arrived by chance even if the randomisation were carried out properly or through some fault in the randomisation process.\" \nhttp://www.nfer.ac.uk/nfer/publications/RCT01/RCT01.pdf", "timestamp": 1368473025}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1368476875899", "service": "gp", "text": "@Simon\n\u00a0Are you proposing to survey everyone twice, to compare before-after? \u00a0Or randomly split the experiment and control group and survey half of each initially and the other half later?", "timestamp": 1368476875}, {"author": "Simon", "source_link": "https://plus.google.com/116669225133133799072", "anchor": "gp-1368477965434", "service": "gp", "text": "The former is what I have in mind, \"survey everyone twice, to compare before-after\"", "timestamp": 1368477965}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1368481752156", "service": "gp", "text": "@Simon\n\u00a0Some quick worries about surveying people twice:\n
\n
\n* You're going to lose a lot of people who don't complete the initial survey. \u00a0If we assume each survey costs us 90% of our responses, then adding an initial survey means we need to place 10x the ads.\n
\n
\n* When you ask people to retake a survey they've done before you'll get effects from their remembering their answers from the past.\n
\n
\n* If the survey is part of a direct flow that takes people to the anti-veg ad, then when they see the survey the second time they may connect it to the ad.\n
\n
\nWith large sample sizes and proper randomization we should be pretty safe from ending up with control and experimental groups that differ much.", "timestamp": 1368481752}, {"author": "Arora", "source_link": "https://www.facebook.com/jefftk/posts/612089066482?comment_id=613547813142", "anchor": "fb-613547813142", "service": "fb", "text": "I'd like to take surveys on why I am a vegetarian so that the information could reach others. I think that true vegans and vegetarians would go for a more altruistic approach while those following such diets on more of a 'trend base' would go for such an incentive. Vegetarianism and Veganism are life-style choices, not reward based trends (for the few anyways)", "timestamp": "1369193132"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/612089066482?comment_id=643029631342", "anchor": "fb-643029631342", "service": "fb", "text": "Jason Ketola and I are now working on making this happen. We had our first conversation about it today. Our notes are here: https://impact.hackpad.com/Conversation-with-Jason-Ketola...", "timestamp": "1388458130"}]}