I want to measure the effect of clicking a link to a site like Who's Against Animal Cruelty on people's rates of being vegetarian. Effective Animal Activism currently recommends these online ads based off of a survey [1] that has several problems including sample size (n=44), sample bias, conflict of interest, and leading questions. I'd like to make a new study that doesn't suffer from these problems, which means I want people to try and poke holes in this proposal below.

We want to measure the effect of buying pay-per-click facebook ads for sites like Who's Against Animal Cruelty on how much people eat meat. We need an experimental group who will see the video, and a control group that won't. Both groups will see the initial facebook ad, and both groups will get followup surveys after some amount of time that ask them whether they eat meat. The difference in meat eating rates would then be attributable to the video.

I'm not sure what exactly the ads should look like, but basing them off of the examples here should be about right. I'm also not sure how they should be targeted. I'm going to write to group that put out the initial survey to find out what they recommend.

For the followup, the ideal thing would be if we could just make people take a survey later. Unfortunately (fortunately?) the internet doesn't let you do that. The best I can come up with is to drop a retargeting cookie after people click the ad, and then later place ads against that cookie to get people to take a survey. To get people to take the survey I could see offering a gift card, maybe a small chance of a $25 one?

The survey should be hosted on a neutral domain, with no apparent connection to the facebook ad click or the video that the experimental group watched. It should ask several apparently unrelated things, so that it's not obvious what's being tested. Current plan, definitely needs work:

1. What is your age?
2. What is your gender?
3. Do you ever purchase products online?
4. Do you live within 20 minutes of your workplace or school?
5. Do you eat meat?
6. Do you have a smartphone?
7. Have you traveled outside the US in the last year?

Cost estimate of running the survey:

• 10,000 ad clicks on facebook @ $0.20: $2000
• Building sites: I'll do that for free; sounds interesting.
• Domain name for video: $10
• Domain name for survey: $10
• Hosting: basically free
• 1000 followup survey clicks @ $2: $2000

I'm thinking of doing a kickstarter for this. I might also be able to get people who think veg ads are valuable to contribute half of the facebook ad spend, as half of those people are seeing veg ads.

Update 2016-04-24: someone ran a study like the one I proposed, and my thoughts on it are here.

[1] I've looked about this study before: 2013-04-15, 2012-03-18, and 2012-03-07.