{"items": [{"author": "Andrew", "source_link": "https://www.facebook.com/jefftk/posts/141926372580128?comment_id=141940455912053", "anchor": "fb-141940455912053", "service": "fb", "text": "I agree that these kinds of experiments are interesting and the results may not be intuitive. One thing I am wondering about in your examples is that there seem to be many changes between A and B, so while it is simple to look at the data to see which works better, it might be hard to know *why* it works better. Would the experiments provide more useful data if the changes were smaller - that is, one change at a time?", "timestamp": "1321710881"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/141926372580128?comment_id=141941559245276", "anchor": "fb-141941559245276", "service": "fb", "text": "@Andrew: it makes sense to do both. In small scale testing you change a lot of minor things and see what works better. Sometimes, however, there is a better design that's only reachable from a worse design via paths that are even worse. So you also have large scale testing where you try a whole new design.", "timestamp": "1321711076"}, {"author": "Andrew", "source_link": "https://www.facebook.com/jefftk/posts/141926372580128?comment_id=141947775911321", "anchor": "fb-141947775911321", "service": "fb", "text": "Yes. Another idea - I think this kind of refinement and testing is a form of evolution and natural selection - I imagine that you make the changes and evaluate them manually. If you had enough software development resources, you could program the tweaks and the feedback automatically, and let the designs just evolve. You could end up looking at your web site and say, \"gee, I wonder how that happened?\"", "timestamp": "1321712148"}, {"author": "Allison", "source_link": "https://plus.google.com/103741579182942078941", "anchor": "gp-1321717565478", "service": "gp", "text": "While I like the new site design better than the old, my bet is that it might have done worse, possibly because only has one buy button. It's really hard to say; there are so many factors. Will you say which one was which at some point?", "timestamp": 1321717565}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1321725287345", "service": "gp", "text": "@allison\n I will, at some point", "timestamp": 1321725287}, {"author": "Kiran", "source_link": "https://www.facebook.com/jefftk/posts/141926372580128?comment_id=142256349213797", "anchor": "fb-142256349213797", "service": "fb", "text": "This makes me wonder, is correctly predicting a skill, or a talent And if it's a talent, how widespread is it?", "timestamp": "1321766622"}, {"author": "Adam Yie", "source_link": "https://plus.google.com/114873051319510815414", "anchor": "gp-1321885383274", "service": "gp", "text": "I'll guess the bottom was an improvement.", "timestamp": 1321885383}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1321891037982", "service": "gp", "text": "@Adam Yie\n \n@allison\n the top one (email) was good, the bottom one (site) was bad.", "timestamp": 1321891037}, {"author": "Allison", "source_link": "https://plus.google.com/103741579182942078941", "anchor": "gp-1321892500832", "service": "gp", "text": "I wonder about long-term impact of changes using short-term metrics, like day-of sales or clicks. If you essentially trick people into buying something or clicking on a link, won't that scare them off in the future, even if you get a good score for a day? Also, what if degradation is a temporary matter--people might just be used to something? In that case, wouldn't it be good to see how sales pan out over a longer period of time? How long is long enough?", "timestamp": 1321892500}, {"author": "Adam Yie", "source_link": "https://plus.google.com/114873051319510815414", "anchor": "gp-1321892617339", "service": "gp", "text": "If anything, it's an example of being drawn or repulsed by first impressions: the phrase \"combination skin\" sounds vaguely Visitors-ish to me (though Google confirms it's just my ignorance) and the picture has more slink to it than I expect in skincare ads. So although everything I've read would justify supporting the top change, it still weirded me out. If only I could convince myself to look at \nlayout\n rather than specifics. =)", "timestamp": 1321892617}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1321892766978", "service": "gp", "text": "@allison\n we try not to trick people into doing things for the reasons you say: it's not good except in the immediate short term.\n
\n
\nFor people getting used to a completely new layout, like the one above, we'll sometimes run the test longer (a week?) if we think that's going on, but a large fraction of people are seeing our pages for the first time, so that can't be a huge effect.", "timestamp": 1321892766}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1321894846261", "service": "gp", "text": "@Adam Yie\n Sorry, I was confusing. What we were testing was the layout, not the content. Each of the deals you see advertised was shown under both sides of the test. I would have preferred to use four screen shots all of the same deal, but I didn't have that available.", "timestamp": 1321894846}, {"author": "Adam Yie", "source_link": "https://plus.google.com/114873051319510815414", "anchor": "gp-1321898243498", "service": "gp", "text": "@Jeff Kaufman\n I don't think you were confusing. I was trying to poke fun at myself in my previous comment--even knowing that you were testing layout, it wasn't enough for me to get past the content in that example.", "timestamp": 1321898243}]}