{"items": [{"author": "Zera", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855207215762", "anchor": "fb-855207215762", "service": "fb", "text": "*adds prediction market regarding what percentage of friends will adopt this tool*", "timestamp": "1489793185"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855207340512", "anchor": "fb-855207340512", "service": "fb", "text": "Robin Hanson", "timestamp": "1489793265"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855209416352", "anchor": "fb-855209416352", "service": "fb", "text": "If you update your prediction you should not be competing against yourself.", "timestamp": "1489794429"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855209416352&reply_comment_id=855212170832", "anchor": "fb-855209416352_855212170832", "service": "fb", "text": "→ You're not? If you create the contract you can't get points until after someone else has made a prediction.
(And if you predict twice in a row the math cancels out)", "timestamp": "1489795441"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855209416352&reply_comment_id=855213079012", "anchor": "fb-855209416352_855213079012", "service": "fb", "text": "→ Let's say there is one prediction at 40%. You think the true score is 80%. Can you go 0%, to move the consensus to 20, and then 80?", "timestamp": "1489795885"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855209416352&reply_comment_id=855222385362", "anchor": "fb-855209416352_855222385362", "service": "fb", "text": "→ You can do that, but you won't earn any more points than just going right to your actual belief", "timestamp": "1489799505"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532", "anchor": "fb-855216312532", "service": "fb", "text": "OK, I went down a rabbit hole. You've got an error in your description of how you do the scoring: \"so 30% and 40%, and I earn (lose) log(30/40) points.\" -> should be log (40/30)", "timestamp": "1489797127"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855232026042", "anchor": "fb-855216312532_855232026042", "service": "fb", "text": "→ Hmm, I think log(30/40) is correct:
The score is log(accuracy / previous_accuracy). If it's resolved true, then accuracy was 70% and previous was 60%, so log(70/60). If it's resolved false then accuracy was 30% and previous was 40% so log(30/40).
The actual code is: https://github.com/.../5b7854dd95711fdc27d4ddec.../app.py...", "timestamp": "1489802376"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855238852362", "anchor": "fb-855216312532_855238852362", "service": "fb", "text": "→ I don't have a good theoretical explanation but empirically it looks like you need to flip the numbers. Plug in some values and see. Try consensus .50, real .60, predictions .65 and .55. Predictions have the same magnitude of error but the score is different your way, the same with the log values flipped as I suggest.", "timestamp": "1489805273"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855241277502", "anchor": "fb-855216312532_855241277502", "service": "fb", "text": "→ There's still a slight difference even if you flip the logs. You could use Brier scoring, which definitely does not have this issue.", "timestamp": "1489806511"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855243807432", "anchor": "fb-855216312532_855243807432", "service": "fb", "text": "→ .6 * log(70/60) - .4 * log (30/40) ~= .09. So your hypothetical predictor is scoring positive points even though the crowd prediction was totally accurate. Should be a negative score.", "timestamp": "1489807485"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855313038692", "anchor": "fb-855216312532_855313038692", "service": "fb", "text": "→ Frederic: that - should be a +. log(30/40) is already negative.", "timestamp": "1489833452"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855316776202", "anchor": "fb-855216312532_855316776202", "service": "fb", "text": "→ (it's still off by 0.009, though)", "timestamp": "1489834344"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855336222232", "anchor": "fb-855216312532_855336222232", "service": "fb", "text": "→ OK, I've read up more on logarithmic scoring. It seems to attach more emphasis to predictions that are closer to certainty. Brier scoring is what I am used to and is symmetrical, so being miscalibrated by the same percentage will result in the same score no matter how close to certainty you are. I don't really understand the implications here, but it seems that you are making some value judgments about more certain predictions by choosing logarithmic scoring.", "timestamp": "1489844764"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855337409852", "anchor": "fb-855216312532_855337409852", "service": "fb", "text": "→ @Frederic: I'm not entirely sure why the scoring works this way (Leah?) but I think this idea is that the difference between \"this event is at 1% vs 2%\" really is much more important in practice than \"this event is at 55% vs 56%\". So we make 1% vs 2% be worth the same as 25% vs 50%.", "timestamp": "1489845489"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855338662342", "anchor": "fb-855216312532_855338662342", "service": "fb", "text": "→ It's not obvious to me why this should be true and it seems to skew things in weird ways. Here's a practical example: if the actual percentage is 80%, predictions of 90% and 66.5% have just about the same value.", "timestamp": "1489845586"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855342020612", "anchor": "fb-855216312532_855342020612", "service": "fb", "text": "→ @Frederic: The amount of evidence needed to move you from 80% to 66.5% or 80% to 90% are the same though. Which I think is related to why people who take a bayesian approach tend to like log-odds formulations of probabilities and log-scoring?", "timestamp": "1489847503"}, {"author": "Mark", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855349600422", "anchor": "fb-855216312532_855349600422", "service": "fb", "text": "→ Intuitively it should be very different to go from 50% to 60% than to go from 90% to 100%", "timestamp": "1489850624"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855349979662", "anchor": "fb-855216312532_855349979662", "service": "fb", "text": "→ Well you never get to 100% because the scoring system can't handle non-bayesians", "timestamp": "1489850825"}, {"author": "Mark", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855350498622", "anchor": "fb-855216312532_855350498622", "service": "fb", "text": "→ Why not? The described scoring algorithm is well defined for 100%. If for some reason it wasn't, subtract an epsilon from 60% and 100%.", "timestamp": "1489851179"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855354730142", "anchor": "fb-855216312532_855354730142", "service": "fb", "text": "→ You end up with log(0) if you are 100% certain but wrong", "timestamp": "1489852232"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855354844912", "anchor": "fb-855216312532_855354844912", "service": "fb", "text": "→ Which is why Jeff doesn't let you enter 0 or 100% certainty", "timestamp": "1489852353"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855355069462", "anchor": "fb-855216312532_855355069462", "service": "fb", "text": "→ With the current scoring rule I shouldn't allow 0.000000001 either. It's probably someone who doesn't understand how the scoring handles being wrong", "timestamp": "1489852557"}, {"author": "Mark", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855355618362", "anchor": "fb-855216312532_855355618362", "service": "fb", "text": "→ whoops misunderstood how the scoring works when you are on the wrong side of the probability", "timestamp": "1489852957"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855357998592", "anchor": "fb-855216312532_855357998592", "service": "fb", "text": "→ You should probably add an error code for someone who enters a probability below 1 in case they are trying to use the range from 0 to 1 rather than a percentage.", "timestamp": "1489853858"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855368911722", "anchor": "fb-855216312532_855368911722", "service": "fb", "text": "→ If you're entering a percentage, you say e.g. \"20%\", which is equivalent to \".2\", which you can also enter.
Everything accepted is in the 0 to 1 range already.", "timestamp": "1489856815"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855370219102", "anchor": "fb-855216312532_855370219102", "service": "fb", "text": "→ As I understand the code if you enter a prediction of .5 it is going to turn that into .5%.", "timestamp": "1489857162"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855371426682", "anchor": "fb-855216312532_855371426682", "service": "fb", "text": "→ No:
if '%' in percentage:
value = float(percentage.replace('%', '')) / 100
else:
value = float(percentage)", "timestamp": "1489857713"}, {"author": "David Chudzicki", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855371511512", "anchor": "fb-855216312532_855371511512", "service": "fb", "text": "→ https://github.com/wave.../predictions/blob/master/app.py...", "timestamp": "1489857762"}, {"author": "Frederic", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855388173122", "anchor": "fb-855216312532_855388173122", "service": "fb", "text": "→ Sorry, you are correct. My coding skills are pretty rudimentary.", "timestamp": "1489862755"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855216312532&reply_comment_id=855397509412", "anchor": "fb-855216312532_855397509412", "service": "fb", "text": "→ David: when posting links to specific lines, generally best to link to a commit instead of a branch: https://github.com/.../5b7854dd95711fdc27d4ddec.../app.py...", "timestamp": "1489867358"}, {"author": "Barbara", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855323662402", "anchor": "fb-855323662402", "service": "fb", "text": "Viliam B\u00far", "timestamp": "1489838465"}, {"author": "Jan-Willem", "source_link": "https://plus.google.com/100580955183019057735", "anchor": "gp-1489845406333", "service": "gp", "text": "Seems like if I move to 0 or 100% I can force the numbers to +-infinity and the math \nwon't\n just work out. If I do that and walk back to my actual prediction I should end up with NaN points.", "timestamp": 1489845406}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1489846897346", "service": "gp", "text": "@Jan-Willem\n We're using logarithmic scoring, so 0% and 100% aren't valid predictions. The tool won't allow you to enter them. (You can probably screw it up by entering 0.999999999999 or something though, and getting into floating point error range, though. If so I should prohibit things within epsilon of absolute certainty, and not just absolute certainty itself.)", "timestamp": 1489846897}, {"author": "Jan-Willem", "source_link": "https://plus.google.com/100580955183019057735", "anchor": "gp-1489848570439", "service": "gp", "text": ":-)", "timestamp": 1489848570}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855397589252", "anchor": "fb-855397589252", "service": "fb", "text": "If anyone wants me to help them get set up to have /predict available on their slack, let me know!", "timestamp": "1489867403"}, {"author": "Adom", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855397589252&reply_comment_id=855419575192", "anchor": "fb-855397589252_855419575192", "service": "fb", "text": "→ I'm currently looking to get this set up for the Event Horizon slack, and running into some trouble early on. When I try to run `pytest .` I get a syntax error in importtestmodule. I'm installing a bunch of python dev tools as I go through this, and I think there might be something going on with python 2.7 vs python 3.
Anyway I'm looking through SO etc., but let me know if you have any ideas about what might be going on.
Thanks!", "timestamp": "1489872987"}, {"author": "Adom", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855397589252&reply_comment_id=855419734872", "anchor": "fb-855397589252_855419734872", "service": "fb", "text": "→ When I run `SLACK_TOKEN=1 python app.py` I get the same error on the same line:
`File \"app.py\", line 339
response = selected_command(*internal_args, *args)
^
SyntaxError: invalid syntax`", "timestamp": "1489873110"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855397589252&reply_comment_id=855420004332", "anchor": "fb-855397589252_855420004332", "service": "fb", "text": "→ Adom: it needs Python 3.6 (see runtime.txt), sorry!", "timestamp": "1489873235"}, {"author": "Adom", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855397589252&reply_comment_id=855420243852", "anchor": "fb-855397589252_855420243852", "service": "fb", "text": "→ Thanks!", "timestamp": "1489873457"}, {"author": "Leah", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855397589252&reply_comment_id=855437329612", "anchor": "fb-855397589252_855437329612", "service": "fb", "text": "→ ", "timestamp": "1489879428"}, {"author": "Adom", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855419979382", "anchor": "fb-855419979382", "service": "fb", "text": "Luke Muehlhauser", "timestamp": "1489873221"}, {"author": "Miranda", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=855686525222", "anchor": "fb-855686525222", "service": "fb", "text": "Satvik Beri (Ruby)", "timestamp": "1489984187"}, {"author": "Gordon", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=936878401052", "anchor": "fb-936878401052", "service": "fb", "text": "Is there a name (other than \"CFAR-style\") for this kind of prediction-scoring? Anyplace where it's been discussed in depth, or where there are example logs of its use on real propositions? I'm wondering how to apply it to maintain incentives to update one's own last p when there's new info, or to avoid inflated tallies where people just repeat themselves in a cycle. (\"80%\" \"No, 20%\" \"No, 80%\" \"No, 20%\"...)", "timestamp": "1523487399"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=936878401052&reply_comment_id=936921110462", "anchor": "fb-936878401052_936921110462", "service": "fb", "text": "→ When you update yourself in a cycle you don't get any more points than if you immediately predicted your best number. You can try it with the scoring.
You still have incentives to update your predictions even if no one has made a prediction in between, because you score points in proportion to how much you improve the group consensus.", "timestamp": "1523492128"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=936878401052&reply_comment_id=936937043532", "anchor": "fb-936878401052_936937043532", "service": "fb", "text": "→ I don't know a name other than cfar-style or where to learn more. Leah? Do you know the origins of this form of market?", "timestamp": "1523494936"}, {"author": "Leah", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=936878401052&reply_comment_id=936938805002", "anchor": "fb-936878401052_936938805002", "service": "fb", "text": "→ ", "timestamp": "1523495778"}, {"author": "Gordon", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=936878401052&reply_comment_id=936939189232", "anchor": "fb-936878401052_936939189232", "service": "fb", "text": "→ Jeff Kaufman Hm, I'll poke a bit more at the code to understand corner cases. (From a quick glance at `show()`, it looks like repeated assertions of the same p, with others' intervening, will be tallied more than once. That's what I was concerned about - two (or more) insistent people just contradicting each other.)", "timestamp": "1523496016"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/855206911372?comment_id=936878401052&reply_comment_id=936981274892", "anchor": "fb-936878401052_936981274892", "service": "fb", "text": "→ Two people repeatedly contradicting each other are continually raising the stakes: one will lose a lot and one will gain a lot.", "timestamp": "1523498553"}]}