{"items": [{"author": "Jacob", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886379256742", "anchor": "fb-886379256742", "service": "fb", "text": "For what it's worth, I am very in favor of AI safety research and (a) assign less than 10% to AGI in 10 years, (b) don't feel like my being in favor of it is dependent on this (though it maybe is dependent on likelihood in the next 100 years, and I would have a greater sense of urgency, and work on different problems, if I believed 10% in 10 years).", "timestamp": "1499789277"}, {"author": "Paul", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886382435372", "anchor": "fb-886382435372", "service": "fb", "text": "10% probability in 10 years seems sufficient but certainly not necessary. I'd definitely endorse altruists working on foreseeable safety problems given 10% probability in 20 years, though obviously the exact break-even point will depend a lot on preferences and alternatives. I think you can also weaken the technical distance claim to \"current techniques will play an important role\"---that's enough to run into the foreseeable safety problems, if other techniques are involved we just might *also* run into other problems.", "timestamp": "1499790707"}, {"author": "Louis", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886382580082", "anchor": "fb-886382580082", "service": "fb", "text": "My impression from skimming MIRI papers has been that they're mostly aiming to be so mathematically general it doesn't matter how AGI is implemented, or whether it looks anything like current ML techniques? (I haven't read very many, though.)", "timestamp": "1499790724"}, {"author": "Taymon", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886382580082&reply_comment_id=886384840552", "anchor": "fb-886382580082_886384840552", "service": "fb", "text": "→ Per http://www.jefftk.com/.../superintelligence-risk-project..., it sounds like this disagreement is primarily between the skeptic view and the prosaic-AGI view. I would guess that, under the MIRI view, if AGI arrives in the next 10 years, we're probably just screwed, because it'll be difficult to build a solid theoretical foundation that quickly and nothing else will suffice. (MIRI presumably doesn't count as \"ML people\".)", "timestamp": "1499791949"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886382580082&reply_comment_id=886386831562", "anchor": "fb-886382580082_886386831562", "service": "fb", "text": "→ Taymon: yup!", "timestamp": "1499793008"}, {"author": "Dario", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886382580082&reply_comment_id=886392844512", "anchor": "fb-886382580082_886392844512", "service": "fb", "text": "→ One thing worth adding here is that I don't think people realize how general ML is as a framework (see below). People often conflate \"ML\" with \"deep neural networks\", and the latter really are just a specific technique, but the former is actually extremely general and we (humanity) have put a lot of work into understanding it theoretically (see for example statistical theory of learning: https://en.wikipedia.org/wiki/Statistical_learning_theory). I'm not an expert on this topic (Paul is), but much of the work we do (and certainly Paul's blog posts) can be interpreted as being about general systems that learn. One thing that would help me understand MIRI's position is if they made clear precisely in what sense they are covering a more general case than statistical learning theory, and what cases they hope to cover, that we don't, with this generality. Certainly there's some sense in which \"systems that take actions and do internal processing\" is broader than \"systems that learn\", but it's never been clear to me how much useful generality is gained this way or whether any useful additional systems will be covered by this generality.", "timestamp": "1499794872"}, {"author": "Dario", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886387355512", "anchor": "fb-886387355512", "service": "fb", "text": "\"If the chance were less than 1%, then while those are still interesting research questions that look useful for today, they're not altruistically important.\". I don't agree with this statement -- in particular I think technical distance is more important than time (so it's entirely possible that it could take 50 years to build AGI but it would still look basically like ML, and in that case I see substantial value to the current line of safety research). I also, in line with Paul, see substantial value to ML safety if current techniques are just one component of AGI -- and this seems very likely to me. For example, it's hard for me to imagine an AGI with a vision system that's not based on some kind of statistical ML. Maybe higher reasoning and other parts of the system work differently, but vision involves fuzzy concepts and it's hard to imagine capturing those concepts without ML. It's then the case that any sensory information you propagate to other parts of the AI system has all the robustness and reliability problems associated with ML systems, and you need to deal with those, or they could cause bad decisions in any part of your system. So it seems like even if AGI is built using some unknown additional components, which need to be made safe in some unknown novel way, you also need to solve the robustness problems we know exist today.", "timestamp": "1499793312"}, {"author": "Dario", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886388563092", "anchor": "fb-886388563092", "service": "fb", "text": "I'd also encourage people to be more explicit about what they mean by \"alternatives to ML\". I know what people mean by \"alternatives to deep learning\" -- for instance, Bayesian networks, SVM's, or MCMC methods, but what do people actually mean by alternatives to ML? Anything I can imagine is either something really similar to ML (that might end up being given a different name for basically branding reasons), or some kind of non-grounded logical reasoning system. The latter dominated AI from 1950 to 1975 or so, which may be why people imagine there are all these alternatives to ML, but really these systems didn't learn and should probably be thought of as more like ordinary programming. Certainly no one thinks such systems will give us AGI. What could happen is that ML-based fuzzy decisions could drive formal reasoning (see the DeepMath work: https://arxiv.org/abs/1606.04442), but this is really ML -- it's a bit like neural-net driven human brain entering digits into a calculator and then using the results in further calculations.", "timestamp": "1499794024"}, {"author": "Geoffrey", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886388563092&reply_comment_id=886487928962", "anchor": "fb-886388563092_886487928962", "service": "fb", "text": "→ Yeah, the gap between symbolic and fuzzy AI is temporary at best.", "timestamp": "1499813889"}, {"author": "Bil", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886401362442", "anchor": "fb-886401362442", "service": "fb", "text": "I notice that they didn't talk about Intelligence at all. They just talked about \"fancy\" programs that can perform simple tasks such as driving a car or moving chess pieces. Wonderful, useful tasks, but no Intelligence.", "timestamp": "1499797959"}, {"author": "Brandon", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886403652852", "anchor": "fb-886403652852", "service": "fb", "text": "re: off the cuff answer: Well I mean, it's the kind of answer you get when there exists some true answer and you have some good information but not the information you need. Similar: \"How many spacefaring civilizations are there in the galaxy?\" \"Will the United States exist in 200 years?\" \"Could you pull off a mohawk?\"", "timestamp": "1499798390"}, {"author": "opted out", "source_link": "#", "anchor": "unknown", "service": "unknown", "text": "this user has requested that their comments not be shown here", "timestamp": "1499815822"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886493842112&reply_comment_id=886510289152", "anchor": "fb-886493842112_886510289152", "service": "fb", "text": "→ Sorry! This post is a quick update on a larger project. The introduction defines the acronyms and gives background: http://www.jefftk.com/p/superintelligence-risk-project", "timestamp": "1499817976"}, {"author": "opted out", "source_link": "#", "anchor": "unknown", "service": "unknown", "text": "→ this user has requested that their comments not be shown here", "timestamp": "1499819047"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542", "anchor": "fb-886625363542", "service": "fb", "text": "We won't even have a working self-driving car in 10 years. Chances of that are under 1%-- even the people working on it say 20 (and that's optimistic). This is an actual self-driving car, not a car that requires a driver to pay attention sometimes, which defeats the purpose. How could we possibly have AGI before that? This whole conversation seems like a bunch of CS majors who want to work on something interesting (nothing wrong with that) trying to convince themselves that the thing they find interesting is actually super important from an altruism perspective.", "timestamp": "1499861843"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886644500192", "anchor": "fb-886625363542_886644500192", "service": "fb", "text": "→ My \"working self-driving cars in 10 years, as in cars that get you from most US addresses to most other US addresses without you having to pay attention\" would be much higher than 1%, maybe 50%. I'd bet you $10 to $90?", "timestamp": "1499867841"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886654764622", "anchor": "fb-886625363542_886654764622", "service": "fb", "text": "→ Jonathan, I'm not saying the same people are trying to develop both with the same technologies, which I'm sure is untrue. Jeff, I would take that bet in a heartbeat. 10 years, no way there's a 10 percent chance of that. The engineer quoted in this article says we'll have \"Level 5\" self-driving cars in 25 years in closed environments. Which is fine, but does not constitute actually having self-driving cars in any meaningful way (unless Uber gets its way and gets the government to completely fund new roads for its \"self-driving\" cars which are like trains but worse in all ways, etc): http://www.eschatonblog.com/.../06/and-500-other-things.html", "timestamp": "1499870069"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886667968162", "anchor": "fb-886625363542_886667968162", "service": "fb", "text": "→ @Dave: My $10 to your $90 then? Evaluated 2027-07-12? Any phrasing tweaks?
(I'm also willing to do larger amounts, up to my $200 to your $1800)", "timestamp": "1499875901"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886669854382", "anchor": "fb-886625363542_886669854382", "service": "fb", "text": "→ Clarification: I'm trying to bet on the tech, not on the regulatory environment. Like, maybe cars are fully capable of doing this but aren't legally allowed to. But for questions like \"when will we have AGI\" the tech is what matters.", "timestamp": "1499876626"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886670592902", "anchor": "fb-886625363542_886670592902", "service": "fb", "text": "→ A way to test this: get into a driverless car and go to a random US address. Then pick another US address within a 1hr drive (not quite right, but otherwise the test is way to slow) and tell it to go there. Bet resolves \"yes\" if the car gets there reasonably without any intervention, \"no\" if it does require intervention. The driver doesn't pay attention (unless they're legally required to).", "timestamp": "1499876929"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886678182692", "anchor": "fb-886625363542_886678182692", "service": "fb", "text": "→ My preferred test would be: get into a driverless car and go to a random US address, then another US address within an hour drive and tell it to go to the second address. Bet resolves \"yes\" if the car gets there reasonably without any intervention and without any driver. Bet resolves \"no\" if not. If you think the bet would resolve in your favor except for a law requiring a driver to be present and paying attention, that's inconclusive in my view. I would also prefer to do the bet in 2017 dollars, buy a mutually acceptable investment with those dollars, and simply have the winner take control of the investment in 10 years. Who knows what will happen to the dollar between now and 2027 (probably not much, but still). I'd go my $900 to your $100; basically I'd buy an investment of some sort, see if there's a way to give you joint control over it, and you could pay me 1/10th the value of the investment now, exclusive control going to the winner in 2027 (or the other way around if you'd rather set it up).", "timestamp": "1499878692"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886680612822", "anchor": "fb-886625363542_886680612822", "service": "fb", "text": "→ Although, come to think of it, maybe it makes more sense for the first address also to be somewhere convenient for both of us, to the extent possible, if there's actual need to test it (in other words, if the technology is arguable there). I mean I only have a hundred at stake, makes no sense for the test to cost well over that.", "timestamp": "1499879576"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886683531972", "anchor": "fb-886625363542_886683531972", "service": "fb", "text": "→ @Dave: maybe we should just pick the addresses now? Harvard sq to the Concord Scout House?", "timestamp": "1499879849"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886683976082", "anchor": "fb-886625363542_886683976082", "service": "fb", "text": "→ Sure, I don't think it's perfect, but easy is better than perfect. Let's start at your dad's house, get into the driverless car, go to Harvard Square, then go to the Scout House. To make it simple, let's say 0 Brattle St as the Harvard Square address?", "timestamp": "1499880086"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886684360312", "anchor": "fb-886625363542_886684360312", "service": "fb", "text": "→ Dave: Things I'm agreed on:
* Route is Dad's house to 0 Brattle St to Scout House
* $100/$900
* Financing where I give you $100 now, you invest my $100 plus your $900, winner gets the value in 10y. Or you could just pay me an amount scaled by the change in VTSMX over that period if you lose.
* Determination of yes/no/inconclusive as you said. If inconclusive money goes back to original owner.
Does that work?", "timestamp": "1499880366"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886685423182", "anchor": "fb-886625363542_886685423182", "service": "fb", "text": "→ Done! I'll let you know this weekend once I've found an investment, etc.", "timestamp": "1499880555"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886691710582", "anchor": "fb-886625363542_886691710582", "service": "fb", "text": "→ Ok, bet confirmed!", "timestamp": "1499882220"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886692653692", "anchor": "fb-886625363542_886692653692", "service": "fb", "text": "→ I think VTSMX (total stock market index fund) is a pretty reasonable default investment", "timestamp": "1499882502"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886692778442", "anchor": "fb-886625363542_886692778442", "service": "fb", "text": "→ Or maybe VTWSX (total world stock market index, including US)
Don't have strong feelings here", "timestamp": "1499882576"}, {"author": "Harris", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886698766442", "anchor": "fb-886625363542_886698766442", "service": "fb", "text": "→ I'm so glad you worked that out in public for all to witness. I am very pleased and entertained.
(I'm going to set a reminder on my phone to check in with you in 10 years about the results of the bet and we'll see if my reminders successfully make it from device to device over that time period.)", "timestamp": "1499884116"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886699175622", "anchor": "fb-886625363542_886699175622", "service": "fb", "text": "→ I hope speech recognition keeps getting better...", "timestamp": "1499884246"}, {"author": "Dave", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886702044872", "anchor": "fb-886625363542_886702044872", "service": "fb", "text": "→ I just set my first reminder ever with Google Assistant, and it worked and even recognized my voice! But it didn't say \"you're welcome\" when I told it, \"Thank you, Google,\" so, minus 1 point for that.", "timestamp": "1499885149"}, {"author": "Sophia", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886728257342", "anchor": "fb-886625363542_886728257342", "service": "fb", "text": "→ Who's betting on whether or not the terms of the bet will still be archived on Facebook?", "timestamp": "1499891182"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886625363542&reply_comment_id=886735263302", "anchor": "fb-886625363542_886735263302", "service": "fb", "text": "→ Sophia: I copied them to an email to Dave; I'm pretty optimistic about still having access to my old email", "timestamp": "1499892591"}, {"author": "Mac", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886662304512", "anchor": "fb-886662304512", "service": "fb", "text": "Interested, but uninformed here. What are: AGI, MIRI, and ML? (I've read the post at the top of this string -- I think.)", "timestamp": "1499872884"}, {"author": "Jai", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886662304512&reply_comment_id=886668701692", "anchor": "fb-886662304512_886668701692", "service": "fb", "text": "→ Artificial General Intelligence: AI that's good at lots of things, probably encompassing the intellectual competencies of most humans
Machine Intelligence Research Institute: A non-profit working on possible mathematical underpinnings of aligning AI goals with human values.
Machine Learning: The euphemism for AI we collectively developed in the 2000's because \"AI\" had negative connotations.", "timestamp": "1499876164"}, {"author": "Brandon", "source_link": "https://www.facebook.com/jefftk/posts/886375748772?comment_id=886662304512&reply_comment_id=886684085862", "anchor": "fb-886662304512_886684085862", "service": "fb", "text": "→ There's a lot of overlap and it's true that the distinction has a lot to do with the academic politics of the term \"artificial intelligence\" but Machine Learning principally concerns itself with regression and classification tasks whereas AI concerns itself with planning tasks. Linear regression, for example, is a very simple Machine Learning technique that no one would consider in the realm of AI.", "timestamp": "1499880153"}, {"author": "David German", "source_link": "https://plus.google.com/111229345142780712481", "anchor": "gp-1499947509394", "service": "gp", "text": "Very clever, slipping a \nsummer\n evaluation date into your bet with Dave.", "timestamp": 1499947509}, {"author": "David German", "source_link": "https://plus.google.com/111229345142780712481", "anchor": "gp-1499947817548", "service": "gp", "text": "Also, technically, your bet is on Level 4 autonomy, not Level 5 as referenced earlier in the thread.", "timestamp": 1499947817}, {"author": "Jeff Kaufman", "source_link": "https://plus.google.com/103013777355236494008", "anchor": "gp-1499951060141", "service": "gp", "text": "@David German\n what's your probability for that bet?", "timestamp": 1499951060}]}