Superintelligence Risk Project Update II

July 18th, 2017
airisk, ea
This is the beginning of my third week looking into potential risks from superintelligence (kickoff, update 1) and I think I'm hitting diminishing returns. I'm planning to wrap up in the next day or so, and go back to figuring out what I should work on next.

Last week:

  • Technical Distance to AGI: I hypothesized (incorrectly) that the main difference between ML researchers who thought we could vs couldn't work on AI risk now was how far off they thought AGI was, in terms of some combination of time and technological distance. Recommended comments: Jacob, Paul, Dario. I also made a 1:9 bet with Dave on whether we'll have driverless cars in the next 10 years.

  • Examples of Superintelligence Risk: I collected the examples I've seen of what a "loss of control of an AI" catastrophe might look like, and tried to figure out why the examples are much less realistic than we see of other existential risks like nukes or bioterror. Recommended comments: Eliezer, Jim, Paul.

  • Conversation with Bryce Wiedenbeck: I talked to an AI professor, main takeaway being that he thinks the technical distance to AGI is very high. Recommended comment: Dario.

  • I found Open Phil's notes on Early Field Growth interesting, especially their section on failure modes in cryonics and molecular nanotechnology. My takeaway was that heavy popularization of a new field prior to scientific success leads scientists on the border of the new field to take on an oppositional stance. The field gets starved of people who could do substantial technical work, makes minimal progress, and I think people also avoid the areas around its edges, like a chilling effect. I see superintelligence risk as just on the edge of this, where it could go either way. Which also makes me (weakly) think that Daniel Dewey's point on the relevant field-building effects of MIRI-style vs prosaic AI-style should maybe go farther. Specifically, you don't want safety to be thought of as a "we don't do that, those people are cranks" sort of thing, so it's a lot better if AI safety develops primarily as a field within ML.

  • Spoke to three other ML researchers, one of which I'm hoping to write up conversation notes from.

  • I had applied for an EA Grant when I thought I might spend longer on this, but withdrew after getting to the phone interview stage.

  • I spent most of Monday working on the house and running errands instead of on this project.

(A big takeaway for me is that I don't like doing this kind of work very much. I think it's a combination of two things: it's isolated work (as I'm doing it) and it's a kind of thinking that I enjoy in moderation, but not for full time work. These two combine pretty strongly: this kind of thinking is much more enjoyable for me when working with someone else, where we can have a lot of conversations to clarify ideas and look for the best areas to make progress. David Chudzicki, one of my housemates, has been helpful here, and we've talked a lot, but it's still something I'm mostly working on alone.)

Comment via: google plus, facebook

Recent posts on blogs I like:

A Big Problem With The Going To Bed Book

One day my dad was reading this book called the "Going to Bed Book" to my sister Nora. The book is basically about a bunch of animals who are getting ready for bed on a boat. They go down the stairs, take a bath, hang their towels on the wall, find…

via Lily Wise's Blog Posts September 18, 2023

Investing in boundaries with young kids

Putting in some work to get the behavior you want The post Investing in boundaries with young kids appeared first on Otherwise.

via Otherwise August 15, 2023

Self-driving car bets

This month I lost a bunch of bets. Back in early 2016 I bet at even odds that self-driving ride sharing would be available in 10 US cities by July 2023. Then I made similar bets a dozen times because everyone disagreed with me. The first deployment to pot…

via The sideways view July 29, 2023

more     (via openring)