• Posts
  • RSS
  • ◂◂RSS
  • Contact

  • Superintelligence Risk Project Update II

    July 18th, 2017
    airisk, giving  [html]
    This is the beginning of my third week looking into potential risks from superintelligence (kickoff, update 1) and I think I'm hitting diminishing returns. I'm planning to wrap up in the next day or so, and go back to figuring out what I should work on next.

    Last week:

    • Technical Distance to AGI: I hypothesized (incorrectly) that the main difference between ML researchers who thought we could vs couldn't work on AI risk now was how far off they thought AGI was, in terms of some combination of time and technological distance. Recommended comments: Jacob, Paul, Dario. I also made a 1:9 bet with Dave on whether we'll have driverless cars in the next 10 years.

    • Examples of Superintelligence Risk: I collected the examples I've seen of what a "loss of control of an AI" catastrophe might look like, and tried to figure out why the examples are much less realistic than we see of other existential risks like nukes or bioterror. Recommended comments: Eliezer, Jim, Paul.

    • Conversation with Bryce Wiedenbeck: I talked to an AI professor, main takeaway being that he thinks the technical distance to AGI is very high. Recommended comment: Dario.

    • I found Open Phil's notes on Early Field Growth interesting, especially their section on failure modes in cryonics and molecular nanotechnology. My takeaway was that heavy popularization of a new field prior to scientific success leads scientists on the border of the new field to take on an oppositional stance. The field gets starved of people who could do substantial technical work, makes minimal progress, and I think people also avoid the areas around its edges, like a chilling effect. I see superintelligence risk as just on the edge of this, where it could go either way. Which also makes me (weakly) think that Daniel Dewey's point on the relevant field-building effects of MIRI-style vs prosaic AI-style should maybe go farther. Specifically, you don't want safety to be thought of as a "we don't do that, those people are cranks" sort of thing, so it's a lot better if AI safety develops primarily as a field within ML.

    • Spoke to three other ML researchers, one of which I'm hoping to write up conversation notes from.

    • I had applied for an EA Grant when I thought I might spend longer on this, but withdrew after getting to the phone interview stage.

    • I spent most of Monday working on the house and running errands instead of on this project.

    (A big takeaway for me is that I don't like doing this kind of work very much. I think it's a combination of two things: it's isolated work (as I'm doing it) and it's a kind of thinking that I enjoy in moderation, but not for full time work. These two combine pretty strongly: this kind of thinking is much more enjoyable for me when working with someone else, where we can have a lot of conversations to clarify ideas and look for the best areas to make progress. David Chudzicki, one of my housemates, has been helpful here, and we've talked a lot, but it's still something I'm mostly working on alone.)

    Comment via: google plus, facebook

    Recent posts on blogs I like:

    The Gift of It's Your Problem Now

    Recently a security hole in a certain open source Java library resulted in a worldwide emergency kerfuffle as, say, 40% of the possibly hundreds of millions of worldwide deployments of this library needed to be updated in a hurry. (The other 60% also …

    via apenwarr January 1, 2022

    The container throttling problem

    This is an excerpt from an internal document David Mackey and I co-authored in April 2019. The document is excerpted since much of the original doc was about comparing possible approaches to increasing efficency at Twitter, which is mostly information tha…

    via Posts on December 18, 2021

    Experiences in raising children in shared housing

    Sometimes I see posts about people’s hope to raise children in a group housing situation, and it often seems overly optimistic to me. In particular they seem to expect that there will be more shared childcare than I think should be expected. Today I talke…

    via The whole sky October 18, 2021

    more     (via openring)


  • Posts
  • RSS
  • ◂◂RSS
  • Contact