• Posts
  • RSS
  • ◂◂RSS
  • Contact

  • Futarchy and Unfriendly AI

    April 3rd, 2015
    future, ideas  [html]
    We have a reasonably clear sense of what "good" is, but it's not perfect. Suffering is bad, pleasure is good, more people living enjoyable lives is good, yes, but tradeoffs are hard. How much worse is it to go blind than to lose your leg? [1] How do we compare the death of someone at eighty to the death of someone at twelve? If you wanted to build some automated system that would go from data about the world to a number representing how well it's doing, where you would prefer any world that scored higher to any world scoring lower, that would be very difficult.

    Say, however, that you've built a metric that you think matches your values well and you put some powerful optimizer to work maximizing that metric. This optimizer might do many things you think are great, but it might be that the easiest ways to maximize the metric are the ones that pull it apart from your values. Perhaps after it's in place it turns out your metric included many things that only strongly correlated with what you cared about, where the correlation breaks down under maximization.

    What confuses me is that the people who warn about this scenario with respect to AI are often the same people in favor of futarchy. They both involve trying to define your values and then setting an indifferent optimizer to work on them. If you think AI would be very dangerous but futarchy would be very good, why?


    [1] This is a question people working in public health try to answer with Disability Weights for DALYs.

    Comment via: google plus, facebook, lesswrong

    Recent posts on blogs I like:

    Who Should Bear the Risk in Infrastructure Projects?

    The answer to the question is the public sector, always. It’s okay to have private-sector involvement in construction, but the risk must be borne by the public sector, or else the private sector will just want more money to compensate for the extra risk. …

    via Pedestrian Observations November 30, 2020

    Fireside Friday, November 27, 2020

    Hey folks! Fireside this week. A bit of a change-up in terms of the coming attractions. I had planned to start “Textiles, How Did They Make It?” next, but I want to do a bit more reading on some of the initial stages of textile production (that is, the pr…

    via A Collection of Unmitigated Pedantry November 27, 2020

    Thoughts you mightn't have thunk about remote meetings

    Welcome to this week's edition of "building a startup in 2020," in which all your meetings are suddenly remote, and you probably weren't prepared for it. I know I wasn't. We started a "fully remote" company back in 2019, but …

    via apenwarr November 23, 2020

    more     (via openring)


  • Posts
  • RSS
  • ◂◂RSS
  • Contact