::  Posts  ::  RSS  ::  ◂◂RSS  ::  Contact

Futarchy and Unfriendly AI

April 3rd, 2015
future, ideas  [html]

We have a reasonably clear sense of what "good" is, but it's not perfect. Suffering is bad, pleasure is good, more people living enjoyable lives is good, yes, but tradeoffs are hard. How much worse is it to go blind than to lose your leg? [1] How do we compare the death of someone at eighty to the death of someone at twelve? If you wanted to build some automated system that would go from data about the world to a number representing how well it's doing, where you would prefer any world that scored higher to any world scoring lower, that would be very difficult.

Say, however, that you've built a metric that you think matches your values well and you put some powerful optimizer to work maximizing that metric. This optimizer might do many things you think are great, but it might be that the easiest ways to maximize the metric are the ones that pull it apart from your values. Perhaps after it's in place it turns out your metric included many things that only strongly correlated with what you cared about, where the correlation breaks down under maximization.

What confuses me is that the people who warn about this scenario with respect to AI are often the same people in favor of futarchy. They both involve trying to define your values and then setting an indifferent optimizer to work on them. If you think AI would be very dangerous but futarchy would be very good, why?


[1] This is a question people working in public health try to answer with Disability Weights for DALYs.

Comment via: google plus, facebook, lesswrong

Recent posts on blogs I like:

How Fast New York Regional Rail Could Be Part 2

In my last post about New York regional rail schedules, I covered the New Haven and Harlem Lines of Metro-North and the Main Line and Hempstead Branch of the LIRR. I was hoping to cover more lines tonight, but due to time constraints only the Hudson Line …

via Pedestrian Observations October 17, 2019

Strong stances

I. The question of confidence Should one hold strong opinions? Some say yes. Some say that while it’s hard to tell, it tentatively seems pretty bad (probably). There are many pragmatically great upsides, and a couple of arguably unconscionable downsides. …

via Meteuphoric October 15, 2019

What do executives do, anyway?

An executive with 8,000 indirect reports and 2000 hours of work in a year can afford to spend, at most, 15 minutes per year per person in their reporting hierarchy... even if they work on nothing else. That job seems impossible. How can anyone make any im…

via apenwarr September 29, 2019

more     (via openring)

More Posts:


  ::  Posts  ::  RSS  ::  ◂◂RSS  ::  Contact