|July 9th, 2017|
|airisk, giving [html]|
Michael believes that weâll probably have machines that are as smart as humans (AGI, for short) at some point. He doesnât think itâs certain, and heâs really interested in the question of whether human-style intelligence is achievable, but he thinks weâll probably figure it out eventually. At the same time, he sees this as very far off. Not just âfarâ in the sense of a long time coming, though he does think thatâs probably the case, but âfarâ in the sense of how much more technical and conceptual work is required before we get there.
For example, I asked him what he thought of the idea that to we could get AGI with current techniques, primarily deep neural nets and reinforcement learning, without learning anything new about how intelligence works or how to implement it (âProsaic AGIâ ). He didnât think this was possible, and believes there are deep conceptual issues we still need to get a handle on. Heâs also less impressed with deep learning than he was before he started working in it: in his experience itâs a much more brittle technology than he had been expecting. Specifically, when trying to replicate results, heâs often found that they depend on a bunch of parameters being in just the right range, and without that the systems donât perform nearly as well.
The bottom line, to him, was that since we are still many breakthroughs away from getting to AGI, we canât productively work on reducing superintelligence risk now.
He told me that he worries that the AI risk community is not solving real problems: theyâre making deductions and inferences that are self-consistent but not being tested or verified in the world. Since we canât tell if thatâs progress, it probably isnât. I asked if he was referring to MIRIâs work here, and he said their work was an example of the kind of approach heâs skeptical about, though he wasnât trying to single them out. 
I asked him what he thought of Concrete Problems in AI Safety (pdf), and Michael told me he hadnât read it in detail but had skimmed it. His response was that getting better at specifying what we care about is a great research problem, and an area heâs working on. Itâs technology that would be useful to us today, and heâs happy to see it developed better. He doesnât see it, however, as useful from a long-term AI safety perspective because we just donât know anything about what AGI would look like yet.
At this point, I was trying to figure out if the key disagreement between him and, say, Dario or Paul was on whether Prosaic AGI was possible. I asked what he would think we should work on if he believed that all we needed to get to AGI was a lot more engineering work on current technologies without discovering anything fundamentally new. Unfortunately, this was far enough from what he actually believes that he found it really hard to get into the hypothetical. He thought that even in this hypothetical he probably wouldnât think âloss of controlâ issues were among the most urgent ones we have, but he really wasnât very sure what he would think if he were actually in that situation.
 Iâm not sure whether Paul Christiano coined this term, but heâs who Iâve heard it from. For more detail on it, see the first section of Prosaic AI Alignment.
 He followed up by email: "At root, I feel that the AI risk community is trying to extrapolate from our cartoonish current understanding of what intelligence is and how it works and making a deeply unsubstantiated leap that more raw computation will give us a great deal more of whatever is powerful about intelligence. This kind of extrapolation has been very successful in areas like physics, where we can infer the structure of far away galaxies and project the impact of what it means for more energy or gravity or temperature to be concentrated in one place. I think it's a mistake to think of intelligence as a physical quantity and to put any faith into similar extrapolations in the cognitive realm. It's simply too easy for us to misinterpret our own introspections about intelligence and the science of the production of creative new ideas is still very very young."