In 1964 [1], I. J. Good wrote Speculations Concerning the First Ultraintelligent Machine. I've only ever seen it cited for a single paragraph:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind ... Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.

This is becoming increasingly relevant, with the AI Futures Project now estimating an intelligence explosion starting in 2027 in response to recent AI progress in automating AI research. Making our 'last invention' relinquishes our influence over the future, we have to get it right, and with the current frantic pace disastrous mistakes are likely.

I think the appropriate reaction to this news is something like "this is completely irresponsible, you're way too likely to get us all killed, do not do that" and marching in the streets (hence yesterday's protest attendance). But as someone who is very interested in how people process ideas and what they thought with the information they had at the time, my reaction also includes wanting to know more about what Good expected six decades ago.

I spent this morning reading his paper and talking about it with Claude (Fable 5.2). I was curious how the ideas had held up, what he got right, and what he got wrong. Some of his ideas were very technical, and not being an ML expert I can't add much there; I've put these in a Claude-authored appendix.

The first sentence reads very differently now:

The survival of man depends on the early construction of an ultraintelligent machine.

I think he was saying that he was worried that without one we'd kill ourselves off (later he writes "it will give the human race a good chance of surviving indefinitely"). This was during the Cold War, a couple years after the Cuban Missile Crisis, and before any agreement to limit (let alone reduce) nuclear stockpiles.

But now I'd interpret it the other way around: our survival depends on 'early construction' the same way that a driver's survival depends on how quickly they drive.

Reading further, the most-quoted paragraph still jumps out, and not just through familiarity. Reading it closely, the "provided that the machine is docile enough to tell us how to keep it under control" is actually pretty prescient: he's essentially describing the plan of record, as much as there is one. Have AI research how to align AI. (This might work! But it also might not.)

He's also uncertain about whether it will go well. In the famous paragraph above he has a "provided that the machine is docile enough", later he says "the sign is uncertain" (math-speak for "could be good or bad"), and speculates whether "the human race will become redundant". He does seem overall optimistic, though, both in the 'survival depends' quote (above) and a few other places. For example, "the machines will create social problems, but they might also be able to solve them in addition to those that have been created by microbes and men."

A live objection at the time was that there wasn't any reason to make AI, and he cites B. V. Bowden's view that "it is easier to construct human brains by the usual method" before responding:

This shows that highly intelligent people can overlook the "intelligence explosion." It is true that it would be uneconomical to build a machine capable only of ordinary intellectual attainments, but it seems fairly probable that if this could be done then, at double the cost, the machine could exhibit ultraintelligence.

I'd actually count this as a bit of a miss: a machine capable of human-level work that was massively cheaper than humans could still be highly economical, and that's essentially what today's deployment of AI is. I suspect he didn't anticipate the extent to which high throughput would precede high intelligence.

I also don't see where his "double the cost" is coming from, and in practice this jump is likely to be much larger. But the critical thing he gets right is that human-level isn't a ceiling, and in a series of domains (games, coding, cybersecurity, math...) we've seen AI, after spending a while at "toy" and some time at "useful tool", quickly pass "human level" to get to something much more capable. I expect this keeps going.

On the technical side, most of the details are in Claude's appendix but at a high level it seems like he was right about a lot of how we got here. He thought it would be "vast artificial neural circuitry" and expected it would learn from experience and reinforcement:

Yet a detailed knowledge of semantics might not be required, since the artificial neural network will largely take care of it, provided that the parameters are correctly chosen, and provided that the network is adequately integrated with its sensorium and motorium (input and output). For, if these conditions are met, the machine will be able to learn from experience, by means of positive and negative reinforcement, and the instruction of the machine will resemble that of a child.

He also anticipated how text-based this would all be:

Whether a machine of classical or ultraparallel design is to be the first ultraintelligent machine, it will need to be able to handle or to learn to handle ordinary language with great facility. This will be important in order that its instructor should be able to teach it rapidly, and so that later the machine will be able to teach the instructor rapidly. It is very possible also that natural languages, or something analogous to them rather than to formal logic, are an essential ingredient of scientific imagination. Also the machine will be called upon to translate languages, and perhaps to generate fine prose and poetry at high speed, so that, all in all, linguistic facility is at a high premium.

He also correctly anticipated that AIs would use traditional computers (still then in their infancy) as tools, with coding and search:

In any event the ultraintelligent machine might as well have a large electronic computer at its beck and call, and also a multimillion dollar information-retrieval installation of large capacity but of comparatively slow speed, since these would add little to the total cost.

And he anticipated a period before AI fully surpassed humans, during which they'd work together:

Until an ultraintelligent machine is built perhaps the best intellectual feats will be performed by men and machines in very close, sometimes called "symbiotic," relationship

This is how many of us work daily, including both my day job and these blog posts.

He was also a strong chess player (on the Bletchley Chess Club during the war) and suggested:

so-called "randomized chess," in which the positions of the white pieces on the first rank are permuted at random before the game begins (but with the two bishops on squares of opposite colors), and then the initial positions of the black pieces are determined by mirror symmetry. This gives rise to 1440 essentially distinct initial positions and effectively removes from the game the effect of mere parrot learning of the openings

This is very close to Chess960 (though without the extra castling work) which Bobby Fischer proposed decades later, and I was initially pretty excited to see it. But reading more it turns out this was a known variant going back centuries. Good's including it here to illustrate that what we really care about is the reasoning component of chess, not memorizing opening lines.

Thinking about things Good got wrong, the timing initially seems like a big one: "more probable than not ... within the twentieth century". But he only put it at 50%, and while it has taken longer it looks like it's going much faster than most others predicted.

A much larger miss is the cost of artificial neurons:

If we could raise say a hundred billion dollars we might be able to simulate all the neurons of a brain, and of a whole man, at a cost of ten dollars per artificial neuron.

Since Moore's law is newer than Good's writing it's not surprising that he didn't fully account for it (though he did expect "great advances in microminiaturization"), but the hardware to power an artificial neuron costs ~10,000x less than this (~$0.01 in current dollars, less if you quantize; Good's $10 would be ~$100 today).

On the other hand, and more importantly, he's in the right ballpark on total cost: the world has spent maybe $1T to date creating AI systems and I expect we spend maybe that again by the end of 2027 as recursive self-improvement increases the returns to AI R&D. Good's $100B is $1T in current dollars, but note that he didn't expect anyone to pay this:

But it seems unlikely that more than say [$300M then; $3B today] would actually be forthcoming, and even this amount might be difficult to obtain without first building the machine!

This is indeed how funding has worked, with investors putting in more money in response to initial progress. That he ended up close to right on the ~$1T looks to me like less of a coincidence than it appears: costs come down over time, and it's only when they're getting into an economically feasible range that someone might actually do it.

The biggest miss, though, is about focus. He was super early in thinking about whether such a machine would be good for the world, but then spends almost the entire paper on how one might be designed. Focusing on the "how" and not the "whether" is very normal for technical people then and now, and as an engineer I feel this pull as well. But it's something people have been getting wrong since the beginning: excitement about the potential, not staring hard at the true scale of the risk.

This paper isn't the last we have of Good's thinking, however, and in 1998 he provided a bio to the IEEE on receiving their Computer Pioneer Award. It doesn't seem to have been used at the ceremony or published, but James Barrat says Good's assistant Leslie Pendleton gave him a copy, and Barrat quotes it in his 2013 Our Final Invention. Note that this is Good writing about himself in the third person:

"The survival of man depends on the early construction of an ultra-intelligent machine." Those were his words during the Cold War, and he now suspects that "survival" should be replaced by "extinction." He thinks that, because of international competition, we cannot prevent the machines from taking over.

How to get countries to coordinate to prevent the machines from taking over is a key question today.

Appendix: Good's technical predictions I asked Claude Fable 5.2 to write an appendix. The judgments are its own, and I haven't checked them:

Most of Good's paper is a theory of how a neural network could store meaning and recall it. Below I go through its main technical claims and say how well I think each matches how current AI systems work. I grade each as a strong match, a partial match, a loose match, or a miss.

Two cautions first. A 58-page speculative paper makes a lot of claims, and picking out the ones that look right in hindsight flatters it. And Good was less of a lone voice than this list might suggest. Neural networks were an active research area in the early 1960s: he cites Rosenblatt's perceptron work, and builds on Hebb, Ashby, and Shannon. His bet was on one of the two live approaches, the one that then went out of fashion for decades and eventually won.

Learning is adjusting connection strengths. Strong match. In his radio-linked machine (p35, p36):

The strength of the connection could be represented by the accuracy of the tuning. ... "Positive reinforcement" would correspond to improved tuning of these filters.

Every connection has an adjustable strength, and learning means changing the strengths. That is what a "weight" is in a modern network, and training is nothing more than adjusting weights.

How the strengths get adjusted. Miss. Good's rule (p80) is that a connection has a small random chance of stepping up in strength whenever it is "successfully used (i.e., contributes to the activation or inhibition of an artificial neuron)". This is a local rule: each connection changes based only on its own activity. Modern networks are trained by backpropagation, which works out how every weight in the network contributed to an error in the final output and adjusts them all together. Nothing in the paper anticipates this, and local rules of Good's kind have not been made to work at scale.

What "reinforcement" means. Partial match. Jeff asked whether Good's "positive and negative reinforcement" from an instructor is what is now called reinforcement learning from human feedback. It does resemble it, and that technique is used to shape how current models behave. But most of what those models know comes from a different source, predicting the next word across a vast amount of text. Good's usage is broader than reward from an instructor. The rule above involves no instructor, he says the machine is educated only "partly" by reinforcement (p80), and his account of the child is about exposure:

The child later associates words with objects and activities, by implicit statistical inference ... At a later stage of linguistic development, similar statistical associations are developed between linguistic elements themselves. (p45, p46)

He has the right raw material, the statistics of which words occur with what, and he discusses predicting words from the preceding ones (p43, p54). He never proposes prediction as the thing to train on.

Words are known by the company they keep. Strong match, though not original to him.

The essential idea is that words can be seen to be associated not merely because they occur frequently together, but because they both occur frequently in conjunction with a third word (p71)

His example is that "ferry" and "fare" are related because both go with words like "bus". This is the basis of how modern systems represent word meaning. Linguists (Harris, Firth) had stated the idea in the 1950s. What Good adds is the argument that a machine needs it, since machines that "rely only on frequencies" (p71) can say nothing about combinations they haven't seen.

Word vectors. Partial match. On p52 and p53 he takes a table of how often each word occurs in each document, takes logarithms, subtracts a constant from each row and column, and proposes decomposing the result so that

This would associate a w-dimensional vector with each index term and with each document. The relevance of index term i to document j could now be defined as the correlation coefficient between the two associated vectors.

This is recognizably the method later known as latent semantic analysis (around 1990), and a relative of the word-embedding methods of the 2010s, which were shown to be doing something close to factoring a similar table. Two things are missing. He keeps as many dimensions as there are words, where the later methods get their power from keeping only a few hundred. And the vectors are computed once, not learned by a network. He also offers it only as "another conceivable method" and judges the computation "exceedingly great".

Generalization is the central problem. Strong match on the diagnosis, partial on the cure.

One of the main difficulties in this subject is how to estimate the probabilities of events that have never occurred. That such probabilities are relevant to intelligence is to be expected, since intelligence is sometimes defined as the ability to adapt to new circumstances. (p32)

This is still the right way to state it. Good had worked on this problem with Turing, and the resulting method (Good-Turing estimation) was a standard component of statistical language models into the 2000s. Current models don't use it. They generalize by learning that similar words and situations behave similarly, which is closer to his "ferry" and "fare" idea, and to his remark (p71) that getting the network to form abstractions is "one of the most vital problems".

What comes next depends on recent context and stored weights. Partial match. Good's unit of thought is the "assembly", Hebb's term for a group of neurons that fire together and stand for a word or idea.

Which assembly becomes active at the next moment must depend on the current sensory input, the current dominant assembly, and the currently active subassemblies. Indirectly, therefore, it depends on the recent assembly sequence, wherein the most recent assemblies will have the greatest influence. It also depends of course on the semipermanent static storage, the "past history." (p60)

A language model has the same division: what comes next is determined by the recent text it is looking at (its "context") and by weights fixed during training. Good's list of memory types (p72) puts short-term memory in current activity and long-term memory in connection strengths, which is the same split, and he says a long "attention span" is "certainly essential for intelligence" (p63). He goes on (p66) to score each candidate next assembly as a baseline plus a contribution from each recent one, with the highest total firing. The last step of a language model has that form, a score for every possible next word. But in a modern model the contributions from earlier words interact in complicated ways before that step, and Good only gestures at this by saying interaction terms will be needed (p67). I wouldn't claim more than a family resemblance.

Concepts are spread across many neurons. Strong match, shared credit.

each neuron would occur in a great many distinct circuits, and would not be indispensable for any of them. (p56)

He contrasts this with "one concept, one neuron". Modern networks store concepts as patterns over many units, with each unit taking part in many concepts, and this is a main reason their workings are hard to read off. The idea came from neuroscience (he cites Lashley and Eccles). Good's contribution is an efficiency argument for it, by analogy with a punched-card coding scheme.

Keep probabilities, not hard decisions. Strong match. Section 3 argues that a system understanding speech should pass along graded probabilities between its levels instead of committing to one answer at each, because "each definite 'decision' loses information" (p39). Modern networks pass continuous values between layers and only commit at the output.

Rejecting a logical "canonical language". Strong match. He considers representing meaning by translating every statement into a standard formal form, and concludes this would be "extremely difficult, perhaps even logically impossible, or perhaps it would require an ultraintelligent machine to do it!" (p41). Large projects later attempted roughly this and did not get far. Relatedly, his guess (p36, quoted in the main post) that natural language "rather than ... formal logic" is needed for scientific imagination is a partial match: current models do much of their reasoning by writing out steps in ordinary language.

Scale can substitute for design. Partial match.

a well-designed machine of say 10⁹ components might be educable to ultraintelligence, but a much more carelessly designed machine of say 10¹³ components might be equally good. (p62)

The history of the field broadly supports this: simple methods given more computation have repeatedly beaten cleverer ones. But Good, following Ashby, bases it on the hope that almost any large, richly connected network will behave intelligently. That turned out to be false. Scale paid off only once there was a training method able to make use of it.

The hard part is ape-level intelligence. Partial match. He says "most of the struggle" will be building "a machine with the intelligence of an ape" (p45), because perception and basic learning are harder than what comes after. In favor: getting networks to learn to see took until about 2012, and fluent language followed within a decade. Against: language arrived without anything like an ape's physical competence, and robotics is still behind.

Sleep and replay. Partial match. His conjecture that sleep serves "to replay the assembly sequences that were of greatest interest during the day in order to consolidate them" (p80) is close to a well-supported view in neuroscience. In AI, replaying stored experience, with priority to the most informative parts, is a standard technique in systems that learn by trial and error. It is not part of how language models are mainly trained, so his suggestion that the machine would "benefit from periods of comparative rest" (p72) is a loose match at best.

Understanding as economy. Loose match. He argues that storing meanings instead of exact wordings saves a great deal of memory, and that this may be "the evolutionary function of meaning and understanding" (p77). There is a real modern parallel, since learning to predict text well and learning to compress it are mathematically the same task. Good's argument is about storage, though, and I think the connection is more suggestive than exact.

Hardware. Mixed. He was right that serial speed alone wouldn't do. He expected computers to become about a million times faster than neurons and still concluded the machine "will need to be ultraparallel" (p35), and current systems do depend on massively parallel chips. He was right that low power consumption "will be valuable" (p33). He was wrong that the neurons would be physical devices. They are simulated in software.

The brain as blueprint. Mostly a miss. The paper opens by saying that to design the machine "we need to understand more about the human brain or human thought or both", and later that brain anatomy is "very relevant" (p63). Current systems borrow the neuron and the adjustable connection and almost nothing else. Of the brain theory itself, Hebb's assemblies remain a respected idea in neuroscience, but the central control system Good relies on (taken from Penfield) has been abandoned, and his own extensions haven't been taken up. To his credit, he says the designer will have to guess which features of the brain matter, and he allows that the first machine might not use a neural network after all (p78).

Theory before experiment. Miss.

The need for further theory is great, since, without advances in theory, the amount of experimentation might be prohibitive. (p64)

Computation became cheap enough to experiment at enormous scale, and the systems we have were found that way. Theory explaining why they work is still well behind.

How much the machine needs to learn from. Miss. He sizes the machine's education to a person's: "A man cannot learn more than ten million statements in a lifetime" (p36). Current models learn much less from each example than people do, and compensate with volume. By my rough estimate they are trained on something like ten thousand times more text than a person could read in a lifetime.

Overall. Good was right about the kind of thing the machine would be: a very large, parallel neural network that learns the meanings of words from statistics, stores them as distributed patterns, treats generalization as the core problem, and works in natural language with conventional computers as tools. He was wrong, or silent, about how such a network could actually be trained, which is the part that took the field another fifty years. I'd put his reputation for foresight on these pages at least as much as on the famous paragraph.

[1] I generally see 1965, including on Wikipedia probably because of the publication date. But I think it's more informative to count by the date he wrote it: "The first draft of this monograph was completed in April 1963, and the present slightly amended version in May 1964."