The people who made this possible

1884 · Gottlob Frege

Mathematician and logician, University of Jena

Laid down the rule that a word's meaning is to be sought in the sentence it sits in, not on its own.

What he contributed

Frege was building the foundations of arithmetic, not studying language. But to do that he had to settle a prior question, what it is for a word to mean something, and the rule he laid down inverted the usual order. Do not begin with a word and ask what it stands for. Begin with the sentence, and ask what the word contributes to it. Meaning, on this account, is not a property a word carries around on its own. It is something a word has only in company.

That is the philosophical license for everything on this page that handles ambiguity. A model stores no definition of the word bank. It stores a position that its neighbors then adjust. The claim that a word has no fixed standalone meaning is what makes such a design coherent rather than obviously broken.

Frege is also the source of the compositionality principle, that the meaning of a sentence is a function of the meanings of its parts and how they are combined. Scholars have argued at length over whether that is consistent with the context principle, which appears to run the other way.

The split matters more than it looks. Formal semantics inherited compositionality and builds meaning upward from parts. Distributional semantics, and every statistical language model this page builds toward, inherited the context principle and derives the parts from the whole. Modern systems are in effect a large empirical bet on the second reading.

Why it matters here

This is the philosophical ancestor of the whole of the embeddings and attention machinery. The reason a model can handle the word bank correctly in two different sentences is that it never assigns the word a meaning on its own. It assigns it a position that then gets adjusted by its neighbors.

The honest qualification

Frege was guarding against confusing logic with psychology in the foundations of arithmetic. He was not making a claim that meaning is constituted by patterns of co-occurrence, and he had no notion of anything like a word vector.

Treating him as an ancestor of distributional semantics is a later interpretive move. It is a defensible one, and this page states it as ours rather than as his. Scholars also dispute whether Frege still held the principle after 1890.

Sources

• Frege, G. Die Grundlagen der Arithmetik: eine logisch mathematische Untersuchung über den Begriff der Zahl. Breslau: Wilhelm Koebner, 1884. The three principles appear in the introduction. NOTE: the archive catalog record for this scan wrongly describes it as a 1986 F. Meiner edition; the scanned title page is the 1884 Koebner printing source
• The English rendering quoted is J. L. Austin's translation, The Foundations of Arithmetic

1943 · Warren McCulloch and Walter Pitts

University of Illinois College of Medicine and University of Chicago

Showed that because a neuron either fires or does not, a network of them can be treated as logic.

What they contributed

A neurophysiologist and a logician asked whether the brain could be described in mathematics. Their move was to strip the neuron down to a single property, that it either fires or it does not, with nothing in between. From that one simplification they showed that the activity of a neuron can be written as a logical proposition, and that a network of such units can compute any logical expression meeting certain conditions.

This is the founding permission slip for the whole field. Before it, the brain was biology. After it, a network of simple units was a legitimate object for mathematics and, eventually, for engineering, and the word network entered the conversation. What they did not supply, and this is the gap the next forty three years are spent filling, is any way for such a network to learn. Their thresholds were set by hand and stayed where they were put.

Their unit computes a threshold function over binary inputs, with an absolute inhibition mechanism that can veto firing outright. That is sufficient for AND, OR and NOT, and therefore for any Boolean function, given enough units arranged in enough layers.

The limits are equally definite. A finite net of such units has finite state, so there are things it cannot compute at all, and the paper handles nets containing cycles separately and with some difficulty. The parallel with the present is closer than it looks: what a fixed-depth transformer can and cannot express is an open research question, asked in the same terms.

In the paper's own words

Because of the 'all-or-none' character of nervous activity, neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms.

McCulloch and Pitts, A Logical Calculus of the Ideas Immanent in Nervous Activity, 1943

The gloss to be careful with

This work is usually summarized as modeling neurons as on or off logic gates. That is a defensible modern paraphrase and it is not the paper's vocabulary. The word gate is not used in that sense anywhere in it, and the paper's own frame is propositional logic rather than circuits.

The more important point: their units had fixed thresholds, no weights and no learning at all. It is a theory of what a network can compute. The question of how a network could come to compute the right thing is the subject of a later era on this page, and it took forty three years.

Why it mattered to what came next

It made the brain a legitimate object for mathematics, and it put the word network into the conversation. Every neural network since is downstream of the claim that neural activity is a thing arithmetic can represent.

Sources

• McCulloch, W. S. and Pitts, W. A Logical Calculus of the Ideas Immanent in Nervous Activity. The Bulletin of Mathematical Biophysics 5(4), 115 to 133, 1943. DOI 10.1007/BF02478259. The linked file is the 1990 verbatim reprint in Bulletin of Mathematical Biology 52(1/2), 99 to 115, so its page numbers differ from the 1943 original source
• Journal record, DOI 10.1007/BF02478259 source

1950 · Alan Turing

Mathematician, University of Manchester

Refused to define thinking, and proposed a game instead.

What he contributed

Asked whether machines can think, Turing refused the question. His argument was that machine and think cannot be defined without quietly assuming the answer, so the question cannot be settled by argument. He replaced it with something you can actually run. An interrogator exchanges written messages with two hidden participants, one a person and one a machine, and has to say which is which.

The contribution is the conversion, not the game. He turned a metaphysical dispute into an experiment about observable behavior, and that move set the field's terms for seventy five years. It also set the trap that this page keeps returning to. A test of whether output passes for human measures the output, and says nothing about what produced it. ELIZA in the chatbot era is the first clear demonstration that those two things come apart, and the current systems are the largest.

The bulk of the paper is Turing anticipating and answering nine objections, and the strongest is the mathematical one drawn from Gödel: any consistent formal system powerful enough for arithmetic contains true statements it cannot prove, so a machine will always have questions it answers wrongly or not at all.

His reply is worth knowing because it is still the reply. He concedes the result and denies it distinguishes anything, on the ground that no one has established that human mathematicians are exempt from the same limitation. The objection assumes the superiority it is trying to prove.

The opening lines, as printed

I propose to consider the question, 'Can machines think?' This should begin with definitions of the meaning of the terms 'machine' and 'think'.

Turing, Mind, Volume LIX, Number 236, October 1950. Quotation marks modernized from the original spaced single quotes

The name he never used

Turing calls it the imitation game throughout, fifteen times. The phrase Turing test appears nowhere in the paper and was applied later by others. This page does not attribute the coinage to anyone, because we did not establish who first used it.

Sources

• Turing, A. M. Computing Machinery and Intelligence. Mind, Volume LIX, Issue 236, October 1950, 433 to 460. Publisher record, DOI 10.1093/mind/LIX.236.433 source
• Caution on sourcing: the most widely circulated online transcription of this paper contains transmission errors including a wrong volume number in its header. Quotations here were read from page images of the original Mind printing.

1951 · Claude Shannon

Bell Telephone Laboratories

Measured how predictable written English is, by asking someone to guess it one letter at a time.

What he contributed

In 1948 Shannon had given information a unit and a measure. Here he turned the measure on English itself and asked a question nobody had made quantitative. How much of a sentence is actually carrying information, and how much is redundant, predictable from what came before? His method was disarmingly plain. Take a passage the subject has not seen, ask them to guess it one letter at a time, tell them when they are right, and record how they do. A person's guessing accuracy is a measurement of the statistical structure of the language they have absorbed.

This is the direct conceptual ancestor of how every model on this page is trained. Pre-training is Shannon's guessing game, played by a machine, at a scale no person could sit through, with the guesses used to adjust the machine rather than to compute a statistic. The underlying insight is the one the entire field rests on: prediction is a measurement of structure, so a system that predicts text well has, in a precise sense, captured how the text is put together.

Shannon put numbers on it. With a 26 letter alphabet and spaces ignored, a uniform distribution gives `log2(26) = 4.7` bits per letter. Letter frequencies alone bring that to 4.14. Adding the statistics of letter pairs brings it to 3.56. Extending the context to about a hundred letters, he estimated, reduces it to about 1 bit per letter, a redundancy of roughly 75 percent.

That is the quantity modern language models are still scored on, though not in the same units. Shannon measured bits per letter. A model's cross-entropy is measured per token, where a token is whatever its tokenizer settled on, and it is usually reported in nats rather than bits. Perplexity is that quantity exponentiated in whichever base matches. Convert with care and the comparison holds, which is why a model's loss can be set against Shannon's roughly one bit per letter at all. Put the two numbers side by side without converting and you are comparing different things.

A detail this page will not assert

The experiment is often described as having been performed with his wife, Mary Elizabeth Shannon, known as Betty, doing the guessing. The paper itself names no participant at all and refers only to the subject. The attribution comes from a later biography rather than from the primary source, so it is recorded here as commonly reported rather than stated as fact.

Two dates, both correct

The paper was published in January 1951 and the manuscript was received on 15 September 1950. Sources giving either year are describing different events.

Sources

• Shannon, C. E. Prediction and Entropy of Printed English. Bell System Technical Journal 30(1), 50 to 64, January 1951. DOI 10.1002/j.1538-7305.1951.tb01366.x. Scan source
• Shannon, C. E. A Mathematical Theory of Communication. Bell System Technical Journal 27, 379 to 423 (July 1948) and 623 to 656 (October 1948). A two part paper across two issues, with a DOI each: 10.1002/j.1538-7305.1948.tb01338.x and 10.1002/j.1538-7305.1948.tb00917.x

1955 · John McCarthy, Marvin Minsky, Nathaniel Rochester and Claude Shannon

The Dartmouth proposal, 31 August 1955

Wrote a funding application that named the field. The name is in the title, a year before the meeting it was applying for.

What they contributed

Four researchers, none of them yet famous, wrote a funding application to the Rockefeller Foundation asking for a summer's work. In doing so they did two things that outlasted anything the meeting itself produced. They gave the field its name, and they wrote its founding assumption into a single sentence.

The name matters more than it sounds. Work that was scattered across cybernetics, automata theory, information theory and psychology acquired one banner, and a banner is what lets money, departments and journals form around a subject. The assumption matters more still. Their conjecture, that every feature of intelligence can in principle be described precisely enough for a machine to simulate it, is not a finding and was never proved. It is a bet, written into a grant application in 1955, and every system described on this page is built on top of it.

The proposal lists seven topics for the study, and the list reads as a table of contents for the next seventy years: automatic computers, how to program a computer to use a language, neuron nets, theory of the size of a calculation, self-improvement, abstractions, and randomness and creativity.

Item two is natural language processing. Item three is connectionism, the thread this page follows. Item four anticipates computational complexity. Item five is machine learning. Four fields, each of which became a discipline with its own conferences, are one bulleted list in a 1955 grant application.

The correction, which is the point of this entry

The term artificial intelligence is very widely said to have been used for the first time at the 1956 Dartmouth workshop. It is in the title and the opening sentence of the written proposal of 31 August 1955, which is a document anyone can read. On the first page alone it appears four times, in the title, the running head, the opening sentence and a section lead-in.

The Oxford English Dictionary, in its 2023 revision, dates the phrase to exactly this document and quotes it as the earliest evidence. Merriam-Webster gives a first known use of 1947 and publishes no supporting citation for it, and the phrase does not appear in Turing's 1947 lecture, his 1948 report or his 1950 paper.

But the stronger version of that claim is wrong

It is tempting to say the phrase had never appeared in print before 1955. It had. The exact words, applied to a machine mind, appear in a science fiction story by Edmond Hamilton published in Startling Stories in January 1951, describing a robot with an artificial intelligence equal to the human.

So the accurate claim is narrower and still worth making. The 1955 proposal is the earliest documented use of the phrase as the name of a research field, and that is what the Oxford English Dictionary is dating.

One further nuance historians have noticed, and this page reports it as an inference rather than a finding. The proposal uses the phrase completely unmarked, with no quotation marks and no explanation. A newly coined term is normally flagged on first use, which suggests it may already have had some currency in conversation before anyone wrote it down.

The opening paragraph, in full

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, 31 August 1955

Who the four were, as printed

J. McCarthy, Dartmouth College. M. L. Minsky, Harvard University. N. Rochester, I.B.M. Corporation. C. E. Shannon, Bell Telephone Laboratories.

The body of the document gives fuller titles. McCarthy is an assistant professor of mathematics, Minsky is a Harvard Junior Fellow in mathematics and neurology, Rochester manages information research at IBM in Poughkeepsie, and Shannon is a mathematician at Bell Labs.

Two things worth noticing. Minsky is at Harvard, not MIT, which is where he is usually placed, and he is a junior fellow rather than a professor. And Shannon, who appeared in the previous era of this page measuring the predictability of English, is one of the four. The two threads on this page were never as separate as the retelling makes them.

The money, and what the money actually bought

The proposal's own budget page totals $13,500. The figure usually quoted, $7,500, is what the Rockefeller Foundation awarded, which is a little over half of what was asked for.

The Foundation also cut the duration, funding five weeks rather than the two months requested. Robert Morison's letter to McCarthy of 30 November 1955, in the Rockefeller Archive Center, gives the reason: there was a great deal of hesitancy about risking any very substantial amount at that stage.

A sourcing note for anyone checking this. The budget page is a separately inserted leaf in the typescript, and it is not included in the 2006 reprint of the proposal in AI Magazine. Citing that reprint for the $13,500 will not work.

What they expected, which is worth noticing

It is also worth noticing what they expected. Ten people. Two months. One summer. Language, abstraction, problem solving and self improvement. Seventy years later the first item on that list is arguably done and the last one is not.

Sources

• McCarthy, Minsky, Rochester and Shannon. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, 31 August 1955. Scan of the original typescript, from Ray Solomonoff's copy, in which the budget appears as a physically inserted page numbered 5a. The Stanford transcription reproduces the same budget content source
• The same proposal as transcribed and hosted by John McCarthy at Stanford source
• Shubinski, B. A Roomful of Brains: Early Advances in Computer Science and Artificial Intelligence. Rockefeller Archive Center, 6 January 2022. Source for the $7,500 award, the five week reduction and the Morison letter of 30 November 1955. Note that this article miscalls Rochester Nathan; his name is Nathaniel source
• Oxford English Dictionary, artificial intelligence, n. Dates the phrase to 1955 and cites this proposal by title as its earliest quotation. SUBSCRIPTION ONLY: this link redirects readers without a subscription to the OED home page, so it cannot be checked by everyone. It is corroboration rather than the load-bearing source, because the proposal itself is linked above and anyone can read the term in its title source
• Hamilton, E. Moon of the Unforgotten. Startling Stories, volume 22 number 3, January 1951, page 120. The 1951 appearance of the exact phrase, applied to a fictional robot source
• Merriam-Webster, artificial intelligence: first known use 1947, with no supporting quotation shown source
• Merriam-Webster on how it dates entries: such dates are provisional and revisable as earlier evidence emerges source

1956 · The Dartmouth Summer Research Project

Hanover, New Hampshire, 18 June to 17 August

The meeting the proposal paid for. It did not go to plan, and it worked anyway.

What it contributed

The meeting the money paid for, and it is worth being honest that its research output was thin. It produced no breakthrough, and nothing resembling the significant advance the proposal had promised. What it produced was a community. Over one summer it put McCarthy, Minsky, Rochester and Shannon in the same place as Ray Solomonoff, Trenchard More, Oliver Selfridge, Arthur Samuel, Allen Newell and Herbert Simon, and most of them went on to run the field for the next thirty years.

So its importance is social rather than technical, and that is not a lesser thing. A subject with a name, a founding document and a roster of people who know each other can attract students and funding. A subject without those cannot. The workshop is where the label acquired a constituency.

The most consequential program demonstrated that summer probably came from two people who were barely there. Allen Newell and Herbert Simon brought Logic Theorist, which proved theorems from Whitehead and Russell's *Principia Mathematica*, and it is widely regarded as the first working artificial intelligence program.

Accounts of how long they attended disagree, and McCarthy's own later recollection is that they came for a few days and were the stars of the show. It is a recurring feature of this history that the working system and the founding document come from different people.

The numbers, which are usually reported wrong

Duration. Very commonly given as six weeks. Trenchard More's contemporaneous record states the project ran from 18 June to 17 August 1956, which is roughly eight weeks. The six week figure traces to a later secondary account.

Attendance. Roughly twenty people passed through. Only three stayed the whole time: McCarthy, Minsky and Ray Solomonoff. Daily sessions typically drew between three and eight people. For the whole of the fifth week there were four.

A ten man study that was four men in its fifth week is a more useful picture of how research actually happens than the founding-moment version.

Why attendance lists disagree

Three lists survive and they count different things, which is the whole explanation for the confusion. Solomonoff's is a contemporaneous notebook page headed people at the summer research project, and it has twenty entries. More's longer list is by his own description ordered by degree of attendance, interest, centrality or suggested attendance. McCarthy's longer list again covers people who participated in, visited, or were known to be interested in the project. McCarthy lost his actual attendance list.

So being named on the two longer lists is not evidence of having been there. Herbert Simon and Allen Newell, who are on the contemporaneous list, attended only briefly, and accounts differ on exactly how briefly.

Why it mattered to what came next

Not for its results, which did not amount to the significant advance the proposal predicted. It mattered because it assembled the people who would run the field for the next thirty years and gave them a shared name to run it under.

The name is the durable output. A research program that had been scattered across cybernetics, automata theory, information theory and psychology acquired one label, and a label is what lets funding, departments and journals form around a subject.

The pattern to notice

This page's second era contains a naming, a spectacular press story about the perceptron, and then a book widely blamed for a decade of lost funding. Ambitious framing, heavy coverage, then a correction that overshoots. It has happened more than once since, and recognizing the shape is more useful than any individual date on this page.

Sources

• Solomonoff, R. J. Handwritten notebook page, August 1956, headed People at Summer research project. The contemporaneous attendance record, twenty entries source
• More, T. Jr. Attendance list with a covering note of 27 April 2002 stating that the project ran from June 18th to August 17th, 1956 source
• Solomonoff, G. Ray Solomonoff and the Dartmouth Summer Research Project in Artificial Intelligence, 1956. Oxbridge Research, 2016. Source for the three who stayed throughout, the three to eight daily figure and the correction of the six week claim source

1957 · John Rupert Firth

Professor of General Linguistics, School of Oriental and African Studies, London

Wrote the one sentence that every paper on word vectors still quotes.

What he contributed

Firth was a linguist working on how meaning behaves in actual use, and he objected to treating a word as a self-contained unit carrying a fixed sense you could look up. His alternative was collocation. Observe what a word habitually appears alongside, and you learn what it does, without ever needing a definition.

Frege had said that meaning must be sought in context. Firth said where to look, and in doing so turned a philosophical principle into something you could count. That is the operational step, and it is why he appears on this page rather than only in linguistics. Word2Vec, fifty six years later, is Firth's instruction carried out on a very large body of text by machine. Every embedding described in the pipeline is, at bottom, a tally of the company a word keeps.

The formal version is the distributional hypothesis, usually credited to Zellig Harris in 1954: difference of meaning correlates with difference of distribution. Operationally that means building a matrix of how often each word co-occurs with each other word, and then reducing it.

There is a result that closes the loop neatly. Levy and Goldberg showed in 2014 that the skip-gram method with negative sampling, which looks like a neural network, is implicitly factorizing a matrix of shifted pointwise mutual information between words. Under the conditions of their derivation, the neural approach and the counting approach turn out to be doing the same arithmetic.

The line, exactly as he wrote it

You shall know a word by the company it keeps!

J. R. Firth, A Synopsis of Linguistic Theory 1930 to 1955, in Studies in Linguistic Analysis, Oxford: Basil Blackwell, 1957, page 11

A small point of accuracy

The exclamation mark is in the original. Almost every citation drops it. If the line is being quoted, it should either be kept or the change should be noted.

What he actually meant

Firth was writing about collocation, one narrow mode of meaning among several he distinguished. His broader notion of context was the context of situation, taking in the speakers, the objects present and the whole setting in which words are used.

Distributional semantics inherited the slogan and not the theory. Firth was the first professor of general linguistics in Britain, and his interest was in what surrounds an utterance, not only in what surrounds a word in a text file.

Why it mattered to what came next

It is the operational form of Frege's principle from the first era on this page. Frege said meaning must be asked after in context. Firth said the company a word keeps is where you look. Word2Vec, fifty six years later, is that instruction carried out by counting.

Sources

• Firth, J. R. A Synopsis of Linguistic Theory, 1930 to 1955, in Studies in Linguistic Analysis (special volume of the Philological Society), Oxford: Basil Blackwell, 1957. The quoted line is conventionally cited to page 11. The only accessible scan is lending restricted, so we could not confirm that page against the volume itself
• Library of Congress name authority record: Firth, J. R. (John Rupert), 1890 to 1960 source

1958 · Frank Rosenblatt

Cornell Aeronautical Laboratory, Buffalo

Built an early system that learned from examples, and then had its capabilities described to the press in terms it could not possibly meet.

What he contributed

Rosenblatt supplied the half that McCulloch and Pitts had left out. Their units had thresholds fixed by hand. His had adjustable weights and, decisively, a rule for changing them. Show the system an example, compare its answer with the correct one, and nudge the weights in the direction that would have been less wrong. Repeat. He also proved a convergence result: for the class of problems a single layer can represent at all, this procedure is guaranteed to find a solution.

That is the first working demonstration that a machine can be trained rather than programmed, and it is the distinction the whole of the build phase in the pipeline section rests on. Nobody wrote down what features to look for. The system found them from examples. What Rosenblatt could not do was train more than one layer, and that limitation is what the next entry made famous and what the third era of this page finally resolves.

The convergence theorem is precise and its condition is the whole story. If the training examples are linearly separable, meaning a single flat boundary can divide them, the perceptron rule is guaranteed to find such a boundary in a finite number of updates. The clean bound, which depends on how wide the margin is, was proved by Novikoff in 1962 rather than in this paper.

If they are not linearly separable, the rule does not converge and it does not degrade gracefully either. It cycles. That is a sharp qualitative limit rather than a matter of needing more data or more time, and it is exactly the limit the next entry formalizes.

The separation that most accounts collapse

The 1958 paper and the Mark I Perceptron machine are two different artifacts, years apart.

The paper, in Psychological Review, reports results from simulations run on an IBM 704, in its own words. The custom hardware, a six rack machine with a 400 unit sensory grid, came later and was first publicly demonstrated on 23 June 1960. It is now in the Smithsonian. Secondary sources date the Mark I to 1958, 1960 and 1962, and 1960 is the only one supported by contemporaneous artifacts, including the machine's own operators' manual.

Sources

• Rosenblatt, F. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review 65(6), 386 to 408, 1958. DOI 10.1037/h0042519. Scan source
• Hay, Lynch and Smith. Mark I Perceptron Operators' Manual, Project PARA, Cornell Aeronautical Laboratory, report VG-1196-G-5, 1960. Catalog record source
• Lefkowitz, M. Professor's perceptron paved the way for AI, 60 years too soon. Cornell Chronicle, 25 September 2019. Carries the 1958 demonstration and the overclaim source

1958 · The New York Times report

A UPI wire story, 8 July 1958, page 25

The moment the field learned what happens when a research result meets a press conference.

What this entry is doing here

The contribution here is negative, which is exactly why it belongs. A wire reporter attended a Navy press conference, watched a program on a rented mainframe learn to sort punched cards, and wrote it up as the embryo of a machine that would walk, talk, see, write, reproduce itself and be conscious of its existence.

This is the first large-scale instance of a pattern that has recurred in every cycle of this field since, including the present one. A real but modest result meets an institution with an incentive to describe it maximally, and the description travels much further than the result. The correction, when it arrives, tends to overshoot in the other direction. The 1969 book two entries below, and the decade of withdrawn funding usually blamed on it, are in part a correction to this article.

The task the reporters watched was distinguishing cards marked on the left from cards marked on the right. That is linearly separable by construction, which is precisely the class the perceptron convergence theorem guarantees it can learn.

So the demonstration was a valid confirmation of a theorem with a known and narrow scope, and no evidence at all for anything printed about walking, talking or consciousness. The gap between the two was available to anyone who read the paper, in 1958.

What was printed

The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

New Navy Device Learns By Doing, The New York Times, 8 July 1958, page 25. The sentence opens the article under the dateline WASHINGTON, July 7 (UPI). The dateline is given here as attribution rather than inside the quotation, because we could not confirm the punctuation the original print edition used to separate it.

What had actually been demonstrated

A program running on an IBM 704, fed punch cards, which learned to tell cards marked on the left from cards marked on the right after about fifty tries. That is a real and interesting result. It is not a machine that will be conscious of its existence.

Three details worth having right

The press conference was on 7 July and the article ran on 8 July. Citations giving 7 July as the publication date have confused the dateline with the print date.

It was a UPI wire report, not a staff written piece, which goes some way to explaining the register.

The famous sentence is the reporter's account of what the Navy expected, not a quotation from Rosenblatt. Attributing those words to him directly, which is common, is not supported.

Why it is on this page at all

Because the gap between what was built and what was reported is the oldest recurring feature of this subject, and the same gap is the reason a page like this one needs to exist. The 1969 book in the next entry, and the decade of lost funding usually attributed to it, are in part a correction to this article.

Sources

• New Navy Device Learns By Doing; Psychologist Shows Embryo of Computer Designed to Read and Grow Wiser. The New York Times, 8 July 1958, page 25. UPI wire report, datelined Washington, July 7. Reproduced in an archival collection, which needs JavaScript to display source
• Note on sourcing: the New York Times archive page for this article could not be opened directly. The wording above was corroborated against two reproductions that agree without variance, an archival collection copy and the Cornell Chronicle retrospective, both linked here.

1969 · Marvin Minsky and Seymour Papert

MIT

Proved something narrower than the legend, and are blamed for something historians do not agree they caused.

What they contributed

Minsky and Papert did for the perceptron what nobody had done in the decade of enthusiasm around it. They analyzed it mathematically instead of arguing about it. They introduced the notion of the order of a problem, meaning the minimum number of input points that any single unit must take in at once, and then proved hard limits. Parity requires the maximum possible order. Connectedness, deciding whether a shape is joined up, requires an order that grows with the size of the image.

That is a permanent and valuable piece of mathematics, and it gave the field its first precise account of what a single layer of adjustable weights can and cannot represent. Their book is nonetheless remembered for two things it did not do. It is remembered as the XOR book, which misdescribes the theorems, and as the book that killed neural network research, which historians dispute. Both are addressed below.

Their central device is the order of a predicate: the smallest number k such that the predicate can be computed as a threshold over functions each looking at no more than k points of the input. Order one means every unit needs to see one point. High order means some unit must take in a large share of the image at once, which defeats the purpose of a local, parallel architecture.

Parity over n points has order n, the maximum. Connectedness has order that grows without bound as the image grows. Both results are exact and both remain true. What their order theorems do not cover is the case where the intermediate units are themselves learned rather than fixed, which is the multilayer case and the one that eventually mattered. They do discuss multilayer machines in the book, and are pessimistic about them. The sharp results are about fixed predicates, and no learning procedure for trained hidden units existed yet for them to analyze.

What it does not prove

The results are about a single layer of adjustable weights. The book proves nothing about multilayer machines. On those it offers a judgment, and the judgment is carefully hedged.

The much quoted passage, in full

The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations. It has many features to attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgment that the extension is sterile.

Minsky and Papert, Perceptrons, MIT Press, chapter 13

Did it kill the field

This is contested among historians and the page does not pick a side.

Against the legend. Mikel Olazaran's study argues the book was a marker of a controversy already ending rather than its cause, noting that most researchers and most funding had already moved toward symbolic approaches before publication.

For the legend. Other participants held that the book worked as a burial, and that a narrow proof was read as a general verdict.

What is not in dispute is that the method which answered the question, training a network with more than one layer, was worked out over the following seventeen years and is the subject of the next era on this page.

Sources

• Minsky, M. L. and Papert, S. A. Perceptrons: An Introduction to Computational Geometry. MIT Press, first edition 1969, corrected printing 1972. The claims on this page rest on those printings. No publisher record for them is offered here, because MIT Press now serves only the 2017 reissue of the 1988 expanded edition (DOI 10.7551/mitpress/11301.001.0001), which carries 1988 front matter that reframes the argument at issue
• Olazaran, M. A Sociological Study of the Official History of the Perceptrons Controversy. Social Studies of Science 26(3), 611 to 659, August 1996. DOI 10.1177/030631296026003005. Paywalled source
• Note on sourcing: the chapter 13 passage was traced to page images of the 1969 and 1972 printings by Quote Investigator rather than read from the book directly by us. The sentence immediately preceding it, which also circulates, is not reproduced here because it was not verified.

1960 · Henry J. Kelley, and Stuart Dreyfus in 1962

Aeronautics and applied mathematics

Working on optimal flight paths, not on learning machines, they produced the gradient methods that the later work would rediscover.

What they contributed

Kelley was working on optimal flight paths and Dreyfus on the numerical solution of variational problems. Neither had any interest in learning machines. But the problem they were solving is structurally identical to the one training a network poses: you have a system with a great many adjustable quantities and a single measure of how badly it is performing, and you need to know which way to move every quantity to improve that measure. Their answer was to compute the gradient, the direction of steepest improvement, and take a step along it. Dreyfus's 1962 derivation uses nothing but the chain rule, which is the form a modern reader recognizes on sight.

So the mathematics of training a neural network was published, in a different discipline, before anyone thought to apply it to one. This is the reason the priority question in this era is hard rather than merely disputed. The idea did not have a single origin. It had several, in fields that were not reading each other's journals.

In modern terms both papers compute the gradient of a cost with respect to control variables in a staged system, by propagating an adjoint variable backward through the stages. Map a stage onto a layer, a control onto a weight and the terminal cost onto the loss, and the equations are the same equations.

That correspondence is exact rather than a loose analogy, and it explains a persistent feature of the literature. Control theory calls the backward quantity the adjoint state. Deep learning calls it the error signal. They are the same object, and control theory reached it first, working on a problem that had nothing to do with learning machines.

Why it mattered to what came next

The mathematics of adjusting many parameters to reduce an error was in place, in a different discipline, before anyone applied it to learning. This is the reason the priority question is hard rather than merely disputed. The idea did not have one origin.

State it carefully

Calling this work a precursor is accurate. Calling it backpropagation is not. It is optimal control, and the people doing it were not trying to build a learning machine.

Sources

• Kelley, H. J. Gradient Theory of Optimal Flight Paths. ARS Journal 30(10), 947 to 954, October 1960. DOI 10.2514/8.5282 source
• Dreyfus, S. E. The numerical solution of variational problems. Journal of Mathematical Analysis and Applications 5(1), 30 to 45, August 1962. DOI 10.1016/0022-247X(62)90004-5 source

1970 · Seppo Linnainmaa

University of Helsinki

A master's thesis in Finnish, about rounding errors, containing the general method that later became known as reverse mode automatic differentiation.

What he contributed

A master's student in Helsinki was working on a problem in numerical analysis. When a long calculation runs in floating point arithmetic, a tiny rounding error is introduced at every step, and those errors accumulate. He wanted to know how much each individual error contributed to the total at the end. Answering that requires an efficient way to compute how one final quantity depends on each of a great many intermediate ones, and the method he set out works backward through the calculation, computing all of those sensitivities in a single pass rather than one at a time.

That is the mathematical engine of backpropagation, in print in 1970. It is not described as such, there is no neural network anywhere in the document, and almost nobody read it, because it was sixty six pages, in Finnish, about rounding error. The efficient method existed and the field that needed it had no idea.

The asymmetry that makes training possible: reverse mode computes the derivatives of one output with respect to all n inputs at a cost of a small constant multiple of one forward evaluation, no matter how large n is. Forward mode costs one pass per input.

For a network with a single scalar loss and hundreds of billions of weights, that is the difference between one backward pass and hundreds of billions of passes. Without this result, training by gradient descent would not be slow. It would be arithmetically out of reach.

Why it mattered to what came next

The efficient method for computing how each of a great many inputs affected a final result was, from this point, in the literature. Almost nobody read it. It was 66 pages, in Finnish, about a numerical analysis problem.

The honest framing

Describing this as reverse mode automatic differentiation is substantively right and is retrospective language. Linnainmaa was not writing about differentiation as his topic and he was certainly not writing about machine learning. Credit for the method is fair. Credit for the application is not.

Sources

• Linnainmaa, S. Algoritmin kumulatiivinen pyöristysvirhe yksittäisten pyöristysvirheiden Taylor-kehitelmänä. Master's thesis, University of Helsinki, 1970, 66 pp. Scan source
• Linnainmaa, S. Taylor expansion of the accumulated rounding error. BIT 16(2), 146 to 160, June 1976. DOI 10.1007/BF01931367 source

1974 · Paul Werbos

Harvard University, doctorate in statistics

A doctoral thesis that independently contained the general method. What his committee would not permit was for neural networks to be a major part of it.

What he contributed

Werbos arrived at the same method independently, and unlike Linnainmaa he arrived at it while thinking about intelligence. His thesis develops what he called ordered derivatives and dynamic feedback: a systematic procedure for working out how a final outcome depends on every parameter in a large connected system, by propagating sensitivity backward through the whole structure.

This is the second independent discovery of the method inside five years, and the first by someone who had learning machines in view from the start. He has said the idea came to him from questions about the phenomenon of intelligence and was applied to statistics only afterward. What his thesis was not permitted to be was a neural network thesis. The degree is in statistics, and the application had to stay in a few lines of chapter 5.

Werbos's notation of ordered derivatives makes explicit something the ordinary chain rule leaves implicit once a system has shared intermediate quantities. If an early variable influences the output along several different routes, the total derivative has to sum the contribution of every route.

That path summation is exactly what the backward pass accumulates as it moves through a network, and it is why the method is as much a bookkeeping result as a calculus one. The mathematics was never hard. Organizing it so it can be computed once rather than re-derived per weight is the contribution.

Why the credit is contested, and stays contested

Werbos states that backpropagation was first spelled out explicitly and implemented in his 1974 thesis, and the IEEE Neural Network Society cited that thesis when it gave him a Pioneer Award in 1994.

Jürgen Schmidhuber, who has documented this history at length, holds that the first neural network specific application came in Werbos's later work rather than in the 1974 thesis.

The detail that settles the narrow question

Harvard responded as follows to this proposal and to later discussions. First, they would not allow ANNs as such to be a major part of the thesis, since I had not found anyone willing to act as a mentor for that part. (I put a few words into chapter 5 to specify essential ideas, but no more.)

Paul Werbos, Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities

So where does that leave it

Werbos's priority for the general method, in a thesis published in 1974, is solid. The stronger claim that the thesis is a neural network paper is contradicted by his own account of what his committee permitted. Both of those things are true at once, which is what a contested attribution actually looks like.

One precision point, because this page has been checked on it. The thesis is not entirely free of the idea. Werbos says he put a few words into chapter 5, and a search of all 453 pages finds a short speculative passage on backward information flow in the brain. So the accurate statement is that neural networks were not permitted to be a major part of it, not that they are absent.

Sources

• Werbos, P. J. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University, August 1974. Scan source
• Werbos, P. J. Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities (his own account). In Automatic Differentiation: Applications, Theory, and Implementations, Springer, 2006, 15 to 34. DOI 10.1007/3-540-28438-9_2. The link is the author manuscript source
• Schmidhuber, J. Who Invented Backpropagation? (the opposing account, by a documented partisan in this dispute) source

1986 · David Rumelhart, Geoffrey Hinton and Ronald Williams

UCSD and Carnegie Mellon

The four page Nature letter that did not invent backpropagation and did something arguably more consequential: it showed the method produces useful internal representations.

What they contributed

By 1986 the method had been discovered independently more than once and the field still was not using it. What this four page letter supplied was the missing evidence. They took a network with a layer of units sitting between the input and the output, units to which nobody had assigned any meaning, trained it with the procedure, and then looked at what those hidden units had turned into. They had organized themselves into representations of features of the problem. Nothing had specified that structure in advance. The network had found it, because finding it made the predictions better.

That is the result that changed the field's mind, and it is worth being exact about why. Minsky and Papert had established what a single layer cannot do, and nobody doubted that more layers would be more powerful. The open question was whether more layers could be trained at all, or whether working out which of millions of weights deserved the blame for a wrong answer made deep networks a permanent theoretical curiosity. This paper answered that question empirically, in Nature, in four pages, with a demonstration others could reproduce. Within a few years the field had reorganized around it.

Everything in the build phase of the pipeline section is this procedure, running. It is the reason a system with billions of parameters can be improved by something other than guesswork, and the reason pre training is a procedure rather than an aspiration.

The mechanism is the generalized delta rule. Replace the perceptron's hard threshold with a differentiable function, in this paper the logistic `y = 1 / (1 + e^-x)`, so the error becomes differentiable everywhere, then apply the chain rule backward layer by layer. They also introduce a momentum term, carrying a fraction of the previous weight change into the next, to speed convergence.

Their two demonstrations are worth knowing because they are diagnostic rather than showy. One network learns to detect mirror symmetry in its input and solves it with just two hidden units. The other is given two isomorphic family trees, one English and one Italian, as a set of relationship triples. Its hidden units, which nobody instructed, come to encode nationality, which generation a person belongs to, and which branch of the family they come from.

And they state the drawback themselves, in the paper: the error surface may contain local minima, so gradient descent is not guaranteed to find a global minimum. Their defense is empirical, that in practice networks rarely get stuck anywhere much worse. Forty years on, that is still roughly the state of the argument.

The correction, first

This paper is very widely described as the invention of backpropagation. It is not, and the priority chain above is why. What is true is that this is the paper after which the field paid attention.

What the paper says about who came first

This is worth being exact about, because it is often reported incorrectly in both directions. The Nature letter carries four references in total and one sentence on priority, quoted below. Werbos, Linnainmaa, Kelley, Dreyfus and Amari appear nowhere in it, and they appear nowhere in the authors' longer chapter in the Parallel Distributed Processing volumes either.

Note the wording carefully. The authors claim independence, not primacy. They did not assert that they were first, and they did not credit those who were.

The only priority sentence in the paper

Variants on the learning procedure have been discovered independently by David Parker (personal communication) and by Yann Le Cun.

Rumelhart, Hinton and Williams, Nature 323, 9 October 1986

One trap worth naming

Frank Rosenblatt used the phrase back-propagating for a different procedure in the early 1960s. Any account that cites Rosenblatt as an early source for backpropagation in the modern sense is making a real error, and it is a common one.

Sources

• Rumelhart, D. E., Hinton, G. E. and Williams, R. J. Learning representations by back-propagating errors. Nature 323(6088), 533 to 536, 9 October 1986. Author-hosted scan of the printed pages source
• Bibliographic record, DOI 10.1038/323533a0 source
• Rumelhart, Hinton and Williams, Learning Internal Representations by Error Propagation, chapter 8 in Parallel Distributed Processing vol. 1, MIT Press, 1986 (machine-searched for the predecessors named above, zero hits), 318 to 362, eds. Rumelhart and McClelland. The chapter has no reference list of its own, so the zero-hit result is a statement about its body text source

1966 · Joseph Weizenbaum

MIT, Department of Electrical Engineering

Wrote ELIZA to demonstrate how shallow machine conversation was, and spent the rest of his life dismayed by how readily people believed it.

What he contributed

Weizenbaum built ELIZA to demonstrate how shallow machine conversation was. The program had no model of meaning and no memory of what had been said in any interesting sense. It matched keywords and applied rules that turned your own sentence back at you as a question. He chose the Rogerian therapist setting deliberately, because it is the one conversational context in which reflecting a statement back is normal rather than evasive. The technique was engineered to sit exactly where its poverty would not show.

Technically almost nothing on this page descends from ELIZA. Its importance is what it revealed about the audience rather than about the program. People who knew precisely what it was formed attachments to it anyway, and Weizenbaum spent much of the rest of his career arguing about what that implied. The gap between how capable a system is and how capable it feels was first measured here, in 1966, on a program with no understanding at all.

The DOCTOR script is a set of decomposition and reassembly rules keyed to ranked keywords. An input is matched against a pattern containing wildcards, the captured fragments are slotted into an output template, and a small table swaps pronouns so that your me becomes its you.

There is no parse, no grammar and no state beyond a short queue holding a few earlier fragments to fall back on when no keyword matches. That is the entire mechanism, and Weizenbaum's point was that people responded to it anyway.

The distinction almost everyone collapses

ELIZA was the program. The therapist personality everyone remembers was a script that ran on it, later given the name DOCTOR. Weizenbaum's own design point was that the script was data rather than part of the program, so the same machinery could be given any role at all.

The name DOCTOR does not appear in the 1966 paper. It is introduced in his follow up paper of 1967. Attributing the name to the 1966 paper is a small error and a common one.

What he found instead, in his own words

My secretary watched me work on this program over a long period of time. One day she asked to be permitted to talk with the system. Of course, she knew she was talking to a machine. Yet, after I watched her type in a few sentences she turned to me and said 'Would you mind leaving the room, please?' I believe this anecdote testifies to the success with which the program maintains the illusion of understanding.

Weizenbaum, Contextual Understanding by Computers, Communications of the ACM, August 1967, pages 477 to 478. The passage straddles the page break and the closing line falls on 478

And what he concluded, a decade later

What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.

Weizenbaum, Computer Power and Human Reason, 1976, introduction

Sources

• Weizenbaum, J. ELIZA, A Computer Program For the Study of Natural Language Communication Between Man And Machine. Communications of the ACM 9(1), 36 to 45, January 1966. Scan. DOI 10.1145/365153.365168 source
• Weizenbaum, J. Contextual Understanding by Computers. Communications of the ACM 10(8), 474 to 480, August 1967. This is the source of the DOCTOR name and of the secretary passage quoted above. DOI 10.1145/363534.363545 source
• Weizenbaum, J. Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman, 1976. The delusional thinking sentence is quoted from the introduction. The exact page could not be confirmed against the book itself, which is access restricted, so no page number is given here

1971 · Kenneth Colby

Psychiatrist, Stanford Computer Science Department

Built PARRY, which unlike ELIZA had an inner state, and then ran the test ELIZA never had to face.

What he contributed

Colby was a practicing psychiatrist, and his interest was not conversation. He wanted a working model of paranoid thinking, something that would let him test whether a formal account of the condition actually reproduced its behavior. PARRY therefore has what ELIZA conspicuously lacks: an internal state. It carries levels of fear, anger and mistrust that shift as a conversation proceeds and then shape what it says next. Ask a hostile question and the fear level rises, and a raised fear level changes the reply.

Thin as that is, it is a model of something rather than a mirror held up to the user. Colby then did what almost nobody in the field was doing at the time. He tested it, having experienced psychiatrists try to tell the program apart from real patients. It is the earliest serious attempt to evaluate a conversational system rather than simply to exhibit one, and the design of that test is still argued about.

PARRY carries numeric affect variables for fear, anger and mistrust which update on each input and then select among classes of response by threshold. It also holds a model of its own delusional narrative, so probing certain topics raises fear and pushes it toward evasion or hostility.

Architecturally this is closer to a finite-state machine with counters than to anything statistical, and it learns nothing. But persistent state that modulates output is a real structural difference from ELIZA, and it is why PARRY holds up over a longer conversation where ELIZA does not.

Two dates that get muddled

PARRY is routinely dated to 1972. The paper describing the working model, Artificial Paranoia by Colby, Weber and Hilf, was accepted on 4 January 1971 and published that year, with interview transcripts already in it. The name PARRY does not appear in that paper. Safer to say it was developed at Stanford in the early 1970s and first described in print in 1971.

What it actually simulated

Commonly described as simulating paranoid schizophrenia. Colby was more careful than that. He restricted his own use of paranoid to a mode of thinking, feeling and acting, and he stated plainly that he was not simulating any actual human case.

The persona was a 28 year old single man working as a post office clerk who, after a quarrel with a bookie, believed the underworld intended to have him injured or killed.

The test, reported accurately

Colby and colleagues ran an indistinguishability test in 1972. Experienced psychiatrists interviewed both real patients and the program over teletype, and a second larger group read transcripts. The overall result was that the judges did no better than chance at telling the program from the patients.

The frequently quoted figure of 48 percent correct belongs to a separate follow on test in which psychiatrists were sent two protocols and asked which was the program. It is worth knowing that this number is normally cited to a review published in 2000 rather than to the original paper, which is not openly accessible.

Sources

• Colby, K. M., Weber, S. and Hilf, F. D. Artificial Paranoia. Artificial Intelligence 2(1), 1 to 25, 1971. Full text. DOI 10.1016/0004-3702(71)90002-6 source
• Colby, K. M., Hilf, F. D., Weber, S. and Kraemer, H. C. Turing-like indistinguishability tests for the validation of a computer simulation of paranoid processes. Artificial Intelligence 3, 199 to 221, 1972. Not open access. Metadata record source
• Saygin, Cicekli and Akman. Turing Test: 50 Years Later. Minds and Machines 10(4), 2000, at pages 500 to 501. This is the accessible source for the experimental design and the 48 percent figure. DOI 10.1023/A:1011288000451 source

1972 · PARRY meets the DOCTOR

Written up by Vint Cerf as RFC 439

The first recorded conversation between two chatbots, conducted over the ARPANET, and it goes about as well as you would expect.

What it contributed

This is an experiment rather than a piece of engineering, and it is one of the more instructive accidents in the field's history. Two programs, each of which produced a reasonably convincing impression against a cooperative human, were connected over the ARPANET and pointed at one another. One was simulating paranoia. The other reflected every statement back as a question.

With the human removed, the exchange collapses, and that is the finding. Whatever competence these systems appeared to have was being supplied by the person at the other end, who was interpreting generously and forgiving the gaps. It is worth carrying into the rest of this page, because the same question can be asked of systems that are very much more capable.

What the transcript shows is how little either program carries forward. ELIZA has almost nothing: the short fallback queue described above and no model of the conversation at all. PARRY has more, the affect variables and the delusional narrative, but that is a handful of hand set numbers rather than a representation of what has been said. Neither builds anything that could be called an account of the exchange, and a transcript can only show that; it cannot prove the absence of state.

Put two such systems in a loop and neither has a model of the other to build on, so the exchange drifts across response templates rather than developing. The contrast with the context window in the pipeline section is the point: a modern model conditions on everything still inside its context window, which is why its exchanges hold together for as long as they do, and why they begin to lose the thread once a conversation outruns that window.

The two dates, both correct

The conversation took place on 18 September 1972. The write up, RFC 439, titled PARRY Encounters the DOCTOR, was published on 21 January 1973. Sources citing either date are right about different events, which is why they appear to disagree.

Sources

• Cerf, V. RFC 439, PARRY Encounters the DOCTOR, 21 January 1973. The document itself records the conversation as having taken place on 18 September 1972. DOI 10.17487/RFC439 source

1995 · Richard Wallace

A.L.I.C.E. and AIML

Showed how far hand written rules could be pushed, by writing a great many more of them than anyone had before.

What he contributed

Wallace took the rule-based approach as far as it can be taken, and documented where the ceiling is. ALICE runs on AIML, a markup language in which every rule pairs an input pattern with a response template, and his real contribution was less the language than the process built around it: an infrastructure that let hundreds of volunteers add rules, watch which ones fired against real conversations, and add more where the coverage failed. The brain grew to roughly 41,000 categories over a decade of that.

It is the definitive negative result for hand written rules, and considerably more useful than a success would have been. Every category was written by a person, so coverage grows in proportion to human effort while language does not. The approach improves steadily and never arrives. Set it against the pipeline section for the contrast that matters: a modern model's language ability contains no hand written rules at all, and its coverage came out of a procedure run over text.

AIML matching walks a trie built from the pattern set, depth first, with several wildcard types ranked by specificity so that more literal patterns win. The matching template may then redirect into another pattern, which is how the rule set composes.

The algorithm is simple and fast. All the complexity lives in the pattern set, which is why it does not scale: extending coverage means a person adding nodes to the trie, and the trie has to anticipate the surface form a user will type rather than what they mean. Every paraphrase is a new node.

The number, stated properly

The figure usually quoted is 41,000 handwritten patterns. Wallace's own words are that the brain contains roughly 41,000 elements called categories, and he explicitly marks it as a count at the time of writing, in a chapter published in 2007.

So it is categories rather than patterns, it is approximate rather than exact, and it describes ALICE after a decade of accumulation rather than at the 1995 release. He also records that more than 500 volunteers contributed, so handwritten by Wallace alone is not right either.

One precision point, because the distinction gets blurred

Worth being precise, because the distinction gets blurred. Hand written instruction has not disappeared, it has moved. A system prompt is hand written, and so are safety policies and tool descriptions. Those now steer a capability the model already has, rather than constituting the capability itself.

Sources

• Wallace, R. S. The Anatomy of A.L.I.C.E., in Parsing the Turing Test, Springer, 181 to 210. Source of the 1995 date, the AIML description and the roughly 41,000 categories figure, in his own words, 2007. DOI 10.1007/978-1-4020-6710-5_13. The linked file is the author preprint, which carries no pagination source

2001 · SmarterChild

Built by ActiveBuddy

Put a chatbot in tens of millions of instant messenger buddy lists, twenty years before it happened again.

What it contributed

Not a research contribution but a distribution one. ActiveBuddy put a conversational program where people already were, in the instant messenger contact list, sitting alongside their actual friends. It answered questions about weather, sports scores and stock prices, and it would trade insults with you.

It is the first time a conversational program was a mass consumer product rather than a research artifact, and it is where tens of millions of people had their first conversation with a machine. It is also the end of the line. Nothing built after it made hand written rules go any further, and the next real advance came from the other thread on this page entirely.

Under the conversational surface it was a broker sitting in front of structured data feeds. It classified an incoming message into one of a modest set of intents, extracted a parameter or two, queried a weather, sports or finance service, and formatted the answer into a template.

That architecture is worth recognizing, because it is tool calling as described at the end of the pipeline section, with the language model removed. Classify the intent, call the interface, format the result. What changed in the twenty years since is not the shape of the system. It is that the classification and the formatting stopped being hand written.

The numbers, and why they are soft

Reported figures include over 17 million users and a billion queries a month at its peak. Those come from an investor account relayed by the Computer History Museum. A separate figure of 30 million is a creator account relayed by a technology publication, and it measures something different, being lifetime rather than peak.

None of it is audited, and the two headline figures are not consistent with each other. This page quotes the 17 million figure with its attribution visible and does not repeat the claim that it was the first chatbot millions used daily, which has no source at all.

Sources

• Hsu, H. SmarterChild: A Chatbot Buddy from 2001. Computer History Museum, 29 May 2025. Source for ActiveBuddy, the 2001 AIM launch, the later Yahoo and post-2004 MSN sequence, and the 17 million figure with its attribution source

1994 · Philip Gage

Software engineer

Published byte pair encoding as a data compression trick. Twenty one years later it became the basis of how most language models read text.

What he contributed

Gage published a compression algorithm with a single idea in it. Find the pair of adjacent bytes that occurs most often in the file, replace every instance of that pair with one byte not otherwise used, and repeat. It compresses because frequent sequences collapse to something short, and it was offered as a simpler alternative to the dominant method of the day at almost the same compression.

Nothing about it was designed for language, and that is precisely why it works so well for language. It counts what is frequent, and nothing else. Run it on text rather than arbitrary bytes and the units it discovers are common character sequences, which turn out to be the right compromise between whole words, of which there are unboundedly many, and single letters, which carry too little on their own. Every tokenizer described in the pipeline section is a descendant of this one procedure.

The merge operation is greedy, and its output therefore depends on the order in which merges were made. That order is saved as a numbered merge table, and applying the tokenizer later means replaying that table in sequence, which makes the process deterministic given the table.

Two consequences follow for anyone using a modern model. Vocabulary size is a chosen stopping point rather than a discovered property, and the merge table encodes the language mix of whatever corpus it was learned from. That is why token counts differ so sharply between English and languages that were thin in the training data.

Sources

• Gage, P. A New Algorithm for Data Compression. The C Users Journal 12(2), 23 to 38, February 1994. No DOI was ever registered. ACM Digital Library record source
• Note on sourcing: the original is a print magazine article and no DOI was ever registered for it. The full text is mirrored in the Dr. Dobb's archive of The C Users Journal source

2013 · Tomas Mikolov and colleagues at Google

Word2Vec, and the analogy that got overstated

Made word embeddings cheap enough to be used by everyone, and produced the demonstration that is quoted more often than it is checked.

What they contributed

The idea that a word could be represented as a position in a space was not new in 2013. What Mikolov and colleagues did was make it cheap. Their models are deliberately, almost aggressively simple, and that simplicity is the contribution: it let them train on billions of words on ordinary hardware in under a day, where comparable approaches had been far more expensive. They then released the trained vectors, so anyone could use them without training anything at all.

That is what moved this idea from a research technique into a standard tool that appeared in ordinary software. It also taught the field an expectation that turned out to matter enormously: that useful structure falls out of a very simple objective when you run it at scale. The same expectation, applied with a vastly larger budget, is what pre training later vindicated.

Two architectures. CBOW predicts a word from its surrounding context, and skip-gram predicts the surrounding context from the word. The speed does not come from the architecture, which is shallow by design, but from the training objective.

A straightforward version would need a softmax over the entire vocabulary at every single step, which is prohibitive. They avoid it with hierarchical softmax over a binary tree, or with negative sampling, which turns one decision over a vocabulary of hundreds of thousands into a handful of yes-or-no comparisons against randomly drawn words. That substitution is what made it cheap enough to matter.

The analogy, and where it really comes from

The famous demonstration is that king minus man plus woman lands near queen. It is usually credited to the Word2Vec papers. It is not in either of them as an original result. The second paper does not contain it at all. The first quotes it as prior work and cites its actual source: a separate 2013 paper by Mikolov, Wen-tau Yih and Geoffrey Zweig, presented at NAACL.

The demonstration is weaker than it looks

The standard scoring code, including Mikolov's own, removes the three input words from the list of candidate answers before choosing the closest match. Put them back and the result usually collapses onto one of the words you started with.

Nissim, van Noord and van der Goot measured this in Computational Linguistics in 2020: macro-accuracy on the standard test set falls from about 0.71 to about 0.21 for the standard method once the constraint is removed, and they use this exact example to show why, reporting that man is to king as woman is to king. Linzen measured the same thing in 2016 and found that unconstrained, the answer returned was the third input word in 93 percent of cases.

What survives the correction, which is most of it

The underlying claim is sound and is not seriously disputed. Embedding spaces really do encode semantic and syntactic relationships as directions, and later work has derived from first principles why roughly linear relationships appear at all, with explicit error terms. What does not survive is the clean arithmetic as a party trick.

The honest summary: the direction from king toward queen is meaningful. The equation is partly an artifact of how the test is scored.

Sources

• Mikolov, Chen, Corrado and Dean. Efficient Estimation of Word Representations in Vector Space. arXiv 1301.3781, 16 January 2013. DOI 10.48550/arXiv.1301.3781 source
• Mikolov, Sutskever, Chen, Corrado and Dean. Distributed Representations of Words and Phrases and their Compositionality. NIPS 2013, arXiv 1310.4546, vol. 26, 3111 to 3119. DOI 10.48550/arXiv.1310.4546 source
• Mikolov, Yih and Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL-HLT 2013, 746 to 751. This is where the king analogy actually appears. No DOI exists, as NAACL 2013 predates ACL DOI assignment source
• Mikolov's own evaluation code, compute-accuracy.c, showing the three input words being skipped in the candidate loop source
• Nissim, van Noord and van der Goot. Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor. Computational Linguistics 46(2), 487 to 497, June 2020. DOI 10.1162/coli_a_00379 source
• Linzen, T. Issues in evaluating semantic spaces using word analogies. RepEval 2016 (the 93 percent figure), 13 to 18. DOI 10.18653/v1/W16-2503 source
• Rogers, Drozd and Li. The (too Many) Problems of Analogical Reasoning with Word Vectors. *SEM 2017, 135 to 148. DOI 10.18653/v1/S17-1017 source
• Ethayarajh, Duvenaud and Hirst. Towards Understanding Linear Word Analogies. ACL 2019 (why the linear structure exists), 3253 to 3262. DOI 10.18653/v1/P19-1315 source

2014 · Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio

Jacobs University Bremen and Université de Montréal

Introduced the attention mechanism, three years before the paper that made it famous.

What they contributed

Translation systems of the day compressed an entire source sentence into a single fixed length vector and then generated the translation from it. This worked for short sentences and degraded badly as they got longer, because everything had to pass through one bottleneck of fixed size. Their fix was to remove the bottleneck. Let the system look back at all of the source positions at each step of producing the output, and learn which positions to weight for the word it is generating now.

That mechanism is attention, and it is the single most consequential component on this page. Here it appears as an improvement bolted onto a recurrent system, which is how everyone used it for the next three years. The transformer paper's move was to notice that the bolt-on was doing the work, and to throw away the thing it had been bolted onto.

Their alignment model scores each source position against the decoder's current state using a small feedforward network, normalizes those scores with a softmax, and builds the context vector as the weighted sum of source annotations. This is known as additive attention.

The transformer's change three years later was to replace that scoring network with a plain dot product divided by the square root of the key dimension. Less expressive per comparison, and far cheaper, because it reduces to a matrix multiplication and runs on hardware built for exactly that. The transformer paper makes the narrower claim, that dot product attention is faster and more space efficient in practice for that reason. What lets every position be scored against every other at once is a different change, the removal of recurrence.

A detail worth having right

The word attention is in the paper, in the original September 2014 version, in section 3.1. It is not in the title or the abstract, and the term was standardized by the work that followed rather than by this paper. So the mechanism and the word both originate here, and the name was settled later.

Sources

• Bahdanau, Cho and Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv 1409.0473, first submitted 1 September 2014, accepted at ICLR 2015 as an oral presentation. DOI 10.48550/arXiv.1409.0473 source

2015 · Rico Sennrich, Barry Haddow and Alexandra Birch

University of Edinburgh

Took Gage's 1994 compression algorithm and applied it to translation, which is how it entered language modeling.

What they contributed

Translation systems worked with a fixed vocabulary and language does not have one. Names, compounds, inflections, technical terms and anything rare fell outside the list and had to be handled by fallbacks that produced visibly worse output. Their move was to stop working in whole words. Break rare words into subword units discovered by Gage's compression algorithm, and the fixed vocabulary problem largely dissolves, because anything unfamiliar can be assembled from pieces the system already knows.

This is the direct ancestor of how every model on this page reads text. The vocabulary is still fixed and frozen before training begins, and it no longer needs to contain every word, only pieces that compose into any word. It is also a good example of how this field actually progresses, which is by borrowing a solved problem from somewhere else.

They test two ways of learning the merges: a separate table for each language, or one joint table over the union of both. The joint table worked better in their experiments, because it keeps the segmentation of a word that appears in both languages consistent, so the system can carry a name or a technical term across without relearning it.

The same choice sits inside most multilingual models today. One shared vocabulary means shared subwords and shared representation across languages, at the cost of spending vocabulary slots on scripts a given user may never touch.

A date to state precisely

The preprint appeared on 31 August 2015 and the paper was published at ACL in 2016. Both dates are correct for different things, which is worth knowing before quoting one of them.

Sources

• Sennrich, Haddow and Birch. Neural Machine Translation of Rare Words with Subword Units. arXiv 1508.07909, 31 August 2015, accepted at ACL 2016, published ACL 2016, 1715 to 1725. DOI 10.18653/v1/P16-1162 source

2016 · Yonghui Wu and colleagues at Google

Google Neural Machine Translation

The state of the art immediately before the transformer, and a clear picture of the limit it hit.

What it contributed

A production system rather than a new idea, and its contribution to this page is as a measurement of the ceiling. It is a deep recurrent network, eight encoder layers and eight decoder layers, using attention and residual connections, and it delivered a substantial jump in translation quality that millions of people noticed.

It shows precisely what the transformer was reacting against. A recurrent network has to process a sequence one position at a time, because position two cannot be computed until position one has been. That is a hard limit on how much training work can be spread out, and no amount of engineering removes it. This paper is the strongest version of the recurrent approach, which is exactly what makes it the right thing to measure the successor against.

The engineering worth knowing: residual connections between the stacked layers, which is what made a network eight layers deep trainable at all, bidirectional processing in the first encoder layer only, and attention computed from the bottom decoder layer to the top encoder layer, which the paper states was done to improve parallelism and so cut training time. The mechanism is worth stating precisely. A stacked recurrent decoder cannot run its layers simultaneously, because each one depends on the layer below it. What the wiring does is shorten the dependency chain, so the attention step does not wait on the top of the decoder stack and more of the work can be overlapped.

Almost all of that survives. The residual connection in particular reappears in the transformer block described in the pipeline section, doing the same job of keeping the gradient signal intact across depth.

Sources

• Wu, Y. et al. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv 1609.08144, 26 September 2016. DOI 10.48550/arXiv.1609.08144 source

2017 · Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin

Attention Is All You Need

Kept the attention mechanism, threw away the recurrence it had been attached to, and made training parallel.

What they contributed

Everyone in the field was adding attention to recurrent networks. This paper proposed removing the recurrent network and keeping only the attention, and then showed the result was better as well as faster to train. The architecture is built from a repeated block combining multi-head attention with a small network applied to each position on its own, plus a way of encoding position, which is needed because attention on its own treats its input as an unordered set.

The significance is not elegance. It is trainability. Dropping recurrence removed the requirement to process a sequence one position after another, so training could run across an entire example at once, and that is what made training at the scale of the following decade possible at all. Almost every system described in public as AI today is a descendant of this paper, which is why it closes the history and opens the pipeline.

One block is multi-head scaled dot-product attention followed by a small network applied at each position independently, with a residual connection and normalization around each. Attention itself is `softmax(Q K^T / sqrt(d_k)) V`, where Q, K and V are the query, key and value matrices.

The division by the square root of the key dimension is not decoration. Without it the dot products grow with dimension, pushing the softmax into a region where its gradients vanish and learning stalls. Position is supplied separately, in this paper by sinusoids of differing frequencies, because attention alone is indifferent to order.

The property the paper argues for is the one that mattered: cost per layer grows with the square of the sequence length, which is expensive, but the number of steps separating any two positions is constant, where a recurrent network needs a number of steps proportional to the distance between them. Constant path length is what makes long-range structure learnable.

In the paper's own words

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.

Vaswani et al., abstract, arXiv 1706.03762 v1, 12 June 2017

Who did what, and the version detail behind it

The published paper carries a footnote stating that all eight contributed equally and that the listing order was random. The same footnote credits Jakob Uszkoreit with proposing that recurrent networks be replaced with self-attention, and Noam Shazeer with scaled dot-product attention, multi-head attention and the position representation.

A detail that matters if you plan to quote it: that long footnote is not in the original June 2017 submission. It first appears in the version posted in December 2017. The June version's footnote says only that contribution was equal and listing order random.

Two claims about this paper that rot, and what to say instead

On citations. The figure is routinely quoted as over 100,000. As of 26 July 2026 Semantic Scholar records 186,264 and Google Scholar shows roughly 271,000. The databases disagree because they index different literature. Any bare number will be wrong within months. The durable statement is that it is among the most cited papers in computer science.

On the authors. The line that all eight have left Google is repeated constantly. It was true by 2023, then stopped being true in August 2024 when Noam Shazeer returned to Google to co-lead Gemini, and he announced a move to OpenAI on 17 June 2026. A sentence that has already flipped twice will flip again. The durable statement is that all eight went on to leave Google at some point, and several founded or now lead major AI companies.

One more detail worth having right

Aidan Gomez is printed on the paper with the affiliation University of Toronto, with a footnote recording that the work was performed while at Google Brain. He was an intern. Describing the paper as the work of eight Google employees is not quite accurate. Eight researchers working at Google is.

Sources

• Vaswani, A. et al. Attention Is All You Need. arXiv 1706.03762, v1 submitted 12 June 2017, 15 pages. DOI 10.48550/arXiv.1706.03762 source
• The published version, 31st Conference on Neural Information Processing Systems, Long Beach, 5998 to 6008. It carries the full equal-contribution footnote, as does arXiv v5 onward. No DOI exists for this volume and that gap is intentional, not an oversight source
• Shazeer's move: he posted the announcement himself on 17 June 2026 ("I'm excited to share that I'll be joining OpenAI"), reported the same day and widely corroborated. His August 2024 return to Google is recorded in the same reporting source
• Citation count observed 26 July 2026 via the Semantic Scholar Graph API source