How can it be relevant to go back so far in history when computing changes so fast? The impression of rapid development is entirely caused by the break-neck speed of hardware development. Compared to this, software moves but slowly. By the mid-1970s the main programming paradigms had been born: imperative, functional, object-oriented, and logic programming. Most of the languages in use today are descendants of these four paradigms. Paul Graham [Note HundredYearLanguage] explains the difference between hardware and software by observing that a programming language is not a technology but a notation. A notation reflects human thought habits and these change slowly.

Functional programming is regarded as a more or less pure form of the lambda calculus; logic programming programming as a more or less pure form of the predicate calculus. In this essay I imagine us back in 1950. Although the calculi existed at the time I imagine us (busy engineers) ignorant of such esoteric matters. But I do imagine us fired up by the invention of the subroutine so that we want to use this invention to the hilt. I will show that we would have invented the high-level languages in the form of subroutines acting on high-level memories.

To get started let me tell how I see the main protagonists of the story.

• Imperative programming

All computation consists of the execution of commands. These are of two types: (1) those that change a state consisting of contents of memory registers and (2) those that transfer control. This branch of the evolutionary tree began with machine code and developed into more and more sophisticated assemblers, Fortran, Algol, and C. A side branch from Algol became Simula, Smalltalk, and later object-oriented languages, where the state is articulated into the separate states of dynamically created objects.

• Functional programming

All computation consists of the evaluation of expressions of the lambda calculus. These are either constants or composite expressions specifying the application of a function to the values of its arguments. The program consists of function definitions and a single expression whose evaluation initiates program executions.

• Logic programming

All computation consists of attempting to prove an instance of a formula (in the sense of predicate logic) with free variables. In case of success the result of the computation is available as a substitution for the variables. A program consists of definitions of predicates and a single formula to be proved.

Of course these brief characterizations omit a lot of detail. But it is remarkable that the rapid development of logic programming in 1971 – 1974 was from the beginning based on predicate calculus. The development of functional programming, which I reckon to have started with Lisp in 1959 and which only reached the ideal of functional programming with Scheme in 1975, followed a more tortuous route.

In 1958, when McCarthy embarked on his new programming language, he had decided on the use of lambda notation, but he ignored the lambda calculus itself. “To use functions as arguments, one needs a notation for functions, and it seems natural to use the lambda-notation of Church. I didn’t understand the rest of the book, so I wasn’t tempted to try to implement his more general mechanism for defining functions.” [Note McCarthyHistory]

Thus the Lisp implementation of 1959 was not intended to be an implementation of the lambda calculus; it is surprising how close it later turned out to be. When it became clear what an implementation of the lambda calculus could look like, Sussman and Steele discovered, by a roundabout route [Note SussmanSteele], that Lisp, modified by replacing dynamic scope with lexical scope, is an implementation of lambda calculus. Hence my dating the origin of functional programming at 1975, preceded by sixteen years of close approximations in the form of the various and widely used dialects of Lisp.

Landin had identified lambda calculus as the basis of a programming language (ISWIM) independently of, and earlier than, Scheme. Although Landin had supplied SECD as an abstract machine for it, I believe that the Scheme of 1975 was the first implementation of functional programming. The SECD machine is in one of the papers of Landin published in 1964 and 1966. I believe the first implementation based on it was Henderson’s Lispkit Lisp of 1980.

By the end of the century functional programming had a variety of widely used forms: Scheme, ML, Haskell, CAML. Yet all were recognized as being based on lambda calculus, a mathematical formalism that existed before computers.

The discrete charm of the subroutine

In spite of this overwhelming display of the power of abstract mathematics over programming I like to imagine how functional and logic programming could have been invented independently of lambda or predicate calculus before even Fortran existed. It could have happened like this. Just like E.W. Dijkstra launched a crusade against the goto statement in 1968, a similarly charismatic guru could have done this in 1948. Just as Dijkstra did in 1968, this guru could have written

For a number of years I have been familiar with the observation that the quality of programmers is a decreasing function of the density of go to statements in the programs they produce.

In 1948 the implied message would have been that the closed subroutineshould be the first choice for transfer of control, with “closed” meaning that the subroutine only interacts with its environment through its parameters.

Turing’s ACE report of 1946 [Note TuringACE] proposes subroutines under the name “subsidiary operation”. What may be the first book on programming [Note WilkesWheelerGill] is organized around the concept of subroutine. As the book was published in 1951 I can imagine this early guru to have had the insight in 1948 that programs be structured around subroutines: that they should consist of subroutine definitions with the sole exception of a single call to initiate the entire program; that each subroutine definition should consist of subroutine calls embedded in a minimum of glue code for the marshalling of arguments in preparation of each call. Let us suppose that the guru would have called this Principled Programming.

There are two ways in which a closed subroutine can interact with its environment: modify the arguments (“procedure” in Algol or “void function” in C terminology) or cause a single result of the body’s computation to replace the subroutine call (“function” in Algol or “nonvoid function” in C). It is in the spirit of Principled Programming to require that a subroutine use only one of these interaction modes. It is in line with human nature that the Principled Programming movement would split up into two warring factions each of which insisting on the superiority of one of these modes: functional versus procedural programming.

When the first Fortran implementation was delivered in 1957, the main innovation was that of the expression. It was this feature that convinced McCarthy of the superiority of Fortran over IPL-V. To continue our minimalist fantasy, let us imagine that the concept of subroutine were only enriched with expressions. In functional programming this eliminates the need for any glue code between subroutine calls in a subroutine body. The body is changed to a single expression and the subroutines calls are changed to subexpressions. Program execution consists expression evaluation only. In effect the entire program consists of a single big expression, broken up into mentally digestible chunks by subroutine definitions. The process of subroutines calling subroutines is occasionally interrupted by a subroutine body consisting of an expression without subexpressions, a literal that directly denotes a value.

Expressions eliminate the need for glue code in functional subroutine programming, thus rendering it pure. The elimination of glue code would leave pure procedural programs with procedure bodies consisting only of procedure calls. In this way the chain of procedures calling procedures can only be stopped by procedures with empty bodies. This is not possible, because their execution would have no effect. Thus the advent of expressions makes functional subroutine programming pure, leaving procedural programming with the need for glue code. Among the devotees of Principled Programming, victory for functional programming.

Yet, with Prolog, pure procedural programming was realized. So there must be procedures with empty bodies, as indeed there are. But calling such procedures causes something to happen because Prolog cheats: when a procedure is called, body execution is not the only thing that happens; it is preceded by unification of the arguments of the call with the parameters of the definition. An argument in a function call is an expression typically containing an unbound variable. Unification with the corresponding parameter in the definition causes this variable to become bound. It is typically bound to an expression containing unassigned variables, making further bindings possible.

The variables of Prolog are called logical variables to indicate that they are created unbound and can only be bound once in the same computation thread. A binding can only be undone by backtracking to restore the computation state at the time of binding and continuing the thread some other way.

Three types of memory

Back to reality. Since 1948 imperative programming has evolved to an eclectic mixture of control constructs such as if-, case-, for- and while statements. Memory has differentiated into global, local, and heap memory. In spite or these differences, they are basically memory registers, albeit with a more convenient naming system than raw addresses would provide. The pure kinds of programming admit only the subroutine call as control construct and can be characterized as

     FUNCTIONAL PROGRAMMING = SUBROUTINES + EXPRESSION MEMORY
          LOGIC PROGRAMMING = SUBROUTINES + RATIONAL-TREE MEMORY

Here “rational-tree” is the name used by Colmerauer for expressions (that is, trees) in which the leaves are constants or single-assignment pointers to rational trees. Here we see that the two types of programming are made possible by assuming a suitable form of high-level memory, a different type for each type of programming. With this insight, let us review the birth of Lisp.

Graham’s account of what McCarthy did

Graham has written a summary of McCarthy’s original Lisp paper. Great as the original undoubtedly is, it benefits from the thirty years of hinds
ight that Graham was able to give it. The summary can be summarized by starting with a number of questions:

1. What is a simple and flexible data structure?
2. What is a textual expre
   ssion to represent such a data structure as simply as possible?
3. What is a small and complete set of operations and tests on the data structure?
4. What is a small and complete set of logical operations?
5. How to represent definitions of subroutines in the data structure?
6. How to represent calls to these subroutines in the data structure?

McCarthy gave the following answers:

1. A possibly empty binary tree.
2. A dotted pair or NIL.
3. CONS, CAR, CDR, EQ, ATOM
4. LAMBDA, LABEL, COND, QUOTE
5. As a binary tree. The left subtree denotes
   the name of function being defined. The right subtree denotes the expression denoting the value of the function. The context provided by reading the source text tells the interpreter which expressions are definitions and which is the expression to be evaluated initially.
6. As a binary tree. The left subtree denotes the function, the right subtree denotes its arguments.

Graham then delivers the punch line: McCarthy’s two mutually recursive functions, apply and eval, defined in terms of the primitive functions that specify the value of an arbitrary expression. Eval served as the first Lisp interpreter.

The method followed by McCarthy can be generalized to a recipe for an abstract machine. What makes McCarthy’s machine abstract is that it assumes all memory to have the form of binary trees. The abstract machine consisting of binary-tree memory with CAR, CDR, CONS, ATOM as instructions is interestingly different from the abstract machines for other languages, such as Prolog and Java. In the latter languages an occasional programmer like myself is not capable of writing a procedure or function in abstract machine code. In Lisp, on the other hand, CAR, CDR, CONS, ATOM are the most common functions in every program. This makes Lisp both low-level and high-level, the key to its versatility. < p> In this way we can reconstruct pure functional programming with subroutines from the ground up. How about pure procedural programming? It is an interesting exercise to accept the above five questions as a recipe for the design and implementation of a logic programming language.

What the above way of telling the story of Lisp suggests is that it is the memory model that makes programming high-level. The experience with logic programming and Prolog suggests that rational trees are a suitable starting point. The fact that rational trees enable pure procedural programming makes this interesting enough to apply McCarthy’s recipe to this choice of memory model. As result we should get an interesting reconstruction of logic programming, one of the few truly novel programming paradigms in the, by now, sixty-year history of the art.

Notes

[Note HundredYearLanguage]
“ The Hundred-Year Language” by Paul Graham.
[Note McCarthyHistory]
“History of Lisp” Stanford AI Laboratory memo 1979 by J. McCarthy, page 6.
[Note SchemeRevisit]
“The first report on Scheme revisited” Sussman and Steele, 1998.
[Note SussmanSteele]
AIM-349 MIT Artificial Intelligence Laboratory, 1975.
[Note TuringACE]
“Proposal for the development in the Mathematics Division of an automatic computing engine”, by A.M. Turing, Report E882 Executive Committee NPL 1946. Reprinted April 1972 as NPL report Com.Sci. 57. (Turing proposed subroutines under the name “subsidiary operation”.)
[Note WilkesWheelerGill]
Wilkes, Maurice V. et al “The Preparation of Programs for an Electronic Digital Computer”, 1951 This book is organized around the concept of subroutine.

” … the neats — those who think that AI theories should be grounded in mathematical rigor — versus the scruffies — those who would rather try out lots of ideas, write some programs, and then assess what seems to be working.”

For a few lines this is a pretty good characterization. But I think it only scratches the surface. In this essay I will explore the contrast in temperament and attitude that exists along several dimensions and is found elsewhere in science.

One of these dimensions is the one in which pious contrasts with naughty. The choice of these terms bears explanation. In reading a scientist like Dijkstra I get the impression that for him science is the highest, most noble calling that a human can aspire to. Hence, “pious”. The other type can be no less excellent as a scientist, but does not exude the pious aura. Consider Richard Feynman who started to wonder why, or whether, the human sense of smell is so much less sensitive than a dog’s. After some encouraging results that showed he was a better sniffer than humans are generally believed to be, he formed the hypothesis that dogs are only better at sniffing trails because they have their nose closer to the ground. So he crawled around the rug on his hands and knees, sniffing, to find out whether he could tell the difference between where he had walked and where he hadn’t [2]. Because of this episode I don’t classify Feynman among the pious scientists. (by the way,  the hypothesis was rejected, even when he had done the walking barefoot. In line with popular belief, the dog was a better sniffer than its owner.)

I might have used “prankish” rather than “naughty”, were it not for a passage in an essay by my favourite venture capitalist. The essay I have in mind [3] discusses what to look for in a start-up company in search of funding. Graham lists five points, among which number four is naughtiness.

Though the most successful founders are usually good people, they tend to have a piratical gleam in their eye. They’re not Goody Two-Shoes type good. Morally, they care about getting the big questions right, but not about observing proprieties. That’s why I’d use the word naughty rather than evil. They delight in breaking rules, but not rules that matter.

It is this that I recognize in accounts of Feynman. As another example in science, consider Irving John (“Jack”) Good. He was the General Editor of a 413-page volume with the title The Scientist Speculates: an Anthology of Partly-Baked Ideas [4]. Many of the 123 entries have intriguing titles. A sampling:

• pbi #48 Explosive Telepathic Fields
• pbi #85 Life in the Sun [i.e. on the surface of]
• pbi #79 Multipurpose Plants
• pbi #76 Steak from Sawdust
• pbi #67 A Problem for the Hedonist
• pbi #67 A Method for Encouraging Clairvoyance in Rats
• pbi #40 Self-Organizing Pumps and Barges
• pbi #39 A Splendid National Investment.

Would any self-respecting person deign to be associated with such a prankish enterprise? Look at this partial list of contributors: Bruno de Finetti, Dennis Gabor, Wassily Leontief, J.E. Littlewood, N.W. Pirie, John Maynard Smith, C.H. Waddington.

And, not surprisingly, we find contributions from Marvin Minsky and Donald Michie. The latter started off the present article indirectly via its predecessor about Michie and Longuet-Higgins, two very different temperaments, of whom it was hoped that their complementary talents would result in fruitful collaboration.

Michie was an experimentalist, but an unusual one. His lack of computer access spurred him in 1965 to invent an experimental set-up named MENACE (“Matchbox Educable Nought-And-Crosses Engine”), a demonstration of the BOXES learning algorithm in the form of a contraption that, on closer inspection, turns out to be an agglutination of 256 salvaged matchboxes requiring the operator to transfer coloured beads between boxes [5].

As I argue in [6], scruffy in AI got its imprint from the MIT hackers as described by Steven Levy [7]. They were not only AI pioneers, but also computer pioneers as well as pranksters. This last aspect showed in their allergy to passwords and locks (of the hardware kind). Back to Feynman, who was their soul brother in this respect [2].

Enough about naughty versus pious. In physics there is a lively animosity between experimentalists and theorists. Here follow some illustrations.

When Edsger Dijkstra was asked how he got into computing, he would tell that he was a student in “theoretical physics”. I suspect that the University of Leiden did not have a theoretical option in its physics program, but that some students had already differentiated in their minds.

When Christopher Longuet-Higgins descended into the Department of Machine Intelligence in Edinburgh, he had earned the freedom to do so by important results in theoretical chemistry.

One of my favourite pop-physics books is the one by Leon Lederman [8]. I recommend it, in spite of the god-awful title (no doubt a concoction of the publisher’s). I learned some physics from it. I also enjoyed his characterization of theorist versus experimentalist. Lederman keeps tracks of points scored by both sides: theorists anticipated the positron, the pion, the antiproton, and the neutrino, particles detected soon after in experiments. These predictions fed the already considerable arrogance of the theorists (“They need us to tell them what they are seeing.”). Theorists failed to anticipate the tau-lepton, the upsilon and the muon. As regards the latter, Isidor Rabi is famous for having remarked, in arch-theorist mode, “Who ordered that?” Lederman, after a dutiful homily about theorists and experimentalists using their complementary skills to uncover the secrets of the universe, is more convincing about the more mundane aspects of the differences between the two tribes:

Today we have two groups of physicists both with the common aim of understanding the universe but with a large difference in cultural outlook, skills, and work habits. Theorists tend to come in late to work, attend grueling symposiums on Greek islands or Swiss mountaintops, take real vacations, and are at home to take out the garbage more often. [...] They tend to worry about insomnia. [...] Experimentalists don’t come in late: they never went home. [...] Sleep is when you can curl up on the accelerator floor for an hour.

You get the flavor. Yet the poor experimentalists are not drop-out theorists. Nor do they leave for easier lives in the financial sector. It looks like experimentalists can’t help being what they are.

It will suffice to remind the reader of the notorious “Pauli effect” [9], the conclusive evidence of the vast difference between theoretical and experimental physicists.

Could it be that the difference is one of temperament and that a similar difference is found in computer science and AI? I think so, and I think that it explains the vast difference between the Longuet-Higginses and someone like Minsky, who not only did a lot of important things in AI, but also invented, built, and patented a new kind of microscope [10].

But, as one of my esteemed correspondents objects, Longuet-Higgins loved, and did, experiments; elegant little experiments. That illustrates my point: for a theorist experiments have to be elegant. If a question can’t be settled this way, the theorist thinks the time is not ripe. An experimentalist can’t wait. If nobody knows how to keep it elegant and little, then the expermimentalist tries to get the money, the facilities, and the people for a big messy experiment.

Sooner or later discussions about clashing temperaments will return to the Snow-Leavis Affair, alias the Two Cultures debate. It started with the 1959 Rede lecture in Cambridge University in which scientist, novelist, and civil servant Charles Snow described and deplored the fact that there seemed to be an unbridgeable gap between the scientific and literary scholars. The literary critic F.R. Leavis reacted to Snow’s lecture with a savage attack. Although fellow literary scholars thought the tone of Leavis’s article deplorable, they tended to agree that Snow’s analysis was shallow.

After half a century it is difficult to appreciate the furore arising from Snow’s lecture. Yet it is worth revisiting in the form of [11], especially for Stefan Collini’s preface, from which I quote:

The literary critic, habitually attending to the fine texture of verbal detail, can at times barely be persuaded that something is being said at all if it is being said badly. It is almost a truism of the critic’s working practice that the conventional distinction between form and content is misleading in literature: a work is those words in that order — one cannot blithely assume some “meaning” behind them which failed to get itself expressed properly but which is nonetheless the “message” of the text.

This fastidiousness has persisted into the twenty-first century. Many of the humanities lectures I have recently attended consist of the lecturer reading the paper, not “talking” to a rapid succession of Powerpoint slides. The verbatim record of the talk (if one would ever come to light) would be a jumble of words depending on a great deal of goodwill for any intelligibility. The text that was read is a joy to read after the occasion. In such fastidiousness I recognise the theorist, the neat: if an experiment cannot be done neatly, then it should not be done at all; we are not ready for it.

Dijkstra’s THE operating system [12], was an elegant little experiment. Of course it took half a dozen people a few years of hard work to write the assembler code for the Electrologica X8. But compared to contemporary operating systems (other than Unix) THE was both elegant and little. In physics there is a constant interaction between the experimentalists and theorists in the sense that a physics experiment only counts as such when theorists agree that it makes sense. In software the scruffies not only dominate, but there do not seem to be any neats around even to criticize, let alone restrain, the ubiquitous big, messy experiments. The reason is of course that, unlike in physics, some of the big, messy experiments generate big, neat piles of money.

References

[0] A Programmer’s Place February 18, 2011.
[1] Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig. Prentice-Hall, First edition 1995, page 21.
[2] “Surely, You’re Joking, Mr Feynman!” by Richard P. Feynman, as told to Ralph Leighton. W.W. Norton, 1985.
[3] What We Look for in Founders
[4] “The Scientist Speculates: an Anthology of Partly-Baked Ideas” Irving John Good, ed. Heinemann, 1962.
[5] “BOXES: an experiment in adaptive control” D. Michie and R.A. Chambers. In: Machine Intelligence vol. 2. Ella Dale and Donald Michie, eds. Oliver and Boyd 1968.
[6] The MIT style in Artificial Intelligence 1958 – 1985
[7] Hackers by Steven Levy. Doubleday, 1984.
[8] The God Particle by Leon Lederman with Dick Teresi. Dell, 1993.
[9] The Pauli Effect
[10] Memoir on Inventing the Confocal Scanning Microscope by Marvin Minsky.
[11] The Two Cultures by C.P. Snow, with introduction by Stefan Collini. Cambridge University Press, 1993.
[12] THE multiprogramming system

In a previous article I illustrated the tension in Artificial Intelligence (AI) between two mentalities, generally referred to as “scruffy” and “neat”. I started by accepting the characterization by Norvig and Russell [1].

“… the neats — those who think that AI theories should be grounded in mathematical rigor — versus the scruffies — those who would rather try out lots of ideas, write some programs, and then assess what seems to be working.”

Critics of my article have opened up so many dimensions of scruffy versus neat that it is hard to know where to start. From my disorganized thoughts one recurrent theme emerged: the distinctive MIT style during the formative years of AI.

In my previous article I explained the Lighthill affair as a conflict between AI proper and those who were stuck in cybernetics. That was in 1972. But cybernetics was there before there was AI and it’s not surprising that that was where some AI people started.

The year was 1946, the place was Cambridge, Massachusetts. The chronicler [2] was Garrett Birkhoff (1911 – 1996). Birkhoff made his reputation in the 1930s as a brilliant young mathematician of the pure variety. His talent was mobilized during World War II and applied to fluid dynamics. Cambridge was abuzz with the developments that Norbert Wiener (1894 – 1964) was soon to synthesize under the label “cybernetics”. As a Harvard mathematician, Birkhoff was not part of this, but confesses himself [2] a fascinated “voyeur”. He identifies as seminal event a public lecture featuring Wiener, John von Neumann (1903 – 1957), and Nicolas Rashevsky (1899 – 1972). The latter was a Chicago professor known for promoting the concept of mathematical biology. The confident organizers had booked an auditorium seating five hundred. Birkhoff estimated an overflow crowd of a thousand. Such was the start of cybernetics. It is safe to assume that the publication of “Cybernetics” by Wiener in 1948 and of “The Mathematical Theory of Communication” by Claude Shannon and Warren Weaver in 1949 were eagerly anticipated.

In the US the second world war had mobilized scientific talent and resources on a large scale. This effort launched novel developments such as operations research, game theory, automata theory, time series analysis, stochastic processes, information theory. By 1945 the innovation machine had gotten up to speed and was to continue unabated for over a decade. Electronics was the technology that made many things practical that had hitherto been confined to theory. In turn, theoretical studies were stimulated by the technological developments. The impact of electronics ranged widely, from the automation of computing to experimental investigation of the nervous systems of animals and humans. To some farsighted researchers there were many interconnections between these new developments. “Cybernetics” was the term coined to characterise the heady brew of synergies.

Hard on the heels of these novelties another wave was coming. Turing talked (and Shannon wrote [3]) about programming a computer to play chess, rather than the trajectory of an artillery shell. Turing famously wrote [4] about the corollary of a computer being programmed to think. In the early 1950s Shannon and McCarthy wanted to convene a symposium, not about cybernetics, but about the new New Thing: programming a computer to display human-level intelligence. McCarthy proposed as title “Artificial Intelligence”. Shannon could hardly disagree, as author of the first article about programming a computer to play chess. But he argued for a less provocative term and got his way: “Automata Studies”. As McCarthy later ruefully observed, “…, and guess what we got … automata studies” [5].

When the next opportunity presented itself, McCarthy had learned not to fly any false flags. The grant application proposed a “Summer Research Conference on Artificial Intelligence”. It was held in Dartmouth College in 1956 [14]. “Artificial Intelligence” stuck. Then, and ever since, many were unhappy with the term, but ended up using it for lack of a better alternative. This time the meeting was successful. In many of the participants and their discussions we recognize the real thing, in addition to some cybernetics and the inevitable automata studies.

From our current vantage point it is difficult to appreciate how insanely ambitious the idea of thinking computerswas compared to the hardware and software technology of the time. In 1956 computers were established as machines for scientific and for administrative applications. Such applications thrived: they were of interest to organizations with lots of money and these applications fitted comfortably inside the constraints imposed by the technology of the time. The hardware and software was developed at Univac and IBM for these purposes and delivered to corporate and university customers accordingly. For people like those who came to the Dartmouth Summer School the software of the time and the way the computers were run was intensely frustrating.

Before anything could happen in the direction of AI much practical development had to be done. In Dartmouth McCarthy was not only aware of this, but had the most advanced software research program. He had correctly identified the incipient Fortran compiler at IBM as the most promising starting point. He extended the compiler to handle the list data structure invented by Newell and Simon at RAND in Santa Monica.

While at Harvard professor Birkhoff was a fascinated bystander there was in his department an equally fascinated person, an undergraduate, who was not content to remain a bystander: he built SNARC, the first randomly wired neural network learning machine. Marvin Minsky continued to a PhD in mathematics in Princeton, with the thesis “Neural Nets and the Brain Model Problem”. He returned to Cambridge to work in Oliver Selfridge’s cybernetics group at the MIT Lincoln Laboratory in 1956 [7].

In 1958 the AI project was launched at MIT, with Minsky and McCarthy as founding faculty. McCarthy had come to the conclusion that souped-up Fortran was not satisfactory for the Advice-Taker project, nor was the Algol 58 that McCarthy was involved in. He needed to develop his own language. Upon his arrival at MIT he found the resources to do so: an IBM 704 and support for his programmers S. Russell and K. Maling [8].

That researchers at MIT had to depend on a machine bought from IBM suggests that MIT was behind IBM in computer development at the time. This is only true of the nonclassified part of MIT. Behind the veil of military security MIT had developed the most advanced computers of its time: Whirlwind, followed by TX0. The latter machine was a test vehicle for novel technology: transistors for the processor and ferrite cores for memory. When the test was successful, a more powerful version was developed, the TX2. The redundant TX0 was transferred in 1958 to the MIT Research Laboratory for Electronics, the home of the AI project.

Up till that time the Tech Model Railroad Club was just one of the student clubs on the MIT campus. TMRC had evolved an elaborate model railroad installation. For one part of the club the installation was a relief from technology. These students delighted in making pretty landscapes and in neatly painting the engines. The invisible part of the installation attracted the other part of the club: the freshest of freshmen and the nerdiest of nerds. They developed the underside of the installation, an insanely elaborate switching system with the complexity of a telephone switching office and indeed built with such equipment [9]. Russell, though not an MIT student, became a member of TMRC.

When the TX0 emerged in 1958 from its security-enforced seclusion it was not long before the more tech-crazed TMRC members discovered it as a playground that was even more interesting than the underside of the TMRC installation. And it was not long before their single-minded obsession with complex systems made some of them into virtuoso programmers and systems engineers at a level as high as anywhere else. Such expertise was in demand, as the typical prof and even grad student at the time was clueless in this area [9].

The IBM installation only allowed experimentation at the level of user programs, which included McCarthy’s work. But it did not allow hooking up cameras or robotic hardware, or the experimentation with systems software that would make this useful. In the TMRC punks hanging out around the TX0 Minsky saw expertise badly needed for the AI laboratory. The hackers, undergraduates in danger of dropping out of their academic program, represented the extreme end of scruffy. They competed for computer access with the representatives of neat: graduate students working on their theses and profs looking for opportunities to publish in the scholarly literature [9].

As an illustration of the kinds of things going on at MIT around 1960, consider David Silver, one of the hackers. He joined the group as a fourteen-year old drop-out from grade six, after twice having skipped a grade backwards. The hackers first tolerated Silver as a sort of mascot. It wasn’t long before he had cobbled together a floor-roving robot controlled from his software that performed tasks that the graduate students were writing about in their theses and pronouncing infeasible on the basis of solid surveys of the scholarly literature. The graduate students were most upset. Experiments without underlying theory were bad enough. And this punk was gobbling up precious machine time for activities that weren’t even experiments (where were the designs? where were the records?). But Minsky felt that the hacker contributions were vital to keeping AI real [9].

McCarthy was not impressed with the TMRC hackers hanging around at the TX0. The feeling was mutual: McCarthy had an ulterior motive behind his programming. For the hackers programming was its own justification; a silly hack was as good as anything else to enjoy programming and to display virtuosity. McCarthy likened them to “ski bums” [9]. Yet, as McCarthy acknowledges, one of the hackers made an important contribution to the implementation of Lisp. But then, this was McCarthy’s own Steve Russell, who had become a hacker in the sense of joining TMRC. The AI project was struggling with a compiler for Lisp, with no end in sight. McCarthy had shown Russell his paper about Lisp as an alternative formalism for specifying computable functions, one based on symbolic expressions rather than numbers. The required universal function was EVAL. Russell suggested programming it on the IBM 704.

… This EVAL was written and published in the paper and Steve Russell said, look, why don’t I program this EVAL and you remember the interpreter, and I said to him, ho, ho, you’re confusing theory with practice, this EVAL is intended for reading not for computing. But he went ahead and did it. That is, he compiled the EVAL in my paper into 704 machine code fixing bugs and then advertised this as a LISP interpreter which it certainly was, so at that point LISP had essentially the form that it has today, the S-expression form … (McCarthy quoted in [8])

I infer that the compiler project went on the back burner for a while.

This feat, destined to become History, did not deflect Russell from his involvement with hacks. In 1962 he programmed the PDP-1 so his friends could play Space War. At the time, a neat hack to show off the graphic display of the PDP-1 and to try out a new symbolic interactive debugger. In retrospect, the neat hack was also the invention of the very concept of something destined to grow into an industry in its own right.

The hackers were totally absorbed in the act of programming and in the exploration of the capabilities of the computer hardware they found and which they urgently needed to modify. They found in their own small group all the audience they needed. No need to publish anything. This attitude, though the exact opposite of the typical academic’s, was congenial to Minsky, who was later to reminisce:

…when it was our strategy in those early days to be unscholarly; we tended to assume, for better or for worse, that everything we did was so likely to be new that there was little need for caution or for reviewing literature or for double-checking anything. As luck would have it, that almost always turned out to be true [7].

Could it be that Minsky was infected by the hackers’ example? It looks like Minsky’s attitude was the root of the specific MIT style in AI, a style that was later to be, if not denounced, at least branded, as “scruffy”.

Compare McCarthy’s Advice-Taker project [12]. His approach was the epitome of “neat”: its knowledge base was to be expressed in formulas of predicate logic; the planning was to be the outcome of logical inference. McCarthy considered the existing programming languages, insofar as there were any, unsuitable for implementing such a system of logic. Hence his effort in developing a new language, the one that became Lisp. McCarthy must have assumed that somehow a suitable inference system would show up once Lisp was available. It didn’t. However, Lisp opened up so many exciting new possibilities that lack of progress in the Advice-Taker didn’t matter. These early years of Lisp were the years of the undocumented firsts that Minsky mentions in [7]; there is also documented work [10].

It was during this period that resolution inference appeared [11]. At first sight it seemed exactly what Advice Taker had been waiting for. The possibilities opened by resolution set off a wide-spread effort to realize the goal of Advice-Taker: given a description in logic of a world state and of a goal to be achieved, generate a constructive proof that the goal is achievable. The constructive nature of the proof implies that the sequence of actions to be performed can be picked out from the proof. In short, a proof as a plan. And automatically generated. That was the dream.

At MIT, however, the AI group may not have tried too hard to harness resolution to this task [13]. After all, if you have Lisp, why would you let logic stand in the way of programming a planning agent? Carl Hewitt’s answer to this question was Planner, untypically for MIT, not a Lisp program, but the design for a language. Planner was intended for implementation in Lisp, and a subset named Micro-Planner was implemented. It was an extension of Lisp, enriching the control structure with the addition of automatic backtracking. This was also the choice in Prolog, which appeared soon after. Sussman, one of the implementers of Micro-Planner, came to the conclusion that the fully automatic backtracking implied in “planning” was unsatisfactory and thought some conniving on the part of the programmer would be a more realistic approach for the automatic generation of plans; hence the language Conniver, created in collaboration with Drew McDermott. When conniving also turned out to be too ambitious, backtracking was thrown out altogether, so that the programmer had to resort to scheming. With Guy Steele, Sussman designed Schemer to achieve the desired level of programmer control. And guess what, Schemer was essentially Lisp, except that it was the long overdue lexically scoped version of Lisp. Thus the planning problem was brought back to what it should have been all the time according to the characteristic MIT approach: a problem to be solved in Lisp. The ITS operating that Sussman and Steele were working with did not allow filenames longer than six characters, so that “Schemer” was truncated to “Scheme”, now a standardized and much-loved programming language, whose roots in planning have mostly been forgotten.

It is of course not wise to try to tie down the heyday of the MIT style in AI to an exactly specified period. In the title I chose 1958 for the start because that was the year that Minsky started the AI project at MIT. That was also the year when he presented a wide-ranging overview of AI [6]. In the beginning Minsky spoke mainly through his students: in 1969 “Semantic Information Processing” appeared, edited by Minsky and with an article contributed by him. The bulk of the book consists of papers based on the PhD theses of Minsky’s first batch of students. The next milestone is an AI Memo co-authored with Seymour Papert [15]. Reading these two, the extreme of Scruffy, makes it hard to realize that Minsky could be as Neat as any. Not long before Semantic Information Processing he had published Computation: Finite and Infinite Machines and the year after, with Seymour Papert, Perceptrons: an Introduction to Computational Geometry. As an undergraduate Minsky had studied with Andrew Gleason, a mathematician’s mathematician [17]. My guess is that Minsky could imagine becoming a mathematician like Gleason, but set his sights, if not higher, at least further.

For the end date in the title I picked that of the publication of Minsky’s “The Society of Mind” [16]. It starts like this:

This book tries to explain how minds work. How can intelligence emerge from nonintelligence? To answer that, we’ll show that you can build a mind from many little parts, each mindless by itself.I’ll call “Society of Mind” this scheme in which each mind is made of many smaller processes. These we’ll agents. Each mental agent by itself can only do some simple thing that needs no mind or thought at all. Yet when we join these agents in societies — in certain very special ways — this leads to true intelligence.

Of course, when you write a book “that tries to explain how minds work”, then you get in the way of the psychologists. For the better part of a century, psychology had tried very hard to be a real science. Physics and chemistry had dominated the intellectual scene so much that “real science” was identified with these fields. Physics and chemistry got great by never accepting apparent complexity at face value: there always had to be something simple hiding behind. As a result, for almost half a century, behavioralism reigned supreme in psychology. Only shortly before the appearance of AI, psychology freed itself from this stranglehold. Minsky dared to ask: what if the mind is not inherently simple? What if, behind this apparent complexity, there is … complexity? What if this complexity takes the form of a changing collection of many, many agents connected in intricate and changing patterns?

Behaviourist psychology, with its urge to conform to “real science” and its emphasis on the Scientific Method, was Neat. Minsky, in his bid to discover how the mind works, no holds barred, was Scruffy.

Acknowledgments

Thanks to Paul McJones, Alan Robinson, and Steve Russell for suggestions and corrections.

References

[1] Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig. Prentice-Hall, First edition 1995, page 21.
[2] Article by Garrett Birkhoff in History of Computing edited by N. Metropolis, J. Howlett, and Gian-Carlo Rota. Academic Press, 1980.
[3] Claude E. Shannon: Programming a Computer for Playing Chess, Philosophical Magazine, Ser.7, Vol. 41, No. 314, March 1950.
[4] Turing, A.M. (1950). Computing machinery and intelligence. Mind, 59, 433-460.
[5] BBC television 1973.
[6] “Some methods of artificial intelligence and heuristic programming” in Mechanisation of Thought Processes Symposium No. 10, National Physical Laboratory, Her Majesty’s Stationery Office, 1959.
[7] “Introduction to the COMTEX Microfiche Edition of the Early MIT Artificial Intelligence Memos” by Marvin L. Minsky, AI Magazine, 1983: 19 – 22.
[8] “Early Lisp History (1956 — 1959)” Herbert Stoyan. LFP ’84: Proceedings of the 1984 ACM Symposium on LISP and functional programming.
[9] “Hackers” by Steven Levy. Doubleday, 1984.
[10] Marvin Minsky: “Semantic Information Processing“, MIT Press, 1968. Slagle’s thesis.
[11] J.A. Robinson: “A Machine-Oriented Logic Based on the Resolution Principle”, Journal of the ACM, 1965.
[12] “Programs with common sense” in Mechanisation of Thought Processes Symposium No. 10, National Physical Laboratory, Her Majesty’s Stationery Office, 1959.
[13] But see an interview with Robinson for an interesting peek behind the scenes.
[14] Dartmouth 1956 proposal
[15] “Artificial Intelligence: Progress Report” by Marvin Minsky and Seymour Papert, Artificial Intelligence Memo No. 252, January 1, 1972.
[16] The Society of Mind by Marvin Minsky. Simon and Schuster, 1985.
[17] See the interview with Andrew M. Gleason in More Mathematical People edited by D.J. Albers, G.L. Alexanderson, and C. Reid, Harcourt Brace Jovanovich, 1990.

… the Lighthill Report, which formed the basis for the decision by the British government to end support of AI research in all but two universities. (Oral tradition paints a somewhat different and more colorful picture, with political ambitions and personal animosities that cannot be put into print.) [1]

In this article I will put in print some of the things hinted at here, and elaborate on the issues that have remained topical.

I first heard the word “Lighthill” in November 1972, soon after I started my research fellowship with the Department of Machine Intelligence at the University of Edinburgh. As Jean Michie said to me: “What, you haven’t heard of Lighthill? That man who wants to do away with us?” I hadn’t, but it wasn’t before long I learned that Jean was right about his wanting to do away with the Department of Machine Intelligence and, what was more, about his having the clout to bring this about.

The “Lighthill” referred in the conversation with Jean came to mean the publication of a report, followed by a televised debate at the Royal Institution in London. The proceedings gave the impression of a movie Crown Prosecutor presenting of his case. Sir James Lighthill, FRS, Lucasian Professor at Cambridge University pronounces from behind a lectern on a platform elevated above the other participants of the debate, seated in the pit below: Donald Michie, John McCarthy, and Richard Gregory. The debate was actually only the second half of the proceedings; the first half consisted of an oration by Lighthill standing on his platform. Science policy or theater?

At the time the Department of Machine Intelligence, Michie’s creation, had become a world centre of AI. To give you an idea, I’ll list (as far as memory serves) of visitors to the department I met over the years I was there [2]. A year later the Department of Machine Intelligence at the University of Edinburgh was reduced to three people: Donald Michie, professor of Machine Intelligence, one secretary, and one technician. The remainder joined the newly created Department of Artificial Intelligence or left for jobs elsewhere. All this on the basis of the report of a single person, ignorant of the research area. It is extraordinary that a body dispensing public funds, such as the UK Science Research Council, could have proceeded in this manner.

If a report were necessary, it should have been written by an American expert. Only in the US was a suitable depth of expertise available. Moreover, the search for a British expert would have been complicated by the difficulty of finding someone not connected via the old boy network. Not only was the reporting “expert” British, but also the old boy connection was present with a vengeance: Lighthill was one of, what I shall call, “The Winchester Four”, described in an interview with Freeman Dyson:

IOPScience: You excelled in mathematics at school. Were those happy days?
Freeman Dyson: Yes, on the whole. We were very lucky because everything was screwed up by the war [World War II] — I remember in my last year at Winchester having only seven hours of classes a week. It was wonderful — we were free to get our own education. The teaching was fairly good, but didn’t make much difference — we learned much more from each other than we did from the teachers. There were four of us, who were about the same age, who became fellows of the Royal Society — the Longuet-Higgins brothers, Sir James Lighthill and I.

The Winchester Four stayed close in their interests: all became mathematical physicists. All four became Fellows of the Royal Society. If there would have been something like Nobel Nominees, then Freeman Dyson and Christopher Longuet-Higgins would have been among those; Dyson in quantum electrodynamics and Longuet-Higgins major (public-school style, to avoid confusing him with the younger brother) in quantum chemistry.

Longuet-Higgins major joined Donald Michie and Richard Gregory in founding the Department of Machine Intelligence in Edinburgh. Soon after this, Gregory left Edinburgh. By the early seventies an acrimonious dispute had developed between Longuet-Higgins (from now on all references to L-H will be to Longuet-Higgins major) and Michie. The latter, who had taken the initiative in the department’s founding, believed that it was time for Longuet-Higgins to assume the admittedly onerous duties as the head of the department. Longuet-Higgins denied that this could be expected of him.

The dispute with Michie was not the only reason for Longuet-Higgins to be dissatisfied with the Department of Machine Intelligence. Artificial Intelligence (from now on “AI”) suffered under the tension between two schools of thought, briefly denoted as the Neats and the Scruffies, described as follows by Russell and Norvig [1]:

“… the neats — those who think that AI theories should be grounded in mathematical rigour — versus the scruffies — those who would rather try out lots of ideas, write some programs, and then assess what seems to be working.”

His mastery of mathematics had given Longuet-Higgins a deep understanding of quantum mechanics, which, in turn made it possible for him to explain certain chemical phenomena that had been missed by earlier followers of Linus Pauling’s program of explaining the nature of the chemical bond in terms of quantum mechanics [8]. In AI or out, Longuet-Higgins was the epitome of Neat. By the early seventies Scruffy had become the dominant mode in AI. To the distress of Longuet-Higgins, Machine Intelligence in Edinburgh was not all Neat.

Such was the situation when it transpired that the UK Science Research Council had selected for its grand review of AI not an AI researcher, nor even a computer scientist, but, astonishingly, one of the Winchester buddies of Longuet-Higgins. One can imagine that we (that is, those whose jobs were about to be terminated) were not unbiased readers of the, to us, notorious Lighthill Report.

Many years have passed. Most of those forced out found other jobs, mostly far away from Edinburgh. I trust that for the others these exciting events have become a distant memory, as they have for me. A memory consisting of Sir James Lighthill as a pompous idiot who lent himself to produce a flaky report to serve as a blatantly inadequate cover for a hatchet job.

Last Christmas I was reading the autobiography of Laurent Schwartz (1915–2002), the great French mathematician. Schwartz was an amazing phenomenon. As a mathematician he was unusually versatile, spanning the entire range from probability and applied mathematics to Bourbaki (he even was a Bourbaki). For his efforts in opposing the Algerian and Vietnam wars I regard him as a French version of Pauling, or Chomsky.

Schwartz is mainly remembered for his discovery of distributions, a theory that mathematically modelled certain hitherto pseudo-mathematical objects that were found indispensable in physics; the Dirac delta function, for example. Schwartz made the discovery in 1944, in recently liberated Paris. It was a few years before the new theory gained a following. As usual it was among the best young mathematicians that it was first appreciated. Schwartz mentions one James Lighthill.

This, and the passage of time, caused me to review my 1973 impression of Sir James as a “pompous idiot”. I found the book by Lighthill [4] on distributions, purchased in 1958 by Victoria College (now University of Victoria). It is a gem: brief, elegant, accessible; a great way not only to learn the advanced concept of distribution, but also a good introduction to basics like Fourier series and integrals. A biography I dug up outlined an impressive career including practical applications in aerodynamics and directing the Royal Aircraft Establishment.

And I got the [Lighthill Report]. Did I find valuable insights that I missed in 1973? Alas not. Yet, with the wisdom of hindsight, and knowing how Neat versus Scruffy has played out over the decades, it was worth re-reading the report. What I now read is a tract by a Neat on the attack, setting out to root out Scruff.

Allow me to elaborate on Neat and Scruffy as described by Russell and Norvig. Yes, the Neats believed that AI theories needed to be grounded mathematically. But I think it is more enlightening to say that to be Neat was to be stuck in cybernetics. “Cybernetics” is the term coined by Norbert Wiener and used as the title of his 1948 book introducing the concept. What inspired the new word was the recently discovered parallel between feedback control in electrical circuits and animal nervous systems. That Wiener mentions digital computers at all in a 1948 book shows that he was unusually well informed and perceptive (at the time “computer” meant, if not a person, an analog electronic computer).

In 1948, Wiener’s book was a conceptual breakthrough. The problem was that by 1970 the next conceptual breakthrough — the computer as new universe — had already happened, but the majority of mathematicians and scientists were still living in a world bounded by cybernetics. Admittedly, by then computers (now unambiguously digital) were vastly more useful than they were in 1948, but they still played a minor role in cybernetic research. For the Neats, the significance of improvements in computer technology was that electrical engineering systems could be automated in a more sophisticated way. This is laid out in the Advanced Automation section (“area A”) of the Lighthill Report. As befits a cybernetician, Lighthill applauds the use of computers for studying natural nervous systems (area C in the Lighthill Report). The report places some AI activities in areas A or C. The Neats had been imaginative enough to include post-Wiener additions to the mathematical arsenal, like resolution theorem-proving as method for problem-solving and planning. Not surprisingly the report was forced to include a third category, which it named area B. Here went everything that didn’t fit the cybernetics framework. Like robotics.

The Neats continued to regard the computer as an instrument useful only for the advancement of Wiener’s vision of cybernetics. I think the Scruffies deserve more credit than their characterization by Russell and Norvig. What distinguishes them is their sense of the computer being more than an instrument subservient to existing disciplines. To them the computer represents a phenomenon in its own right, one that promises to shed new light on endlessly debated conundrums like the nature of intelligence, thought, and consciousness. As Archimedes is reputed to have said: “Give me a lever long enough and a fulcrum on which to place it, and I shall move the world,” so the implied motto of the Scruffy was “Give me a PDP-10 with Lisp and I will blast wormholes through the space of the traditional sciences.”

McCarthy disdained decades of psychology research in learning. “I am interested in the phenomenon where you teach, not by multiple training actions, but by telling, once, and the subject then knowing it forever — something that only happens in humans.” Michie exulted in bypassing control theory, much approved of by Lighthill as a gloriously successful and sophisticated branch of applied mathematics, to produce controllers for unstable systems via the admittedly stupid BOXES algorithm.

As a whole, AI at MIT was Scruffy and particularly so when it came to representing and using knowledge in computers. In this respect Stanford and Edinburgh were Neat: they embraced mathematical logic as a formal framework. They welcomed resolution because it was within mathematical logic as well as easy to implement on a computer. MIT viewed mathematical logic as a distraction. They believed that the resolution system could not accommodate the necessary pragmatics. Accordingly they advocated “procedural embedding of knowledge” by bypassing mathematical logic and creating a new language [6].

The disagreement led to a confrontation at the 1970 Machine Intelligence workshop in Edinburgh. The workshop programme announced “The Irrelevance of Resolution”, a talk by Seymour Papert, one of the leaders of the AI group at MIT. Instead, one Gerry Sussman showed up with the message that he was sent by Papert to give the talk in his place. It looked like Sussman had no experience giving talks in faraway countries. He seemed nervous about going into the bastion of resolution theorem-proving and telling the assembled experts how totally misguided their work was. As a result he started by overdoing the confrontational aspect. A few minutes into the talk, Longuet-Higgins walked out of the room, pensively puffing his pipe.

In 1972 Minsky and Papert published what I view as the manifesto of Scruffy [AIM-252]. The summary on page 2 intones:

Thinking is based on the use of
SYMBOLIC DESCRIPTIONS
and description-manipulating processes
to represent a variety of kinds of KNOWLEDGE
— about facts, about processes,
about problem-solving, and about computation itself,
in ways that are subject to
HETERARCHICAL CONTROL STRUCTURES
— systems in which control
of the problem-solving programs
is affected by heuristics
that depend on the meaning of events.

The words are verbatim from AIM-252; I have been so free as to add ventilation [7], as called for by the solemnity of the occasion. Since that time we have learned that this is compatible with Neat; at the time it seemed the essence of anti-Neat.

The manifesto brings the computer beyond the role of enabler of cybernetics, which is where Neat had become stuck. It is a bold step putting the computer at the centre of a new world (let’s call it AI) where, for instance, it becomes possible to think (productively) about thinking. To establish AI as a subject in its own right it was initially necessary to repudiate heritage, even where it could be useful. In the early 1970s the Scruffies had to be unnecessarily confrontational, just as a teenager needs to be unnecessarily obnoxious. But the author of the Lighthill report did not have the strength of a wise parent. He was enraged by what he saw and constructed a report calculated to kill as much of Scruffy as possible.

Acknowledgments

Thanks to Alan Robinson for helpful criticisms, to Michael Levy for finding the Lighthill Report, and to André Vellino for finding the televised debate. The text benefited greatly by editorial input from Eva van Emden (http://editing.vanemden.com/)

References

[1] Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig. Prentice-Hall, First edition 1995, page 21.
[2] Here follows a list of AI researchers of whom I remember right now that I met them at the Department of Machine Intelligence between 1969 and 1975: Harry Barrow, Woodrow Bledsoe, Daniel Bobrow, Robert Boyer, Peter Buneman, Alan Bundy, Alain Colmerauer, John Darlington, Edward Elcock, Cordell Green, Patrick Hayes, Carl Hewitt, Gérard Huet, Robert Kowalski, Christopher Longuet-Higgins, David Luckham,  John McCarthy, Zohar Manna, Donald Michie, Robin Milner, Ugo Montanari, J Moore, Nils Nilsson, Seymour Papert, Ira Pohl, Alan Robinson, Laurent Siklossy, Aron Sloman, Gerald Sussman, Austin Tate, Richard Waldinger, and David Warren.
[3] I asked a friend whose life is in quantum chemistry, unsullied by AI, whether he had heard of H.C. Longuet-Higgins. His reply: “In my opinion Longuet-Higgins has made a very important contribution (1963) to theoretical chemistry by explaining that the symmetry of a molecule is not described by a point group, but by a permutation-inversion group. The latter group is in general unnecessarily large, and L-H has shown that only the subgroup of ‘feasible operations’ (a concept invented by L-H) is significant. This is a very subtle piece of theory that remains a mystery to most chemists. … The point of L-H is that you should see the nuclei of atoms as quantum mechanical particles, and most chemists cannot do that, they are so used to the structure of molecules that they cannot understand that nuclei can tunnel through a potential barrier (whereas most chemists can understand that for electrons). Chemists say: you have trans-butadiene and cis-butadiene and those molecules are different. L-H says, no they are two wave functions with maxima in the trans and the cis configuration and the wave functions overlap: trans-butadiene is a ‘little bit’ cis and vice versa.”
[4] An Introduction to Fourier Analysis and Generalised Functions by M.J. Lighthill. Cambridge University Press, 1958.
[6] See Wikipedia article.
[7] See Ventilated Prose.
[8] The Nature of the Chemical Bond by Linus Pauling, 1939.

Primes: from tables to programs

First some background. Prime numbers have been bugging (some) humans for many centuries. The great Gauss himself was among those who fell under the spell of the primes. For lack of a published table of primes he made one up for himself; he pored over its columns in search of a pattern. In despair he gave up, concluding that it looked like the numbers were generated by some demon throwing dice. And, bingo, this probabilistic model led him [7] to the famous conjecture about the distribution of primes that turned out to be correct, but took almost a century to be proved.

Given this history, it is not surprising that those bitten by the bug of the primes welcomed the computer as an alternative to printed tables. Jacob Philip Kulik (1793 – 1863) devoted twenty years to a manuscript from which a printed table of the prime numbers up to eleven million was prepared [1]. Errors were corrected in the process; unkind souls allege that many remain. By the time computers arrived, more dependable sources existed, though few and far between. The process by which they were created inspired more confidence than poor Kulik slaving away in solitude, but worries about accuracy remained.

With the computer, a new phenomenon arose: the possibility of a program that generated in a matter of minutes (now seconds) all primes up to eleven million, the range of Kulik’s table. Now, rather than rely on authority, we can convince ourselves of the correctness of the table by studying the page or so of code that it takes to generate the first N primes. Still, even now there are trusting souls who put their faith in tables. For example, I found on the web an applet claiming to list the primes up to a hundred million, with that range served up in convenient chunks of a thousand at a time. But wouldn’t you rather rely on twenty lines of code that do the same job? (Appendix A).

The function in Appendix A is simple, and wasteful. For example, soon after eliminating all factors 2, it tries to divide by 4. It continues spending much of its time performing such superfluous operations. Yet it is quite effective for undemanding tasks like serving up the factorizations of, say, a thousand numbers. For reasons of uniformity, the code in Appendix A has been dumbed down to just indicating primality.

Trial Division versus Sieve of Eratosthenes

When we eliminate the redundancy of the function in Appendix A, we arrive at the method of Trial Division, which only tries the successive primes as potential divisors. The primes that it needs are generated by itself in an earlier iteration. Briefly, Trial Division works like this. Starting from the last prime found, generate as candidate primes the successive odd numbers and test them for divisibility by all the primes in succession. If no divisor is found by the time you reach the prime that is just greater than the square root of the candidate, you can stop and add the candidate as next prime to the table (Appendix B).

Trial Division is one of the stock examples in programming courses. Another is the Sieve of Eratosthenes, another algorithm that generates the sequence of primes. It is one of the earliest algorithms and was invented thousands of years ago. The name of the algorithm suggests that you imagine a sieve into which you drop all the numbers. You shake the sieve, and then the prime numbers come falling through. It helps to imagine the sieve as a superposition of sub-sieves: S2, S3, S5, S7, … Sub-sieve S2 catches all multiples of 2, S3 catches all multiples of 3, and so on for 5, 7, 11, … Each sieve keeps back the multiples of the prime it is dedicated to, so that only primes get through.

The process can be illustrated by the following bit of ASCII art, depicting a sort of Morse-code rendering of the Sieve of Eratosthenes.

   ____________________________________________________________________________

       0000000011111111112222222222333333333344444444445555555555666666666677
       2345678901234567890123456789012345678901234567890123456789012345678901

   S2  -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.
   S3  .-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..-..
   S5  ...-....-....-....-....-....-....-....-....-....-....-....-....-....-.
   S7  .....-......-......-......-......-......-......-......-......-......-.
   S11 .........-..........-..........-..........-..........-..........-.....

   Figure 1.
   The Sieve of Eratosthenes consisting of superposition of the sieves S2, S3,
   S5, S7, and S11. Each column consisting of dots only corresponds to a prime.
   ____________________________________________________________________________

Read the top two lines vertically, giving the numbers to be sieved: 02, 03, …, 10, 11, …, 70, 71. By default, numbers are rendered as dots. In the line for S2 the dot at each multiple of 2 has been replaced by a dash; the one after that has a dash at each multiple of 3, and so on for 5, 7, and 11. As a result, any column with dots only indicates that the number labeling the column is a prime.

The number theorists Derrick Lehmer (father and son) built devices based on bicycle chains with rods at certain significant intervals. The assembly could be set up in such a way that it acted as a sieve of Eratosthenes, or as a generator of solutions of a Diophantine equation [2].

To simulate the Sieve on a computer you could use an array starting with all zero bits and write ones every p locations, for p equal to 2, 3, 5, 7, and so on. Where Trial Division uses divisions, the Sieve generates the various sieves by means of additions. The Sieve used to be preferred because divisions were slower than additions. In the old days not all computers had hardware to perform the divisions required by Trial Division, so there could be a huge difference in speed.

The Assembly-Line algorithm

If you want to generate all primes up to N, then the Sieve method requires an amount of memory proportional to N. Trial Division needs memory for the table of the first primes, and this table need only have a size of the square root of N. So here is the stand-off: the Sieve is faster, while Trial Division needs less memory.

The difference between the two methods is an example of a phenomenon that is not restricted to computing. Imagine that you have to manufacture a large number N of items. Suppose that each item requires M operations A, B, C, … to be performed on it. Imagine N in the thousands and M to be about the square root of N. Suppose furthermore that each operation requires some start-up time. For example you need to consult the cheat sheet on how to perform the operation and you have to get the tools ready. As a result it takes considerably less time to repeat the same operation on successive items than the same number of different operations on the same item.

This situation suggests two strategies:

1. minimize time — line up all N items and perform on each in turn A, then on each in turn operation B, and so on.
2. minimize number of items in process — perform operations A, B, C, … on the same item, and get it out of the way.

Of course we want a method that combines the advantages of both approaches.

Primes to be generated can be likened to items to be manufactured. The operations are ensuring non-divisibility by 2, by 3, by 5, and so on. The Sieve method is like laying out all items simultaneously. As a result, ensuring non-divisibility can be done fast: only additions are needed. Trial Division is like performing all operations on a single item before proceeding to the next. Because ensuring non-divisibility by a given prime is done in isolation from the same operation on the other items, the division operation is needed, which is more expensive than addition. As in manufacturing, the disadvantages of both strategies suggest looking for a way that combines the advantages of both approaches.

In manufacturing the assembly line has been invented to combine speed with reducing the number of in-process items. In an assembly line there is one worker for each operation, who only performs that operation on the items as they are successively presented by a traveling belt. Operations are performed at maximum speed. The device of the assembly line reduces the number of in-process items from N to M.

Let us translate the idea of the assembly line to the Sieve of Eratosthenes as illustrated in Figure 1. Suppose we want to check whether N = 36 is a prime. The workers on the assembly line are represented as the set of multiples of 2, 3, 5, 7, and 11 that are in column N or the first such to the right. For N = 36 these are 36, 40, 42, and 44. As one of the multiples is equal to N, N is dismissed as non-prime. N is incremented. Any multiples that are to the left of N are incremented by the corresponding prime; in this case 36 is incremented to 38 in its role as multiple of 2 and to 39 in its role as multiple of 3. Now we have that N = 37 is less than the least multiple under consideration; this proves it to be prime.

The operations here are addition and identification of the least of the multiples. This latter operation is potentially significant: for N at eleven million there are over three thousand multiples. Therefore the multiples should be organized as a heap.

See Appendix C for further explanation and a C++ implementation.

Whence the Assembly-Line algorithm?

What should we acknowledge as the source of the Assembly-Line algorithm? Before I noticed Dijkstra’s hidden gem, the only source I knew for the assembly-line algorithm was a 2008 paper by Melissa O’Neill [3]. O’Neill refers to the analogue of the assembly line as “just-in-time”. This sounds like an analogy with manufacturing, but it is not. “Just-in-time” manufacturing refers to coordination between factories, not to what happens within a single factory. Apart from this discrepancy, “just-in-time” is an apt characterization of a balanced assembly line.

O’Neill gives no source for the Assembly-Line algorithm, so I assume she is one of the inventors of it. The book by Crandall and Pomerance [4] seems unaware of it. Their pseudo-code for the Sieve is unclear about details. I surmise they were not aware of the Assembly-Line algorithm because of their comment on page 121 “The biggest computer limitation on sieves is the enormous amount of space they can consume.”

The earliest description of the Assembly-Line algorithm that I know of is Dijkstra’s essay “Notes on Structured Programming” [5]. Dijkstra’s discovery may not have received the attention it deserved because it was not presented as a novel solution for an important problem in prime-number generation. Dijkstra just wanted to describe in detail how to go about systematically developing an algorithm, some algorithm. He picked the generating of prime numbers as an example that is somewhat interesting, yet short enough to fit in the report he was writing.

For his example of systematic program development Dijkstra chooses the Trial Division algorithm. In an aside he remarks that the friends to whom he showed the draft objected that “Everybody knows that the Sieve is more efficient”, and refused to read further. After meticulously motivating each tiniest step, Dijkstra almost completes an algorithm for the Trial Division method. The last step is the actual division.

At the last step, implementing how to test for divisibility the candidate for the next prime, Dijkstra fills in the obvious modulo instruction. This would have resulted, if transposed to the C programming language, in something like the code in Appendix B. But then he says (page 37): “To give the algorithm an unexpected turn we shall assume the absence of a convenient remainder computation.” He continues with a brief outline of how he could continue the development of his algorithm to one that maintains a location for each prime and ensures that each of these locations contains the least multiple of the prime that is not less than the number under consideration. These multiples need only an addition to be updated; a division is never needed. Yet he never needs more multiples of primes in memory than the square root of the size of the table to be produced.

There it is: among many other things, the Master invented prime-number generation according to the assembly-line principle. The only thing that is missing is a mention of the need to repeatedly find the smallest of these multiples; that is, the need of a suitable data structure such as a heap.

The Assembly-Line algorithm in action

It may be that in the early seventies the expected speed of the Sieve was only attributed to the speed of addition. Since then all algorithms have been subjected to asymptotic analyses. As [3] describes, the Sieve is not just faster by a constant factor, but is also asymptotically faster. Encouraged by this scientific nugget, I implemented (Appendix C) the Assembly-Line algorithm, using STL, the standard library for C++, for the required heap data structure. I compared its results with the those of the program in Appendix B on the re-computation of Kulik’s table, all prime numbers up to 11 million.

To my surprise, Trial Division, though asymptotically slower, does it about five times as fast as the Assembly-Line algorithm. Of course, being asymptotically faster does not mean faster on small inputs. Apparently, for this problem, 11 million is still a small input, too small for the asymptotic difference in speed to prevail.

Winding up

In winding up I need to report that, to prevent this from getting even longer, I omitted another fascinating skirmish of Dijkstra’s with prime numbers. This is Chapter 17 (“An exercise attributed to R.W. Hamming”) in A Discipline of Programming [6]. Here he uses again what I call the assembly-line idea, but without acknowledging any connection with prime numbers and missing the opportunity to use addition.

Another time, perhaps.

References

[1] Waclaw Sierpinski (Andrzej Schinzel, ed.): Elementary theory of numbers. Elsevier, 1988, page 119.
[2] http://en.wikipedia.org/wiki/Lehmer_sieve (December 31, 2010)
[3] O’Neill, Melissa E., “The Genuine Sieve of Eratosthenes”, Journal of Functional Programming, Published online by Cambridge University Press 9 October 2008.
[4] Richard E. Crandall and Carl Pomerance: Prime numbers: a computational perspective. Springer-Verlag, 2005.
[5] O.J. Dahl, E.W. Dijkstra, and C.A.R. Hoare: Structured Programming. Academic Press, 1972.
[6] E.W. Dijkstra: A Discipline of Programming. Prentice-Hall, 1976.
[7] Marcus du Sautoy: The Music of the Primes. HarperCollins, 2003.

Appendix A

A C++ program to print the prime numbers in a block of one thousand.

#include <iostream>

using namespace std;
typedef unsigned nat; // natural number

bool prime(nat n) {
  nat n0 = n, factor = 2, product = 1;
  while (factor*factor <= n) {
    while (n%factor == 0) {
      n /= factor; product *= factor;
    }     factor++;
  }
  return ((n != 1) && (n == n0));
}
void primes(nat n) {   if (n >= 1000)
  for (nat i = n-999; i <= n; i++)
    if (prime(i)) cout << i << endl;
}

Appendix B

typedef unsigned nat; // natural number

void prTable(nat p[], nat n) {
// Place the first n prime numbers in p[0..n-1]
// using Trial Division.
  nat k;
  assert(n > 1);
  p[0] = 2; p[1] = 3;
  k = 2;
  while (k < n) {
  // p[0..k-1] are first k primes
    nat cand = p[k-1]+2;
    nat j = 0;
    while (p[j]*p[j] <= cand) {
    // p[0..j] do not divide cand
      j++;
      if (cand%p[j] == 0) {
        j = 0; cand += 2;
      }
    }
    p[k++] = cand;
  }
  // p[0..k-1] are first k primes
  // k == n
}

Appendix C

The Assembly-Line algorithm.

To determine whether N is a prime, we maintain as a heap S(N), a set containing a multiple of each of the primes up to the ceiling of the square root of N. We can increment or decrement a multiple; this means increment or decrement (if possible) by the prime corresponding to the multiple. We say that a multiple in S(N) is “too big” (with respect to N) if it can be decremented without making it less than N. We say that a multiple in S(N) is “too small” (with respect to N) if it is less than N. The theorem that is at the basis of the Assembly-Line algorithm says that N is prime if no multiple in S(N) is too big and if N is less than the least multiple; that is, less than the top of the heap.

An algorithm that starts with S(N) that does not contain a multiple that is too big can maintain this property by only incrementing multiples that are too small. This suggests as algorithm for determining the next prime at or after N.

Ensuring none of its multiples is too big,
create S(N) as a heap; // smallest multiple at top

while (N >= top of heap) {
// There may be a multiple that is too small.
  while (N > top of heap) {
    // There is a multiple that is too small.
    increment top of heap
  } // No multiple is too big.
  if (N == top of heap) {
    // N is not a prime; try next N.
    N = N+2;
    // No multiple is too big.
  }
}
// No multiple is too big and N < top of heap
// Therefore N is prime.

C++ code:

#include <iostream>
#include <vector>
#include <algorithm>
#include <utility>

using namespace std;
typedef unsigned nat; // natural number
typedef pair<nat,nat> pnn;

bool gt(pnn x, pnn y) {
  return x.first > y.first;
}
void primes(nat table[], nat n, nat lim) {
// place first n primes in table[0..n-1]
// ensure p[n-1] <= lim
  table[0] = 2;
  table[1] = 3;
  vector<pnn> heap; vector<pnn>::iterator i;
  nat max = 0; // max prime for heap
  while (max*max <= lim) max++;
  heap.push_back(pnn(2*2,2));
  heap.push_back(pnn(3*3,3));
  push_heap(heap.begin(), heap.end(), gt);
  nat N = 5, count = 2;
  while (count < n) {
    while (N >= heap.begin() -> first) {
      while (N > heap.begin() -> first) {
        // increment least multiple by corresponding prime
        pop_heap(heap.begin(), heap.end(), gt);
        vector<pnn>::iterator last = heap.end()-1;
        last -> first +=  last -> second;
        push_heap(heap.begin(), heap.end(), gt);
      }
      if (N == heap.begin() -> first)
      // N not a prime; try next candidate
        N+=2;
    }
    // N < heap.begin() -> first
    // N is prime
    table[count++] = N;
    if (N <= max) {
      heap.push_back(pnn(N*N,N));
      push_heap(heap.begin(), heap.end(), gt);
      // new pair absorbed in heap
    }
    // N is a prime; try next candidate anyway
    N += 2;
  }
}

Thanks to Paul McJones and to Tomas Bednar for pointing out an error in the first posted version.

Many textbooks in numerical analysis start by showing that in certain situations the small errors caused by the use of floating-point arithmetic can rapidly grow and render the result of the computation meaningless. Such textbooks describe conditions under which mathematical algorithms can safely be transplanted to floating-point hardware: that the problem be well-conditioned and that the algorithm be stable. In the early days computing centres employed numerical analysts to make sure that none of the scarce processor cycles got wasted on meaningless results caused by ill-conditioned problems or unstable algorithms.

Much has changed since then. Thousands of scientists and engineers have on their desks gigaflops of computing power, and there is not a numerical analyst in sight. Although an even larger part of problem-solving is entrusted to the computer, the same fragile methodology is followed. And still the only justification is that conventional numerical computation seems to work most of the time.

Robust alternatives have existed for decades. These are not magical cures for ill-conditioned problems and unstable algorithms. They are robust in the sense that it is impossible, or near-impossible (depending on the alternative), that an incorrect result is returned without a clear warning. A responsible response in such a case is to critically examine the model (is it well-conditioned?) and the algorithm (is it stable?).

Such robustness comes at the price of increased demands on the processor(s) and memory. So far practitioners have invariably preferred to use the abundant increase in computing capacity for running the same problem with a finer mesh rather than for the security attainable by robust methods.

In science this preference is not likely to change: by nature, scientists feel at home with Rube Goldberg devices held together by sealing wax and string. Engineers operate in a different world. In a court of law, a professional engineer can only appeal to his “best effort” in using software that is not worse than what everybody else uses. In turn, these widely accepted packages can only appeal to being best efforts in selecting algorithms that are numerically stable. It is up to the user to select an algorithm appropriate to the problem and to ensure that the problem is well-conditioned. Even for a numerical analyst this is sometimes hard to tell; harder than solving the problem itself. The time may eventually come that these facts penetrate to the engineer’s mind.

The above should not be a revelation. A promotional document of Intel says it well:

The standard doesn’t provide a guarantee that the answers are the “right” answer. That may require careful error analysis. However the standard provides many mechanisms that make it easier for a program to get the “right” answer on conforming computers from different vendors. — John Crawford [1]

Like many manufacturers who warn that you “may need to consult your physician” for safe use of the product, Intel says that your problem “may require careful error analysis”. Yes, and who can tell you whether your problem actually does fall into this dread category? That would have to be one of those rare and expensive numerical experts.

Before we continue, let me try to state briefly what I mean by “conventional numerical computation”. Let’s say we are modeling an equilibrium state. It could be the state of a petroleum refinery or of an aircraft in flight. Although these are dynamical systems, suppose we are interested in the static state of equilibrium. The state variables have to satisfy a system of equations (let’s say, nonlinear algebraic equations), and we need to solve the system. Unless the system is trivial, an algorithm for solving it will be iterative. That is, the algorithm itself is a dynamical system. For the algorithm to be considered at all, it has to be a dynamical system that evolves by its own dynamic law to an equilibrium state, and this needs to coïncide with the solution of the system of equations.

This is what numerical analysts do for a living: invent such dynamical systems in the form of algorithms. In our example, this could be the n-dimensional version of Newton’s method. A proper textbook will not only describe the algorithm, but also conditions under which it will converge. What’s missing from the conditions is the assumption that holds for most of the entire textbook: all variables range over the real numbers and all operations are the ones defined for the real numbers. That is, the algorithm itself is an abstract dynamical system; its properties hold in the abstract setting. The properties have been proved with some manipulation of formulas. In this manipulation it is taken for granted that certain laws, like associativity of addition and multiplication, hold. And they do: the proof is given for the abstract dynamical system.

So far so good. Now comes the sleight of hand: every real variable is replaced by a floating-point variable. Moreover, every operation on the reals is replaced by … by what? That’s hard to say more exactly than: the operation on floating-point numbers that has the same name as one of the operations on the reals. This is what you are up to: prove that the resulting concrete dynamical system, the one that takes the computer through its paces, converges, and converges to a state near to the state of its abstract counterpart.

In the most favourable case, the abstract dynamical system has been proved correct. Most people don’t even realize that what goes on in the computer is not what has been proved correct. That it is not just a matter of the abstract dynamical system’s variables being jiggled a bit by rounding, but where even the operations themselves are perturbed. This is what I call unprotected numerical computation. The fact that it works most of the time is a bloody marvel.

Before this is getting too long, I should do two things. One is to show some concrete examples of computations going off the rails. The other is to give you some idea of safe alternatives to unprotected numerical computation.

All users appreciate that answers can be off because rounding errors, though small, can accumulate. Some reason like “My computation takes about half an hour, say a thousand seconds. At a gigaflop per second that adds up to 10^12 operations. Double-length arithmetic gives 10^-15 relative error. Hmm … could I get a total relative error of 10^-3? Not good … but wait! These errors cancel each other most of the time, so I’ll end up not much worse than 10^-15 relative error”. As chapter one of the numerical analysis textbook explains, this error model is not satisfactory. Appendix A shows an example (attributed to S. Rump of Hamburg) where a few dozen double-length operations are enough to wreak total havoc.

What gives away the evil expression in Appendix A is the fact that the results for single length and double length are different. However, as Parker [2] shows, this is no guarantee against error. See Appendix B.

Are there alternatives to the Conventional Method? I’m aware of two: interval arithmetic [4] and Monte Carlo arithmetic [2, 3]. Let’s start with the latter. Remember that what motivated the competing computer manufacturers to collaborate on a standard for floating-point arithmetic was that their products gave different answers for the same problem while none of the manufacturers was confident that their own were any better. As long as everyone’s results were the same, all would be well, at least on the sales front.

The big boys in numerics in pre-standard time were CDC, IBM, and Cray. All three had bad floating-point arithmetic compared with the standard to come, and bad in different ways. At least theoretically it was possible for an inmate of an IBM installation to go over to the neighbours with a CDC and ask for the favour of running the program. If the results didn’t differ too much, then you could be reasonably confident that the results were not far from correct. But now your IBM is IEEE standard and whatever the neighbours have, theirs is IEEE standard too. To the industry’s great relief, it speaks with One Voice.

Admittedly, running over to the neighbours for a check is not practical. And, as Intel’s Palmer said in the above quote, the standard makes it easier to develop safe alternatives to unprotected computation. Monte Carlo arithmetic [2,3] is a case in point. It uses the idea of running over to the neighbours for a check, but thanks to the standard, you can get a mild version of a different arithmetic right in your own processor, even without interrupting your computation. Monte Carlo arithmetic makes use of the standard’s provision to change rounding mode under program control while the program is running. Repeated runs can then be regarded as a sample and analysed statistically. I count this as a safe alternative because an unduly high standard deviation casts doubt on the validity of the result.

The other safe alternative is interval arithmetic [4]. Here one operates on intervals of numbers rather than on individual numbers. Let [x,y] assume x not greater than y and let it stand for the interval of all numbers from x to y. Then, for example, interval subtraction turns out to be [a,b]-[c,d] = [a-d,b-c]. The bounds a-d and b-c in the result are computed in floating-point arithmetic and are in general subject to rounding error. The standard makes it possible to compute a-d in rounding mode towards minus infinity and b-c in rounding mode towards plus infinity. Rounding introduces uncertainty about a result. By picking the rounding modes in this way, we are assured that, in spite of the unavoidably introduced uncertainty, the result interval contains all possible values. In this sense interval arithmetic is a safe alternative.

Of course, in a problematic problem, neither Monte Carlo nor interval arithmetic can give you the solution. But either will tell you that you have a problem. In Monte Carlo arithmetic in the form of an excessive standard deviation; in interval arithmetic in the form of an excessively wide interval. For example, interval evaluation of function f1 in Appendix A results in an interval with a width in the order of 10^21; a clear warning, if nothing else.

The excessive width is due to catastrophic cancellation of figures. Suppose we do not have significant cancellation, but the typical large number of operations that we have our gigaflops for. Then the accumulation of the small rounding errors does become a problem, because in interval arithmetic they are all in the same direction for each of the bounds. As a consequence, the result interval, though correct, is much too wide. It is so wide because it reflects the (unlikely) worst case of the unprotected version of the same computation. In this respect Monte Carlo arithmetic is better: it gives a single result with an error under statistical control.

I would not recommend merely intervalized versions of conventional algorithms. Interval arithmetic is of great interest, but only because of the algorithms it makes possible that are not even dreamt of in conventional computation. For example, algorithms that give intervals for all zeros of a system of nonlinear equations; algorithms for the hardest case of mathematical programming: constrained global optimization for objective functions with large numbers of local extrema. When used with the variant of interval arithmetic that is known as interval constraints [5] one can even mix in integer variables in the type of optimization just mentioned.

It is time to dump the fiction that we can perform the error analysis that would be necessary to justify unprotected numerical computation. The safe alternatives are safe in that they warn of trouble and that they can be relied on in the absence of warning. To get this certainty we have to live with the possibility that the safe method gives a warning while unprotected execution gives a correct result.

References

[1] www.intel.com/standards/floatingpoint.pdf
[2] “Monte Carlo Arithmetic: exploiting randomness in floating-point arithmetic” by D. Stott Parker, Research Report CSD-970002, UCLA Computer Science Department, 1997.
[3] “Monte Carlo arithmetic: how to gamble with floating-point and win” by D. Stott Parker, Brad Pierce, and Paul R. Eggert; Computing in Science and Engineering, Vol. 2, No. 4, pp. 58-68, July/August 2000.
[4] Introduction to Interval Analysis by R.E. Moore, R.B. Kearfott, and M.J. Cloud; SIAM, 2009.
[5] Numerica: A Modeling Language for Global Optimization by P. Van Hentenryck, Y. Deville, L. Michel; MIT Press, 1997.

Acknowledgements

Thanks to Dr E.J. van Kampen of the Technical University in Delft for checking out some results in interval arithmetic.

Appendix A

In the C++ program below the function f1 requires a few dozen floating-point operations. The expression in f1 happens to simplify to the one in function f0. If the two functions give different results, as they do here, then the latter is more likely correct.

#include <stdio.h>

template<class flpt>
flpt f1(flpt x, flpt y) {
  flpt x2 = x*x;
  flpt y2 = y*y, y4 = y2*y2, y6 = y2*y4, y8 = y4*y4;
  return (333.75-x2)*y6 + x2*(11*x2*y2-121*y4-2) +
         5.5*y8 + x/(2*y);
}
template<class flpt>
flpt f0(flpt x, flpt y) { return (-2+x/(2*y)); }
int main() {
  printf("%e  %e  %e\n",
         f1<float>(77617,33096),
         f1<double>(77617,33096),
         f0<double>(77617,33096)
        );
}
/* Output:
-2.341193e+29  1.172604e+00  -8.273961e-01
*/

Appendix B

#include <iostream>
#include <iomanip>
using namespace std;

template<class flpt>
flpt f(flpt x0, flpt x1) {
  return (111.0 - 1130.0/x1 + 3000/(x0*x1));
}

int main() {
  cout << setiosflags(ios::fixed) << setprecision(8);
  float f0, f1, tempf; double d0, d1, tempd;
  const int N = 30;
  f0 = d0 = 2;
  f1 = d1 = -4.0;
  for (int i = 1; i <= N; i++) {
    tempf = f1; tempd = d1;
    f1 = f<float>(f0,f1); d1 = f<double>(d0,d1);
    f0 = tempf; d0 = tempd;
  }
  cout << f1 << "  " << d1 << endl;
}
/* Output:
100.00000000  100.00000000
*/

This C++ program computes the same short iteration in both single and double length, with good agreement. However, the true value is

990176025870222717970867 / 164874117215934539909207 =

6.0056484877…

See Parker [2].

Though the standard was only ratified in 1985, most of the work had been done several years before. Up till the late 1970s the computing community was resigned to the situation in which the manufacturers that dominated the market for scientific computers fielded processors with execrably bad implementations of the main tool: floating-point arithmetic. Users had to live with “features” such as:

• Numbers that tested as nonzero in a comparison, but behaved as zero in division.
• Numbers that underwent non-negligible change when multiplied by 1.0.
• X-Y could evaluate to zero for different X and Y.

The whole idea of floating-point arithmetic was that you could just replace the real variables in a mathematically verified algorithm by floating-point numbers. In reality programmers had to insert code like X := (X+X)-X; at critical spots [1].

At the time there was no shortage of experts who could tell what was wrong and how to improve it. One of the best was William Kahan at Berkeley, who was in fact consulting for IBM, with little effect on the egregious shortcomings of their processors, as quality of floating-point processing was not regarded as a competitive advantage. The major players at the time, IBM and CDC, both had bad floating-point arithmetic.

Here we have a glimpse of the situation:

The [CDC] 6600 was the first commercial supercomputer, outperforming everything then available by a wide margin. While expensive, for those that needed the absolutely fastest computer available there was nothing else on the market that could compete. When other companies (namely IBM) attempted to create machines with similar performance, he [Seymour Cray] increased the challenge by releasing the 5-fold faster CDC 7600. [2]

As you can see, speed was all that matters. Speed was something that the VPs in charge of multimillion dollar purchases could understand. They neither knew nor cared what was done at that speed.

To find state-of-the-art floating-point arithmetic one had to look elsewhere, in the unlikely direction of a cheap mini-computer, in this case the PDP-11 from DEC. To the relief of those who cared about quality in computer arithmetic, the subsequent VAX (bigger and faster) had arithmetic that was no worse. But the big players like CDC and IBM, and, by this time, also Cray Computer, felt no need to do anything about their deficient floating-point arithmetic.

While DEC could safely be ignored around 1980, Intel was not even noticed as a potential source of processors. One of the few companies that took Intel seriously was Intel. They planned a floating-point co-processor and retained Kahan as consultant. He advised them to adopt the specifications of the arithmetic of the DEC VAX. This was state-of-the-art, and much better than that of the big players. It was also insanely ambitious for a single chip with a mere 40,000 transistors, which is what Intel had available.

However, Intel declined to follow Kahan’s advice. They did not want the state of the art: they want the best possible. Kahan, working with J. Coonen and H. Stone, produced a specification of a floating-point arithmetic that was the experts’ dream. By the early eighties, when the dinosaurs still roamed the earth [4], Intel had implemented this in 40,000 transistors.

By 1985, when the IEEE standard 754 was finally ratified, there were half a dozen implementations on the market, ominously for the then supercomputers, all these were single-chip CMOS (co-)processors.

In this success story Kahan [1] and other participants emphasize the altruism displayed by their colleagues. The IBM representative was supportive, even though the products of his company at the time were nowhere near compliant. But I can’t help taking a less rosy view of the situation: all participants had a vested interest in avoiding the situation that the same numerical program would give different results on their different computers. I can’t help thinking that they were more worried about results seen to be wrong than about them actually being wrong. As long as the machines wouldn’t contradict each other, all would be fine.

A program giving significantly different results on different machines is likely to be unsound numerically; it may well be that the different results are all wrong, including the result obtained by an IEEE-754 compliant machine. Such a program needs to be analyzed by an expert and rewritten. This would be expensive. The “old anarchy”, much deplored by those who celebrate IEEE 754, could actually be a valuable warning that the algorithm underlying the program is not stable or that the problem is ill-conditioned. Not welcome news. The Brave New World of IEEE 754 prevents such warnings from ever arising.

The Standard Methodology is:

Design an algorithm in terms of real numbers that is stable. Replace the reals by floating-point numbers; replace operations on reals by the ones on the floating-point numbers that have the same name. Apply to a model that is numerically well-conditioned.

The problems with this are (a) that the stability depends on properties of the reals (like associativity of operators) that their floating-point counterparts do not have and (b) that it is often not easy, even for an expert, to determine whether a model is well-conditioned. For example, for some computations the condition depends on a certain eigenvalue of a matrix. And it may be that determining that eigenvalue is harder than the problem you want to solve in the first place.

Thus, a valuable supplement to the Standard Methodology may be to run your program program on a CDC 6600 as well as on an IBM 360, both bad, and in different ways. If the results do not differ significantly, then this can be used to boost confidence. Otherwise, it should be “back to the drawing board”. Intel, in a document celebrating their heroic role in IEEE 754, includes a very apt warning:

“The standard doesn’t provide a guarantee that the answers are the “right” answer. That may require careful error analysis. However the standard provides many mechanisms that make it easier for a program to get the “right” answer on conforming computers from different vendors.” — John Crawford [3]

Yes, and you need an expert for the required careful analysis. Every engineer has gigaflops on his desktop, but there’s not an expert in sight.

Enough of these dark thoughts. What is so wonderful about IEEE 754? Admirers of the standard may give different answers; here I’ll give mine.

• The reasonable accuracy achievable by a guard digit, which had been known for decades, but ignored by most of the industry, is guaranteed. The guarantee can be expressed as: if the floating-point result of an operation coincides with the real-valued result, then that is the result; otherwise it is one of the bounds of the smallest floating-point interval containing the real-valued result. And you can pick which of the bounds you want; see next.
• Rounding mode is not fixed. It can be upward, downward, towards zero, or to nearest. This can be changed at runtime.
• One of the quirks of number systems is that you have to have an asymmetric collection of numbers if you, correctly, want a single zero. Then you necessarily have a zero in the middle and symmetrically on either side the positive and negative numbers. If you want symmetry, then you must have either no zeroes, or two, a positive and negative one. This latter option seems to be inherent in the notion of a floating-point number system, so it’s not surprising that IEEE 754 has a positive and a negative zero.But the standard provides for the natural counterparts of the zeros, which are the infinities, positive and negative, so that division by a zero of a nonzero finite number gives you infinity of the right sign and so that division of such a number by infinity gives the zero of the right sign. In this way computation can continue after division by zero if that makes sense in your model, or can be trapped if not.

In a future post I hope to be able to say something about the Standard Methodology of numerical computation and its alternatives. In the Appendix I present in 27 lines the salient details of IEEE 754. I feel that these are also the most important details of IEEE 754-2008, the current version.

References

[1] http://www.cs.berkeley.edu/~wkahan/ieee754status/754story.html
[2] http://en.wikipedia.org/wiki/Seymour_Cray
[3] www.intel.com/standards/floatingpoint.pdf
[4] The last of Seymour Cray’s companies, Cray Computer Corporation, filed for bankruptcy in 1995 (from Wikipedia article “Seymour Cray”). Behind the current “Cray” computers is a succession of other companies that passed on the rights to the “Cray” marque.

Appendix

By now the 1985 IEEE 754 has been replaced IEEE 754-2008. It’s a fat wad of paper not particularly encouraging for those whose curiosity has been aroused by the story so far. Apart from some niggling details, the original is a subset of the current standard. It so happens that I advocate a model of computing, about which more later, that allows one even to ignore most of the much smaller old standard. All that I want to know about floating-point arithmetic is the part of the standard described in the following few lines of ASCII-art, lifted straight out of my code:

/* Precis of the IEEE standard 754 format
Layout of fields:
Single-length  31-31: sign  30-23: exponent  22-0: significand
Double-length  63-63: sign  62-52: exponent  51-0: significand
Least significant bit: 0
Most significant bit: 31 (single-length), 63 (double-length)

Single-length                Double-length               Value
------------------------   ---------------------------
 exponent    significand   exponent        significand
U     U-Bias              U      U-Bias
=====================================================================
0     -127        0       0      -1023         0         0
0     -127      nonzero   0      -1023        nonzero   see Denormalized
1-254 -126-+127 anything  1-2046 -1022-+1023  anything  see Normal
255   +128       0        2047   +1024         0        infinity
255   +128      nonzero   2047   +1024        nonzero   NaN

Sign bit:  1 ~ negative    0 ~ positive
Significand: the bits after the binary point
Bias:    single-length, 127       double-length, 1023
Emin:    single-length, -126      double-length, -1022
Columns U: value of exponent field as unsigned integer
Columns U-Bias: as above, with Bias subtracted
Normal: (1+0.significand)*2^(exponent-bias)
Denormalized: (0.significand)*2^Emin
*/

The single-length format occupies 32 bits, numbered from 31 for the most significant to 0 for the least. Double length has 64 bits, numbered similarly. Emin is the least value of the exponent. Bias is the number to be subtracted from the exponent field (interpreted as unsigned number) to get the exponent.

The exciting thing about this little table is that it lets you have a peek under the hood of the computer on your desk. Let’s have a look at what one tenth looks like in IEEE 754. In the following C program one tenth in single-length floating-point is aliased to an unsigned integer and then printed out in hexadecimal. This way we get to see the bits, which we then interpret with above table.

#include<stdio.h>

int main() {
union fltInt {float F; unsigned U;} x;
x.F = 0.1;
printf("%f %x\n", x.F, x.U);
}
/* Output:
0.100000 3dcccccd
*/

Hm … can we make sense of 3dcccccd? First lay it out in bits:

 3    d    c    c    c    c    c    d
0011 1101 1100 1100 1100 1100 1100 1101

Then re-arrange according to single-length floating-point format:

sign  exponent    significand
0     01111011    10011001100110011001101

To make sense of the significand (the fractional part after the binary point) we have to realize that what we see here is not the one that belongs to one tenth, because you cannot represent it as a binary numeral. As one tenth is a rational number, it is an infinitely repeating binary numeral. As ten only has small prime factors, there have to be several repetition cycles here. Hence 3dcccccd has to be the rounded version of

sign  exponent    significand
0     01111011    10011001100110011001100110011001...ad inf

The sign bit is 0, which means positive (see table). The exponent is 123 as unsigned integer; -4 after subtracting the bias of 127 (see table). According to formula Normal in the table, the value is

    + 2^{-4} * 1.10011001100110011001100110011001...ad inf
=   + 2^{-4} * (1 + (1 + 9/16 + 9/(16^2) + ...)/16)
=   + 2^{-4} * (1 + 3/5)
=   + 1/10.

The infinite binary expansion of 1/10 is between 3dcccccc and 3dcccccd. It is closer to the latter. The default rounding mode is “round to nearest”. As the program did not change the rounding mode, we should find 3dcccccd, which we did.

Initially, computers reflected the age of the formula: FORTRAN comes from FORmula TRANslator. But FORTRAN was found more useful for writing algorithms in which formulas played a minor role.  Used in the right way, an algorithm can be self-explanatory in a way that the old-style formulas were not. The formulaic paradigm was authoritarian in the sense that the user did not get an explanation of the formula found in the book. Thus the algorithmic paradigm is less authoritarian: access to the source code implies access to an explanation, not of a formula, but of the number you are getting. Yet relics from the age of formulas linger. In this article I discuss two of these and use them to illustrate the power of the algorithmic approach.

Calendrical calculations are, more than those in other areas, encrusted in ancient lore and antiquated methods. Such anachronisms were the motivation for Dershowitz and Reingold to write Calendrical Calculations. They set out to make the subject computational and use an algorithmic language. Yet they remain to some extent under the thrall of the formulaic paradigm. For example, for the standard (“Gregorian”) calendar, they derive from more complex formulas, after considerable algebraic manipulation, that the number of days in months 1,…,m-1 is given by the floor of (367m-362)/12. To prevent all this algebra becoming more complicated, the assumption was made that February always has 30 days, so this still needs to be corrected later.

Let us compare this approach with the algorithmic alternative. To start with, the formula for the number of days in the completed months before a given date was needed as stepping stone to the serial number of the day in the year, starting with January 1 as day 1. In the algorithmic approach we might as well tackle that directly, for example with:

int dayNum(int d, int m, int y) {
  while (m > 1) { m--; d += monthLen(m, y); }
  return d;
}

My claim is that this is easier to explain than the formula. By “explain” I don’t mean a mathematical correctness proof. Many pages of formalism have been expended on justifying such code mathematically, with formally defined Variants and Invariants. The formalism tends to be more forbidding than the algebra of Calendrical Calculations. No, the power of the algorithmic approach is that the thinking behind the algorithm does not need to be formalized. For a programmer the justification does not even become conscious as the code is written. If the reader needs help, it can be given in the form of prose in the vein of:

The sum of d and the number of days in the months 1,…,m-1 is constant. Before entry into the loop this constant is the number that we want the function to compute. The loop maintains the value of the constant while driving m down. As the loop exits we have m equal to one, so that d is equal to the constant.

Of course the algorithm needs a function for the lengths of the months. Our grandparents (as did their grandparents before them) remembered these by:

Thirty days hath September,
April, June, and November;
All the rest have thirty-and-one,
Save February, with twenty-eight days clear,
And twenty-nine in each leap year,

which is almost C:

    int monthLen(int m, int y) {
      switch (m) {
        case 4: case 6: case 9: case 11: return 30;
        case 2: return leap(y) ? 29 : 28;
        default: return 31;
      }
    }

Of course we still need a function for telling leap years. Even after deciding to use C rather than algebra, we have a rearguard action to fight. In textbooks one often finds something like

int leap(int y) {
  return (y%400 == 0) || (y%4 == 0 && y%100 != 0);
}

Whether it is helpful to express it this way depends on how one remembers the rule. A common way to remember it is as a rule with exceptions, which in turn have exceptions. To be specific: a year is normally not a leap year, except when its number is divisible by 4, but it’s normal again when divisible by 100, except when divisible by 400. The following is a direct translation of this way of remembering the Gregorian calendar:

   int leap(int year) {
     if (year%400 == 0) return 1;
     if (year%100 == 0) return 0;
     if (year%4 == 0) return 1;
     return 0;
   }

This completes what I want to say about an algorithmic alternative to that particular formula of Calendrical Calculations. That formula at least has an easily accessible justification. In my next example this is not the case, though the temptation to use a formula is more understandable. Googling “Zeller’s Congruence” gives evidence of the use in programs of a formula as a quick and easy way to determine the day of the week for a given date in <day,month,year> format.

In the 19th century Christian Zeller [G] invented such a formula to compute the day of the week with y the year, m the month, and d the day of the month:

    ((d+(((m+1)*26)/10)+k+(k/4)+(j/4))-(2*j))%7.

Here k = y%100 and j = y/100; all divisions are integer divisions. The value of the formula represents Saturday as 0, and the other days of the week accordingly. (Actually, the month and year may need to be tweaked a bit; see the code further on.) As the formula maps an infinity of dates to a finite domain of seven values, it can be thought of as establishing an equivalence relation among dates. It’s not clear where “congruence” comes from, but that is what it seems to be called.

Zeller’s formula represents the formulaic paradigm: it solves the problem with a minimum of computation, it is not easy to prove correct, and, I guess, it required considerable ingenuity to invent. In the formulaic paradigm this effort is locked up in a black box to be preserved as intellectual capital for future use. Its use requires mindless and obedient substitution of values followed by the prescribed arithmetical operations.

I propose to use algorithms in two ways, minor and major. The minor one is to prove the correctness of Zeller’s formula; the major one is to do away with the formula and substitute an algorithm for it. This substitution has the advantage that the algorithm is more intelligible than the formula. It will turn out to have the disadvantage that more computation is needed to obtain the answer. But the difference in computational requirement is not noticeable when a computer is used to execute the algorithm.

How can I hope to prove the correctness of Zeller’s formula when I despair of even understanding it? It turns out to be easy to show that only a finite number of dates need to considered and that it is, for contemporary computers, a negligible computational task to evaluate the formula for each of these dates.

To see that the checking of the formula can be reduced to a finite number of cases, it useful to view the rule for leap years in the Gregorian calendar as a system of nested cycles rather than nested exceptions:

• The innermost cycle consists of three normal years followed by a leap year; let’s call this a quad. The length of the quad is 4*365+1, which is 1461 days.
• Wrapped around the quads is the next cycle: the century. It consists of 24 normal quads followed by one that is exceptional in being one day shorter. Let’s call this cycle a cent. The length of the cent is 25*|quad|-1, which is 36524 days.
• The outermost cycle consists of three normal cents followed by an exceptional one, which is one day longer. I call this cycle a clav, in honour of Clavius [G], the astronomer of Pope Gregory XIII. Clavius was, I suspect, the brains behind the Gregorian calendar. The length of the clav is 4*|cent|+1, which is 146097 days.

Let me note, if I may indulge in a short digression, that this gives 146097/400 = 365.2425 days as the length of a year averaged over a clav, and, by the cyclical nature of the calendar, this is also the average over all eternity. As the average length of the year is actually 365.2422 days, we see that Clavius, with his simple rule, managed to create a calendar that is, on average, only off by 0.0003 days per year: it takes around three thousand years to be off by a single day.

After this digression I am ready to state

Theorem 1: The formula of Zeller gives the day of week for every day of the Gregorian calendar.

Proof:
We first reduce the theorem, which makes an assertion for an infinite number of dates, to a finite number of special cases. This can be done by observing that the length of a clav is divisible by 7. Hence two dates fall on the same day of the week if they differ only in the year and if this is in such a way that the difference in year numbers is divisible by 400 (you can call this Van Emden’s Congruence, if you must). Thus the correctness of Zeller’s Congruence only needs to be checked for the 146097 days of a single clav.

In mathematics it is a well-known stratagem to make a proof easier by splitting into special cases, like n<0, n=0, and n>0. But mathematicians take a dim view of, say, a dozen special cases. When Appel and Haken [G] split the Four-Color Theorem into 1936 special cases (each to be checked with a computer program), there were lots of people who objected. You can imagine what these people will think of puny Theorem 1 and the need for 146097 cases to be checked by a computer.

After noting this objection, the proof: as the program below terminates normally with as output a single line containing “146097″, the Theorem is proved.

#include<iostream>

int Zeller(int day, int month, int year) {
  int d = day, m = month, y = year;
  assert(!(m<3 && year == 0));
  if (m  Saturday etc
  int h =
    ((d+(((m+1)*26)/10)+k+(k/4)+(j/4))-(2*j))%7;
/* From Wikipedia article "Zeller's Congruence"
   observed September 15, 2010.
   Convert from h to 1 ~ Monday, ..., 7 ~ Sunday
*/
  return (h+5)%7 + 1;
}
int proof() {
/* Returns the number of days for which the Zeller formula
   gives the true value as obtained by counting backwards
   from a known day. Fails at first case, if any, in which
   the formula does not give the true value.
*/
  int year0 = 1, year1 = 400;
  int dayOfWk = 7; // Dec 31, 400 was a Sunday
  int count = 0;
  for(int year=year1; year>=year0; year--)
    for(int month=12; month>=1; month--)
      for(int day = monthLen(month, year);
          day >= 1; day--
         ) { assert(Zeller(day,month,year) == dayOfWk);
             dayOfWk--; if (dayOfWk < 1) dayOfWk += 7;
             count++;
           }
  return count;
}
int main() {
  printf("%d\n", proof());
  return 0;
}

My reason for preferring the algorithmic to the formulaic paradigm is that I find writing the proof a straightforward exercise, whereas understanding Zeller’s formula seems a daunting task to me. I trust that my readers will not find my code unduly difficult to understand and will find it less mysterious than Zeller’s formula. Alan Robinson has called for the need of  “intelligibility engineering”; I hope he accepts this as a contribution.

So far the minor application of the algorithmic paradigm in matters calendrical. Minor, because the day of the week for distant dates belongs in trivia-land, like the storming of the Bastille happening on a Tuesday. Perhaps a bit more significant is that Zeller’s formula gives Sunday for December 7, 1941. The formula may also be used to assure someone born on December 13, 1973 that this day was not a Friday, so that subsequent mishaps have to be attributed to other causes.

My major example of demonstrating the algorithmic alternative to formulas is to replace Zeller’s formula by an algorithm. The reason for doing so is that the algorithm, although occupying many more times the number of lines of the formula, is intelligible: it can be read straight through compared to the need to puzzle over the single line of the formula.

What is an algorithmic alternative to a single-line formula for determining the day of the week? A promising starting point is the essence of a calendar: a system to name the days. In an algorithmic approach, a calendar uses the fact that the days form a sequence. This suggests to use integers for the names. In this way the calender does not only name the days, but does also count them. And counts not only the days themselves, but the number of days between any two dates.

I follow this approach. The result is a building block that can be used for many calendrical computations. The algorithm for numbering the days will be more intelligible than Zeller’s formula. The algorithm will also more widely applicable, while the day of week is just one of many uses of converting days to integers. Thus the algorithmic alternative to Zeller’s formula will take the form of a function that converts a date in <day,month,year> format to an integer. As a side effect, knowing the day of the week of just one date (like, say, today) implies the equivalent of Zeller’s formula via the remainder on division by seven.

I follow Dershowitz and Reingold in calling the integer name of a day rata diem and I abbreviate it to rd. These authors assign the rd of 1 to January 1 of year 1. Perhaps the idea is that the symmetry that the integers have with respect to 0 can be used in the rd naming scheme. This would be a promising approach, were it not for the fact that the Gregorian calendar, with its positive and negative years, is not symmetric: for positive years time moves away from the origin; for negative years it moves in the same absolute direction, which is towards the origin. As computer integers have a limited range anyway, a computerized calendar has to have an earliest date as well as a latest date. Accordingly, I use unsigned integers for the rata diem. As 32 bits is a commonly available size of unsigned integers and gives a generous number of years into past and future, I choose this size. As there is an even number of integers, there are two midpoints: 7fffffff and 80000000 in hexadecimal. I choose the later one of these as the rd of January 1 of year 1. I call this date the epoch, adopting another bit of calendrical terminology from [DR97].

Here is the first bit of the program:

typedef unsigned uint;
const uint epoch = 0x80000000; // Rata Diem of January 1, 1 C.E.

const int yLen = 365;               // length of normal year
const int quadLen = 4*yLen + 1;     // length of normal quad
const int centLen = 25*quadLen - 1; // length of normal century
const int clavLen = 4*centLen + 1;  // length of clav

Notice that I have switched to the C++ compiler. Not because I want to do any object-oriented programming, but because of the minor civilities that this improved form of C affords. Remember that I relied on my shaky pencil-on-back-of-the-envelope calculations to come to the conclusion that a clav has 146097 days? A man shouldn’t have to do that in this day ‘n age, but C forbids anything except a literal in the initialization of a constant. (I didn’t actually mind using a pencil. The real reason is that I want to replace magic numbers by their provenance, as documentation).

A promising strategy for converting Rata Diem to <day,month,year> is to split the rd into the days coming from the completed years (if any) in the past and the days coming from the current year. The latter number is computed by our old friend:

int dayNum(int d, int m, int y) {
  while (m > 1) { m--; d += monthLen(m, y); }
  return d;
}

Function dayNum() was written with positive years in mind. For negative years the completed years are greater than the current year and the days in the current year are not counted forward from the beginning, but backwards from the end of the year. Rather than tweak dayNum() to a form that works in negative years as well as positive years, I use the fact that the Gregorian calendar is the same for every clav. So I shift every date, irrespective of whether the year is positive, negative, or in between, to the clav that begins on January 1, 1 C.E. and ends on December 31, 400 C.E. I implement this shift by:

 while (year < 1) {
    year += 400; rd -= clavLen;
  }
  while (year > 400) {
    year -= 400; rd += clavLen;
  }

This takes linear time. Yes, it could be done in constant time. But the rule for modulo a negative number is less clear for me, so I prefer the slower alternative. After all, the goal is intelligibility.

See the Appendix for a complete set of C++ functions.

Zeller’s formula, though unintelligible, has the advantage of being computationally efficient, more so than my algorithmic alternative. In my experience the most extreme example of an unintelligible formula is Matijasevic’s polynomial [G] that is claimed to be a representation of the prime numbers. Not only is it a mystery to all except the few initiated why the polynomial should do so, but it is also computationally inefficient in the extreme. The polynomial is of degree 25 and has 26 variables. It is claimed to represent the prime numbers in the sense that, as the 26 variables range over the non-negative integers, all positive values of the polynomial are prime and vice versa. Contrast this, for example, with Dijkstra’s prime-number generator [DDH72], which is intelligible as well as reasonably efficient.

In this article I have concentrated on examples where algorithms are preferable to formulas. I’m not forgetting that there is a whole world of programming that exists on the strength of the opposite: in functional programming one is forced to abstain from algorithms and to express everything in formulas. Of course, the formulas allowed in functional programming form a less restrictive language than the one to which Zeller was limited. A modern language of formulas is enriched with conditionals and with recursion. These facilities introduce possibilities for explanation that were not open to Zeller.

References

[1] Information on this was found with Google.

[2] Calendrical Calculations by Nachum Dershowitz and Edward Reingold; Cambridge University Press 1997.

[3] Structured Programming by O.-J. Dahl, E.W. Dijkstra, and C.A.R. Hoare; Academic Press 1974.

[4] The ability to tell the day of week of distant dates occurs in several calculating prodigies; see The Great Mental Calculators by Steven Smith, Columbia University Press 1983. Another study is an essay by Oliver Sacks (“The Twins”, in The Man Who Mistook His Wife for a Hat, Picador, 1986) describing twin brothers with this faculty. Sacks reports (page 187) that the men had an IQ of sixty, could not do addition or subtraction with any accuracy, and could not comprehend the meaning of multiplication or division. This rules out Zeller’s formula as their secret weapon. In the Postscript on page 200 Sacks quotes one Israel Rosenfeld, apparently consulted for his supposedly superior knowledge of matters numerical:

Their ability to determine the days of the week within an eighty-thousand-year period suggests a rather simple algorithm. One divides the total number of days between ‘now’ and ‘then’ by seven. …

One wonders whether Mr Rosenfeld has tried out his “rather simple algorithm”.

Appendix

#include<iostream> // for assert and printf

typedef unsigned uint;
typedef struct{int day; int month; int year;} Date;
void printDate(Date d) {
  printf("%u, %u, %d\n", d.day, d.month, d.year);
}
Date mDate(int d, int m, int y) {
  Date date;
  date.day = d; date.month = m; date.year = y;
  return date;
}

const int yLen = 365;               // length of normal year
const int quadLen = 4*yLen + 1;     // length of normal quad
const int centLen = 25*quadLen - 1; // length of normal century
const int clavLen = 4*centLen + 1;  // length of clav
const uint epoch = 0x80000000; // Rata Diem of January 1, 1 C.E.

int leap(int year) {
  if (year%400 == 0) return 1;
  if (year%100 == 0) return 0;
  if (year%4 == 0) return 1;
  return 0;
}
int monthLen(int m, int y) {
  switch (m) {
    case 4: case 6: case 9: case 11: return 30;
    case 2: return leap(y) ? 29 : 28;
    default: return 31;
  }
}
int dayNum(int d, int m, int y) {
  while (m > 1) { m--; d += monthLen(m, y); }
  return d;
}
uint Date2RD(Date argDate) {
  int day = argDate.day, month = argDate.month;
  int year = argDate.year;
// Converts date to Rata Diem.
  int rd = epoch-1; // Rata Diem of December 31, 0
// Normalize date to one in standard clav.
  while (year < 1) {
    year += 400; rd -= clavLen;
  }
  while (year > 400) {
    year -= 400; rd += clavLen;
  }
  assert(1 <= year && year <= 400);
  rd += dayNum(day, month, year);
  year --;
  // year contains number of whole years, if any
  if (year == 0) return rd;
  assert(0 < year);
  rd += (year/100)*centLen; year %= 100;
  rd += (year/4)*quadLen;    year %= 4;
  rd += year*yLen;
  return rd;
}
int dayOfWeek(Date date) {
  uint y = 6+Date2RD(date)%7; // day 0 was a Saturday
  return y>7 ? y-7 : y;
}
int main() {
  printf("%d\n", dayOfWeek(mDate(14,7,1789)));
  return 0;
}

In the following interview we try to cover at least a little bit of this wide and varied terrain.

APP
I first heard of you as part of the team of the CAL time-sharing system in Berkeley. That must have been around 1969. Is that right? Were you still a student at that time?

McJones
Yes, I came to Berkeley in the fall of 1967 as a freshman.

APP
Did you get started on programming in connection with a course? Which?

McJones
No, it wasn’t in a university course. At high school a dedicated physics teacher taught me programming.

APP
Ah! I’m sure that’s a great way to get started. Tell me more.

McJones
Actually, my first taste of programming was even earlier, again because of a dedicated teacher. In the seventh grade, a teacher (now Dr. Marilyn Fendrick; then Mrs. Benefield) encouraged me to pursue my curiosity about digital logic and computers. I read a book called Basic Computer Programming by Theodore G. Scott. This exposed me to the basic ideas of computers, but without a real machine to program, I explored analog and digital electronics for the next few years.

In high school, I took a course called Analytical Techniques in the Physical Sciences, designed and taught by physics teacher Carl Duzen. It included some linear algebra, solving differential equations on an analog computer, and programming a digital computer in FORTRAN IV and assembly language. We used keypunches at a company nearby and an IBM 7094 at an aerospace company in Los Angeles. Duzen only had time to drive to the aerospace company about once a week, so the turn-around time was very long!

Later that year, Jack Perrine, an independent software consultant, visited Duzen’s class and hired me and another student as after-school interns. Perrine had been a member of the team at Computer Sciences Corporation that wrote the original UNIVAC 1107 FORTRAN compiler. Eventually Perrine tired of corporate life, and founded ATHENA Programming to support the compiler. My assignments were to fix bugs in the runtime and to write utility routines such as snapshot dumps — this was my real introduction to assembly language programming. So by the time I entered Berkeley that fall, my experience at ATHENA helped me get an entry-level programming job at the campus Computer Center.

APP
What kind of work did you do there?

McJones
At first it was similar to what I’d done at ATHENA: writing a library subroutine to read very long magnetic tape records, and a utility program to catalog the contents of a tape. The spring of my freshman year, I got interested in Snobol3. It so happened that Charles Simonyi had entered Berkeley the same time as I did. He already had a great deal of experience writing compilers in Hungary and at Regnecentralen in Denmark. At Berkeley he proposed to the Computer Center that he write a from-scratch implementation of Bell Lab’s brand-new Snobol4. The Bell implementation used a virtual machine implemented in macros that was very portable and elegant, but too inefficient in memory and cycles for running student jobs. Charles’s plan was to write in assembly language (for the CDC 6400), base the compiler on the GIER design, and keep the entire system “lean and mean”. Someone suggested I work with Charles; he agreed, and I jumped at the opportunity.

APP
So you were off and running, working with one of the greats in programming, and that in the first year of University! Did you have any time to take courses?

McJones
I was enrolled full-time in the electrical engineering program, enjoying the social life in a dormitory with new friends with diverse backgrounds, riding my motorcyle around the Bay Area, and spending lots of time programming. To be honest, my grades suffered the first year. Late in the year, I met my future wife (at the Xerox machine, while copying the Snobol3 manual), and her good influence helped my grades recover.

APP
What was the fate of the Snobol system? Such projects often get lost in management muddles.

McJones
We built a successful system — if I remember correctly, it was limping along that fall — that was used at Berkeley and other CDC installations well into the 1970s because of its efficiency and despite its lack of full compatibility. Charles created a sound architecture and wrote a prodigious amount of code. (It just occurred to me that he wrote a prototype of the pattern matcher in ALGOL 60). I wrote I/O routines that took advantage of the asynchronous I/O capability of the SCOPE operating system APIs, so a one-line copy program written in our Snobol would keep tapes running at full speed, unlike some of the standard utilities. While our implementation omitted a number of standard Snobol4 features (e.g., the TABLE data type), it also added some new ideas. For example, we wanted to be able to write a symbolic debugger in Snobol4, as one could do in LISP. This required being able to decompose every compound data value, such as the elaborate Snobol4 patterns. So we added appropriate built-in predicates and selectors.

Snobol4 was used in computer science courses since it allowed creating data structures — a role later subsumed by Pascal. It was also used in a course for humanities students since it had good tools for processing text; the lecture notes included an appendix serving as the language reference manual.

A couple of years ago, through my interest in the history of software, I was able to look through the electronic archives of the late Ralph Griswold, one of the Snobol designers. I was amused and pleased to find a copy of the source code to our Snobol system, and even more amused when I was able to assemble and run it on an emulator for the CDC 6600.

APP
When you tell of keypunch days with long turnarounds, I wondered what else you did in addition to working over listings. Text with paper and pencil, of course. Did you use diagrams? Blackboards?

McJones
Schooled as I was in Fortran and assembly code, Algol control structures were a revelation. I wrote Algol-like pseudo code for my initial tasks at the Berkeley Computer Center and inserted it as comments interspersed with the actual assembly language. We drew lots of data structure diagrams with boxes and arrows. Somewhere I still have a photocopy of the ones Charles did for our Snobol’s runtime system: strings, numbers, patterns, and so on. We probably used blackboards some, but I think we often just wrote on the back of an old listing. Here‘s an example of a finished design from those days, complete with lots of diagrams.

APP
After interactive, paper-based teletypes and such the video terminal was an important transition for coders. Was there composition on the keyboard right away? Or was it still via pencil and paper first?

McJones
I first used video terminals at IBM, around 1975. They were in a common terminal room, so I’m pretty sure I often composed ahead of time via pencil and paper. I did have an IBM 2741 Selectric-style terminal in my office, but I think I tended to walk down to the terminal room most of the time — it was a useful way to “share state” with other team members. In late 1976 I went to Xerox. We used Altos for program development from the beginning. There was a brief attempt to have us share them, but we quickly achieved one Alto per person. I still wrote out my programs on paper; among other things, that meant I could work on them in the evening, many miles away from Xerox.

APP
Was the transition to IDE’s significant?

McJones
Most contemporary programmers love them, but I think IDEs always have a struggle dealing with the size and complexity of the largest systems. For example, it may still take 30 minutes to rebuild a complex operating system, even after making a small change.

APP
Because the tools of work have changed so much, one wonders how that influenced productivity, quality, or even thought itself?

McJones
In many programming shops today there is such a focus on speed: very frequent releases; programmers are measured by how much code they write or checkins they perform — do people have time to think? Software developers have become very good at testing, but notions of specification and correctness at design time often seem to be ignored.

APP
Speaking of tools, how about languages. In the first decades, there seem to have been two great transitions: in numerical work from assembler to FORTRAN, and in systems programming from assembler to C. How do these transitions compare?

McJones
They are hardly comparable. The transition to FORTRAN for numerics was extremely fast. Here‘s a September 1957 memo from Backus to his management with statistics showing a very impressive adoption rate only about six months after FORTRAN I shipped. FORTRAN allowed scientists and engineers to work at a fairly natural level of abstraction (numbers and arrays) without paying a performance penalty because of excellent code generation. FORTRAN quickly spread to most computers (rarely with the same quality of code generation), giving rise to the notion of portability. The first edition of McCracken’s book on FORTRAN programming came out in 1961; college courses taught the language to engineers and scientists. All this created a pretty strong positive-feedback loop.

APP
And why is the transition from assembler to C in systems programming hardly comparable?

McJones
It took much longer — roughly two decades — and involved many intermediate languages. I’m not sure quite why, but I’ll speculate that some of the sources of the delay were:

1. Evolution of machine architectures toward the eventual standard byte-addressed memory, uniform integer and floating data formats, and banks of general-purpose registers (the C machine!).
2. Evolution of language features allowing expressive, efficient programming of algorithms (e.g., iteration and selection statements), data structures (record/structure definitions; pointer types and pointer arithmetic, etc.), and management of complexity (preprocessors, modules, abstract data types, polymorphism of various kinds, etc.).
3. Realization by professional programmers (compared with scientists and engineers who program) of the advantages of a standard, portable programming language.

Along the way, people tried out ALGOL 58 dialects (NELIAC, MAD, and JOVIAL), ALGOL 60 extensions (Burroughs ESPOL, SAIL, etc.), ALGOL 60 follow-ons (ALGOL W, Pascal, Mesa, Modula), PL/I, Bliss, BCPL, and even FORTRAN. By the early 1980s, C was emerging as the standard: it was a good match for the standard machine architecture, and expressive enough for disciplined programmers to use for a wide variety of applications and systems. It spread along with Unix, and was relatively simple to port or reimplement on non-Unix operating systems including MS-DOS and Macintosh OS. Since then, C++ has joined C both by being a “better C” and by providing powerful tools for user-defined types.

APP
How about the transition to Object-Oriented Programming?

McJones
While Simula-67 introduced classes, inheritance and virtual procedures, Smalltalk popularized them. In the 1980s, many programmers wanted to try object-oriented programming and many language designers obliged them. Pundits convinced management this would solve all their problems. Java and object-oriented scripting languages came along in the 1990s with garbage collection and huge libraries: for a time, programming appeared to be a simple matter of copy-and-paste. But I think the pendulum is swinging the other way: people once again realize they need to focus on algorithms and data structures as well as “glue”. Inheritance is a programming technique, good for some things but not others.

For the gist of Atwood’s criticism I quote:

Instead, many chapters just reprint a few pages of code and conclude — see, it is beautiful! Many times I was unable to grasp the problem — what was it that required that so-called beauty to emerge? I couldn’t see the whole picture, but the authors presume I do. Any possible appreciation of beauty requires deep understanding.

I can believe that, about the deep understanding! So here we have a software guru, like Brian Kernighan, or Jon Bentley, pointing at a piece of gobbledygook, and saying: “This is beautiful!” They are like many other gurus, through the ages, pointing at this:

μῆνιν ἄειδε θεὰ Πηληϊάδεω Ἀχιλῆος
οὐλομένην, ἣ μυρί’ Ἀχαιοῖς ἄλγε’ ἔθηκε,
πολλὰς δ’ ἰφθίμους ψυχὰς Ἄϊδι προί̈αψεν
ἡρώων, αὐτοὺς δὲ ἑλώρια τεῦχε κύνεσσιν
οἰωνοῖσί τε πᾶσι, Διὸς δ’ ἐτελείετο βουλή,
ἐξ οὗ δὴ τὰ πρῶτα διαστήτην ἐρίσαντε
Ἀτρεί̈δης τε ἄναξ ἀνδρῶν καὶ δῖος Ἀχιλλεύς.

and saying “This is beautiful!”. I both cases I’m not likely ever to undertake the arduous journey to the required deep understanding. But in both cases I am willing to believe the gurus and to respect those who do make the effort.

Atwood likens code to the paint of the painting. If you can only see the paint of Van Gogh’s Irises, and not the painting, then you don’t experience beauty. Similarly, if you only see the code, and not what’s behind it, then you don’t see beauty. A difference is that most people are endowed with an innate capacity to appreciate “Irises”, but do not have this capacity for text. Text needs to be worked at, to a greater or lesser extent.

If Van Gogh would talk to us about the idea in “Irises” it’s unlikely we would experience beauty. Beauty needs to be embodied, and for paintings embodiment takes the form of paint on canvas. But the paint can get in the way if you look at it the wrong way. Similarly, if the authors collected in “Beautiful Code” would only talk about “the idea, the algorithm”, then it wouldn’t be enough. The code is needed, just as the paint is needed.

With “Irises” it’s so that the artist can only present it on a take-it-or-leave-it basis: you either see it or you don’t. When asked what the meaning of a painting was, Picasso is reputed to have said: “If I could tell you, Madam, I would be a writer. As it happens, I am a painter.” Code does not need to be presented on a take-it-or-leave-it basis. The author happens to be a writer (she had better be), so can work at facilitating appreciation of the code. In this article I will try to demonstrate the phenomenon of beauty in code. Taking Atwood’s criticisms to heart, I make the examples drastically smaller and I add plenty of context.

I’m going to experiment with the possibility of supplementing beautiful code with the required understanding with two related examples. The first is the task of reversing an array segment a[p..q]. To be concrete, the task is that of writing a function declared as

void rev(int a[], int p, int q);

assuming one has available

void swap(int a[], int i, int j) {
  int temp = a[i]; a[i] = a[j]; a[j] = temp;
}

To reverse an array segment, one swaps each element in the left half of the segment with its mirror image in the right half, for example like this:

void rev(int a[], int p, int q) {
  for(int i = (p+q)/2; i >= p; i--) swap(a, i, q-(i-p));
}

Even in this simple example the student author must have experienced a few annoyances. Is the halfway index OK for both odd and even segment lengths? Should I introduce another variable to save recomputation in the argument expression in the call to swap?

Compare this student’s effort with that of another:

void rev(int a[], int p, int q) {
  for(; p<q; p++,q--) swap(a, p, q);
}

This was my favourite until I saw the comment from Samuel Tardieu:

void rev(int a[], int p, int q) {
  while(p<q) swap(a, p++, q--);
}

Isn’t this more than just better code? Don’t you experience this “more” as, er, beauty? The function definitions embody the same idea and the same algorithm. According to Atwood one cannot be more beautiful than the other. But just look … code can make a difference.

As the other example I choose the task of cyclically shifting the contents of a one-dimensional array. I used this example in the first course I ever taught, which was for novice programmers. The problem has besides some obvious solutions an interesting one. How could I keep my students away from the obvious ones? Well, I showed them these in advance, so they would know what I did not want them to do.

I explained that the array can be copied shifted to a temporary other array and that the contents can be copied back, like this:

void cycShift(int a[], int n, int k) {
  int b[n];
  for(int j=0; j<n; j++)
    if (j-k < 0) b[j-k+n] = a[j];
    else b[j-k] = a[j];
  for(int j=0; j<n; j++) a[j] = b[j];
}

The novice needs to be made aware of the dire disadvantages of this simplistic solution: all that wasted space. OK, so how about no extra space, like here:

void cycShift(int a[], int n, int k) {
  while(k-- > 0) {
    int temp = a[0];
    for(int i=1; i<n; i++) a[i-1] = a[i];
    a[n-1] = temp;
  }
}

The novice needs to be made aware of the dire disadvantages of this simplistic solution: all that wasted time.

By denying the novice these easy ways out, I was hoping to light the spark of inspiration. I suggested using mathematics, showing that there is a kind of algebra of sequences. What the operations of such an algebra are depends on who you talk to. But all would agree that concatenation and reversal are worth including, if only because of the following property: (x.y)’ = y’.x’ where the dot is concatenation and the prime is reversal. And note how the order of x and y is reversed.

I left it at this, hoping that they would continue with something like the following. Let s be the sequence to be shifted by k places to the left. Define x and y such that s = x.y with the length of x equal to k. Let us denote by s|k the result of cyclical shift of s by k places to the left. Then we have s|k = y.x.

How do we get y.x? Let’s set s = x.y so that s’ = (x.y)’ = y’.x’. Then an algorithm for cyclical shift is obtained by

s|k = (x.y)|k = y.x = (y.x)'' = (x'.y')'

It is an algorithm because the latter expression consists of easily implementable operations. The algorithm is apparently: first reverse left and right segments separately, then reverse the entire sequence. In code:

void cycShift(int a[], int n, int k) {
  rev(a, 0, k-1); rev(a, k, n-1); rev(a, 0, n-1);
}

This steers the way between the Scylla of excessive memory use and the Charybdis of excessive processing. It needs a constant amount of memory beyond that for the array to be shifted, so it is well clear of Scylla. It needs n swaps, each of which require 4 array accesses (2 reads, 2 writes), so it is well clear of Charybdis. By showing my students two ways how not to do it, I hoped to give them a push on the way to finding this piece of beautiful code.

I wasn’t sure whether I could expect any of the students to get this, the bonus question. After all, one needs some experience and sophistication to relate algebraic manipulations with code. I was in for a surprise. Actually, a double surprise. One part of the surprise was that there was a student who submitted what he claimed was working code that needed only a constant amount of extra memory and an amount of processing that was independent of k. The other surprise was that he did it not in the way I believed to be the one right and elegant way. What did he do?

I found the code puzzling. By the time I figured out what he had been doing, I realized that he only used one half of the number of array accesses compared to mine. The idea that I found was that if one has merely one free location in the array, then one can have this free location jump all around the array to receive every element in turn at the new location required by the cyclic shift. Is this idea beautiful? Maybe it can only become so by being realized in every detail so that a machine can do it this way fully automatically. That is, to code it. But first I need to make the above hand-waving sketch of the idea more concrete by an example.

Here is an array with indexes 0..9 containing the first ten letters of the alphabet:

          0 1 2 3 4 5 6 7 8 9
         ---------------------
          a b c d e f g h i j

We are asked to shift it by three locations to the left. Index 0 is the initial free space. It is “free” because its content is saved away outside the array:

          0 1 2 3 4 5 6 7 8 9
         ---------------------
   a        b c d e f g h i j

In the shifted situation this location has to have the d in it. So we move it there, thereby freeing the location where the d was before. We continue in this way until all nine remaining elements are in their new locations:

          0 1 2 3 4 5 6 7 8 9
         ---------------------
   a        b c d e f g h i j
   a      d b c   e f g h i j
   a      d b c g e f   h i j
   a      d b c g e f j h i
   a      d b   g e f j h i c
   a      d b f g e   j h i c
   a      d b f g e i j h   c
   a      d   f g e i j h b c
   a      d e f g   i j h b c
   a      d e f g h i j   b c
          d e f g h i j a b c

In the penultimate line all nine remaining elements have been moved to their shifted positions. The location of the hole in the penultimate line has to be the place for the element that was initially shunted away. So that’s where it went in the last line.

This example is simple because the distance to be shifted, 3, and the length of the array, 10, are mutually prime. As a result, everything is moved in a single cycle. In general, gcd(n,k) cycles are needed for shifting by k in an array of length n. Here is the example with shifting by 2:

          0 1 2 3 4 5 6 7 8 9
         ---------------------
   a        b c d e f g h i j
   a      c b   d e f g h i j
   a      c b e d   f g h i j
   a      c b e d g f   h i j
   a      c b e d g f i h   j
          c b e d g f i h a j    end of first cycle
   b      c   e d g f i h a j
   b      c d e   g f i h a j
   b      c d e f g   i h a j
   b      c d e f g h i   a j
   b      c d e f g h i j a
          c d e f g h i j a b    end of second cycle

Time for coding.

Executing a single cycle is a self-contained task:

void cycle(int a[], int n, int k, int start) {
// executes single cycle starting from "start"
    int i = start;
    int j = (i+k)%n;
    int saved = a[i];
    // save element at first location of cycle
    while (j != start) {
      a[i] = a[j]; // shift element
      i = j; j = (j+k)%n; // shift indexes
    }
    a[i] = saved;
}

To complete the cyclic shift, gcd(n,k) cycles are needed; hence the function:

void cycShift(int a[], int n, int k) {
// shift a[0..n-1] to the left by k places, cyclically
  assert((0 < k) && (k  0));
  int start = 0; // index where cycle begins
  for (int i = gcd(n,k); i > 0; i--)
    cycle(a, n, k, start ++);
}

Is this Beautiful Code? In this case again I agree with Atwood that the beauty is not an attribute that applies to code. “Beauty is in the eye of the beholder”. A more precise formulation of this piece of wisdom is that beauty is a binary relation. One that may or may not exist between the beholder and the object beheld. Beauty as a unary predicate is erroneously derived from this binary relation. The patter that preceded the code in this article is intended to establish that relation.

This brings me back, way back, to when I first used this exercise. C did not exist; the language was Algol 60. The student who got this solution was Dik Winter. His code ended with the comment:

Like as the waves make towards the pebbled shore,
So do our minutes hasten to their end,
Each changing place with that which goes before
In sequent toil all forwards do contend.

From sonnet LX by William Shakespeare

Dik was a famously uncommunicative person. I take it that in this way he wanted to tell me that he had experienced beauty.

Let this article be dedicated to the memory of this remarkable man, who died on December 28, 2009.

PS Paul McJones was so kind as to provide some scholarly background to this article:

Your reverse and cycShift examples are nice. I believe that the earliest published version of the algorithm that Dik Winter discovered was Fletcher and Silver in 1966 (Algorithm 284: Interchange of two blocks of data. CACM 9(5): 326). Actually, Dik’s is slightly better: his cycle uses n+1 assignments, whereas theirs uses n swaps. Note Dik’s algorithm requires general indexing, whereas the algorithm based on rev requires only ++ and — on indices. There is another efficient variation that requires only ++ (e.g., it will work on a singly-linked list); it was first published by Gries and Mills in 1981 (Swapping Sections. Tech Report 81-452, Department of Computer Science, Cornell University). Perhaps unsurprisingly, GCD comes up again with it. Our chapter 10 (“Rearrangements”) covers these things.

PPS The “Chapter 10″ in the PS is in “Elements of Programming” by Alexander Stepanov and Paul McJones; Addison-Wesley, 2009.