Software Engineering: “From Craft to Industry”?

For brevity you can’t beat the following recap of software history:

Grizzled old hackers tell of going into insurance companies in the 1960s. The typical computer cost at least $500,000 and held data of great value. When Cromwell & Jeeves Insurance needed custom software, they didn’t say, “Maybe we can save a few centimes by hiring a team of guys in India.” They hired the best programmers they could find from MIT and didn’t balk at paying $10,000 for a week of hard work. Back in those days, $10,000 was enough to hire a manager for a whole year, a fact not lost on managers who found it increasingly irksome.

Managers control companies, and hence policies that irk managers tend to be curtailed. Nowadays, companies have large programming staffs earning, in real dollars, one-third of what good programmers earned in the 1960s. When even that seems excessive, work is contracted out to code factories in India. Balance has been restored. Managers are once again earning three to ten times what their technical staff earn. The only problem with this arrangement is …

This is from Philip Greenspun’s “Philip and Alex’s Guide to Web Publishing”, page 318. In case some background is required, read on.

What’s Behind “Design Patterns”

One of the few true bestsellers in the computer world is Design Patterns by Gamma, Helms, Johnson and Vlissides (“The Gang of Four”). Not only did it sell well, but its success also triggered the appearance of dozens of books with title and content conspicuously linked to the original. As the Gang acknowledged, the original itself is, if not an imitation, at least strongly inspired by another book. Their product is highly original in having its source of inspiration in an area far away from computers and software. That source is Design Patterns by Christopher Alexander, Sara Ishikawa, and Murray Silverstein, and the area is design in architecture. Not computer architecture, not software architecture, but architecture. To avoid confusion, I will refer to the Gang-of-Four book as “DP” and accord to the original original the honour of Design Patterns in full.

What is not widely acknowledged, and does not even seem to be known among the aficionados of DP, is that Design Patterns is but a companion to the main volume, which is The Timeless Way of Building with Christopher Alexander as sole author. That the epiphenomenon makes a great furore in the software world while the main item remains unknown is, it seems to me, remarkable. The present essay may be useful in introducing the unknown main item and arguing its relevance for the building of software.

I remember Donald Michie (1923 – 2007)

On July 7, 2007 Dame Anne McLaren and Donald Michie died in a car accident. Last time I had met Anne was in the 1970s when I had stayed at their house in Edinburgh. Both were launched on scientific careers that were to lead them to positions rivalling each others’ eminence in their respective fields.

The last time I met Donald (henceforth “DM”) was when my wife and I visited him in Oxford in November 2004. He demo’ed the SOPHIE chatbot system, asking us what we thought of his choice of accent for the speech-generating software. He was intrigued by the way his current choice, labeled “Southern California Trash”, blended with the other personality attributes of Sophie. The stereo was playing DM’s current favourite, “Harper Valley PTA” sung by Jeannie C. Riley, another sassy American female.

Although excited by his current project, DM was depressed by the gloomy British weather; he would depart shortly to spend the winter in Gibraltar. “Gibraltar??”. “Yes, Gibraltar. I trust it will be sunny, and it is British.” In the following pages I have noted, roughly in chronological order, some experiences with this extraordinary man, one of the great pioneers in Artificial Intelligence.

Against Structured Programming

In 1968 the Communications of the ACM published a Letter to the Editor from E.W. Dijkstra (volume 11 (1968), pp 147–148). The Editor, perhaps a bit puzzled by the contents, summarized them by supplying as title “Goto Statement Considered Harmful”. The mere thought was sufficiently provocative to ensure that the idea spread like wildfire, initially to be dismissed, but soon to be blessed with universal approval. “Structured Programming” was born.

The most enthusiastic converts were the computer science professors. These people were doing research on esoteric topics like formal languages or graph theory and were gang-pressed into teaching introductory programming. With little programming experience, they welcomed a clear rule (no GOTOs!) and welcomed the casuistry required for forcing non-conforming code into the straight jacket of the officially approved formats.

This essay is a thought experiment: suppose one had never heard of Structured Programming, that one wanted to solve a little programming problem by common sense reasoning; that this solution process not only had to supply the conviction of correctness, but also had to help discover the algorithm. My conclusion will be that these goals, though ambitious, are achievable, but that Structured Programming is at best a handicap.

Remember Software Verification?

Maybe not. It used to be taken for granted that in the more or less remote future it would no longer be necessary to debug software because it had been verified.

That time is long past. What we now take for granted is that, for example, the two pieces of software essential to the health and safety of the internet, MS Windows and MS Internet Explorer, “vulnerabilities” are discovered at such a rate that the repair sessions are saved up to the second Tuesday of each month. Rarely does such a Tuesday pass without an update; sometimes five or six. Another exhibit for the prosecution: many pieces of software that invite being downloaded distinguish between the “latest version” and the “latest stable version”. Forget correctness: “stability” is the goal. Diminished expectations …

Ventilated Prose

In the 1930s Buckminster Fuller (he of the domes, but also of many other things) was doing research for the Phelps Dodge Corporation. His boss could not read Fuller’s reports, but found them perfectly intelligible when read aloud by the author. Fuller thought he remedied the problem by breaking up the text in the same way he read it aloud. Though this made the written text readable, it was not acceptable: it looked like … eh, well, poetry, and the Phelps Dodge Corporation was not into poetry. As a compromise, Fuller called his text format “ventilated prose”. In this article I show examples of Fuller’s writing and report how I use it to get over “writer’s block” in my own practice.

Rewriting a Utility

I can’t leave well enough alone. The most recent manifestation of this problem is the following program from Kernighan and Ritchie (“The C Programming Language”), which is perfectly OK code for converting a number to the equivalent decimal numeral.

No Silver Bullet? Get a Humble Programmer.

Consider this:

1. Software reliability is rapidly becoming more
   important.
2. Software development technology is inadequate and
   is only slowly improving.

The question arises:

Is disaster inevitable?

This has been debated for some time in the computer
science community. Recently it has begun to attract
attention from the general public.
For example a newspaper ran a story entitled: “A Lemon Law for
Software?” Here the author observed:

If Microsoft were liable for product defects the same
way automobile manufacturers are, then it would have
been driven out of business a long time ago.

As we all know, this has not happened. And this is only
because the law exempts the software industry from
product liability.

The question I’m now asking is: Should it be exempted in
this fashion?

Any suggestion that it is time for the software industry
to grow up and accept responsibility for its products is
greeted by an indignant chorus asserting that software is
special and that not recognizing it as such will stifle
innovation.

Clearly Microsoft stands to lose most in a change from
the status quo. But the rest of industry, including those
who love to hate Microsoft, do not dare to contemplate a
world where creators of software are responsible,
seriously responsible, for their products.

While in industry there is agreement on this topic,
academia, in so far as it has an opinion at all, is
divided. Some argue that software is special in such a
way that the current unreliability of software is
inevitable. They argue that there is no prospect of
drastic improvement. I will call this camp “The Real
World”. Their most eloquent, and most widely quoted,
spokesman is Frederick P. Brooks, Jr., in his paper “No
Silver Bullet”.

Others argue that techniques for producing reliable
software are known, that these are practical, and that
some of them have been known for decades. Whenever one
brings these to the attention of industry, one is
dismissed as being sadly out of touch with reality. Hence
I will denote the proponents of the known techniques for
producing reliable software as “The Ivory Tower”.

In this introduction I have sketched the problem,
which is:

1. Software reliability is rapidly becoming more
   important.
2. Software development technology is inadequate and
   is only slowly improving.

I have also sketched the main points of view, and called
them “The Real World” and “The Ivory Tower”. In this
lecture I will elaborate further on these two opposing
positions, starting with the real-world people, and
continuing with those who inhabit the Ivory Tower.
Although I will avoid technical matters, few such
concepts need a brief review, which I will do next.

Cargo-cult Engineering

There is an old song about software engineering. I thought it had died until the other day I came across a post by Jack Ganssle (Embedded.com; 05/12/08). Under the title “Programmer or Engineer?” he argues that the word “programmer” should be banned and that embedded applications should by built by electrical or computer engineers.

It may well be that engineers are satisfactory for the “embedded applications” that Mr Ganssle has in mind. But another category of people is needed to meet the greater challenges posed by computers. Neither of the existing categories: “engineer” nor “computer scientist” can be relied on to fit the bill.

A Possible Future History of Logic Programming

(from “The ALP Newsletter”, vol. 17, no. 1, Feb. 2004)

While it is well-known that it is hard to know anything about the future, it is less widely realized that it is also difficult to understand the present. It is therefore paradoxical that it sometimes helps to understand the present by means of a fictitious future history. In this essay I am going to exploit this paradox.

I place the viewpoint well away into the future. Again this may seem paradoxical, because in computing everything is supposed to happen fast. Not everybody thinks so. Paul Graham has embarked on a project that he calls “The Hundred Year Language” , pointing out that it is only hardware that changes fast. Programming languages change slowly. This is because they are part of culture; they reflect how human minds tackle problems. Hence Graham’s “Hundred Year Language”.

It so happens that Graham is concerned with Lisp. This is not the only paradigm in need of the long-term perspective. Accordingly, I imagine a history of logic programming as it may be written in the 2020s.

As I’m already exercising your indulgence with speculation, I will not attempt the fictitious future history itself, but instead bring out the salient points in an equally fictitious review of this nonexistent
history.