Software Engineering: “From Craft to Industry”?

For brevity you can’t beat the following recap of software history:

Grizzled old hackers tell of going into insurance companies in the 1960s. The typical computer cost at least $500,000 and held data of great value. When Cromwell & Jeeves Insurance needed custom software, they didn’t say, “Maybe we can save a few centimes by hiring a team of guys in India.” They hired the best programmers they could find from MIT and didn’t balk at paying $10,000 for a week of hard work. Back in those days, $10,000 was enough to hire a manager for a whole year, a fact not lost on managers who found it increasingly irksome.

Managers control companies, and hence policies that irk managers tend to be curtailed. Nowadays, companies have large programming staffs earning, in real dollars, one-third of what good programmers earned in the 1960s. When even that seems excessive, work is contracted out to code factories in India. Balance has been restored. Managers are once again earning three to ten times what their technical staff earn. The only problem with this arrangement is …

This is from Philip Greenspun’s “Philip and Alex’s Guide to Web Publishing”, page 318. In case some background is required, read on.

That background is the transition from Craft to Industry. The Grizzled Old Hackers, who at the time were neither old nor grizzled, represented Craft. The managers, in wresting back control, represented Industry. What powered the managers in this struggle is the momentum of several centuries of Industry beating back Craft. The ideology was provided by the Enlightenment philosophers in the 18th century with the movement around d’Alembert’s Encyclopédie, which saw the guilds with their master craftsmen as remnants of the Dark Ages to be conquered by the enlightening forces of Science. Accordingly, the Encyclopédie published manufacturing processes, preferably with scientific explanations, thus far guarded as trade secrets by the guilds.

By and large Industry triumphed. The actors in this process were too busy to notice the large-scale tide of history that they were engaged in. It took outsiders to recognize it and to celebrate it. For example, Richard Buckminster Fuller published in 1962 a book with the title “Untitled Epic Poem On the History of Industrialization”. For another example, let us go to Peter Drucker, who started his career as a management guru in the 1940s with a study of the General Motors corporation.

At the time GM was one of the wonders of the world. Perhaps hard to imagine now, but at the time it was the most successful actor in the most successful paradigm, which was Industry with a capital “I”. On the way to guru status Drucker got a head start out of describing the phenomenon and celebrating its quintessential actor, the Manager. If you ever wondered where the mystique of the MBA comes from, this is it.

A good way to get the flavour of Industry with capital I, and its triumph over Craft, is to look at a specific episode, the Norden bombsight. Up to World War II Germany reigned supreme in optics, an industry concentrated around the firms of Carl Zeiss in Jena and of Ernst Leitz in Wetzlar. Nothing outside of Germany came close. Because of the crucial importance of accurate bombsights, the German military were aware of this and considered optics one of their prime strategic assets. In 1940 they could safely assume that the Allies could not even come close. The German military also believed that this situation was not going to change anytime soon: it would take a generation to produce the highly trained optical craftsmen that produced the high-quality German bombsights.

In the early 1940s the US had a promising prototype bombsight, the Norden. The problem was, in the absence of a suitable optical industry, to manufacture thousands of high-quality copies. Drucker, in his autobiography “Adventures of a Bystander”, recounts how Dreystadt, the head of the Cadillac Division of GM, bid on the nastiest defense contract around, the production of the Norden bombsight. Everybody “knew” that the job required highly-skilled mechanics. At the time there was no unemployed labour, let alone mechanics at any level of skill. Over to Drucker for the delicate wording of this part of the story:

The only labor to be found in Detroit were superannuated Negro prostitutes. To everybody’s horror Nick Dreystadt hired some 2,000 of them. “But hire their madams too,” he said. “They know how to manage the women.” Very few of the women could read and the job required following lengthy instructions. “We don’t have time to teach them to read,” said Nick, “and few would learn anyhow.” So he went to the workbench and himself machined a dozen of the bombsights. When he knew how to do it, he had a movie camera take a film of the process. He mounted film frames separately on a projector and synchronized them with a flow diagram in which a red light went on to show the operator what she had already done, a green light for what she had to do next, and a yellow light to show what to make sure of before taking the next step. By now this is standard procedure for a great many assembly processes; it was Dreystadt who invented it. Within a few weeks these unskilled illiterates were turning out better work and in larger quantity than highly skilled machinists had done before.

This is one of the many defeats suffered by Craft in the past centuries. It was Drucker who developed the ideology of Industry and the central role of the manager in it. Together with the work W. Edwards Deming in quality control, this represented a neat package ready for export. In due time the US market was flooded by innovative consumer items powered by precision miniature electric motors designed by Japanese engineers and assembled by unskilled illiterate Chinese peasant women. The engineers not only designed the artifact, but also the process of manufacturing it, down to the assembly steps and the training of the workers. This is called industrial engineering, a branch of engineering dating back to Frederick Taylor in the early 1900s.

Drucker’s story does not have a happy ending. Dreystadt’s victory over the unions was only a temporary one. When the war was over, the women were sent back to their former lives.

It was against the background of the triumph of Industrial Engineering that the “Software Crisis”, so named by professor Bauer from Munich, was discovered. Not long after E.W. Dijkstra, in his Turing Award lecture, summarized the situation as

… as long as there were no machines, programming was no problem; when we had a few weak computers, programming became a mild problem, and now that we have many powerful computers, programming has become a huge problem.

In response to the Software Crisis the NATO Science Committee sponsored a meeting in Garmisch, Germany in 1968; frugally in October, before the skiing and after the summer tourist season. The title of the report is

SOFTWARE ENGINEERING

At the time the effect was electric: outside the in-crowd at Garmisch the term “software” had barely ceased to be a startling novelty; to combine it with “engineering” was a bit of a taunt, like “artificial intelligence” a decade earlier. The discussion at the conference reveals glimpses of the prevailing mindset. Here is a remark from Douglas McIlroy, soon to become the godfather of Unix:

We undoubtedly produce software by backward techniques. We get the short end of the stick in confrontations with hardware people because they are the industrialists and we are the crofters. Software production today appears in the scale of industrialization somewhere below the more backward construction industries.

Whether the good people at Garmisch were conscious of it or not, they were under the spell of the overbearing success of the Industry paradigm. They took for granted that the way ahead was to follow it. Thus the report has as Chapter 4 “Design”, followed by Chapter 5 “Production”. Note that “Production” means the programming of the system.

The analogy is not valid: when you go into “Production” with bombsights, you have a prototype. A working, tested, documented, prototype. Apparently the Garmisch people forgot that when you have a working, tested, documented prototype of your software system there is only something left to do for the marketing people. Industrial Engineering is a legitimate branch of engineering because of the difficulties that attend the accurate and economical duplication of a working, tested, and documented prototype. To substitute the “Industrial” by “Software” in Industrial Engineering is to fall into the cargo cult trap.

For those who did not hear or read the late Richard Feynman’s CalTech 1974 commencement address, or the quote of it in my essay of August 2008: Feynman explains why, even though they perfectly emulate science, certain studies in psychology are not. All the right moves are there: Null Hypotheses, Type I and Type II errors, …, the works. But though the form of these studies is perfect, the exercise is vacuous. He called this Cargo Cult Science, after the phenomenon observed shortly after the second world war on some of the islands in the Pacific. From Feynman’s address:

During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antenna’s — he’s the controller — and they wait for the planes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No planes land.

Early technology was about providing power (as in horsepower) by machines. When this was well established, the stage was set for Industry, which was about the organization of routine work: assembly lines and all that. This is where Industrial Engineering lives and where Industry has triumphed over Craft.

By the time Industrial Engineering was discovered, clerical work such as the processing of mail orders and bank transactions was done on a large scale. This motivated the organization of clerical work along the same principles as those of mass production in manufacturing. During the second world war scientific computations reached such a scale that the same organizational techniques were needed that had earlier been applied to the processing of mail orders and bank transactions. Enter the computer, and all the carefully engineered routine work was swept away.

The computer introduces a new game. While the processing of material leaves an irreducible residue of work for humans, in the processing of information any work that is routine instantly vanishes. Extracting the routine part from an information processing task is a creative endeavour. It is called programming. In the building of a software system any time you think you have something routine to be handed over to managed cubicle-dwelling drones, you are missing an opportunity for automation. In the building of a software system there is only room for creative work. It is Craft, irreducibly so.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: