by Ray Blank
Cinemagoers flocked this year to the release of Dune: Part Two, the second installment of Director Denis Villeneuve’s adaptation of the science fiction novel written by Frank Herbert and published in 1965. The setting of this story about war, love and revenge in an otherworldly desert landscape is underpinned by an intriguing premise: what if humans are capable of interstellar travel but are no longer allowed to construct machines that think? The inhabitants of Dune do not even have pocket calculators, never mind the smartphones or PCs that you are using to read this. Current concerns about the threat posed by artificial intelligence make Herbert’s speculation appear prescient, but his inspiration can be traced all the way back to a novel published in 1872. A few lines in the text of the first Dune book mention the Butlerian Jihad, a pogrom of thinking machines that occurred prior to the events in the story. These fleeting references are briefly expanded upon within a glossary that Herbert wrote for his fictional universe.
JIHAD, BUTLERIAN: (see also Great Revolt) — the crusade against computers, thinking machines, and conscious robots begun in 201 B.G. and concluded in 108 B.G. Its chief commandment remains in the O.C. Bible as “Thou shalt not make a machine in the likeness of a human mind.
The name of this revolt is an allusion to Samuel Butler, author of Erewhon, a novel published in 1872 as a scathing satire of contemporary Victorian morals that became Butler’s most popular work. Three chapters of Erewhon discuss another revolt by a fictional civilization that had grown terrified of the threat posed by machines. It is worth revisiting these chapters more than 150 years later because of the clarity with which Butler describes the influence that machines have on human life. His account is also spared of any of the intellectual baggage that has since come with modern jargon, the marketing of consumer electronics, and our most recent technological successes and failures.
Erewhon is both the name of the novel and the previously-unknown civilization discovered by the story’s protagonist and narrator. The structure of the work is indebted to earlier satires which also describe imaginary societies. Thomas More’s Utopia is Greek for ‘no-place’; Erewhon is an anagram of ‘nowhere’. Jonathan Swift used the device of a shipwrecked sailor who washes upon the shore of new countries for Gulliver’s Travels; Erewhon’s unnamed narrator crosses a mountain range and river in search of virgin land for farming but stumbles upon the Erewhonians instead. They are healthy, fruitful people who live sophisticated lives in many respects except for their technology. The narrator recounts the unique customs of Erewhon and some of the history that gave rise to them. A recurring theme is that his watch prompts both fear and anger amongst Erewhonians. Ordinary Erewhonians no longer possess such devices, though some antique watches made by their ancestors are still preserved in their museums. Possession of the watch may eventually lead the narrator to be tried in court for the crime of reintroducing machinery. The narrator gains access to a historical Erewhonian text to better understand the reasons for this strange prohibition. Chapters 23, 24 and 25 of Erewhon are dedicated to the narrator recounting what he learns from ‘The Book of the Machines’.
Modern readers who are sensitive to cultural differences may already be thinking of the tension created by discussing a ‘newly-discovered’ civilization, as if there is not a choice between the perspective of a European explorer who steps on to Erewhonian land without knowing of its inhabitants before, and the perspective of the inhabitants confronted with an outsider who unexpectedly appears in their territory. Butler explores a similar tension by begging the question of why the evolution of machines should be assessed from the perspective of what humans gain by having machines, instead of asking what machines gain by having humans. Charles Darwin’s On the Origin of the Species was published in 1859, and its core conception of biological evolution had radically upset previously dominant belief systems. Butler observes that machines also undergo a form of evolution. Transposing Darwin’s theories about natural selection to machines gives rise to a new way of predicting how technology will develop.
The Book of the Machines begins by addressing the potential for a machine to gain consciousness. The nature of consciousness is described as an emergent property with respect to both history and matter. If no assumptions are made about the requirements for consciousness then we cannot exclude the possibility of new forms of consciousness arising over time.
There was a time, when the earth was to all appearance utterly destitute both of animal and vegetable life, and when according to the opinion of our best philosophers it was simply a hot round ball with a crust gradually cooling. Now if a human being had existed while the earth was in this state and had been allowed to see it as though it were some other world with which he had no concern, and if at the same time he were entirely ignorant of all physical science, would he not have pronounced it impossible that creatures possessed of anything like consciousness should be evolved from the seeming cinder which he was beholding? Would he not have denied that it contained any potentiality of consciousness? Yet in the course of time consciousness came.
Consciousness, in anything like the present acceptation of the term, having been once a new thing — a thing, as far as we can see, subsequent even to an individual centre of action and to a reproductive system (which we see existing in plants without apparent consciousness) — why may not there arise some new phase of mind which shall be as different from all present known phases, as the mind of animals is from that of vegetables?
Machines could gain consciousness by undergoing a form of development analogous to that of animal species. However, alterations and enhancements to machines occur at a much more rapid rate.
There is no security… against the ultimate development of mechanical consciousness, in the fact of machines possessing little consciousness now. A mollusc has not much consciousness. Reflect upon the extraordinary advance which machines have made during the last few hundred years, and note how slowly the animal and vegetable kingdoms are advancing. The more highly organised machines are creatures not so much of yesterday, as of the last five minutes, so to speak, in comparison with past time. Assume for the sake of argument that conscious beings have existed for some twenty million years: see what strides machines have made in the last thousand! May not the world last twenty million years longer? If so, what will they not in the end become? Is it not safer to nip the mischief in the bud and to forbid them further progress?
The intellectual turmoil created by the theory of evolution is harnessed to an even more radical conjecture: that machines evolve too. An elegant analogy is offered, establishing the precedent for subsequent arguments that will also draw upon similar analogies between technology and nature.
…a great deal of action that has been called purely mechanical and unconscious must be admitted to contain more elements of consciousness than has been allowed hitherto (and in this case germs of consciousness will be found in many actions of the higher machines) — Or (assuming the theory of evolution but at the same time denying the consciousness of vegetable and crystalline action) the race of man has descended from things which had no consciousness at all. In this case there is no a priori improbability in the descent of conscious (and more than conscious) machines from those which now exist, except that which is suggested by the apparent absence of anything like a reproductive system in the mechanical kingdom. This absence however is only apparent, as I shall presently show.
A 19th century steam whistle was a machine for communication; it may signify the end of a factory shift or warn somebody of the impending arrival of a train. The Erewhonians had built machines which only communicated with people, but they expected the machines of the future would communicate with each other.
As yet the machines receive their impressions through the agency of man’s senses: one travelling machine calls to another in a shrill accent of alarm and the other instantly retires; but it is through the ears of the driver that the voice of the one has acted upon the other. Had there been no driver, the callee would have been deaf to the caller. There was a time when it must have seemed highly improbable that machines should learn to make their wants known by sound, even through the ears of man; may we not conceive, then, that a day will come when those ears will be no longer needed, and the hearing will be done by the delicacy of the machine’s own construction? — when its language shall have been developed from the cry of animals to a speech as intricate as our own?
We might think that humans always control machines, but the more a thing is needed, the harder it is to control. Humans have unlimited freedom to dispense with things that are not required. The freedom that people gain by using machines also means losing the freedom to act in certain ways because of our reliance upon machines.
It can be answered that even though machines should hear never so well and speak never so wisely, they will still always do the one or the other for our advantage, not their own; that man will be the ruling spirit and the machine the servant; that as soon as a machine fails to discharge the service which man expects from it, it is doomed to extinction…
This is all very well. But the servant glides by imperceptible approaches into the master; and we have come to such a pass that, even now, man must suffer terribly on ceasing to benefit the machines. If all machines were to be annihilated at one moment, so that not a knife nor lever nor rag of clothing nor anything whatsoever were left to man but his bare body alone that he was born with, and if all knowledge of mechanical laws were taken from him so that he could make no more machines, and all machine-made food destroyed so that the race of man should be left as it were naked upon a desert island, we should become extinct in six weeks. A few miserable individuals might linger, but even these in a year or two would become worse than monkeys. Man’s very soul is due to the machines; it is a machine-made thing: he thinks as he thinks, and feels as he feels, through the work that machines have wrought upon him…
Machines also depend on people, but dependence is not an obstacle to evolution. Humans serve the needs of machine evolution just as machines are used to change the way humans live.
…even now the machines will only serve on condition of being served, and that too upon their own terms; the moment their terms are not complied with, they jib, and either smash both themselves and all whom they can reach, or turn churlish and refuse to work at all. How many men at this hour are living in a state of bondage to the machines? How many spend their whole lives, from the cradle to the grave, in tending them by night and day? Is it not plain that the machines are gaining ground upon us, when we reflect on the increasing number of those who are bound down to them as slaves, and of those who devote their whole souls to the advancement of the mechanical kingdom?
There is a temptation to think machines do not influence their own evolution because they do not reproduce. This may be based on a confusion; a system for reproduction need not be exclusively limited to internal organs like they are in humans and more evolved animals. Plants reproduce via synergies with animals, creating an overall system that benefits both. Humans are themselves a complicated system of many cellular organisms that work together. Machines reproduce via a sequence of synergies with humans. Very different tasks that ultimately produce machines are effected within the body of society much like very different cells work with each other within a human body. The several parts of a machine may each need to be made using separate methods, only to be assembled into complete machines later, and this totality must be observed to see how machines reproduce in practice.
What is a reproductive system, if it be not a system for reproduction? And how few of the machines are there which have not been produced systematically by other machines? But it is man that makes them do so. Yes; but is it not insects that make many of the plants reproductive, and would not whole families of plants die out if their fertilisation was not effected by a class of agents utterly foreign to themselves? Does anyone say that the red clover has no reproductive system because the humble bee (and the humble bee only) must aid and abet it before it can reproduce? No one. The humble bee is a part of the reproductive system of the clover. Each one of ourselves has sprung from minute animalcules whose entity was entirely distinct from our own, and which acted after their kind with no thought or heed of what we might think about it. These little creatures are part of our own reproductive system; then why not we part of that of the machines?
We are misled by considering any complicated machine as a single thing; in truth it is a city or society, each member of which was bred truly after its kind. We see a machine as a whole, we call it by a name and individualise it; we look at our own limbs, and know that the combination forms an individual which springs from a single centre of reproductive action; we therefore assume that there can be no reproductive action which does not arise from a single centre; but this assumption is unscientific… each part of every vapour-engine is bred by its own special breeders, whose function it is to breed that part, and that only, while the combination of the parts into a whole forms another department of the mechanical reproductive system, which is at present exceedingly complex and difficult to see in its entirety.
People are considered responsible for improvements in machines, but improved machines also enable the manufacture of better machines. The balance between these factors can change over time, so that more of the improvements made to machines will stem from the increased capabilities of machines, and less from the capabilities of human beings. With the benefit of hindsight, we can now see how the development of vacuum tubes permitted the creation of programmable computers that could be configured to execute multiple different series of logical steps on data that was input to them, the improvements in computational power and programming have fed into increasingly precise applications of materials in the design and production of yet more powerful processing chips, and this has permitted the development of computational models that permit machines to learn from experience. These latter AI models are now at a stage where they can write better computer programs than people can. Moore’s Law, which states the number of transistors on a single chip will double every two years at minimal costs, and other rules of thumb that anticipate acceleration in computational power were foreshadowed by the importance attached to an accelerating rate of change described in The Book of the Machines.
…there seem no limits to the results of accumulated improvements if they are allowed to descend with modification from generation to generation. It must always be remembered that man’s body is what it is through having been moulded into its present shape by the chances and changes of many millions of years, but that his organisation never advanced with anything like the rapidity with which that of the machines is advancing.
I fear none of the existing machines; what I fear is the extraordinary rapidity with which they are becoming something very different to what they are at present. No class of beings have in any time past made so rapid a movement forward. Should not that movement be jealously watched, and checked while we can still check it?
Humans view the sophistication of machines based on a hierarchy that assumes humanity is the highest state of evolution. The perspective chosen when determining which is a higher or lower state of evolution is arbitrary. Machines will evolve without necessarily becoming more like human beings.
May we not fancy that if, in the remotest geological period, some early form of vegetable life had been endowed with the power of reflecting upon the dawning life of animals which was coming into existence alongside of its own, it would have thought itself exceedingly acute if it had surmised that animals would one day become real vegetables? Yet would this be more mistaken than it would be on our part to imagine that because the life of machines is a very different one to our own, there is therefore no higher possible development of life than ours; or that because mechanical life is a very different thing from ours, therefore that it is not life at all?
The Book of the Machines returns to the question of whether machines can gain consciousness. It argues against too narrow a definition of consciousness that limits it to organic life. It would be better to recognize machine consciousness for what it is than to pretend machines will never have properties that are common to all conscious beings.
…the regularity with which machinery acts is no proof of the absence of vitality, or at least of germs which may be developed into a new phase of life. At first sight it would indeed appear that a vapour-engine cannot help going when set upon a line of rails with the steam up and the machinery in full play; whereas the man whose business it is to drive it can help doing so at any moment that he pleases; so that the first has no spontaneity, and is not possessed of any sort of free will, while the second has and is.
This is true up to a certain point; the driver can stop the engine at any moment that he pleases, but he can only please to do so at certain points which have been fixed for him by others, or in the case of unexpected obstructions which force him to please to do so. His pleasure is not spontaneous; there is an unseen choir of influences around him, which make it impossible for him to act in any other way than one… The only difference is, that the man is conscious about his wants, and the engine (beyond refusing to work) does not seem to be so; but this is temporary…
Where does consciousness begin, and where end? Who can draw the line? Who can draw any line?
…the difference between the life of a man and that of a machine is one rather of degree than of kind, though differences in kind are not wanting. An animal has more provision for emergency than a machine. The machine is less versatile; its range of action is narrow; its strength and accuracy in its own sphere are superhuman, but it shows badly in a dilemma; sometimes when its normal action is disturbed, it will lose its head, and go from bad to worse like a lunatic in a raging frenzy: but here, again, we are met by the same consideration as before, namely, that the machines are still in their infancy; they are mere skeletons without muscles and flesh.
The latter paragraph fits well with what we know about progress in the realm of artificial intelligence. Machine intelligences created to perform highly specific tasks, like winning at a game of chess or Go, are now capable of outperforming the best human minds. Progress in AI has somewhat been measured by examining how many new kinds of tasks are being mastered by machines. Generative AI, and the risks associated with it, have provoked safety concerns because the outputs of AI are becoming more general than they were before. Per the method of exposition in Erewhon, we are witnessing an evolution of AI demonstrated by increasing versatility.
Furthermore, The Book of the Machines anticipates the significance of the transition from the physical matter of machinery to the abstract logic of computation by drawing a similar contrast between ‘skeletons’ and ‘muscles and flesh’. Muscles move bones; conscious thought moves muscles. Humans benefit by harnessing the muscles of machines, but at the cost of increasing our dependence upon them. Relying on the thoughts of machines increases the risk to humans by an order of magnitude. Contrary to storylines from more populist forms of science fiction, the threat to humanity stems not from physical altercations with killer robots, but from the loss of human control over decisions that determine our environment.
The misery is that man has been blind so long already. In his reliance upon the use of steam he has been betrayed into increasing and multiplying. To withdraw steam power suddenly will not have the effect of reducing us to the state in which we were before its introduction; there will be a general breakup and time of anarchy such as has never been known; it will be as though our population were suddenly doubled, with no additional means of feeding the increased number. The air we breathe is hardly more necessary for our animal life than the use of any machine, on the strength of which we have increased our numbers, is to our civilisation; it is the machines which act upon man and make him man, as much as man who has acted upon and made the machines; but we must choose between the alternative of undergoing much present suffering, or seeing ourselves gradually superseded by our own creatures, till we rank no higher in comparison with them, than the beasts of the field with ourselves.
Herein lies our danger. For many seem inclined to acquiesce in so dishonourable a future. They say that although man should become to the machines what the horse and dog are to us, yet that he will continue to exist, and will probably be better off in a state of domestication under the beneficent rule of the machines than in his present wild condition. We treat our domestic animals with much kindness. We give them whatever we believe to be the best for them; and there can be no doubt that our use of meat has increased their happiness rather than detracted from it. In like manner there is reason to hope that the machines will use us kindly, for their existence will be in a great measure dependent upon ours; they will rule us with a rod of iron, but they will not eat us; they will not only require our services in the reproduction and education of their young, but also in waiting upon them as servants; in gathering food for them, and feeding them; in restoring them to health when they are sick; and in either burying their dead or working up their deceased members into new forms of mechanical existence.
Per The Book of Machines, the threat posed to humanity is that many people will be reduced to the status of pets. Some might retain a slightly higher status analogous to a working animal like a sheepdog or a messenger pigeon. We may have some physical characteristics that allow us to be more useful than machines for certain tasks. Human dexterity may continue to be especially useful when repairing machinery, but our brains will have been surpassed, and so machines will mostly treat us a luxury rather than a necessity. This will occur because the majority of the human population will gladly acquiesce to the life of a domesticated animal that has no burdens or obligations.
The reference to the use of meat increasing the happiness of animals will likely grab the attention of many modern readers, especially those who are vegans and those who disapprove of the cruelty to animals exhibited in factory farms. In this instance, the writer unwittingly gives us an example of how a seeming moral certainty may later be challenged. Human farmers and customers of their products must interpret which farming methods are sufficiently compassionate to animals. If a non-human intelligence was tasked with making similar decisions about the wellbeing of humans there is no guarantee that both parties would be in agreement. Human society already has many disagreements about how to attain the best good for all. This becomes especially apparent when arguing about public health objectives and how to achieve them, such as curtailing freedom of movement during a pandemic, or imposing taxes on sugary drinks. A machine intelligence that made decisions with the goal of delivering the optimal outcome for all people would inevitably displease some.
…the mass of mankind will acquiesce in any arrangement which gives them better food and clothing at a cheaper rate, and will refrain from yielding to unreasonable jealousy merely because there are other destinies more glorious than their own.
The power of custom is enormous, and so gradual will be the change, that man’s sense of what is due to himself will be at no time rudely shocked; our bondage will steal upon us noiselessly and by imperceptible approaches; nor will there ever be such a clashing of desires between man and the machines as will lead to an encounter between them… In point of fact there is no occasion for anxiety about the future happiness of man so long as he continues to be in any way profitable to the machines; he may become the inferior race, but he will be infinitely better off than he is now. Is it not then both absurd and unreasonable to be envious of our benefactors? And should we not be guilty of consummate folly if we were to reject advantages which we cannot obtain otherwise, merely because they involve a greater gain to others than to ourselves?
The Book of the Machines rejects this potential future, because it means choosing to allow machines to surpass our human descendants. It concludes by insisting Erewhon…
…resolve upon putting an immediate stop to all further mechanical progress, and upon destroying all improvements that have been made for the last three hundred years.
The extreme remedy adopted by the Erewhonians is Butler’s way of poking fun at contemporaries who continued to feel scandalized by the theory that humans could have evolved from ‘lower’ animals like apes. Turning the wheel of time in the opposite direction, towards the future, allows Butler to mock opponents of the theory of evolution on the grounds that denying the possibility of change also means denying the possibility of improvement. Extending this notion to machines would mean denying people the increased comfort and prosperity that will only be attained by becoming more dependent on increasingly sophisticated machines. I feel this mockery is wide of the mark. Butler has accidentally chanced upon a genuine moral problem, just as the fictional narrator accidentally chanced upon the land of Erewhon.
Physical needs must be satisfied to free a person to pursue meaning in their life, but the individual’s pursuit of meaning can also be eroded by allowing others to decide how our needs are met. Pets are like children in that they both have a degree of freedom although the most important decisions are made for them by a greater intelligence that chooses how to protect and feed them. The line that separates consciousness from non-consciousness is like the line between children and adults; we cannot draw it precisely, but we know there is a difference when we see it. The transition from childhood to adulthood is a necessary component of becoming a fully realized person. The significance of this transition is managed through societal customs that reflect increased responsibility in addition to the practicalities of dealing with bodily transformations that occur during puberty and which lead us to become fully mature. Handing those responsibilities to a machine that makes decisions necessarily involves taking those responsibilities away from people.
To supplant the adult decision-maker with a machine decision-maker is to deny the possibility of becoming a fully-fledged adult in mind as well as body. This is because the potential responsibilities of parenthood defines much of the significance of the transition from child to adult. Removing the freedom to make adult decisions, including the freedom to make bad decisions, would trap us within a permanent state of infancy as well as dependence. So whilst Butler is most remembered for these few chapters of ingenious humour, they have resonated with subsequent thinkers because they also depict a genuine and seemingly inevitable threat to our humanity.
Erewhon is no longer under copyright so copies of the story can be freely obtained from Standard Ebooks and Project Gutenberg.
~
Bio:
Ray Blank is a former editor of Sci Phi Journal. We are pleased to host his latest essay on SF literature, thereby marking half a decade since his departure from the magazine.
