Emerging technologies may eventually enable us to shape the desires, motivations and values of future beings. This essay invites us to consider the ethical questions raised by that prospect: whether satisfying constructed desires is sufficient to make a life good, whose vision of the future ought to prevail, and to what extent we can—or should—direct what we become.
Those interested in reading a somewhat more pleasantly formatted PDF edition of the essay are welcome to purchase one at the bottom of the page.
“All man’s troubles arise from the fact that we do not know what we are, and do not agree on what we want to be.” — Vercors (Jean Bruller), You Shall Know Them
Consider the prospect of a world in which the power to sculpt the content of our values is evermore significant. A world in which, by means of technological intervention, we can shape the very things that the world’s inhabitants—or future inhabitants—value, and determine their motivations, drives, morals and the things that make them feel satisfied or dissatisfied. Such powers may lie a hundred years into the future, or a thousand, but should they become available, what role might ethical reflection assume in deciding how—or whether—they ought to be exercised?
Several emerging technologies give us reason to consider this prospect seriously, by opening up avenues for reshaping the architecture within which values are formed. The question “What ought we to value?” may thereby acquire an unprecedented degree of practical significance. Supposing we could build an entirely new world—or simulate one, or bring an entirely new kind of lifeform into existence, what features ought it to possess? With what values, dispositions, or purposes ought we to imbue it? And by what standards ought we to assess the result?
Prominent amongst the technological domains that portend to render these questions tangible is, of course, artificial intelligence. Readers will likely be familiar with the “alignment problem”, which concerns bringing the behaviour of artificial systems into accordance with intended values or objectives.
Yet the effort to specify those very objectives raises questions about whose interests ought to count, how competing interests ought to be reconciled, and whether human interests ought even to determine the ultimate purpose of such systems at all, or instead be superseded. Even perfect success in aligning a system with its assigned ends leaves open the question of which ends to assign it in the first place.
Other technological domains extend the range of possibilities: bioengineering, gene-editing, human-machine augmentation, and the creation of simulated worlds, amongst others. These technologies differ considerably in their present capabilities, and some of the possibilities relevant to this discussion remain hypothetical. Nonetheless, to a degree that is without historical precedent, our increasing mastery over these domains situates us on the precipice of an age in which the evaluative endowment of valuing-agents, and the conditions within which their values acquire practical expression, are themselves objects of deliberate choice.
Douglas Adams illustrates this predicament in The Restaurant at the End of the Universe, in which the Ameglian Major Cow has been bred to want to be eaten, and to communicate this desire clearly to its prospective diners. What might otherwise appear an objectionable fate is thus presented as the fulfilment of the creature’s own wishes. Supposing that its desire is entirely sincere, and that it understands what its satisfaction entails, ought we to regard the ethical difficulty as resolved? There remains a distinction between respecting the wishes of a being already in existence and justifying the decision to bring into existence a being disposed to wish for precisely this fate.
Comparable possibilities might, in principle, extend to human beings, or to mechanical beings capable of consciousness, supposing such a thing is possible. We might seek to constitute future beings so that they find fulfilment in conditions we presently regard as undesirable, or willingly serve purposes we would resist having imposed upon ourselves. Neither the unfamiliarity of their values nor their deliberate contrivance would—necessarily—establish that such arrangements were objectionable; our own evaluative dispositions also have biological and social causal-origins.
Nevertheless, the prospect of such contrivance invites us to ask whether the satisfaction of values is sufficient as a guiding principle for ethics when the values themselves become objects of choice. Even if we could ensure that future beings obtained precisely what they wanted, there would remain the question of what we ought to cause them to want. These questions typify the kind of ethical scrutiny that may become increasingly consequential in an age of mutable valuations.
To be clear, it is the emerging degree of our ability to modify these qualities that is historically unprecedented; we have been shaping our evaluative dispositions since long before the invention of contemporary technologies. In large part, morality functions as a social technology to cultivate, restrain, transmit and amend our evaluative endowment; those things that we are cast into the world already disposed to value. Education, custom, tradition and punishment have all participated in this process. For instance, it has been proposed that coordinated punishment may have contributed to evolutionary selection against impulsive aggression. Seen this way, morality may be in part understood as an instrument of human self-domestication, cultivating and restraining our dispositions in accordance with the demands of collective life; a process forming part of what we broadly call civilisation.
Whilst such evolutionary accounts require their own empirical defence, the broader point is that the deliberate formation of values has a history, including quite self-conscious and often disastrous programs of eugenics, and the novelty before us concerns the extent and character of the powers through which it might be undertaken. Empirical investigation can inform us about what an intervention is likely to achieve, the costs it may impose, and the risks attending its use. But choosing which consequences ought to be pursued—or whether such interventions ought be permitted at all—requires evaluative commitments whose rationale is itself a subject of disputation.
It is here that ethics is poised to assume a task of considerable importance: to scrutinise and steward the values of the future, in terms more concrete than ever, and in so doing exert an influence over the course of the emerging world. Here, Friedrich Nietzsche has proven prophetic, although he could scarcely imagine that his call for a “transvaluation of all values” might find practical expression in the creation of entirely new forms of valuing life. In heralding the dawn of the Übermensch—an “over-man” or “beyond-man”—Nietzsche is amongst the first to have glimpsed the advent of a nascent post-human future. Similarly, in Thus Spoke Zarathustra, Nietzsche forewarned of the age of the “last man”, a type of man driven solely by desire-satisfaction and contented merely to pursue hedonic pleasures. A prescient observation, if one considers that our pursuit of material abundance and technological ease may truly usher in an age of “last men”, in which mankind itself is rendered obsolete.
The profundity of what is at stake, and the sheer irreconcilable diversity of the paths open to us is likely to be a source of considerable ideological contention and material conflict. Our species does not possess a single will through which a considered vision of the future can be enacted. Wars of the future might be fought over whether human beings ought to merge with machines, whether consciousness ought to be transferred into simulated worlds, or whether humanity ought to be preserved at all, or superseded by new forms of being. What some regard as advancement, others may regard as degradation or extinction. Such disagreements concern both the desirability of these futures and the authority to bring them about, particularly when one group’s chosen course constrains the possibilities open to others. Vercors’s observation, that mankind’s troubles emerge from our inability to agree on what we ought to be, thus acquires an increasingly tangible significance: we may obtain the power to alter what we are without having reached agreement on what we are to become.
The need to appraise the values with which we intend to sculpt the future does not guarantee that our inquisitions will prove decisive, and ethical consideration may be fated to take a back-seat to the relentless unfolding of material facts, driven by those with the power to determine them. Values may acquire practical expression without having undergone sustained examination, and a project may prevail because it attracts investment, confers power, or satisfies an immediate demand. Its success need bear little relation to the adequacy of the reasons offered in defence of its ultimate purpose. Moreover, the interaction of economic incentives, political competition, individual ambitions and societal inertia may produce a course that nobody has chosen as a whole. Perhaps we will ultimately have little individual or collective authority over the matter, just as our prehuman ancestors had little authority over whether or not they would become human. Humanity was no destination they had conceived and set themselves the task of reaching. Likewise, we may have little conception of what will follow us, or of how our present activities will appear from the standpoint of whatever beings eventually inherit their consequences. The growth of our power to intervene in the world need not bring with it an equivalent understanding of what those interventions will ultimately produce.
There is nevertheless a distinction between lacking knowledge of our eventual destination and lacking all influence over the course. We may be unable to determine what humanity—or whatever succeeds it—will ultimately become, while still exercising consequential choices about what we bring into existence now. Even supposing that the entire course of events is determined, our deliberations and the arguments that influence them could themselves belong amongst its causes.
The pace of change may impose a further limitation on our ability to evaluate the desirability of a given future. A transformation may become established before its implications are adequately understood, and choices made provisionally may acquire consequences that are difficult to reverse. Under such conditions, ethical reflection might recognise the significance of a decision only after the opportunity to influence it has passed. Its questions would remain pertinent, but its answers might arrive too late. The consequences of our evaluative commitments may thus become more extensive even as the opportunity to examine them narrows.
Moreover, we needn’t presume that ethical investigation must invariably recommend a greater effort at exercising authority over the future. It may give us reasons for restraint, for pausing to contemplate possibilities we do not yet understand, or for accepting that our successors—rather than ourselves—ought to wield the reins of the future. The ambition to direct the course of events is itself an ambition whose justification deserves examination. We might, upon reflection, choose to relinquish control, or in acceptance of the futility of steering the course of the ship, permit ourselves to be carried by ungovernable currents to shores beyond our horizon.
Thank you for reading. Those interested in a PDF edition of the essay may purchase a subscription below: