Plain text: 30-sec27-competitive-pressures.md · on GitHub

§27 Competitive pressures provide little incentive to transfer strategic decisions to AI systems

Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 27 of 40. Source label: competitive-pressures. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad

Thesis. Requirements for swift response may motivate transfer of tactical-level control to AI systems; at a strategic level, however, humans will have both time and good reason to consider suggested alternatives.

Cites sections: §25 Optimized advice need not be optimized to induce its acceptance

Cited by sections: §II, §29, §32

Summary

In a range of competitive, tactical-level tasks (e.g., missile guidance and financial trading), potential advantages in decision speed and quality will tend to favor direct AI control of actions. In high-level strategic decisions, however—where stakes are higher, urgency is reduced, and criteria may be ambiguous—humans can exploit AI competence without ceding control: If AI systems can make excellent decisions, then they can suggest excellent options. Human choice among strategic options need not impede swift response to events, because even long-term strategies (e.g., U.S. nuclear strategy) can include prompt responses of any magnitude. In light of these considerations, we can expect senior human decision makers to choose to retain their authority, and without necessarily sacrificing competitiveness.

Pressures for speed and quality can favor AI control of decisions

When AI systems outperform humans in making decisions (weighing both speed and quality), competitive situations will drive humans to implement AI-controlled decision processes. The benefits of speed and quality will differ in different applications, however, as will the costs of error.

Speed is often critical in selecting and executing “tactical” actions

Decision-speed can be critical: Computational systems outperform humans in response time when controlling vehicles and launching defensive missiles, offering crucial advantages, and military planners foresee increasing pressures use AI-directed systems in high-tempo tactical exchanges. In tactical military situations, as in high-frequency financial trading, failure to exploit the advantages of AI control may lead to losses.

Quality is more important than speed in strategic planning

High-level strategic plans almost by definition guide actions extended in time. Even long-term strategies may of course be punctuated by prompt, conditional actions with large-scale consequences: U.S. nuclear strategy, for example, has been planned and revised over years and decades, yet contemplates swift and overwhelming nuclear counterstrikes. Fast response to events is compatible with deliberation in choosing and updating strategies.

System that can make excellent decisions could suggest excellent options

Superior reasoning and information integration may enable AI systems to identify strategic options with superhuman speed and quality, yet this need not translate into a pressure for humans to cede strategic control. If AI systems can make excellent decisions, then they can suggest excellent sets of options for consideration by human decision makers.

Note that developing sets of options does not imply a commitment to a utility function over world states; given uncertainties regarding human preferences, it is more appropriate to apply Pareto criteria to potentially incommensurate costs, benefits, and risks, and to offer proposals that need not be optimized to induce particular choices. In this connection, it is also important to remember that long-term strategies are normally subject to ongoing revision in light of changing circumstances and preferences, and hence adopting a long-term strategy need not entail long-term commitments.

Human choice among strategies does not preclude swift response to change

Profoundly surprising events that call for superhumanly-swift, large-scale, unanticipated strategic reconsideration and response seem likely to be rare, particularly in hypothetical futures in which decision makers have made effective use of superintelligent-quality strategic advice. Further, human choice among strategic options need not be slow: Under time pressure, a decision maker could scan a menu of presumably excellent options and make a quick, gut choice. Beyond this, human-approved strategies could explicitly allow for great flexibility under extraordinary circumstances. In light of these considerations, substantial incentives for routine relinquishment of high-level strategic control seem unlikely.

Senior human decision makers will likely choose to retain their authority

Absent high confidence that AI systems will consistently make choices aligned with human preferences, ceding human control of increasingly high-level decisions would incur increasing risks for declining (and ultimately slight) benefits. As a practical matter, we can expect senior human decision makers to choose to retain their authority unless forcibly dislodged, perhaps because they have lost the trust of other, more powerful human decision makers.