Plain text: 35-sec32-world-stability.md · on GitHub

§32 Unaligned superintelligent agents need not threaten world stability

Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 32 of 40. Source label: world-stability. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad

Thesis. A well-prepared world, able to deploy extensive, superintelligent-level security resources, need not be vulnerable to subsequent takeover by superintelligent agents.

Cites sections: §12 AGI agents offer no compelling value; §23 AI development systems can support effective human guidance; §26 Superintelligent-level systems can safely provide design and planning services; §27 Competitive pressures provide little incentive to transfer strategic decisions to AI systems; §31 Supercapabilities do not entail “superpowers”

Cited by sections: §II, §6

Summary

It is often taken for granted that unaligned superintelligent-level agents could amass great power and dominate the world by physical means, not necessarily to human advantage. Several considerations suggest that, with suitable preparation, this outcome could be avoided:

In other words, applying SI-level capabilities to ensure strategic stability could enable us to coexist with SI-level agents that do not share our values. The present analysis outlines general prospects for an AI-stable world, but necessarily raises more questions than it can explore.

General, SI-level capabilities can precede AGI agents

As has been argued elsewhere, the R&D-automation/AI-services model of recursive improvement and AI applications challenges the assumption that the pursuit of general, SI-level AI capabilities naturally or necessarily leads to classic AGI agents. Today, we see increasingly automated AI R&D applied to the development of AI services, and this pattern will (or readily could) scale to comprehensive, SI-level AI services that include the service of developing new services. By the orthogonality thesis (Bostrom 2014), high-level AI services could be applied to more-or-less any range of tasks.

SI-level capabilities could be applied to strengthen defensive stability

World order today—from neighborhood safety to the national security—is imperfectly implemented through a range of defensive services, e.g., local surveillance, self-defense, and police; military intelligence, arms control, and defensive weapon systems. A leading strategic problem today is the offensive potential of nominally defensive systems (deterrence, for example, relies on offensive weapons), engendering the classic “security dilemma” and consequent arms-race dynamics.

Bracketing thorny, path-dependent questions of human perceptions, preferences, objectives, opportunities, and actions, one can envision a state of the world in which SI-level competencies have been applied to implement impartial, genuinely defensive security services. Desirable implementation steps and systemic characteristics would include:

  1. Preparatory, SI-level red-team/blue-team design competition

  2. Anticipatory deployment of well-resourced, SI-level security services

  3. Ongoing application of effective, physically-oriented surveillance

  4. Ongoing application of effective, physically-oriented security measures

In other words, one can envision an AI-stable world in which well-prepared, SI-level systems are applied to implement services that ensure physical security regardless of the preferences of unaligned or hostile actors. (Note that this does not presuppose a solution to AGI alignment: AI-supported design and implementation of policies for security services need not be equivalent to utility maximization by an AGI agent.)

Unopposed preparation enables strong defensive capabilities

A background assumption in this discussion is that, given access to SI-level capabilities, potentially enormous resources (indeed, literally astronomical) could be mobilized to achieve critical civilizational goals that include AGI-compatible strategic stability. In other words, we can expect civilizations as a whole to pursue convergent instrumental goals (Bostrom 2014, 109), and to apply the resulting capabilities.

In this connection, recall that what Bostrom (2014, 93) terms “superpowers” are contextual, being properties not of agents per se, but of agents that have an effective monopoly on the capability in question (Bostrom 2014, 104): In a prepared world, mere superintelligence would not confer superpowers. (Regarding the “hacking superpower” (Bostrom 2014, 94), note that, even today, practical operating systems can provide mathematically provable, hence unhackable, security guarantees. [fn: https://sel4.systems/])

Strong defensive capabilities can constrain problematic agents

The above points offer only an abstract sketch of a development process and objective, not a map of a road or a destination. A closer look can help to clarify key concepts:

1) Preparatory, SI-level red-team/blue-team design competition

can explore potential attacks while exploring the conditions necessary for security services to block attack capabilities with an ample margin of safety. Adversarial exercises could readily employ physical simulations that are qualitatively biased to favor hypothetical attackers, while assigning arbitrarily large, highly-asymmetric quantitative advantages to proposed security services. As noted above, enormous resources could potentially be mobilized to support SI-level exploration of hypothetical red-team threats and proposed blue-team security measures; thorough exploration would call for a good working approximation to what Bostrom (2014, 229) terms “technological completion”, at least in a design sense.

2) Anticipatory deployment of well-resourced, SI-level security services

would implement systems that reflect the results of stringent red-team/blue-competitions, and hence would employ more-than-adequate physical and computational resources. Note that preparatory, selective development and deployment of security systems strongly embodies what Bostrom (2014, 230) terms “differential technology development”.

3) Ongoing application of effective, physically-oriented surveillance

calls for collection of information sufficient to establish reliable (yet not excessively conservative) upper bounds on the scope of potentially threatening physical capabilities of potentially untrustworthy actors. Recognition of threats can be informed by risk-averse generalizations of worst-case red-team strategies.

4) Ongoing application of effective, physically-oriented security measures

calls for the application of ample (yet not unnecessarily conservative) resources to forestall potential threats; policies can be informed by amply (yet not excessively conservative) risk-averse generalizations of robust blue-team security measures. Crude security measures might require either strong interventions or stringent constraints on actors’ physical resources; well-designed security measures could presumably employ milder interventions and constraints, optimized for situation-dependent acceptability conditioned on global effectiveness.

This brief analysis necessarily raises more questions than it can explore

The concept of impartial and effective AI-enabled security services raises questions regarding the deep underpinnings of a desirable civilizational order, questions that cannot be explored without raising raising further questions at levels that range from security policies and physical enforcement to the entrenchment of constitutional orders and the potential diversity of coexisting frameworks of law. Prospects for a transition to a secure, AI-stable world raise further questions regarding potential paths forward, questions that involve not only technological developments, but ways in which the perceived interests and options of powerful, risk-averse actors might align well enough to shape actions that lead to widely-approved outcomes.

A familiar alternative scenario, global control by a value-aligned AGI agent, presents several difficulties

Discussions of superintelligence and AI safety often envision the development of an extremely powerful AI agent that will take control of the world and optimize the future in accord with human values. This scenario presents several difficulties: It seems impossible to define human values in a way that would be generally accepted, impossible to implement systems that would be trusted to optimize the world, and difficult to take control of the world (whether openly or by attempted stealth) without provoking effective, preemptive opposition from powerful actors. Fortunately, as outlined above, the foundational safety challenge—physical security—can be addressed while avoiding these problems.

Further Reading