# §II Overview: Questions, propositions, and topics *Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section II (introductory). Source label: `intro2`. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad* **Cites sections:** §1 *R&D automation provides the most direct path to an intelligence explosion*; §2 *Standard definitions of “superintelligence” conflate learning with competence*; §3 *To understand AI prospects, focus on services, not implementations*; §4 *The AI-services model includes both descriptive and prescriptive aspects*; §5 *Rational-agent models place intelligence in an implicitly anthropomorphic frame*; §6 *A system of AI services is not equivalent to a utility maximizing agent*; §7 *Training agents in human-like environments can provide useful, bounded services*; §8 *Strong optimization can strongly constrain AI capabilities, behavior, and effects*; §9 *Opaque algorithms are compatible with functional transparency and control*; §10 *R&D automation dissociates recursive improvement from AI agency*; §11 *Potential AGI-enabling technologies also enable comprehensive AI services*; §12 *AGI agents offer no compelling value*; §14 *The AI-services model brings ample risks*; §15 *Development-oriented models align with deeply-structured AI systems*; §16 *Aggregated experience and centralized learning support AI-agent applications*; §17 *End-to-end reinforcement learning is compatible with the AI-services model*; §18 *Reinforcement learning systems are not equivalent to reward-seeking agents*; §19 *The orthogonality thesis undercuts the generality of instrumental convergence*; §20 *Collusion among superintelligent oracles can readily be avoided*; §21 *Broad world knowledge can support safe task performance*; §22 *Machine learning can develop predictive models of human approval*; §23 *AI development systems can support effective human guidance*; §24 *Human oversight need not impede fast, recursive AI technology improvement*; §25 *Optimized advice need not be optimized to induce its acceptance*; §26 *Superintelligent-level systems can safely provide design and planning services*; §27 *Competitive pressures provide little incentive to transfer strategic decisions to AI systems*; §28 *Automating biomedical R&D does not require defining human welfare*; §29 *The AI-services model reframes the potential roles of AGI agents*; §30 *Risky AI can help develop safe AI*; §31 *Supercapabilities do not entail “superpowers”*; §32 *Unaligned superintelligent agents need not threaten world stability*; §33 *Competitive AI capabilities will not be boxed*; §34 *R&D automation is compatible with both strong and weak centralization*; §35 *Predictable aspects of future knowledge can inform AI safety strategies*; §36 *Desiderata and directions for interim AI safety guidelines*; §38 *Broadly-capable systems coordinate narrower systems*; §39 *Tiling task-space with AI services can provide general AI capabilities*; §40 *Could 1 PFLOP/s systems exceed the basic functional capacity of the human brain?* --- ## Summary This document outlines topics, questions, and propositions that address: 1. Prospects for an intelligence explosion 2. The nature of advanced machine intelligence 3. The relationship between goals and intelligence 4. The problem of using and controlling advanced AI 5. Near- and long-term considerations in AI safety and strategy The questions and propositions below reference sections of this document that explore key topics in more depth. From the perspective of AI safety concerns, this document offers support for several currently-controversial propositions regarding artificial general intelligence: - That AGI agents have no natural role in developing general AI capabilities. - That AGI agents would offer no unique and substantial value in providing general AI services. - That AI-based security services could safely constrain subsequent AGI agents, even if these operate at a superintelligent level. ## Reframing prospects for an intelligence explosion ### Does recursive improvement imply self-transforming agents? Ongoing automation of tasks in AI R&D suggests a model of asymptotically-recursive technology improvement that scales to superintelligent-level (SI-level) systems. In the R&D-automation model, recursive improvement is systemic, not internal to distinct systems or agents. The model is fully generic: It requires neither assumptions regarding the content of AI technologies, nor assumptions regarding the pace or sequence of automation of specific R&D tasks. Classic self-transforming AGI agents would be strictly more difficult to implement, hence are not on the short path to an intelligence explosion. - §1 *R&D automation provides the most direct path to an intelligence explosion* - §10 *R&D automation dissociates recursive improvement from AI agency* - §11 *Potential AGI-enabling technologies also enable comprehensive AI services* ### Would self-transforming agents provide uniquely valuable functionality? Suites of AI services that support SI-level AI development—working in consultation with clients and users—could provide a comprehensive range of novel AI services; these would presumably include services provided by adaptive, upgradable, task-oriented agents. It is difficult to see how the introduction of potentially unstable agents that undergo autonomous open-ended self-transformation would provide additional value. - §3 *To understand AI prospects, focus on services, not implementations* - §12 *AGI agents offer no compelling value* ### Can fast recursive improvement be controlled and managed? Recursive improvement of *basic AI technologies* would apply allocated machine resources to the development of increasingly functional building blocks for AI applications (better algorithms, architectures, training methods, *etc.*); basic technology development of this sort could be open-ended, recursive, and fast, yet non-problematic. *Deployed AI applications* call for careful management, but applications stand outside the inner loop of basic-technology improvement. - §23 *AI development systems can support effective human guidance* - §24 *Human oversight need not impede fast, recursive AI technology improvement* ### Would general learning algorithms produce systems with general competence? The application of an idealized, fully general learning algorithm would *enable* but not *entail* the learning of any particular competence. Time, information, and resource constraints are incompatible with universal competence, regardless of *ab initio* learning capacity. - §2 *Standard definitions of “superintelligence” conflate learning with competence* - §21 *Broad world knowledge can support safe task performance* - §39 *Tiling task-space with AI services can provide general AI capabilities* ## Reframing the nature of advanced machine intelligence ### Is human learning an appropriate model for AI development? Action, experience, and learning are typically decoupled in AI development: Action and experience are aggregated, not tied to distinct individuals, and the machine analogues of cognitive change can be profound during system development, yet absent in applications. As we see in AI technology today, learning algorithms can be applied to produce and upgrade systems that do not themselves embody those algorithms. Accordingly, using human learning and action as a model for AI development and application can be profoundly misleading. - §2 *Standard definitions of “superintelligence” conflate learning with competence* - §7 *Training agents in human-like environments can provide useful, bounded services* - §16 *Aggregated experience and centralized learning support AI-agent applications* ### Does stronger optimization imply greater capability? Because optimization for a task focuses capabilities on that task, strong optimization of a system acts as a strong constraint; in general, optimization does not extend the scope of a task or increase the resources employed to perform it. System optimization typically tends to reduce resource consumption, increase throughput, and improve the quality of results. - §8 *Strong optimization can strongly constrain AI capabilities, behavior, and effects* ### Do broad knowledge and deep world models imply broad AI capabilities? Language translation systems show that safe, stable, high-quality task performance can be compatible with (and even require) broad and deep knowledge about the world. The underlying principle generalizes to a wide range of tasks. - §21 *Broad world knowledge can support safe task performance* ### Must we model SI-level systems as rational, utility-maximizing agents? The concept of rational, utility-maximizing agents was developed as an idealized model of human decision makers, and hence is inherently (though abstractly) anthropomorphic. Utility-maximizing agents may be intelligent systems, but intelligent systems (and in particular, *systems of agents*) need not be utility-maximizing agents. - §5 *Rational-agent models place intelligence in an implicitly anthropomorphic frame* - §6 *A system of AI services is not equivalent to a utility maximizing agent* - §17 *End-to-end reinforcement learning is compatible with the AI-services model* - §18 *Reinforcement learning systems are not equivalent to reward-seeking agents* ### Must we model SI-level systems as unitary and opaque? Externally-determined features of AI components (including their development histories, computational resources, communication channels, and degree of mutability) can enable structured design and functional transparency, even if the components themselves employ opaque algorithms and representations. - §9 *Opaque algorithms are compatible with functional transparency and control* - §15 *Development-oriented models align with deeply-structured AI systems* - §38 *Broadly-capable systems coordinate narrower systems* ## Reframing the relationship between goals and intelligence ### What does the orthogonality thesis imply for the generality of convergent instrumental goals? Intelligent systems optimized to perform perform bounded tasks (in particular, episodic tasks with a bounded time horizon) need not be agents with open-ended goals that call for self preservation, cognitive enhancement, resource acquisition, and so on; by Bostrom’s orthogonality thesis, this holds true regardless of the level of intelligence applied to those tasks. - §19 *The orthogonality thesis undercuts the generality of instrumental convergence* ### How broad is the basin of attraction for convergent instrumental goals? Instrumental goals are closely linked to final goals of indefinite scope that concern the indefinite future. Societies, organizations, and (in some applications) high-level AI agents may be drawn toward convergent instrumental goals, but high-level intelligence *per se* does not place AI systems within this basin of attraction, even if applied to broad problems that are themselves long-term. - §21 *Broad world knowledge can support safe task performance* - §25 *Optimized advice need not be optimized to induce its acceptance* ## Reframing the problem of using and controlling advanced AI ### Would the ability to implement potentially-risky self-transforming agents strongly motivate their development? If future AI technologies could implement potentially-risky, self-transforming AGI agents, then similar, more accessible technologies could more easily be applied to implement open, comprehensive AI services. Because the service of providing new services subsumes the proposed instrumental value of self-transforming agents, the incentives to implement potentially-risky self-transforming agents appear to be remarkably small. - §11 *Potential AGI-enabling technologies also enable comprehensive AI services* - §12 *AGI agents offer no compelling value* - §33 *Competitive AI capabilities will not be boxed* ### How can human learning during AI development contribute to current studies of AI safety strategies? We can safely predict that AI researchers will continue to identify and study surprising AI behaviors, and will seek to exploit, mitigate, or avoid them in developing AI applications. This and other predictable aspects of future knowledge can inform current studies of strategies for safe AI development. - §35 *Predictable aspects of future knowledge can inform AI safety strategies* ### Can we avoid strong trade-offs between development speed and human oversight? Basic research, which sets the overall pace of technological progress, could be safe and effective with relatively little human guidance; application development, by contrast, requires strong human guidance, but as an inherent part of the development task—to deliver desirable functionality—rather than as an impediment. Support for human guidance can be seen as an AI service, and can draw on predictive models of human approval and concerns. - §22 *Machine learning can develop predictive models of human approval* - §23 *AI development systems can support effective human guidance* - §24 *Human oversight need not impede fast, recursive AI technology improvement* ### Can we architect safe, superintelligent-level design and planning services? Consideration of concrete task structures and corresponding services suggests that SI-level AI systems can safely converse with humans, perform creative search, and propose designs for systems to be implemented and deployed in the world. Systems that provide design and planning services can be optimized to provide advice without optimizing to manipulate human acceptance of that advice. - §26 *Superintelligent-level systems can safely provide design and planning services* - §28 *Automating biomedical R&D does not require defining human welfare* ### Will competitive pressures force decision-makers to transfer strategic decisions to AI systems? In both markets and battlefields, advantages in reaction time and decision quality can motivate transfer of tactical control to AI systems, despite potential risks; for strategic decisions, however, the stakes are higher, speed is less important, advice can be evaluated by human beings, and the incentives to yield control are correspondingly weak. - §27 *Competitive pressures provide little incentive to transfer strategic decisions to AI systems* ### What does the R&D-automation/AI-services model imply for studies of conventional vs. extreme AI-risk concerns? Increasing automation of AI R&D suggests that AI capabilities may advance surprisingly rapidly, a prospect that increases the urgency of addressing conventional AI risks such as unpredictable failures, adversarial manipulation, criminal use, destabilizing military applications, and economic disruption. Prospects for the relatively rapid emergence of systems with broad portfolios of capabilities, including potentially autonomous planning and action, lend increased credence to extreme AI-risk scenarios, while the AI-services model suggests strategies for avoiding or containing those risks while gaining the benefits of high- and SI-level AI capabilities. - §12 *AGI agents offer no compelling value* - §14 *The AI-services model brings ample risks* ### What can agent-oriented studies of AI safety contribute, if risky AI agents are optional? People will want AI systems that plan and act in the world, and some systems of this class can naturally be modeled as rational, utility-directed agents. Studies of systems within the rational-agent model can contribute to AI safety and strategy in multiple ways, including: - Expanding the range of safe AI-agent architectures by better understanding how to define bounded tasks in a utility-directed framework. - Expanding safe applications of utility-directed agents to less well-bounded tasks by better understanding how to align utility functions with human values. - Better understanding the boundaries beyond which combinations of agent architectures and tasks could give rise to classic AGI-agent risks. - Better understanding how (and under what conditions) evolutionary pressures could engender perverse strategic behavior in nominally non-agent-like systems. - Exploring ways access to high-level AI services could could help to avoid or mitigate classic agent-centered AI risks. - §10 *R&D automation dissociates recursive improvement from AI agency* - §29 *The AI-services model reframes the potential roles of AGI agents* ## Reframing near- and long-term considerations in AI safety and strategy ### Could widely available current or near-term hardware support superintelligence? Questions of AI safety and strategy become more urgent if future, qualitatively SI-level computation can be implemented with greater-than-human task throughput on affordable, widely-available hardware. There is substantial reason to think that this condition already holds. - §40 *Could 1 PFLOP/s systems exceed the basic functional capacity of the human brain?* ### What kinds of near-term safety-oriented guidelines might be feasible and useful? Current technology presents no catastrophic risks, and several aspects of current development practice align not only with safety, but with good practice in science and engineering. Development of guidelines that codify current good practice could contribute to near-term AI safety with little organizational cost, while also engaging the research community in an ongoing process that addresses longer-term concerns. - §4 *The AI-services model includes both descriptive and prescriptive aspects* - §34 *R&D automation is compatible with both strong and weak centralization* - §35 *Predictable aspects of future knowledge can inform AI safety strategies* - §36 *Desiderata and directions for interim AI safety guidelines* ### How can near-term differential technology development contribute to safety? Concerns with AI safety and strategy should influence research agendas intended to promote broad societal benefits. Directions that deserve emphasis include work on concrete problems in AI safety, on predictive models of human approval and disapproval, and on capabilities that could facilitate the detection of potential bad actors. - §22 *Machine learning can develop predictive models of human approval* - §23 *AI development systems can support effective human guidance* ### In the long term, are unaligned superintelligent agents compatible with safety? A well-prepared world, able to deploy extensive, superintelligent-level security resources, need not be vulnerable to subsequent takeover by superintelligent agents. - §20 *Collusion among superintelligent oracles can readily be avoided* - §30 *Risky AI can help develop safe AI* - §31 *Supercapabilities do not entail “superpowers”* - §32 *Unaligned superintelligent agents need not threaten world stability* ## Conclusions We can expect to see AI-enabled automation of AI research and development continue to accelerate, both leveraging and narrowing the scope of human insights required for progress in AI technologies. Asymptotically-recursive improvement of AI technologies can scale to a superintelligent level, supporting the development of a fully-general range of high-level AI services that includes the service of developing new services in response to human demand. Because general AI-development capabilities do not imply general capabilities in any particular system or agent, classic AGI agents would be potential *products* of SI-level AI development capabilities, not a path to uniquely valuable functionality. Within the space of potential intelligent systems, agent-centered models span only a small region, and even abstract, utility-directed rational-agent models invite implicitly anthropomorphic assumptions. In particular, taking human learning as a model for machine learning has encouraged the conflation of intelligence-as-learning-capacity with intelligence-as-competence, while these aspects of intelligence are routinely and cleanly separated AI system development: Learning algorithms are typically applied to train systems that do not themselves embody those algorithms. The service-centered perspective on AI highlights the generality of Bostrom’s Orthogonality Thesis: SI-level capabilities can indeed be applied to any task, including services that are (as is typical of services) optimized to deliver bounded results with bounded resources in bounded times. The pursuit of Bostrom’s convergent instrumental goals would impede—not improve—the performance of such services, yet would be natural for those same instrumental goals to be hotly pursued by human organizations (or AI agents) that *employ* AI services. Prospects for service-oriented superintelligence reframe the problem of managing advanced AI technologies: Potentially-risky self-transforming agents become optional, rather than overwhelmingly valuable, the separation of basic research from application development can circumvent trade-offs between development speed and human oversight, and natural task architectures suggest safe implementations of SI-level design and planning services. In this connection, distinctions between tactical execution and strategic advice suggest that even stringent competitive pressures need not push decision-makers to cede strategic decisions to AI systems. In contrast to unprecedented-breakthrough models that postulate runaway self-transforming agents, prospects for the incremental emergence of diverse, high-level AI capabilities promise broad, safety-relevant experience with problematic (yet not catastrophic) AI behaviors. Safety guidelines can begin by codifying current safe practices, which include training and re-training diverse architectures while observing and studying surprising behaviors. The development of diverse, high-level AI services also offers opportunities for safety-relevant differential technology development, including the development of common-sense predictive models of human concerns that can be applied to improve the value and safety of AI services and AI agents. The R&D-automation/AI-services model suggests that conventional AI risks (*e.g.*, failures, abuse, and economic disruption) are apt to arrive more swiftly than expected, and perhaps in more acute forms. While this model suggests that extreme AI risks may be relatively avoidable, it also emphasizes that such risks could arise more quickly than expected. In this context, agent-oriented studies of AI safety can both expand the scope of safe agent applications and improve our understanding of the conditions for risk. Meanwhile, service-oriented studies of AI safety could usefully explore potential applications of high-level services to general problems of value alignment and behavioral constraint, including the potential architecture of security services that could ensure safety in a world in which some extremely intelligent agents are not inherently trustworthy.