# §11 Potential AGI-enabling technologies also enable comprehensive AI services

*Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 11 of 40. Source label: `agent-is-service-tech`. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad*

**Thesis.** If future AI technologies can implement self-transforming AGI agents, then similar capabilities could more easily be applied to implement open, comprehensive AI services.

**Cites sections:** §8 *Strong optimization can strongly constrain AI capabilities, behavior, and effects*; §10 *R&D automation dissociates recursive improvement from AI agency*; §12 *AGI agents offer no compelling value*; §15 *Development-oriented models align with deeply-structured AI systems*; §22 *Machine learning can develop predictive models of human approval*; §23 *AI development systems can support effective human guidance*; §24 *Human oversight need not impede fast, recursive AI technology improvement*; §26 *Superintelligent-level systems can safely provide design and planning services*; §35 *Predictable aspects of future knowledge can inform AI safety strategies*

**Cited by sections:** §II, §1, §4, §14, §29

---
## Summary

In catastrophic runaway-AI scenarios, systems capable of self-improvement lead to—and hence precede—opaque AGI agents with general superhuman competencies. Systems capable of self-improvement would, however, embody high-level development capabilities that could first be exploited to upgrade ongoing, relatively transparent AI R&D automation. Along this path, transparency and control need not impede AI development, and optimization pressures can sharpen task focus rather than loosen constraints. Thus, in scenarios where advances in technology would enable the implementation of powerful but risky AGI agents, those same advances could instead be applied to provide comprehensive AI services—and stable, task-focused agents—while avoiding the potential risks of self-modifying AGI-agents.

## In runaway-AGI scenarios, self-improvement precedes risky competencies

Self-improving, general-purpose AI systems would, by definition, have the ability to build AI systems with capabilities applicable to AI development tasks. In classic AGI-takeover scenarios, the specific competencies that enable algorithmic self-improvement would precede more general superhuman competencies in modeling the world, defining world-changing goals, and pursuing (for example) a workable plan to seize control. Strong AI development capabilities would precede potential catastrophic threats.

## “Self”-improvement mechanisms would first accelerate R&D

In any realistic development scenario, highly-capable systems will follow less-capable predecessors (*e.g.*, systems with weaker architectures, smaller datasets, or less training), and developers will have practical knowledge of how to instantiate, train, and apply these systems. [Along paths that lead to systems able to implement more capable systems with little human effort](→§10), it will be natural for developers to apply those systems to specific development tasks. Developing and packaging an opaque, self-improving AI system might or might not be among those tasks.

## “Self”-improvement mechanisms have no special connection to agents

The option to develop and apply self-improving AGI systems would be compelling only if there were no comparable or superior alternatives. Given AI systems able to implement a wide range of AI systems, however, there would be no compelling reason to package and seal AI development processes in an opaque box. Quite the opposite: [Practical considerations generally favor development, testing, and integration of differentiated components](→§15). Potential AGI-level technologies could presumably automate such processes, while an open system-development architecture would retain system-level transparency and process control.

## Transparency and control need not impede the pace of AI development

Because the outputs of basic research—the building blocks of technology improvement—need not directly affect the world, [the need for human intervention in basic research is minimal and need not impede progress](→§24). Meanwhile, in applications, guiding development to serve human purposes is not a burden, but an [inherent part of the task of providing beneficial AI services](→§23). Note that developing and applying AI systems to help humans guide development is itself a task within the scope of [comprehensive AI R&D automation](→§22). [→ §23 *AI development systems can support effective human guidance* and §22 *Machine learning can develop predictive models of human approval*]

## Optimization pressures sharpen task focus

Thinking about prospects of applying high-level AI capabilities to the design and optimization of AI systems has been muddied by the tacit assumption that optimizing performance implies relaxing constraints on behavior. For any bounded task, however, this is exactly wrong: [The stronger the optimization, the stronger the constraints](→§8). In the context of AI-enabled AI-development, effective optimization of a development system will tend to minimize resources spent on off-task modeling and search. Regardless of their internal complexity, optimized components of AI development systems will have no spare time to daydream about world domination.

## Problematic emergent behaviors differ from classic AGI risks

Systems of optimized, stable components can be used to implement fully general mechanisms, hence some configurations of components could exhibit problematic emergent behaviors of unexpected kinds. We can expect and encourage developers to note and avoid architectures of the kind that [produce unexpected behaviors](→§35), perhaps aided by [AI-enabled analysis of both AI objectives](→§22), and [proposed implementations](→§26). Avoiding problematic emergent behaviors in task-oriented systems composed of stable components is inherently more tractable than attempting to confine or control a self-modifying AGI system that might by default act as superintelligent adversarial agent.

## Potential AGI technologies might best be applied to automate development of comprehensive AI services

In summary, an AI technology base that could implement powerful self-improving AGI agents could instead be applied to implement (or more realistically, upgrade) increasingly automated AI development, a capability that in turn can be applied to implement a comprehensive range of AI applications. Thus, swift, AI-enabled improvement of AI technology [does not require opaque self-improving systems](→§10), and [comprehensive AI services need not be provided by potentially risky AGI agents](→§12).

## Further Reading

- §10 *R&D automation dissociates recursive improvement from AI agency*

- §12 *AGI agents offer no compelling value*

- §15 *Development-oriented models align with deeply-structured AI systems*

- §24 *Human oversight need not impede fast, recursive AI technology improvement*

- §30 *Risky AI can help develop safe AI*

- §33 *Competitive AI capabilities will not be boxed*