# §1 R&D automation provides the most direct path to an intelligence explosion

*Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 1 of 40. Source label: `most-direct-path`. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad*

**Thesis.** The most direct path to accelerating AI-enabled progress in AI technology leads through AI R&D automation, not through self-transforming AI agents.

**Cites sections:** §5 *Rational-agent models place intelligence in an implicitly anthropomorphic frame*; §10 *R&D automation dissociates recursive improvement from AI agency*; §11 *Potential AGI-enabling technologies also enable comprehensive AI services*; §12 *AGI agents offer no compelling value*; §16 *Aggregated experience and centralized learning support AI-agent applications*; §23 *AI development systems can support effective human guidance*

**Cited by sections:** §II, §4, §34

---
## Summary

AI-enabled development of improved AI algorithms could potentially lead to an accelerating feedback process, enabling a so-called “intelligence explosion”. This quite general and plausible concept has commonly been identified with a specific, challenging, and risky implementation in which a predominant concentration of AI-development functionality is embodied in a distinct, goal-directed, self-transforming AI system—an AGI agent. A task-oriented analysis of AI-enabled AI development, however, suggests that self-transforming agents would play no natural role, even in the limiting case of explosively-fast progress in AI technology. Because the mechanisms underlying a potential intelligence explosion are already in operation, and have no necessary connection to unprecedented AGI agents, paths to extremely rapid progress in AI technology may be both more direct and more controllable than has been commonly assumed.

## AI-enabled AI development could lead to an intelligence explosion

As suggested by Good (1966), AI systems could potentially outperform human beings in the task of AI development, and hence “could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”

## Risk models have envisioned AGI agents driving an intelligence explosion

The widespread (yet questionable) assumption that this feedback process—“recursive improvement”—would entail [self-transformation of a particular AI system that engages with the world as an agent](→§5) has motivated a threat model in which “the machine” might not be “docile enough to tell us how to keep it under control” (Good 1966, 33). The surprisingly complex and difficult ramifications of this threat model have been extensively explored in recent years (*e.g.*, in (Bostrom 2014)).

## Self-transforming AI agents have no natural role in recursive improvement

[Advances in AI technology emerge from research and development](→§10), a process that comprises a range of different technical tasks. These tasks are loosely coupled, and none requires universal competence: Consider, for example, the technology-centered tasks of training-algorithm development, benchmark development, architecture search, and chip design; and beyond these, application development tasks that include [human consultation](→§23) and [application prototyping, together with testing, monitoring, and customer service](→§16).

Accordingly, general-purpose, self-transforming AI agents play no natural role in the process of AI R&D: They are potential (and potentially dangerous) *products*—not *components*—of AI development systems. To the extent that the concept of “docility” may be relevant to development systems as a whole, this desirable property is also, by default, [deeply ingrained in the nature of the services they provide](→§23).

## The direct path to an intelligence explosion does not rely on AGI agents

It may be tempting to imagine that *self*-improvement would be simpler than loosely-coupled *systemic* improvement, but drawing a conceptual boundary around a system does not simplify its contents, and to require that systems capable of open-ended AI development *also* exhibit tight integration, functional autonomy, and operational agency would [increase—not reduce—implementation challenges](→§11). To attempt to place competitive R&D capabilities inside an agent presents difficulties, yet [provides no compensating advantages in performance or utility](→§12).

The pervasive assumption that an intelligence explosion must await the development of agents capable of autonomous, open-ended self improvement has encouraged skepticism and complacency [fn: See (Chollet 2017)] regarding prospects for superintelligent-level AI. [fn: See (Kelly 2017)] If we think that any given set of human tasks can be automated, however, than so can any—and in the limit, all—AI R&D tasks. This proposition seems relatively uncontroversial, yet has profound implications for the likely trajectory of AI development.

AI-enabled automation of fundamental AI R&D tasks is markedly accelerating. [fn: For example, see (Zoph et al. 2017; Bello et al. 2017; Jaderberg et al. 2017; Chen et al. 2017; Schrimpf et al. 2017)] As the range of automated tasks increases, we can expect feedback loops to tighten, enabling AI development at a pace that, while readily controlled, has no obvious limits.

## Further Reading

- §11 *Potential AGI-enabling technologies also enable comprehensive AI services*

- §12 *AGI agents offer no compelling value*

- §23 *AI development systems can support effective human guidance*

- §24 *Human oversight need not impede fast, recursive AI technology improvement*

- §30 *Risky AI can help develop safe AI*

- §40 *Could 1 PFLOP/s systems exceed the basic functional capacity of the human brain?*