# §10 R&D automation dissociates recursive improvement from AI agency

*Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 10 of 40. Source label: `automation-not-agents`. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad*

**Thesis.** In automation of AI research and development, AI agents are useful products, not necessary components.

**Cited by sections:** §II, §1, §3, §4, §5, §11, §14, §17, §21, §24, §25, §28, §29, §31, §33, §34

---
## Summary

AI development processes can be automated by applying specialized AI competencies to AI-development tasks, and incremental automation of AI development systems can continue to reflect the familiar organization of research and development processes. Asymptotically recursive technology improvement requires neither self-improving components nor agents that act in the world, and can provide general AI functionality without recourse to general AI agents. The R&D-automation model offers a range of potential affordances for addressing AI safety concerns.

## R&D automation can employ on diverse, specialized AI tools

AI research and development, like other areas of software technology, exploits a growing range of automated tools. As AI technologies approach or exceed human competence in AI development tasks, we can expect to see incremental automation throughout the development process, asymptotically enabling recursive technology improvement. The development-automation model differs from classic models of recursive improvement in that it does not call for self-improving general-purpose agents.

## AI R&D automation will reflect universal aspects of R&D processes

The R&D process links the discovery of general principles and mechanisms to the construction of complex systems tailored to specific functions and circumstances. The R&D-automation model describes AI development today, and increasing automation of AI development seems unlikely to obliterate this deep and universal task structure.

> **Figure 3.** Schematic model of the current AI research and development pipeline.
> *(Image `fig2.png` in the source; not reproduced here. The description below is editorial — a model-written first pass — not the author's text.)*
>
> The diagram shows five boxes connected left to right by gray block arrows, with one gray arrow looping back from the last box to the second box. The first box, "Basic research," points rightward into the second box, "Design resources," which lists five items: "Design methods," "Training methods," "Trainable components," "Trained resources," and "Development tools." Three parallel arrows lead from "Design resources" into the third box, "Application development." From "Application development," three parallel arrows lead into the fourth box, "Products and tools," which lists three items: "Delivered products," "Trained resources," and "Development tools." A separate feedback arrow, labeled "Improved design resources," runs from "Products and tools" back to "Design resources," closing a loop from the output of the pipeline to its second stage.

## AI R&D automation will reflect the structure of AI development tasks

In AI R&D, the elements of development tasks are organized along lines suggested by the following diagram:

> **Figure 4.** Schematic model of an AI-enabled application-oriented system development task that draws on a range of previously developed components.
> *(Image `fig3.png` in the source; not reproduced here. The description below is editorial — a model-written first pass — not the author's text.)*
>
> The diagram shows three boxes connected left to right by two large block arrows, depicting a pipeline from inputs to a builder to a finished product.
>
> The leftmost box, labeled "Design inputs," contains two sub-boxes. The first, "Task-specific inputs," lists "Performance metrics" and "Training corpora." The second, "Generic inputs," lists "Trainable components," "Training methods," and "Trained resources." A block arrow points from the "Design inputs" box to the middle box.
>
> The middle box, labeled "AI builder," contains two sub-boxes, "System architecting" and "Training and testing," arranged vertically with a pair of curved arrows forming a loop between them (one arrow from "System architecting" down to "Training and testing," and one arrow from "Training and testing" back up to "System architecting"), indicating an iterative cycle between the two activities. A block arrow points from the "AI builder" box to the rightmost box.
>
> The rightmost box, labeled "Delivered products," contains one sub-box, "Task-specific component," positioned above three sub-boxes labeled "Resource," "Resource," and "Resource" (one centered below and two flanking it). Double-headed arrows connect the "Task-specific component" box to each of the three "Resource" boxes, indicating two-way relationships between the task-specific component and each resource.

In architecting AI systems, application developers can draw on relatively generic sets of trainable components and training methods, and can compose these with previously developed resources. The resulting architectures are trained with task-specific performance metrics and corpora, and revised based on results.

Incremental automation does not change the fundamental structure of development tasks: The R&D automation model accommodates AI-enabled innovation in components, methods, resources, training corpora, and performance metrics; the model also accommodates ongoing involvement of human developers in any role. Over time, we can expect human roles to shift from technical implementation to general task description and performance evaluation.

## AI R&D automation leads toward recursive technology improvement

In the schematic diagram above, an AI builder is itself an R&D product, as are AI systems that perform exploratory AI research. Pervasive and asymptotically complete AI-enabled automation of AI R&D can enable what amounts to recursive improvement, raising the yield of AI progress per unit of human effort without obvious bound. In this model there is no locus of activity that corresponds to recursive “self” improvement; as we see in today’s AI R&D community, loosely coupled activities are sufficient to advance all aspects of AI technology.

## General SI-level functionality does not require general SI-level agents

The classic motivation for building self-improving general-purpose superintelligent agents is to provide systems that can perform a full range of tasks with superintelligent competence. The R&D-automation model, however, shows how to provide, on demand, systems that can perform any of a fully general range of tasks without invoking the services of a fully general agent.

## The R&D-automation model reframes the role of AI safety studies and offers potential affordances for addressing AI safety problems

In the present framework, agent-oriented AI safety research plays the dual roles of expanding the scope of safe agent functionality and identifying classes of systems and applications (including tightly coupled configurations of R&D components) in which radically unsafe agent behaviors might arise unintentionally. In other words, agent-oriented safety work can both find safe paths and mark potential hazards.

The R&D-automation model describes component-based systems that are well-suited to the production of component-based systems, hence it invites consideration of potential safety-relevant affordances of deeply-structured AI implementations. In particular, the prospect of safe access to superintelligent machine learning invites consideration of predictive models of human approval that are trained on large corpora of human responses to events and actions, and subsequently serve as components of structured, approval-directed AI agents.

## Further Reading

- §12 *AGI agents offer no compelling value*

- §15 *Development-oriented models align with deeply-structured AI systems*

- §22 *Machine learning can develop predictive models of human approval*

- §24 *Human oversight need not impede fast, recursive AI technology improvement*

- §30 *Risky AI can help develop safe AI*

- §35 *Predictable aspects of future knowledge can inform AI safety strategies*