# §5 Rational-agent models place intelligence in an implicitly anthropomorphic frame

*Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 5 of 40. Source label: `implicitly-anthropomorphic`. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad*

**Thesis.** It is a mistake to frame intelligence as a property of mind-like systems, whether these systems are overtly anthropomorphic or abstracted into decision-making processes that guide rational agents.

**Cites sections:** §10 *R&D automation dissociates recursive improvement from AI agency*; §12 *AGI agents offer no compelling value*; §15 *Development-oriented models align with deeply-structured AI systems*; §16 *Aggregated experience and centralized learning support AI-agent applications*; §21 *Broad world knowledge can support safe task performance*; §23 *AI development systems can support effective human guidance*; §28 *Automating biomedical R&D does not require defining human welfare*; §33 *Competitive AI capabilities will not be boxed*

**Cited by sections:** §II, §1, §6, §7, §29, §30

---
## Summary

Concepts of artificial intelligence have long been tied to concepts of mind, and even abstract, rational-agent models of intelligent systems are built on psychomorphic and recognizably anthropomorphic foundations. Emerging AI technologies do not fit a psychomorphic frame, and are radically unlike evolved intelligent systems, yet technical analysis of prospective AI systems has routinely adopted assumptions with recognizably biological characteristics. To understand prospects for AI applications and safety, we must consider not only psychomorphic and rational-agent models, but also a wide range of intelligent systems that present strongly contrasting characteristics.

## The concept of mind has framed our concept of intelligence

Minds evolved to guide organisms through life, and natural intelligence evolved to make minds more effective. Because the only high-level intelligence we know is an aspect of human minds, it is natural for our concept of mind to frame our concept of intelligence. Indeed, popular culture has envisioned advanced AI systems as artificial minds that are by default much like our own. AI-as-mind has powerful intuitive appeal.

The concept of AI-as-mind is deeply embedded in current discourse. For example, in cautioning against anthropomorphizing superintelligent AI, Bostrom (2014, 105) urges us to “reflect for a moment on the vastness of the space of possible minds”, an abstract space in which “human minds form a tiny cluster”. To understand prospects for superintelligence, however, we must consider a broader space of potential intelligent systems, a space in which *mind-like systems themselves* form a tiny cluster.

## Studies of advanced AI often posit intelligence in a psychomorphic role

Technical studies of AI control set aside explicitly human psychological concepts by modeling AI systems as goal-seeking rational agents. Despite their profound abstraction, however, rational-agent models originated as idealizations of human decision-making, and hence place intelligence in an implicitly anthropomorphic frame. More concretely, in rational-agent models, the *content* of human minds (human values, goals, cognitive limits…) is abstracted away, yet the *role* of minds in guiding decisions is retained. An agent’s decision-making process fills an inherently mind-shaped slot, and that slot frames a recognizably psychomorphic concept of intelligence.

This is problematic: Although the rational-agent model is broad, it is still too narrow to serve as a general model of intelligent systems.

## Intelligent systems need not be psychomorphic

What would count as high-level yet non-psychomorphic intelligence? One would be inclined to say that we have general, high-level AI if a coordinated pool of AI resources could, in aggregate:

- Do theoretical physics and [biomedical research](→§28)

- Provide a [general-purpose conversational interface](→§21) for discussing AI tasks

- [Discuss and implement](→§23) designs for self-driving cars, spacecraft, and [AI systems](→§12)

- Effectively [automate development](→§10) of next-generation AI systems for AI design

None of these AI tasks is fundamentally different from translating languages, learning games, driving cars, or designing neural networks—tasks performed by systems not generally regarded as mind-like. Regarding the potential power such a coordinated pool of AI services, note that [automating the development of AI systems for AI design](→§10) enables what amounts to recursive improvement.

## Engineering and biological evolution differ profoundly

Rather than regarding artificial intelligence as something that fills a mind-shaped slot, we can instead consider AI systems as products of increasingly-automated technology development, an extension of the R&D process that we see in the world today. This development-oriented perspective on AI technologies highlights profound and pervasive differences between evolved and engineered intelligent systems (see [table](#eng_evol)).

|                               |                       |                            |
|:------------------------------|:----------------------|:---------------------------|
| *Organization of units:*      | Distinct organisms    | Systems of components      |
| *Origin of new capabilities:* | Incremental evolution | Research, development      |
| *Origin of instances:*        | Birth, development    | Downloading files          |
| *Basis for learning tasks:*   | Individual experience | Aggregated training data   |
| *Transfer of knowledge:*      | Teaching, imitation   | Copying models, parameters |
| *Necessary competencies:*     | General life skills   | Specific task performance  |
| *Success metric:*             | Reproductive fitness  | Fitness for purpose        |
| *Self-modification:*          | Necessary             | Optional                   |
| *Continuity of existence:*    | Necessary             | Optional                   |
| *World-oriented agency:*      | Necessary             | Optional                   |

## Studies of AI prospects have often made tacitly biological assumptions

Although technical models of artificial intelligence avoid overtly biological assumptions, it is nonetheless common (though far from universal!) to assume that advanced AI systems will:

- Exist as individuals, rather than as [systems of coordinated components](→§15)

- Learn from individual experience, rather than from [aggregated training data](→§16)

- Develop through self-modification, rather than being [constructed](→§21) and [updated](→§33)

- Exist continuously, rather than being [instantiated on demand](→§33)

- Pursue world-oriented goals, rather than [performing specific tasks](→§10)

These assumptions have recognizable biological affinities, and they invite further assumptions that are tacitly biomorphic, psychomorphic, and even anthropomorphic.

## Potential mind-like systems are situated in a more general space of potential intelligent systems

It has been persuasively argued that rational, mind-like superintelligence is an attractor in the space of potential AI systems, whether by design or inadvertent emergence. A crucial question, however, is the *extent of the basin of attraction* for mind-like systems within the far more general space of potential AI systems. The discussion above suggests that this basin is far from coextensive with the space of highly-capable AI systems, including systems that can, in aggregate, [provide superintelligent-level services across an indefinitely wide range of tasks](→§12). We cannot chart the space of potential AI problems and solutions solely within the confines of rational-agent models, because most of that space lies outside.

## Further Reading

- §2 *Standard definitions of “superintelligence” conflate learning with competence*

- §7 *Training agents in human-like environments can provide useful, bounded services*

- §10 *R&D automation dissociates recursive improvement from AI agency*

- §12 *AGI agents offer no compelling value*

- §15 *Development-oriented models align with deeply-structured AI systems*