Plain text: 08-sec05-implicitly-anthropomorphic.md · on GitHub

§5 Rational-agent models place intelligence in an implicitly anthropomorphic frame

Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 5 of 40. Source label: implicitly-anthropomorphic. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad

Thesis. It is a mistake to frame intelligence as a property of mind-like systems, whether these systems are overtly anthropomorphic or abstracted into decision-making processes that guide rational agents.

Cites sections: §10 R&D automation dissociates recursive improvement from AI agency; §12 AGI agents offer no compelling value; §15 Development-oriented models align with deeply-structured AI systems; §16 Aggregated experience and centralized learning support AI-agent applications; §21 Broad world knowledge can support safe task performance; §23 AI development systems can support effective human guidance; §28 Automating biomedical R&D does not require defining human welfare; §33 Competitive AI capabilities will not be boxed

Cited by sections: §II, §1, §6, §7, §29, §30

Summary

Concepts of artificial intelligence have long been tied to concepts of mind, and even abstract, rational-agent models of intelligent systems are built on psychomorphic and recognizably anthropomorphic foundations. Emerging AI technologies do not fit a psychomorphic frame, and are radically unlike evolved intelligent systems, yet technical analysis of prospective AI systems has routinely adopted assumptions with recognizably biological characteristics. To understand prospects for AI applications and safety, we must consider not only psychomorphic and rational-agent models, but also a wide range of intelligent systems that present strongly contrasting characteristics.

The concept of mind has framed our concept of intelligence

Minds evolved to guide organisms through life, and natural intelligence evolved to make minds more effective. Because the only high-level intelligence we know is an aspect of human minds, it is natural for our concept of mind to frame our concept of intelligence. Indeed, popular culture has envisioned advanced AI systems as artificial minds that are by default much like our own. AI-as-mind has powerful intuitive appeal.

The concept of AI-as-mind is deeply embedded in current discourse. For example, in cautioning against anthropomorphizing superintelligent AI, Bostrom (2014, 105) urges us to “reflect for a moment on the vastness of the space of possible minds”, an abstract space in which “human minds form a tiny cluster”. To understand prospects for superintelligence, however, we must consider a broader space of potential intelligent systems, a space in which mind-like systems themselves form a tiny cluster.

Studies of advanced AI often posit intelligence in a psychomorphic role

Technical studies of AI control set aside explicitly human psychological concepts by modeling AI systems as goal-seeking rational agents. Despite their profound abstraction, however, rational-agent models originated as idealizations of human decision-making, and hence place intelligence in an implicitly anthropomorphic frame. More concretely, in rational-agent models, the content of human minds (human values, goals, cognitive limits…) is abstracted away, yet the role of minds in guiding decisions is retained. An agent’s decision-making process fills an inherently mind-shaped slot, and that slot frames a recognizably psychomorphic concept of intelligence.

This is problematic: Although the rational-agent model is broad, it is still too narrow to serve as a general model of intelligent systems.

Intelligent systems need not be psychomorphic

What would count as high-level yet non-psychomorphic intelligence? One would be inclined to say that we have general, high-level AI if a coordinated pool of AI resources could, in aggregate:

Do theoretical physics and biomedical research
Provide a general-purpose conversational interface for discussing AI tasks
Discuss and implement designs for self-driving cars, spacecraft, and AI systems
Effectively automate development of next-generation AI systems for AI design

None of these AI tasks is fundamentally different from translating languages, learning games, driving cars, or designing neural networks—tasks performed by systems not generally regarded as mind-like. Regarding the potential power such a coordinated pool of AI services, note that automating the development of AI systems for AI design enables what amounts to recursive improvement.

Engineering and biological evolution differ profoundly

Rather than regarding artificial intelligence as something that fills a mind-shaped slot, we can instead consider AI systems as products of increasingly-automated technology development, an extension of the R&D process that we see in the world today. This development-oriented perspective on AI technologies highlights profound and pervasive differences between evolved and engineered intelligent systems (see table).


Organization of units:	Distinct organisms	Systems of components
Origin of new capabilities:	Incremental evolution	Research, development
Origin of instances:	Birth, development	Downloading files
Basis for learning tasks:	Individual experience	Aggregated training data
Transfer of knowledge:	Teaching, imitation	Copying models, parameters
Necessary competencies:	General life skills	Specific task performance
Success metric:	Reproductive fitness	Fitness for purpose
Self-modification:	Necessary	Optional
Continuity of existence:	Necessary	Optional
World-oriented agency:	Necessary	Optional

Studies of AI prospects have often made tacitly biological assumptions

Although technical models of artificial intelligence avoid overtly biological assumptions, it is nonetheless common (though far from universal!) to assume that advanced AI systems will:

Exist as individuals, rather than as systems of coordinated components
Learn from individual experience, rather than from aggregated training data
Develop through self-modification, rather than being constructed and updated
Exist continuously, rather than being instantiated on demand
Pursue world-oriented goals, rather than performing specific tasks

These assumptions have recognizable biological affinities, and they invite further assumptions that are tacitly biomorphic, psychomorphic, and even anthropomorphic.

Potential mind-like systems are situated in a more general space of potential intelligent systems

It has been persuasively argued that rational, mind-like superintelligence is an attractor in the space of potential AI systems, whether by design or inadvertent emergence. A crucial question, however, is the extent of the basin of attraction for mind-like systems within the far more general space of potential AI systems. The discussion above suggests that this basin is far from coextensive with the space of highly-capable AI systems, including systems that can, in aggregate, provide superintelligent-level services across an indefinitely wide range of tasks. We cannot chart the space of potential AI problems and solutions solely within the confines of rational-agent models, because most of that space lies outside.