Returns on Cognitive Reinvestment for Scaling Machine Intelligence

Eliezer Yudkowsky has looked at the returns on investment (91 pages, Intelligence Explosion Microeconomics) for a sufficiently advance machine intelligence could build a smarter version of itself, which could in turn build an even smarter version of machine intelligence.

Eliezer identifies the key issue as returns on cognitive reinvestment—the ability to invest more computing power, faster computers, or improved cognitive algorithms to yield cognitive labor which produces larger brains, faster brains, or better mind designs. There are many phenomena in the world which have been argued to be evidentially relevant to this question, from the observed course of hominid evolution, to Moore’s Law, to the competence over time of machine chess-playing systems, and many more. He go into some depth on some debates which then arise on how to interpret such evidence. He proposes that the next step in analyzing positions on the intelligence explosion would be to formalize return-on-investment curves, so that each stance can formally state which possible microfoundations they hold to be falsified by historical observations.

Eliezer uses the analogy of neutron multipliers in a nuclear bomb with an intelligence multiplier.

The work has other interesting analogies, historical case studies and interesting questions and definitions.

Eliezer’s Intelligence Definition

Eliezer usually use as notion of “intelligence == efficient cross-domain optimization,” constructed as follows:

1. Consider optimization power as the ability to steer the future into regions of possibility ranked high in a preference ordering. For instance, Deep Blue has the power to steer a chessboard’s future into a subspace of possibility which it labels as “winning,” despite attempts by Garry Kasparov to steer the future elsewhere. Natural selection can produce organisms much more able to replicate themselves than the “typical” organism that would be constructed by a randomized DNA string—evolution produces DNA strings that rank unusually high in fitness within the space of all DNA strings.

2. Human cognition is distinct from bee cognition or beaver cognition in that human cognition is significantly more generally applicable across domains: bees build hives and beavers build dams, but a human engineer looks over both and then designs a dam with a honeycomb structure. This is also what separates Deep Blue, which only played chess, from humans who can operate across many different domains and learn new fields.

3. Human engineering is distinct from natural selection, which is also a powerful cross-domain consequentialist optimizer, in that human engineering is faster and more computationally efficient. (For example, because humans can abstract over the search space, but that is a hypothesis about human intelligence, not part of my definition.)

In combination, these yield a definition of “intelligence efficient cross-domain optimization.

Intelligence Explosion Microeconomics: An Open Problem
Eliezer proposes a project of intelligence explosion microeconomics that can be summarized as
follows:

Formalize stances on the intelligence explosion in terms of microfoundational growth curves and their interaction, make explicit how past observations allegedly constrain those possibilities, and formally predict future outcomes based on such updates.

This only reflects one particular idea about methodology, and more generally the open problem could be posed thus:

Systematically answer the question “What do we think we know and how do we think we know it?” with respect to growth rates of cognitive reinvestment.

If you liked this article, please give it a quick review on ycombinator or StumbleUpon. Thanks