Algorithmic Intelligence Quotient

I covered the Shane Legg talk on Algorithmic Intelligence Quotient back in 2010. I am revisiting it with a video of the talk and a paper on it. It is important because Shane Legg works in Deepmind and it is how they guided improvement in their algorithms.

Measuring universal intelligence: Towards an anytime intelligence test”

This was the development the idea of a universal anytime intelligence test. The meaning of the terms “universal” and “anytime” is manifold here: the test should be able to measure the intelligence of any biological or artificial system that exists at this time or in the future. It should also be able to evaluate both inept and brilliant systems (any intelligence level) as well as very slow to very fast systems (any time scale). Also, the test may be interrupted at any time, producing an approximation to the intelligence score, in such a way that the more time is left for the test, the better the assessment will be. In order to do this, our test proposal is based on previous works on the measurement of machine intelligence based on Kolmogorov complexity and universal distributions, which were developed in the late 1990s (C-tests and compression-enhanced Turing tests). It is also based on the more recent idea of measuring intelligence through dynamic/interactive tests held against a universal distribution of environments. We discuss some of these tests and highlight their limitations since we want to construct a test that is both general and practical. Consequently, we introduce many new ideas that develop early “compression tests” and the more recent definition of “universal intelligence” in order to design new “universal intelligence tests”, where a feasible implementation has been a design requirement. One of these tests is the “anytime intelligence test”, which adapts to the examinee’s level of intelligence in order to obtain an intelligence score within a limited time.

Intelligence is a key concept in the quest for artificial intelligence, and more generally the singularity. Remarkably, relatively little work has gone into developing general, encompassing and theoretically founded definitions of intelligence for machines. This leaves us without a clear foundation for either theoretical research or developing empirical measures of progress. In this talk I outline the major perspectives on the nature of intelligence and some of the informal definitions that have been put forward. I then sketch the main ideas behind the universal intelligence measure. This is a formal definition of intelligence based on Hutter’s AIXI model of theoretically optimal machine intelligence. Based on this a number of researchers, including myself, are developing practical tests of machine intelligence. I describe some of the challenges faced when doing this and share some recent results from testing various artificial agents. Might we soon be able to measure our progress towards machine intelligence?

Here is a 2007 paper by Shane Legg on Universal Intelligence (47 pages)

He is reviewing the computations per second from the best supercomputers by year. He notes that it is slightly curving upward on a log graph. It is projected that we have exaflops around 2018 and 100 exaflops by 2025.

But he wants to see machine intelligence against the year.

Will you use a human model of intelligence or a more ideal or normative model of intelligence

Does it have to have thought, emotions, etc…
Or is it just external behaviors (quack and walk like a duck)


                     Human                         Ideal

Internal                niel block                Aristotal 
Properties



External            Turing                Will look                        
Properties                                here


Theese are different questions not just different answers.
Look at ideal external because if it cures cancer then don’t care how it does it.

But if you cared about uploading then the Internal human measure would matter.

Work well in many different environments
cluster of cognitive abilities that enable success in different areas
be able to adapt and meet goals.

Intelligence is
* property of an agent
* that interacts with its environment
* so as to successfully achieve goals
* across a wide range of environments
* Occam’s razor
formally defines with an equation. Summation across environments with occam razar multiple of success measure …

formally defined, captures essence of many informal definitions, orders imple agents correctly, upper limit is AIXI, continuously measure, non-anthropocentric

Algorithmic Intelligence Quotient (AIQ)

Testing against algorithms is placing them in the expected and correct order.
Q lambda and func approx result AIQ 477
Q lambda result AIQ 405
Q(0) result AIQ 303
Freq agent result AIQ 277
random result AIQ 0