Will robots outsmart us? by the late Stephen Hawkins

There is a interesting article, "Will robots outsmart us?" in today's Sunday Times Magazine. While I don't accept all Stephen's predictions I was most interested to read:

When an artificial intelligence (AI) becomes better than humans at AI design, so that it can recursively improve itself without human help, we may face an intelligence explosion that ultimately results in machines whose intelligence exceeds ours by more than ours exceeds that of snails. When that happens, we will need to ensure that the computers have goals aligned with ours.

Later he says:

In short, the advent of super-intelligent AI would be either the best or the worst thing ever to happen to humanity. The real risk with AI isn’t malice, but competence. A super-intelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours we’re in trouble. You’re probably not an evil ant-hater who steps on ants out of malice, but if you’re in charge of a hydroelectric green-energy project and there’s an anthill in the region to be flooded, too bad for the ants. Let’s not place humanity in the position of those ants. 

Of course we know what happened last time a super-intelligence came into existence. About half a million years ago the Earth was populated by a great variety of animals of a comparatively low intelligence. All the higher animals had brains that worked in roughly the same way, and how much they could learn was limited because everything they learnt was lost when they died. Then one species, which we call Homo sapiens, discovered a way to recursively increase its own intelligence. It was already good at making tools but for several million years the cost of trail and error learning had limited what it could do. But then it invented a tool to boost intelligence, which we call language.  Language not only made it possible to make better tools, but also it made it possible to recursively build a better language generation by generation. So some 5000 generations later the Earth is home to a super-intelligent species ...

And are the goals of this species aligned with the the goals of the millions of other species? Of course not. Billions of animals are kept as slaves to be killed and eaten, while the homes of countless more have been, or are being, destroyed. 

If we invent a super-intelligent AI system why should it treat us with more respect than we have  shown for our animal relatives.

A new book "Brief Answers to the Big Questions," by Stephen Hawkins is published later this week

For the background to my observation see 

An Evolutionary Model of Human Intelligence