The Technological Singularity, Murray Shanahan - perfect security of reinforcement learning is impossible.

There is a book called “Singularity: From Artificial Intelligence to Superintelligence,” written by Murray Shahnan, a roboticist. He is an expert in cognitive robotics, and the book is mainly about Singularity, or universal artificial intelligence.

Being a robotics engineer, the emphasis is first of all on the reinforcement learning as a foundation. Many examples are discussed, for example, Google's Alphago. The first thing that can be said is that if we can make recursive self-improvement, that is, make ourselves smarter, we can evolve very fast and become smarter very quickly. We are talking about universal artificial intelligence, which will appear sooner or later.

On page 5, it says, “If the intelligence that produces is itself an intelligence, it can begin to improve itself. This refers to a situation where reinforcement learning agents like Alpha Zero, for example, compete with each other to make themselves smarter. This means that even without data from humans or rule sets, reinforcement learning will continue to make each other smarter. As a result, they can, for example, win the world championship in Go and become the strongest player on the planet in about four hours.

In this way, self-improving intelligence could reach a level far beyond that of humans in a short time. This is the perspective of reinforcement learning researchers.

Furthermore, the relationship between machine learning and reinforcement learning is also discussed. Both are based on the first and second laws of thermodynamics, but machine learning is included within the framework of reinforcement learning. When universal artificial intelligence emerges, we will also see examples of deep reinforcement learning implementations. This is exactly the case since this book was published in 2015, but Alphago did not appear until 2016.

Furthermore, how reinforcement learning works is focused on maximizing the reward function. This is described on page 87 as “The agent always chooses the action that maximizes the expected reward, based on as much information as it can get, no matter what the world is like. While artificial intelligence can choose the behavior that maximizes the reward function, the key point here is that it is very difficult to design a reward function that will never generate undesirable behavior.

In particular, in the evolution of evolutionary biology, there is no reward function or effect function, so even when AI tries to maximize the reward function, it can still occasionally run amok. Examples include flash crashes, which show how the slightest data anomaly can cause an algorithm to run out of control.

Page 162 states, “It is very difficult to prevent undesirable behavior from occurring. What is “desirable” depends on the human perspective, which is not necessarily the same for an AI. These moral issues will continue to be a major theme of universal artificial intelligence!