1.2 Processing Power

Understanding how Neural Networks work

An extremely interesting feature that provided some insight into how Neural Networks actually work inside is that when it was trained on some binary information that was originally designed for the configuration of a high performance Application Specific Integrated Circuit (ASIC), as expected, the Neural Network would eventually learn all the relationships between the binary Input information and desired binary Output information of the ASIC.

Now, when the Neural Network and all the associated weighted connections were very thoroughly analyzed, it became apparent the Neural Network had automatically built what is known in ASIC electronics design as a Karnaugh Map (K-Map) for the entire digital circuit [24]. This K-Map is something that defines all the boolean algebra and logic gates needed to design the connections of all the digital circuitry of an ASIC. The Neural Network figured out how to build this K-Map entirely by itself. Until this time, K-Maps had always required Human Intelligence to be built, often combined with very advanced ASIC design tools for very large and complex ASICs. Therefore, Neural Networks are unbelievably Intelligent at building K-Maps.

So even though many people say Neural Networks are “black boxes that cannot be understood", this is not entirely true if you are able to deeply analyze sections of them. Notably however, as Neural Networks have become increasingly larger and learned more complex relationships between sets of information, it has become exponentially more difficult to understand the intricacies of the circuit designs a Neural Network has built by itself as it learns. It's reasonable to say the Neural Networks today are so incredibly large, both wide and deep, they cannot be understood by Humans. Simply, Humans don't have the Intelligence this requires.

Notably, the K-Map is a purely binary construction with only 0’s and 1’s, whereas Neural Networks also often operate over a continuous analog range, usually from 0 through to 1, and so they can form significantly more complex multivariate non-linear multidimensional algebraic [28] functions.

As part of this particular degree, the class was also learning analog and digital control theory, analog and digital signal processing (DSP), and what felt like every kind of mathematical and multi-dimensional transform ever devised... all these years later there’s still some traumatic Brain injury. However, these theories were really useful for creatively designing different kinds of Neural Networks using multilayer feedback loops, which today are used in some Recurrent Neural Networks (RNNs). These were very hard to reliably train.

Overall, it turns out these Neural Networks were really quite a simple and very different type of computing system that did work in a massively parallel way. It became immediately apparent that it is possible to build a Computer that learns, almost by itself, by building multiple layers of these neural network components and creating different kinds of looped-feedback connections between the deeper and earlier layers. 

Neural Networks are computationally intense

The backpropagation of errors with gradient descent was really just an error correction and optimization algorithm that was very computationally expensive, and actually not very good at global optimisation. Getting the Neural Network error to below 1% could sometimes be really difficult because of what is called local minima. 

Local minima are something conceptually akin to small valleys across a vast mountain range (built in mathematics), and you are trying to find the lowest possible point in the lowest of all valleys within the entire mountain range (the lowest possible mathematical error), but you are only able to see what's around you from where you are standing (your current error value), so a lot of your view can be obscured by various mountains of different sizes around you. Depending on where you happen to be standing at the time (your current error value), you can't always see the lowest point, so you don't know which direction to head (mathematically) to get to the absolute lowest possible point in the entire mountain range. 

Local minima are arguably one of the biggest problems with Neural Networks. The problem is generically called Neural Network optimization, and backpropagation is one approach, but it gets stuck in local minima. This can be EXTREMELY annoying.

Today, there are other optimisation algorithms and Neural Network designs that improve these issues by orders of magnitude, but it seems most Neural Networks today still use some variant of the good’ol backpropagation gradient descent algorithm, which is really quite sloooooow, and computationally intensive.

Frustratingly, over lots of experiments, no matter how many different Neural Network designs were developed, there's always this one big damn problem! Through years of ongoing experimentation with Neural Networks it became very obvious that there is a definite Hard Limit on AI that is essentially defined by just one thing: Computer processing power.

There was just not enough computing power to do the things that looked really interesting, like getting a Neural Network to learn to model the entire world’s stock exchange system using the last 5 or so years of stock price back-test information, and then predict future stock prices at any exchange [23] in realtime. Well, you can guess why that's interesting. :)  After all, the Neural Network only needs to be a bit more right than wrong most of the time, and it is generally performing statistically better than potentially every investment fund manager and day trader in the world.

It became necessary to begin exploring a range Neural Network types and early parallel computing technologies such as Vector Processors, Transputers, and various Harvard Architecture processors, clustered Linux machines, through to various generations of GPUs made by NVIDIA. Despite the fact these were all very computationally powerful, and expensive at useful scale, even these have Hard Limits on Computer processor power. 

Until the introduction of datacentre server infrastructure, offered as ‘cloud computing’ from around 2010, this Hard Limit has been really damn annoying. ie. basically unworkable for really serious AI applications with Neural Networks.

Genetics is everything

Immediately after graduating with an Applied Science degree in Digital Technology majoring in AI, the first job out of the gate was with a large multinational software company called Logica, headquartered in the UK. Before even officially starting with Logica, the company provided the opportunity to go to the Logica Cambridge laboratories where all of the company's pioneering AI work was being researched and developed. It was very surprising to learn that many of the largest banks in the world used Logica software for nearly everything they did. Logica already had early generation AI systems in place at major global banks for credit card fraud detection using an early AI technology called Expert Systems, which is just a special database with thousands of conditional Human rules turned into software code. In continuing its commercial AI software R&D, Logica Cambridge had started building more advanced Neural Networks for some enterprise grade commercial software applications.

The subject of Neural Network error optimization for faster training became the key point of discussion during the time at Logica Cambridge, where eventually the head of the AI lab handed over a book and simply said: "Read this and see what you can do with it, and then let's talk further". The book was titled: 'Genetics Algorithms in Search, Optimisation and Machine Learning - combinatorial optimization, algorithms, machine learning' written by David E. Goldberg [7].

Well after reading and converting the strange new ideas contained in the book into software, it was staggering to realize that genetic algorithms are extremely powerful at solving seemingly intractable problems. It seems that life uses an absolutely ingenious system of countless numbers of variable genes to encode and distribute the programmed information necessary for intelligent life to survive and grow. This genetic algorithm is shockingly brilliant and totally terrifying, at the same time.

Intelligence automatically develops from a special type of program contained within and across all of genetics.

The importance of this point and how it relates to Hyperselfish Intelligence and AI will be become clearer as you read further.

The Hard Limit of Computer Information Processing 

Clearly, there have been some stunning breakthroughs in Neural Network designs and learning algorithms, particularly in the past 10 years, but quite seriously, these are all basically variations built from the same foundational ideas going back more than 50 years. AI researchers are building idea upon idea, but are really creating variations on a theme and progressive fractional improvements, with occasional small but very important discontinuous jumps. So, apologies it this offends some heavily dedicated Neural Network researchers and cognitive scientists, etc.

Fundamentally, it is only through raw increases in Computer processing power that AI has really been able to learn the relationships within larger sets of information and truly become incrementally better over the years. That’s it. Again, sorry.

Now if you know anything about something called “Moore’s Law”, taken from the late genius and Intel chip designer Gordon Moore, you’ll know that the power of Computer information processing has been doubling roughly every 1.5 - 2 years for around the same cost. Give or take. Although in recent years this rate of doubling has been slowing.

So in relation to AI’s Capabilities, this leads to the following obvious logical outcome: 

The Hard Limit of AI Capabilities is actually rising with time as the power of Computer information processing continually increases.