Highly Trainable Neural Network Configuration

What is this patent about?

’320 is related to the field of computer-based neural networks, specifically addressing the challenge of training very deep networks. Traditional deep neural networks, while theoretically powerful, suffer from optimization difficulties as the number of layers increases. This is due to issues like vanishing gradients, where the error signal diminishes as it propagates back through the network during training, hindering the learning process, especially in early layers.

The underlying idea behind ’320 is to introduce a gating mechanism within each neuron that controls the flow of information. Instead of always applying a non-linear transformation to the input signal, the neuron can selectively allow the original input signal to pass through, unmodified, or pass a transformed version of the input, or a mix of both. This is achieved using transform and carry gates , which determine the weighting between the transformed and non-transformed components.

The claims of ’320 focus on a computer-based method, a neural network architecture, and a computer-readable medium implementing this architecture. The core element is a neuron that receives an input signal, applies a first non-linear transform to produce a 'plain' signal, and then uses two additional non-linear transforms (gates) to produce a 'transform' signal and a 'carry' signal. The neuron then calculates a weighted sum of the original input signal and the 'plain' signal, where the weights are determined by the 'transform' and 'carry' signals.

In practice, this architecture allows for the creation of 'information highways' through the network. By initializing the network to favor the 'carry' behavior, the original input signal can propagate through many layers without significant attenuation. This makes it easier to train very deep networks because the error signal can propagate back more effectively, even through many layers. The transform gate learns to regulate information flow, allowing the network to dynamically choose between transforming the input or simply passing it through.

This approach differs from prior solutions that rely on careful initialization schemes or complex training techniques to overcome the optimization challenges of deep networks. By introducing the gating mechanism, ’320 provides a more robust and flexible way to train networks with virtually arbitrary depth, using standard Stochastic Gradient Descent (SGD) with momentum. This allows for the exploration of deeper architectures and potentially more complex problem-solving capabilities without the limitations imposed by traditional training difficulties.