Notes on Neural Networkds

Derivatives

It is important to understand derivatives, when talking about neural networks. The chain rule is particularly important. This is a useful reminder: https://www.mathcentre.ac.uk/resources/uploaded/mc-ty-chain-2009-1.pdf

Neurons

There is a very useful video by 3Blue1Brown: https://youtu.be/tIeHLnjs5U8?si=s949GBePEVvWQt_x

If we use the following notation:

We would optimize parameters of a neuron at the layer L using the following derrivative:

The derivative for the previous layer looks very similar:

Backpropagation

We can use calculations from previous layers to optimize parameters of the current layer. One useful video on this is from Mikael Laine: https://youtu.be/8d6jf7s6_Qs?si=AzgXwRXL7okh5z4O

Experimentation

With neural networks we can build logic operations. Let's build the XOR operation:

We can first build the operations: and, or and not-and, and then put them together:

X1 OR X2

X1 AND X2

NOT (X1 AND X2)

Combining these parameters together:

The diagram looks as follows:

The result using the optimization is slightly different, but produces very similar values.

More links:

https://blog.abhranil.net/2015/03/03/training-neural-networks-with-genetic-algorithms/