Neural Networks
Second playlist with Dr. Francisco Boldt (the first was Pattern Recognition), now in his Neural Networks course. Same style as always: code by hand, live, in class, no pre-baked formula slides. If you've already read the previous playlist, you know exactly what to expect.
The reference book changes: here I use Neural Networks and Deep Learning: A Textbook, by Charu Aggarwal (2018), to play the role Bishop played in the previous playlist, filling in the foundation the notebook only shows in code. A lot of the material also connects straight back to what I've already covered: the delta rule and linear regression already showed up in Pattern Recognition, and loss functions already showed up in Andrew Ng's specialization, so whenever it fits I'll pull those threads instead of reteaching from scratch.
The notebooks come from the course repository, pablobelmiro/aulasann, a fork of Dr. Boldt's own repository, fboldt/aulasann, where he publishes each lecture's code. Same convention as the previous playlist: whoever "wrote" the code is always him, the professor; the foundational explanation is mine.
One important detail this time: this course is being taught right now, live, and the repository only has the beginning of the course as of this moment (perceptron, Adaline, cost functions, multiclass, the XOR limit, the MLP that finally solves that limit (first with hand-picked hidden weights, then randomly drawn ones via ELM), and finally the hidden layer actually learning through backpropagation, generalized to any depth). Unlike the Pattern Recognition playlist, which I only started once the whole course had already ended, this one is a living playlist: it grows every time the professor publishes a new lecture, and I come back to keep going. We start at the very beginning: the simplest neuron there is.
Playlists · 11 posts
- Perceptron: the First Neuron That Learns on Its Own
Lecture 1a: the professor implements the Perceptron Learning Algorithm from scratch, no bias yet. I tell the story of two papers that came before it and find a real bug hiding in the algorithm's stopping condition.
- Bias and Vectorization: Unlocking the Perceptron
Lectures 2a and 2b: the professor adds bias to the perceptron (solving exactly the limit that closed last lecture) and then vectorizes the whole weight update, swapping the point-by-point loop for a single computation over the entire dataset.
- From Classification to Regression, and the One-Line Solution
Lectures 2c and 2d: the professor swaps the binary sign for a continuous line (linear regression) and then replaces the whole gradient descent loop with a single closed-form computation, the normal equation via pseudo-inverse.
- Adaline: Train on the Line, Classify on the Sign
Lectures 2e and 2f: the professor trains Adaline by fitting a continuous line to the ±1 labels (instead of fitting directly on the sign), and I connect this to the delta rule that already showed up in the other playlist. In the end, a deliberately biased dataset reveals the problem was never the training algorithm.
- Swapping the Cost Function Like Changing Clothes
Lectures 3a and 3b: the professor generalizes training to accept any pluggable cost function, and each choice (Widrow-Hoff, perceptron criterion, log-likelihood, hinge) recovers a different algorithm from this playlist. Along the way, I found a real mismatch between activation function and label encoding.
- Multiclass: When a Weight Becomes a Matrix
Lecture 3c: reusing the binary classifier directly on a 4-class problem lands exactly at random-guess accuracy, 25%. The fix is generalizing the weight from a vector to a matrix, one set of weights per class, and swapping sign() for argmax.
- XOR: Where a Single Neuron Just Hits a Wall
Lecture 3d: the same perceptron that learns OR and AND effortlessly gets stuck on XOR, pinned at 50% accuracy, weights collapsed to nearly zero. I close this batch of the playlist right at the threshold before the MLP, the cliffhanger the next post already resolves.
- The Sigmoid Got Fixed, the Hinge Lost Its Margin
Lecture 4a: the professor comes back and fixes the exact sigmoid-versus-±1-label mismatch I found last lecture. But checking it line by line, the hinge loss fix hid a new problem: it accidentally turned back into the perceptron criterion.
- Two Hidden Lines Solve XOR (and Don't Even Need Training)
Lectures 4c, 4d, and 4e: the hook I left hanging in the XOR post finally pays off. A real MLP solves the problem, first with hand-picked hidden weights, then with randomly drawn hidden weights (ELM), the output layer always solved by the same normal equation from post 3.
- The Hidden Layer Learns Too (Backprop's "Pulo do Gato")
Lectures 5a and 5b: after seeing the hidden layer drawn at random (ELM) or hand-picked, it's finally its turn to actually learn. The final error gets split up among the hidden neurons and flows back to adjust each one. The professor himself names this mechanism, right there in the code, the pulo do gato.
- Generalizing to Any Depth: the Same Loop Solves Everything
Lecture 5c: the last post's pulo do gato only handled a single hidden layer. Here it becomes a loop that works for zero, one, two, or any number of layers, and a much harder dataset shows why, sometimes, a single layer just isn't enough.