The Hidden Layer Learns Too (Backprop's "Pulo do Gato")

The last post closed with two ways to set up an MLP's hidden layer: pick the weights by hand, knowing in advance which lines would separate XOR, or draw them randomly and trust that, with enough neurons, some combination would work (ELM). Neither one learns the hidden layer from the error. One is fixed because someone already knew the answer, the other is fixed because it doesn't even try to know. This lecture closes that gap: what if, instead of picking or drawing, the hidden layer could also adjust itself, looking at the error it helped cause? That question has a name, and the professor left it written right in the middle of the code.

Lecture 5a: the last layer learns by gradient, not by pseudo-inverse

Before touching the hidden layer, there's an intermediate step worth taking: keep the same architecture from the last post (two fixed hidden lines, hand-picked) and train only the output layer, this time with a step-by-step update instead of the pseudo-inverse's closed-form solve.

class LastLayer(BaseEstimator, ClassifierMixin):
  def __init__(self, max_iter=100, learning_rate=0.01):
    self.max_iter = max_iter
    self.learning_rate = learning_rate

  def fit(self, X, y):
    self.Xb = include_bias(X)
    self.ihw = np.array([[0.5,-1,1],[0.5,1,-1]]).T
    self.pH = self.Xb @ self.ihw
    self.H = np.tanh(self.pH)
    self.Hb = include_bias(self.H)
    if len(y.shape) == 1:
      y = y.reshape(-1,1)
    self.how = np.random.uniform(-1, 1, size=(self.Hb.shape[1], y.shape[1]))
    for _ in range(self.max_iter):
      logits = self.Hb @ self.how
      y_pred = np.sign(logits)
      error = y - y_pred
      delta = self.Hb.T @ error
      self.how += self.learning_rate * delta
    return self

Look at the first half of fit: ihw is still exactly [[0.5,-1,1],[0.5,1,-1]], the same two hidden lines as always, hand-picked and never touched inside the loop. What changes is the second half: how no longer comes out of a pseudo-inverse solved in one line, it starts random (np.random.uniform) and gets corrected, iteration after iteration, by delta = self.Hb.T @ error. If that shape looks familiar, it's because it's the same delta rule from Adaline, input.T @ error, except the "input" here isn't raw x0/x1 anymore, it's Hb, the hidden layer's output with bias already tacked on. The data this math sees has already gone through the space transformation; the adjustment rule on top of it is the same as always.

Output: accuracy 1.0, output weights [[0.439], [-0.529], [-0.514]].

The result shouldn't be surprising: as I already saw in the last post, once it goes through the two fixed hidden lines, XOR becomes linearly separable on top of H. If a problem is linearly separable, a delta rule running long enough finds a boundary that separates it, no matter whether that boundary got solved all at once (pseudo-inverse) or bit by bit (gradient). The interesting question isn't "does this work?", it's the next one.

Lecture 5b: what if the hidden layer could also be wrong and correct itself?

The notebook reuses that same LastLayer class from above as a starting point (same result, accuracy 1.0, different output weights because how's random draw changes every run), except now it grows two new, separate methods, called from inside fit:

class SingleHiddenLayer(BaseEstimator, ClassifierMixin):
  def __init__(self, n_hidden=2, max_iter=10000, learning_rate=0.005):
    self.max_iter = max_iter
    self.learning_rate = learning_rate
    self.n_hidden = n_hidden
    self.activation = np.tanh

  def forward(self, X):
    self.Xb = include_bias(X)
    self.pH = self.Xb @ self.ihw
    self.H = self.activation(self.pH)
    self.Hb = include_bias(self.H)
    logits = self.Hb @ self.how
    return logits

  def backward(self, y, y_pred):
    ho_delta = y - y_pred
    ho_grad = self.Hb.T @ ho_delta
    ### PULO DO GATO ###
    tanh_grad = (1 - self.H**2)
    ih_delta = ho_delta @ self.how[1:].T * tanh_grad
    ### #### ## #### ###
    ih_grad = self.Xb.T @ ih_delta
    self.ihw += self.learning_rate * ih_grad
    self.how += self.learning_rate * ho_grad
    return self

  def fit(self, X, y):
    self.ihw = np.random.uniform(-1, 1, size=(X.shape[1]+1, self.n_hidden))
    if len(y.shape) == 1:
      y = y.reshape(-1,1)
    self.how = np.random.uniform(-1, 1, size=(self.ihw.shape[1]+1, y.shape[1]))
    for _ in range(self.max_iter):
      logits = self.forward(X)
      self.backward(y, logits)
    return self

Two changes at once compared to LastLayer. First, ihw is also drawn at random (np.random.uniform) inside fit, no longer hard-coded in __init__: nobody picks the hidden lines by hand anymore. Second, and this is the part that matters: inside backward, after computing ho_grad for the output layer (the exact same math as always, Hb.T @ error), there are three more lines, which the professor himself boxed in with a comment worth pausing on.

What these three lines do, without skipping a single one

ho_delta = y - y_pred is the output's error, one number per point, the same difference between truth and prediction as always. The question these three lines answer is: how much of that error does each hidden neuron actually own?

ih_delta = ho_delta @ self.how[1:].T * tanh_grad

Think of each hidden neuron as someone giving an opinion, and the output layer as whoever weighs the two opinions (how[1:], the output weights without the bias) to reach a final call. If the final call was wrong, it makes sense that whichever opinion weighed the most in that decision carries more blame, and whichever barely weighed at all (a weight near zero) walks away nearly innocent. That's exactly the split ho_delta @ self.how[1:].T computes: it takes the output's error and hands it out between the two hidden neurons, in the exact proportion of the weights the output layer gave each one. Notice the neat symmetry: going forward, how takes two hidden numbers and combines them into one output number. Going backward, it runs the trip in reverse, taking one error number and splitting it into two, one for each neuron that helped produce that output. That's why the .T: it's the very same matrix, used the other way around.

But splitting the blame isn't enough on its own. A neuron can only actually change its mind if it still had "room" to move, and that's where tanh_grad = (1 - self.H**2) comes in, tanh's derivative computed straight from the activation H itself (a tanh identity guarantees tanh'(z) = 1 - tanh(z)², so there's no need to keep the pre-activation around separately). When a neuron is saturated, H close to +1 or -1, that derivative sits near zero: no matter how much blame it carries, the adjustment it gets is tiny, because it already made its call with full conviction and nudging its weights now barely moves the output. When H sits near 0, right in the middle, the derivative is at its peak (1): that neuron is still "undecided", and that's exactly where a push makes the biggest difference.

Multiply the two together (split blame × leftover room), and out comes ih_delta, and from there it's Adaline all over again: ih_grad = self.Xb.T @ ih_delta is the same delta rule, input.T @ error, now applied per hidden neuron, using the error that was just computed specifically for it.

There's a name for this in math: multiplying the derivatives of chained functions (the output depends on how, which depends on H, which depends on ihw) to find how much something deep in the chain affects the final result is the chain rule. ho_delta @ self.how[1:].T is the derivative of the output with respect to H; tanh_grad is the derivative of H with respect to the hidden pre-activation; multiplying the two, element by element, is literally applying the chain rule to figure out how much each hidden neuron's pre-activation weighed on the final error. Backpropagation doesn't invent any new math, it just organizes the chain rule in a practical order: compute from the end backwards, reusing each layer's own math to find the one behind it.

This block of three lines, finding the error each hidden neuron owns and adjusting it only as much as the room it had left, is what the professor named right in the code ### PULO DO GATO ### ("the cat's jump", a Brazilian idiom for the crucial trick that makes something click). He's not exaggerating: this is, literally, the entire idea behind backpropagation. Training only the output layer (lecture 5a) is plain delta rule. Training the whole network is the delta rule applied layer by layer, each one using the error the next layer passed back to it. There's no new magic anywhere, just the same math, chained.

Output: accuracy 1.0, output weights [[-1.396], [2.145], [-1.883]].

Watching the pulo do gato happen

Try it yourself. The model below starts with the checkbox unchecked, exactly lecture 5a: the two hidden lines are born already at their hand-picked values, and stay fixed the whole time, only the output layer adjusts on every click. Check the box and train again: now it's lecture 5b, and the two hidden lines also move on every step, pulled by the error the output layer sends back to them.

With the hidden layer frozen, the decision region always starts from the same two parallel lines, only the final boundary's slope between them shifts from click to click. With the box checked, notice that the hidden lines themselves change position while training, sometimes even crossing over, the gradient reaching them through ih_delta is genuinely redrawing the space transformation, not just re-adjusting whatever comes after it. And since the initial draw of ihw/how is random, sometimes training gets stuck at a bad minimum before reaching accuracy 1.0, that's real too: gradient descent doesn't carry the exact-solution guarantee ELM's pseudo-inverse had. If it gets stuck, the reset button draws fresh weights and tries again.

Wrapping up

Practical Application

In the last post, I tested ELM (hidden layer drawn at random, solved by pseudo-inverse) on the two-moons dataset and watched accuracy climb up to 50 neurons and drop again at 100, classic overfitting. Now that I know how to actually train the hidden layer, the obvious question is: does training beat drawing at random? I ran the same SingleHiddenLayer (learning_rate tuned per size, 3000 steps, 10 initialization seeds per size) on the same dataset, same 70/30 train/test split:

Two things jump out. First: with just 1 neuron, backprop already hits 0.88, well above ELM's 0.71, because here that single neuron isn't stuck pointing in a randomly drawn direction, the gradient pushes it toward the direction that actually separates the moons. Second, more subtle: backprop doesn't crash at 100 neurons the way ELM did. ELM's pseudo-inverse always finds the exact solution for the training set, fitting even the noise perfectly, and with a network that's too big that turns into pure overfitting. Gradient descent, running a fixed number of steps, almost never reaches that exact solution, it stops partway there, and stopping partway there, here, acts as a brake against memorizing the training set that the pseudo-inverse's closed-form math doesn't have.

Now train it live, yourself, right in the browser: the model below is the same SingleHiddenLayer, actually running on the 300 points of the two moons (the same ones I generated for the table above, train and test together, since the point here is watching the gradient work, not measuring generalization again):

Click "train more steps" a few times: you can watch the boundary go from a near-random tangle to a curve wrapping around the two moons, with nobody hand-picking those 20 hidden directions or drawing them once and locking them in, the gradient itself found them. The accuracy shown here is on the live training set itself, not directly comparable line-by-line with the table above (which measured on a held-out test set), but the behavior is the same: a handful of clicks already solves most of it, and after a while the curve saturates.