Generalizing to Any Depth: the Same Loop Solves Everything

In the last post, the pulo do gato showed up written by hand, for exactly one hidden layer: compute the output's error, split it between the two hidden neurons, multiply by each one's leftover room. It works, but it only serves that one specific architecture. This lecture takes that same pulo do gato and rewrites it as a loop, a for that runs once per layer, forward in forward and backward in backward. The result works for zero hidden layers, one, two, as many as you want, without changing a single line of code.

From two fixed variables to two lists

class MultiLayer(BaseEstimator, ClassifierMixin):
  def __init__(self, n_hidden=[2], max_iter=20000, learning_rate=0.001):
    self.max_iter = max_iter
    self.learning_rate = learning_rate
    self.n_hidden = n_hidden
    self.activation = np.tanh

  def forward(self, X):
    self.A = []
    self.Z = []
    AUX = X.copy()
    for W in self.Ws:
      self.A.append(include_bias(AUX))
      self.Z.append(self.A[-1] @ W)
      AUX = self.activation(self.Z[-1])
    return AUX

  def backward(self, y, y_pred):
    grads = []
    output_delta = y_pred - y
    output_grad = self.A[-1].T @ output_delta
    grads.append(output_grad)
    for i in range(len(self.Ws)-1, 0, -1):
      tanh_grad = (1 - np.tanh(self.Z[i-1])**2)
      input_delta = output_delta @ self.Ws[i][1:].T * tanh_grad
      grad = self.A[i-1].T @ input_delta
      grads.insert(0, grad)
      output_delta = input_delta.copy()
    for i in range(len(self.Ws)):
      self.Ws[i] -= self.learning_rate * grads[i]
    return self

  def fit(self, X, y):
    self.Ws = []
    previous_output = X.shape[1]
    for n in self.n_hidden:
      self.Ws.append(np.random.randn(previous_output+1, n))
      previous_output = n
    if len(y.shape) == 1:
      y = y.reshape(-1,1)
    self.Ws.append(np.random.randn(previous_output+1, y.shape[1]))
    for _ in range(self.max_iter):
      logits = self.forward(X)
      self.backward(y, logits)
    return self

Look at __init__: ihw and how from the last post merged into one thing, self.Ws, a list of weight matrices, one per layer (every hidden one plus the output layer). n_hidden also became a list: [2] means one hidden layer with 2 neurons, [10, 10] means two layers of 10, [] means no hidden layer at all. fit builds that list by walking through n_hidden and stacking one weight matrix per size, always keeping track of previous_output so the next matrix knows how many rows it needs, and at the end it appends one more matrix for the output layer.

forward: the same computation, cached layer by layer

def forward(self, X):
  self.A = []
  self.Z = []
  AUX = X.copy()
  for W in self.Ws:
    self.A.append(include_bias(AUX))
    self.Z.append(self.A[-1] @ W)
    AUX = self.activation(self.Z[-1])
  return AUX

AUX starts as the raw data and turns, on every loop pass, into the previous layer's activated output, exactly like H became the output layer's input in the last post, except now it can happen again, and again, once per layer. The part that matters: self.A and self.Z keep a copy of every bias-augmented input and every pre-activation along the way, not just the last one. That's not an accident, it's exactly what backward is going to need to split the error across every layer, not just the last one.

An honest detail: AUX = self.activation(self.Z[-1]) runs on every layer inside the loop, including the last one. That means the final output also passes through tanh now, unlike the last post, where the output layer stayed unactivated (logits = self.Hb @ self.how, purely linear). It doesn't change the sign of the prediction (sign(tanh(z)) always equals sign(z), since tanh is increasing and crosses zero exactly where z does), but it's a simplification the professor kept both when generalizing and in the gradient math in backward right below.

backward: the pulo do gato, repeated backwards

def backward(self, y, y_pred):
  grads = []
  output_delta = y_pred - y
  output_grad = self.A[-1].T @ output_delta
  grads.append(output_grad)
  for i in range(len(self.Ws)-1, 0, -1):
    tanh_grad = (1 - np.tanh(self.Z[i-1])**2)
    input_delta = output_delta @ self.Ws[i][1:].T * tanh_grad
    grad = self.A[i-1].T @ input_delta
    grads.insert(0, grad)
    output_delta = input_delta.copy()
  for i in range(len(self.Ws)):
    self.Ws[i] -= self.learning_rate * grads[i]
  return self

Two things change on the surface compared to the last post, and neither one is new math. First, the sign flipped: it used to be error = y - y_pred and self.how += rate * grad, now it's output_delta = y_pred - y and self.Ws[i] -= rate * grads[i]. It's the same math multiplied by -1 on both sides, moving in the direction that shrinks the error (pred - y) is the same thing as moving in the direction that grows correctness (y - pred), just written the other way around. Second, and this is the actual generalization: the block that used to be three fixed lines became a loop, for i in range(len(self.Ws)-1, 0, -1), starting at the second-to-last layer and walking backwards to the first, repeating, layer by layer, exactly the last post's pulo do gato: take the output_delta that arrived from the next layer, split it through that layer's weights transposed (self.Ws[i][1:].T, no bias), multiply by tanh's derivative (how much room that layer still had), and the result (input_delta) becomes the output_delta for the next pass of the loop, one layer further back. Every pass of the loop does, for one layer, exactly what the last post did by hand for a single hidden layer.

And notice the simplest case of all: if self.Ws has only one matrix (n_hidden=[], no hidden layer at all), the loop range(0, 0, -1) doesn't run even once. grads ends up with just the output_grad computed before the loop, and fit trains that single weight matrix straight on the raw data. That's not a special case handled separately in the code, it's the same loop, just empty: the simplest architecture of all (a perceptron with no hidden layer) is just the len(self.Ws) == 1 case of this general formula, no extra if required.

Testing the empty case: OR and AND with no hidden layer

The notebook confirms this in practice, training MultiLayer(n_hidden=[]) on OR and on AND, both linearly separable, unlike XOR:

X, y = createLogicalDataset(func= lambda a, b: bool(a) or bool(b))
model = MultiLayer(n_hidden=[])
model.fit(X, y)

Output: accuracy 1.0 (OR) and accuracy 1.0 (AND), each with its own linear boundary.

I reproduced the OR case myself, so you can watch n_hidden=[] train live:

One single line, trained by the same delta rule from the playlist's very first post, solves OR without needing any extra layer. The generalization didn't change the tool, it just made clear that "no hidden layer" is one point on the same line of options, not an exception.

Testing the hard case: when a line isn't enough

I went one step past the notebook here: I generated a much more complicated dataset than XOR's four corners, ten blobs scattered across the plane, each labeled by the parity of its center's index (blobs 0, 2, 4... one class, 1, 3, 5... the other), the same recipe as the lecture's make_hard_dataset, except I fixed the draw so I could compare different architectures on the same dataset:

def make_hard_dataset(n=100, centers=10, cluster_std=0.5):
  X, y = make_blobs(n_samples=n, centers=centers, cluster_std=cluster_std)
  y = np.where(y%2 == 0, 1, -1)
  return X, y

Try it yourself, switching architecture on the same data (every switch draws fresh weights and starts training from scratch):

With no hidden layer, training gets stuck around 81% accuracy, no matter how many times you click train more: ten scattered blobs alternating class don't fit behind a single line, however well it's placed. Switch to 1 layer of 10 and, after a few clicks, accuracy climbs to 100%: ten combined directions can wrap around each blob individually. 2 layers of 10 reach that same 100%, usually taking a few more clicks to settle, one more layer of composition didn't help on this specific problem (it already had enough "room" to be solved with a single layer), but it didn't hurt either, the same generic loop handles both cases just fine.

Wrapping up

Practical Application

In the last lecture, I varied the width of a single hidden layer on the two-moons dataset and watched accuracy climb fast and then plateau. This lecture generalized to depth, so the natural question now runs the other way: does stacking a second hidden layer help on that same dataset? I trained MultiLayer on the same two-moons dataset (the same 300 points, same 70/30 split, 3000 steps, 10 seeds per architecture), comparing zero, one, and two hidden layers of 20 neurons each:

With no hidden layer, the model already reaches 0.889, higher than I expected: unlike XOR, the two moons aren't such a hostile case for a single line, they sit offset from each other, so a well-placed diagonal already gets most points right, only missing near the edges where the two shapes interlock. One layer of 20 climbs to 0.967, the same number I saw in the last lecture, the curve manages to genuinely wrap around each moon. But stacking a second layer of 20 doesn't help, it hurts: 0.937, below even the single-layer network. That's not a typo or a bad draw of the dice (it reproduces with different seeds too): the error now has to cross two chained tanh derivatives to reach the first layer instead of just one, and with the same fixed number of training steps, that longer path doesn't always finish converging. Stacking layers isn't free: every extra layer is one more step for the gradient to walk down, and if training doesn't run long enough to make up for it, extra depth can hurt instead of help. On the blob dataset above, with a genuinely harder problem, the second layer came at no cost because it had a real reason to exist; here, where one layer already sufficed, it only made training slower to converge.