Two Hidden Lines Solve XOR (and Don't Even Need Training)

I'm closing this first big batch of the playlist exactly where it got stuck: the XOR post ended with a single neuron banging its head against the wall, and a hook for whenever the professor published the MLP lecture. There's no 04b in the repository (he jumped straight from 04a to 04c), but the rest showed up, and it pays off the whole promise: first an MLP with hand-picked hidden weights solves XOR, then the same architecture swaps the hidden activation, and finally a third way of setting the hidden weights shows up, no training at all.

The architecture: two hidden questions instead of one

class XORClassifer(BaseEstimator, ClassifierMixin):
  def __init__(self):
    self.ihw = np.array([[0.5, -1, 1], [0.5, 1, -1]]).T
    self.how = np.array([0.5, -1, -1])

  def fit(self, X, y):
    pass

  def predict(self, X):
    self.Xb = include_bias(X)
    self.pH = self.Xb @ self.ihw
    self.H = np.sign(self.pH)
    self.Hb = include_bias(self.H)
    logits = self.Hb @ self.how
    y_pred = np.sign(logits)
    return y_pred

Notice fit: there's just a pass in there. This model doesn't train, the weights already come ready-made in __init__. ihw (input-to-hidden weights) is a (3,2) matrix: each column is one hidden-layer neuron, with its own bias and its own two weights, the same pre-activation as always, except now there are two neurons computing it in parallel, not just one. how (hidden-to-output weights) is the third neuron, which takes the first two's output as if it were two brand-new input variables.

This is the first time in this entire playlist that one output becomes another one's input. That's what separates an MLP (multilayer perceptron) from everything I've seen so far: a hidden layer sitting between input and output, each of its neurons seeing the raw data, but nothing outside it seeing what it computes directly, only the next layer does.

What each hidden neuron is actually asking

The two hidden neurons, written as boundaries: 0.5 - x0 + x1 = 0 (x1 = x0 - 0.5) and 0.5 + x0 - x1 = 0 (x1 = x0 + 0.5). Two parallel lines (same slope, 1), one shifted half a unit above the other. I computed the pre-activation (pH) of both hidden neurons at exactly the four corners XOR uses:

Notice something remarkable: the two false corners land on the exact same hidden point, [1, 1]. Each hidden neuron is, in practice, a yes/no question ("am I above this line?"), and XOR turns out true exactly when the two answers disagree (one yes, one no), and false when they agree (both yes, in this case, since no corner falls below both lines). That's not a dataset coincidence, it's the definition of XOR reappearing one level up: the output layer only needs to separate "the two answers agree" from "they disagree", and that, look at that, is XOR again, just now between two binary answers instead of two continuous variables. The difference is that summing two disagreeing answers always gives 0, and summing two agreeing ones always gives 2 or -2: that is linearly separable (just split on the sum), which the original continuous variable wasn't.

With how = [0.5, -1, -1], the output is 0.5 - H₀ - H₁, that is, 0.5 minus the sum of the two hidden answers. False corners ([1,1], sum 2): 0.5 - 2 = -1.5, negative. True corners (sum 0): 0.5 - 0 = 0.5, positive. Matches the label exactly, with no training at all, because the weights were already chosen knowing this.

Output: accuracy 1.0.

The final boundary: two lines become a band

The blue region is no longer a half-plane cut by one line, it's a diagonal band between the two hidden neurons' parallel lines. That's exactly the shape a single neuron could never draw: a band has two boundaries, a single line only has one. Two linear boundaries, combined, produce a shape neither of them could alone.

Lecture 4d: swapping sign for tanh

class XORClassifer(BaseEstimator, ClassifierMixin):
  def __init__(self):
    self.ihw = np.array([[0.5, -1, 1], [0.5, 1, -1]]).T
    self.how = np.array([0.75, -0.9, -0.9])

  def predict(self, X):
    self.Xb = include_bias(X)
    self.pH = self.Xb @ self.ihw
    self.H = np.tanh(self.pH)
    self.Hb = include_bias(self.H)
    logits = self.Hb @ self.how
    y_pred = np.sign(logits)
    return y_pred

Same ihw, same two hidden lines. The only change is H = np.tanh(self.pH) instead of H = np.sign(self.pH) (and how adjusted, from [0.5,-1,-1] to [0.75,-0.9,-0.9], to compensate for the different scale). As I already saw with Adaline, the difference between pre-activation and post-activation isn't just theoretical: here it decides whether the final boundary has a sharp edge or is smooth.

Output: accuracy 1.0.

Compare with the chart above: same overall diagonal band, but now with rounded edges instead of sharp cuts. tanh saturates smoothly near ±1 instead of jumping straight there, so the transition between "inside the band" and "outside the band" is smooth too.

What the hidden layer is actually doing: it transforms the space

The notebook plots the same point cloud after passing through the two hidden neurons (H, not x0/x1 anymore), and that's where the trick becomes obvious:

In this new space, the two "false" clusters (which were opposite corners in x0/x1) collapse to practically the same spot, and the two "true" clusters split into two clearly distinct groups, one on each side. And look at the boundary: it's a line, exactly the same old linear boundary, just drawn over H instead of x. That's the central idea behind every network with a hidden layer: it doesn't "learn to bend a line", it transforms the space until the problem is linearly separable again, and then solves it with the simplest tool there is.

Lecture 4e: what if the hidden weights were just drawn at random?

class ELM(BaseEstimator, ClassifierMixin):
  def __init__(self, n_hidden=2):
    self.ihw = np.random.randn(3,n_hidden)

  def fit(self, X, y):
    self.Xb = include_bias(X)
    self.pH = self.Xb @ self.ihw
    self.H = np.tanh(self.pH)
    self.Hb = include_bias(self.H)
    self.how = np.linalg.pinv(self.Hb) @ y
    return self

This is an ELM (Extreme Learning Machine). The difference from everything I've seen so far: ihw (input-to-hidden weights) is purely random, np.random.randn, and it never changes, fit never adjusts it at all. Only how (output layer) actually gets computed, and look how: np.linalg.pinv(self.Hb) @ y, the very same pseudo-inverse from the normal equation, back in post 3. ELM doesn't train the whole network, it draws a random space transformation and solves only the last layer, in one shot, no gradient at all.

Notebook output (n_hidden=4): accuracy 1.0, output weights [-216.55, -14.43, -13.51, -6.63, 232.97].

I reproduced this myself, with a seed that also hits 1.0, so I could plot the final boundary (notice the output weights, also quite large, 265.96, -158.78, 3.17, 61.15, -167.19, the same pseudo-inverse behavior I already saw before: it always finds an exact answer, even if that means huge weights):

Nothing like the tidy band above, it's an irregular shape, four hidden neurons pointing in random directions, each slicing the plane a different way. But it separates all four XOR corners just the same, without me (or the professor) hand-picking a single direction.

That raises the obvious question: how many random hidden neurons does it take to have a decent shot at solving XOR? I ran 15 different seeds for each hidden-layer size:

With 1 or 2 neurons, the draw almost never lands two directions that, combined, separate XOR, mean accuracy sits near chance (0.5). From 3 or 4 onward, the odds of at least two random directions landing "well positioned" grow fast, and with 8 they never failed, across the 15 seeds I tried. Sheer number of random directions makes up for the lack of smart choice, the central idea behind the whole family of random-projection methods.

Wrapping up

Practical application

I tested ELM on the two-moons dataset (make_moons, the same one from the XOR post, noise=0.25, the same one already used in the last post), varying the number of hidden neurons, 10 seeds per size, always with a 70/30 train/test split:

With just 1 random neuron, ELM ends up worse than the properly trained perceptron: a single drawn direction, with no adjustment at all, is rarely as good as a direction the algorithm itself optimized. From 5 neurons on, ELM already catches up to the perceptron, and at 20 to 50 it pulls well ahead, several random directions combined manage to bend the boundary around the moon shape, something no single line can do. But at 100 neurons accuracy drops: a hidden layer too big for only 210 training points becomes an easy space to fit exactly, the pseudo-inverse nails the training set "perfectly" at the cost of worse generalization, the same complexity-versus-generalization trade-off that already showed up in the other playlist.

The code that produced this table is the same ELM from above, just now inside a loop varying n_hidden and the seed:

elm = ELM(n_hidden=50)
elm.fit(X_train, y_train)

Taking the best of the 10 seeds for 50 neurons (test accuracy 0.956), plotting the boundary over the 90 test points, never seen during training:

50 random directions combined really do manage to curve around the two moons, nothing like the perceptron's single line or the XOR's tidy two-line band: it's an irregular boundary, full of folds, but one that follows the data's real shape in a way no single line ever could. Comparing with the two-hidden-line mlp-decision-region-chart from above, line by line: it's the same component, the same math (tanh on the hidden pre-activation, sign on the final logit), just with 50 hidden weight columns instead of 2, and those 50 columns came from np.random.randn, not hand-picked numbers.