Adaline: Train on the Line, Classify on the Sign

Lectures 2e and 2f, and the model's name changes to Adaline (ADAptive LInear NEuron), by Bernard Widrow and Ted Hoff, 1960, just two years after Rosenblatt. Their core idea is subtle, but it splits apart two things I'd been treating as one so far: what the model optimizes during training and what it computes at prediction time.

Pre-activation and post-activation: the distinction Adaline introduces

class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.linalg.pinv(X) @ y
    return self

  def predict(self, X):
    X = include_bias(X)
    return X @ self.w_

Notice: this is exactly last post's normal equation, without changing a line, except now y is -1 or +1 instead of continuous. The professor is treating classification as if it were regression: fitting the line to land as close as possible to -1 on one class's points and +1 on the other's, never applying sign().

Output: RMSE 0.4148, weights [-1.76, 1.01, 2.74].

RMSE makes sense here because predict returns a continuous number, not a class. But to actually classify, there's one last step missing:

class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.linalg.pinv(X) @ y
    return self

  def pre_activation(self, X):
    X = include_bias(X)
    return X @ self.w_

  def predict(self, X):
    return np.sign(self.pre_activation(X))

Now there are two methods: pre_activation (the continuous value, before any threshold) and predict (applies sign() on top). Aggarwal calls exactly these two things the pre-activation value and the post-activation value (chapter 1): everything a neuron computes happens in two steps, first the weighted sum, then the activation function on top of it. Adaline trains on the pre-activation (it's continuous, so you can measure "how far off" each prediction landed from the target) and only applies the activation (sign) when deciding the final class.

Output: accuracy 1.0, the same [-1.76, 1.01, 2.74] weights as before (it's the same computation, just evaluated by accuracy instead of RMSE this time).

This is the delta rule, and I've already seen it before, under that exact name: "update the weight proportionally to the error times the input" is Widrow-Hoff's signature. The difference from Rosenblatt's perceptron (which I covered two posts ago) is exactly this: the perceptron measures error after applying sign() (error in {-2,0,+2}), Adaline measures it before, on the continuous pre-activation. That sounds like a small detail, but it changes everything: a continuous error gives a real, smooth gradient that points toward the better direction even when a prediction is already on the right side but still a bit "unsure". The perceptron's binary error, by contrast, only fires on an outright misclassification, with no notion of "how wrong."

Interactive: nudging the pre-activation and watching RMSE (and accuracy) move

Instead of training automatically, drag the w0, w1, and bias sliders by hand and watch two readouts at once: RMSE (continuous, changes smoothly with every drag) and accuracy (discrete, only jumps when a point crosses the decision boundary).

Notice how RMSE almost always keeps changing a little even after accuracy already hit 100%: you can push the boundary further into the empty gap between the two classes (RMSE drops more) without gaining or losing a single point (accuracy stays put). That's precisely the difference between "finding some line that separates" (what the perceptron does) and "finding the line that separates with room to spare" (what you get by optimizing RMSE instead of just counting mistakes).

Lecture 2f: the same computation, just iterating (and the notebook calls it "SGD")

class Adaline(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.zeros(X.shape[1])
    for _ in range(self.max_iter):
      y_pred = X @ self.w_
      error = y - y_pred
      self.w_ += self.learning_rate * error @ X
    return self

One honest note about the notebook's name (aula02f adaline with SGD): the code shown here is batch gradient descent (the same X.T @ error as always), computing the error over the entire dataset every iteration, not real SGD (which would update on one example at a time, in shuffled order). It's a common informal way of talking ("it's kind of like gradient descent, so I call it SGD"), but the technical difference is worth noting, since the names carry precise meaning.

Output: accuracy 1.0 on training, weights [-2.92, 3.32, 2.63]. Tested on 1000 new points: accuracy 0.953.

Once again the notebook only keeps the image from this test, not the points. Same trick as the last post: I took the real trained weights [3.32, 2.63], bias -2.92, and drew my own batch of 75 new points from the same generator rule, just to visualize where the learned boundary gets it right and where it doesn't:

A clean boundary, decent margin, only a few stray misses near the x + y = 1 line. Keep this image in mind, it's the "before" in the comparison coming up next.

The "bad" dataset: what it actually proves

X_bad = np.concatenate((X,np.ones_like(X)))
y_bad = np.concatenate((y,np.ones_like(y)))
X_bad = np.concatenate((X_bad,np.ones_like(X)))
y_bad = np.concatenate((y_bad,np.ones_like(y)))
clf_bad = Adaline()
clf_bad.fit(X_bad, y_bad)

The professor concatenates the original dataset with two extra blocks of artificial points: everyone at (1,1), everyone labeled +1. This isn't noise, it's a deliberate bias, nudging training to "believe" the region near (1,1) is even more strongly class +1 than it really is.

Output: accuracy 0.967 on training (over the biased dataset), but only 0.811 on the same 1000 clean test points as before.

The "after" of the comparison: same batch of 75 test points from above, same generator rule, now under the boundary the biased dataset taught (w = [1.94, 0.59], bias -1.48):

Compare the two charts: the blue region (class +1) grows, pushed by the 40 phantom points training saw glued to (1,1). The boundary rotated to "protect" a corner that, in the real world, doesn't deserve that much confidence, and that costs genuinely -1 points that now fall on the wrong side.

Dropped from 0.953 to 0.811. Before writing this post, my working hypothesis was that this cell would show the normal equation (pseudo-inverse) breaking on this problematic dataset, with SGD holding up better. I reproduced the experiment myself, comparing the pseudo-inverse against batch gradient descent on the same biased dataset, and that hypothesis didn't hold up:

Both drop by the exact same amount, with essentially identical weights between them. The real lesson of this cell isn't about which training algorithm is more robust, it's about training data quality: biasing the training distribution (even without adding "noise" in the random-error sense) shifts the learned boundary somewhere that no longer represents the real distribution, and it hurts both methods equally, because both are solving the exact same optimization problem under the hood. This echoes a lesson I already saw in the other playlist: training on a distribution that doesn't match the real world is a data problem, not an algorithm problem.

Wrapping up

Practical application

I reproduced pseudo-inverse vs. batch gradient descent on Iris (setosa vs. versicolor), by now a familiar face in this playlist, but this time training as Adaline, on the continuous ±1 target, not as a perceptron.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
w_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train

On the first attempt, I used the notebook's original learning_rate=0.01 for batch gradient descent, and it diverged (weight became NaN) straight away on raw Iris data. No surprise at this point: it's the same scale lesson from the normal equation post, just rediscovered again, this time needing a much smaller rate (0.001) to not blow up.

With the rate adjusted, same result from both methods again, RMSE identical to the fourth decimal, and both accuracies hitting 100%, the same generous margin from Iris that already favored the perceptron with bias. The real difference between the methods, on this easy dataset, remains just convergence speed (and sensitivity to the choice of learning_rate), not the quality of the final solution.

Comparing with the AdalinePseudoInverse/Adaline from above, line by line: include_bias(X_train) is the same function as always, just applied on top of two real columns (petal length and width) instead of the synthetic dataset. np.linalg.pinv(...) @ y_train is literally the same math as AdalinePseudoInverse.fit, unchanged, and the batch gradient descent's fit_gd is also the same self.w_ += self.learning_rate * error @ X from Adaline, just outside a class and with learning_rate adjusted. The only code that actually changed was the learning_rate, everything else is total reuse.

Pseudo-inverse weights: bias = -1.774, w = [0.485, 0.527]. Plotting just the 30 test points, never seen during training:

Zero test points on the wrong side, same as the perceptron with bias already showed on this dataset. Adaline doesn't need anything extra to get this one right, Iris's generous margin hides its real advantage (a boundary with breathing room, not just "some boundary that separates"), which only shows up when the data gets harder, like the biased dataset above.