Perceptron: the First Neuron That Learns on Its Own

First lecture of the living playlist, and it starts exactly where any neural networks course should: at the simplest neuron there is.

Before the code: two papers, 15 years apart

Neural networks weren't born as code, they were born as a biophysics question. In 1943, Warren McCulloch and Walter Pitts published "A Logical Calculus of the Ideas Immanent in Nervous Activity", proposing a lean mathematical model for a biological neuron: sum up the input signals, and fire a binary (all-or-nothing) signal if that sum crosses a threshold. No learning yet, it's just a fixed logic circuit, each "neuron" solves a logical function someone already decided ahead of time.

The missing leap came 15 years later, with Frank Rosenblatt in 1958: what if, instead of someone hand-picking the weights, the neuron itself learned the right weights by looking at examples? That's the Perceptron, and it's exactly what the professor's lecture 1a implements: the Perceptron Learning Algorithm (PLA), in its rawest form, no bias yet (that's next lecture).

The dataset: two groups separable by a line

def createDataset(n=20):
  X = np.random.rand(n,2)
  coefs = np.array([1, -1])
  labels = X @ coefs
  y = np.array(labels>0, dtype=int)*2-1
  return X, y

The professor generates random 2D points and labels each one by the sign of X @ coefs, the dot product between the point and the vector [1, -1]. Geometrically, X @ coefs is positive on one side of the line that passes through the origin and is perpendicular to [1, -1], and negative on the other side, so the label y (-1 or +1) is linearly separable by construction: a straight line exists that separates the two classes perfectly, because that exact line is what generated the labels.

def plotHyperplan(vector):
  xs = np.array([0,1])
  ys = -(vector[0]*xs)/vector[1]
  plt.plot(xs, ys)

This function draws the line where vector[0]*x + vector[1]*y = 0, the decision boundary for any weight vector w. Same equation as always, w · x = 0 defines a hyperplane, except here without a bias, so the hyperplane is forced to pass through the origin (0,0). Keep that detail in mind, it matters in a bit.

The professor even tests a DummyClassifier with fixed weights [1, -1], the exact same ones that generated the dataset, and of course it hits 100%: that's the answer key fed straight back in. You can see this for real, by plotting the boundary that vector [1, -1] draws on top of the same 20 points that keep showing up for the rest of the lecture:

Notice the line passes right through the origin (0,0), exactly what plotHyperplan's math guarantees, and separates the two colors without missing a single point. There's no magic to it: the vector [1, -1] is the ruler that decided every label back inside createDataset. The real question is a different one: can an algorithm learn that same vector on its own, just by looking at the colored points, without me handing over [1, -1] on a plate?

The algorithm: nudge the weight toward the error

class PLA(BaseEstimator, ClassifierMixin):
  def __init__(self, max_iter=10):
    self.max_iter = max_iter

  def fit(self, X, y):
    self.w_ = np.random.rand(X.shape[1])
    for _ in range(self.max_iter):
      cost = 0
      idx = np.arange(X.shape[0])
      np.random.shuffle(idx)
      for i in idx:
        logits = X[i] @ self.w_
        y_pred = np.sign(logits)
        error = y[i] - y_pred
        if error != 0:
          cost += error**2
          self.w_ += error*X[i]
        if cost == 0:
          break
    return self

Before going line by line, the whole idea fits in one sentence: show the model a point, if it got it wrong, nudge the weight a little toward whatever would have gotten that point right, repeat until nothing's wrong anymore. That's it. No gradient computed, no learning rate to tune, it's about the most direct recipe you could imagine.

Now taking the fit above apart line by line:

• self.w_ = np.random.rand(X.shape[1]): the weight vector starts completely random, a number between 0 and 1 for each variable. The model opens the game with no idea where the right boundary is.
• for _ in range(self.max_iter):: gives the algorithm up to max_iter chances to correct itself. Each of those chances is an epoch: one full pass through the 20 points.
• cost = 0: a counter that resets at the start of every epoch. It only exists to answer one question at the end: "did any mistake happen this time?"
• idx = np.arange(X.shape[0]); np.random.shuffle(idx): shuffles the order of the 20 points before visiting them. Without this, the model would always see the same points first in every epoch, which can bias training.
• for i in idx:: now it visits each point, one at a time, in that shuffled order.
• logits = X[i] @ self.w_: the dot product between the point and the current weight, a single number, positive or negative.
• y_pred = np.sign(logits): turns that into a decision. Positive says "this is class +1", negative says "this is class -1".
• error = y[i] - y_pred: compares against the true label. Since both live in {-1, +1}, error can only be one of three values: 0 (correct), +2 (should've been +1, the model guessed -1) or -2 (should've been -1, the model guessed +1).
• if error != 0: cost += error**2; self.w_ += error*X[i]: only touches w_ when it got it wrong. error**2 is always 4 whenever there's an error (the exact value doesn't matter, only that it's nonzero), and self.w_ += error*X[i] is the actual update.

To see that update with real numbers instead of the abstract version: imagine that, on one of these passes, w_ sits at [0.5, -0.2], and the current point is x = (0.6854, 0.9534), which in the real dataset belongs to class -1. logits = 0.6854×0.5 + 0.9534×(-0.2) ≈ 0.152, positive, so y_pred = +1. But the true label is -1, so error = -1 - 1 = -2. The update becomes w_ += -2×(0.6854, 0.9534) ≈ (-0.871, -2.107), a strong shove in the direction opposite to x.

That makes sense geometrically: w is the vector normal to the decision hyperplane, it points toward the side the model calls "class +1". The model wrongly thought this point was +1, so pushing w away from it makes w · x more negative next time, correcting in the right direction. The opposite happens when the error goes the other way (error = +2): then w gets pushed toward the point, to make w · x more positive. It's a local, cheap adjustment, every mistake nudges the decision boundary a little in the direction that would have gotten that specific point right (this is exactly what Aggarwal calls the "perceptron criterion" in chapter 1: a heuristic update rule that looks a lot like gradient descent, but was designed directly on top of the classification error, before anyone formalized which smooth loss function it was implicitly optimizing).

And Rosenblatt proved, back in 1958, something strong: if the data really is linearly separable (as it is here, by construction), PLA always converges to a zero-error solution, in a finite number of steps. Not "usually works", a mathematical guarantee.

A real bug, hiding in the stopping condition

But looking closely at if cost == 0: break reveals a problem. That check sits inside the loop that walks through the points, not after it. That means: as soon as cost (which only grows on error) hits zero, the loop stops immediately, even if there are still points left to check that epoch.

The catch is that at the very start of every epoch, cost already starts at zero. So if the first sampled point in that epoch happens to already be classified correctly, cost stays zero, and the break fires right there, without checking the other 19 points. The epoch ends thinking "everything's fine", when really only one point got checked.

I ran this exact code, byte for byte, with a fixed seed (np.random.seed(10)), instrumenting just one thing: a counter that records how many of the 20 points each epoch actually processed before the break fired (or the loop finished on its own).

np.random.seed(10)
X, y = createDataset(20)

w_ = np.random.rand(X.shape[1])
counts = []
for _ in range(10):
  cost = 0
  idx = np.arange(X.shape[0])
  np.random.shuffle(idx)
  n_seen = 0
  for i in idx:
    n_seen += 1
    logits = X[i] @ w_
    y_pred = np.sign(logits)
    error = y[i] - y_pred
    if error != 0:
      cost += error**2
      w_ += error*X[i]
    if cost == 0:
      break
  counts.append(n_seen)

Output: [20, 1, 1, 20, 1, 1, 1, 1, 1, 1]. Only epochs 1 and 4 checked the whole dataset, the other 8 stopped after looking at a single point. Final accuracy: 0.95, still short of the perfect separation Rosenblatt guarantees, weights w = [2.228, -1.847].

Notice the isolated orange point sitting inside the blue region, near (0.63, 0.75): that's the one that stayed misclassified. It never got a chance to correct the model, because in every epoch where it came up in the queue, the algorithm had already given up too early to reach it.

Moving just the if cost == 0: break outside the inner loop (checking zero errors after going through all 20 points, not after each one), same seed, same data, same initial w:

Output: [20, 20, 20, 20], four full epochs, and done: converged with accuracy 1.0, exactly what the theorem promises for linearly separable data, weights w = [2.106, -2.082].

That same orange point at (0.63, 0.75) now falls on the right side: with the chance to see the whole dataset every epoch, the algorithm had time to correct the boundary enough to fit it too.

A real, non-hypothetical finding: the obvious intent of the code is "stop once there's no more error this epoch", but the way the break got positioned makes it stop as soon as a single favorable point shows up, even if more error is lurking further down the queue. With luck (as with most seeds I tested), that doesn't change the final outcome because other epochs make up for it. But with this specific seed, the algorithm declares success prematurely eight times in a row, and closes training with 1 point still wrong, when it should have reached the perfect separation.

Interactive: training the perceptron point by point

I rebuilt the same dataset (that seed 10 above, the same 20 points) in a component that runs the correct version of the algorithm, one point at a time. Click "Process next point" and notice: every time a colored point falls on the wrong side of the background region, that's a mistake, and the next click pushes the boundary toward it.

Notice that, with no bias, the boundary is always a line through the origin (0,0), it can only rotate, never slide. It works on this dataset because the data was drawn around the origin on purpose. But what if the point cloud were shifted far away from the origin? I get to that in the Practical Application.

Wrapping up

Practical application

I tested the same no-bias PLA (bug-fixed version, without the break issue) on a real dataset: Iris, the two easiest classes to separate (setosa vs. versicolor), using petal length and petal width as the two variables.

from sklearn.datasets import load_iris
iris = load_iris()
mask = iris.target < 2
X = iris.data[mask][:, [2, 3]]  # petal length and width
y = np.where(iris.target[mask] == 0, -1, 1)

Comparing this with the createDataset from above, line by line: iris.data[mask][:, [2, 3]] plays the role of np.random.rand(n,2), except instead of drawing random points, it grabs two columns from a real dataset (petal length and width). And np.where(iris.target[mask] == 0, -1, 1) plays the role of np.array(labels>0, dtype=int)*2-1: same destination (labels in {-1, +1}), different path to get there. Above, I invented the labels by applying a mathematical rule (X @ coefs) on top of the points; here the labels already exist (it's each flower's real species), it just needs converting from sklearn's 0/1 format to the -1/+1 format PLA expects. The PLA class itself, fit and predict, doesn't change a single line, proof the algorithm doesn't know (or need to know) whether the data it's looking at is synthetic or real.

These two classes are genuinely linearly separable (it's one of the most-cited "actually separable" examples in introductory ML material). But petal length and width are never negative, so the entire point cloud lives far from the origin, quite unlike the synthetic dataset above.

Without bias, PLA never declares convergence within the 50-epoch cap, because the line that would truly separate the two classes doesn't pass through the origin, and without bias that line is simply out of the model's reach. I ran 5 different seeds and none went past 93% accuracy. Adding a single parameter (the bias, which shifts the hyperplane instead of just rotating it around the origin), the same algorithm converges in just 2 epochs with perfect accuracy, weights w = [3.549, 5.515], bias -9.397:

The gap between the two clouds is huge compared to the synthetic dataset above, so pretty much any reasonable line through that empty corridor already separates the two species. That's the exact limit next lecture tackles head-on: data that doesn't live near the origin needs bias to stand any chance.