Swapping the Cost Function Like Changing Clothes

Lectures 3a and 3b. So far I've seen 4 different algorithms (perceptron, vectorized perceptron, Adaline via pseudo-inverse, Adaline via gradient) as somewhat separate things. This lecture shows they're actually the same recipe, just swapping one ingredient: the cost function.

First, the training algorithm becomes pluggable

class TrainingAlgorithm(ABC):
  @abstractmethod
  def get_w(self, X, y):
    pass

class PseudoInverse(TrainingAlgorithm):
  def get_w(self, X, y):
    return np.linalg.pinv(X) @ y

class NeuralNetwork(BaseEstimator, ClassifierMixin):
  def __init__(self, training_algorithm=PseudoInverse()):
    self.training_algorithm = training_algorithm

  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = self.training_algorithm.get_w(X, y)
    return self

  def predict(self, X):
    X = include_bias(X)
    return np.sign(X @ self.w_)

This is the Strategy design pattern: NeuralNetwork no longer knows how the weights get computed, just that there's a training_algorithm object with a get_w method. Swapping PseudoInverse() for SGD() in the constructor swaps out the entire training algorithm, without touching NeuralNetwork.

Output (PseudoInverse): accuracy 0.95, weights [-1.905, 2.656, 1.049]. Output (SGD): accuracy 0.95, essentially identical weights.

Confirms again, now with more cleanly organized code, what I already saw in the last two posts: pseudo-inverse and gradient descent solve the exact same problem.

Now the cost function becomes pluggable too

class CostFunction(ABC):
  @abstractstaticmethod
  def get_cost(y, y_pred):
    pass
  @abstractstaticmethod
  def get_gradient(X, y, y_pred):
    pass

class WidrowHoff(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    return np.mean((y-y_pred)**2)
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y-y_pred)

SGD now also takes a cost_function, and uses self.cost_function.get_gradient(...) instead of computing the gradient by hand. WidrowHoff is exactly the delta rule from the last post: continuous error (y - y_pred, no sign()) times the input. Swapping the cost function here means swapping what "error" means, without touching the training loop.

Before jumping to four more classes, it's worth watching this machinery turn with small numbers, by hand. Imagine just 2 points, X = [[1, 0], [1, 2]] (bias column already attached) and labels y = [1, -1], with a still-raw weight w = [0, 0]. y_pred = X @ w = [0, 0] (with zeroed weights, every prediction comes out zero). get_cost computes mean((y - y_pred)**2) = mean([1, 1]) = 1, the usual mean squared error. get_gradient computes X.T @ (y - y_pred) = X.T @ [1, -1]: multiplying it out, X.T's first row is [1, 1] (the bias column), so the gradient's first entry is 1×1 + 1×(-1) = 0; X.T's second row is [0, 2] (the real variable), so the second entry is 0×1 + 2×(-1) = -2. The gradient comes out [0, -2]: SGD uses that vector to nudge w, exactly like the perceptron nudged w_ with error * X[i] earlier, except now summing both points' contribution at once (the same vectorized X.T @ error from the perceptron-with-bias post), instead of one point at a time. That same computation, get_cost and get_gradient, is exactly what the next four classes rewrite, each with its own idea of "what counts as error".

Every cost function recovers a different algorithm

class SmoothedSurrogate(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y - np.sign(y_pred))

Notice the np.sign(y_pred) inside the gradient: this goes back to measuring error after the threshold, exactly like Rosenblatt's perceptron. The name SmoothedSurrogate matches what Aggarwal calls the perceptron criterion: $L = \max(0, -y \cdot \hat{y})$, zero once the point is on the right side, growing linearly when it's wrong. No coincidence the accuracy hits 1.0: this cost function, plugged into this generic framework, is the original perceptron again, just expressed in the language of "cost function" instead of "update rule".

class LogLikehood(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y - expit(y_pred))

Output: accuracy 0.65, weights [-60.78, 27.44, -24.79]. Much worse than anything I've seen so far, and the weights got huge.

Two things wrong here, worth separating. First, a detail that doesn't affect the outcome: this class's get_cost uses the hinge loss formula (max(0, 1 - y·ŷ)), not an actual log-likelihood formula. That doesn't break anything in practice because get_gradient is the only thing SGD calls, get_cost never gets used during training, it's leftover residue from copying and pasting from another cell.

The second problem is real, and explains the bad accuracy: expit (the sigmoid function) only returns values between 0 and 1, but the label y is -1 or +1. For class y=-1, the error y - expit(y_pred) can never get close to zero, because expit never goes negative: even with an infinitely confident correctly-classified prediction, -1 - expit(y_pred) stays pinned near -1, never 0. I checked this by hand: expit(-1000) = 0.0, so -1 - expit(-1000) = -1.0 exactly, not 0. The gradient for half the points never vanishes, so training never settles, and the weights keep growing trying to compensate for an error that's structurally impossible to zero out.

class LogLikehood(CostFunction):
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y - tanh(y_pred))

model = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))

Output: accuracy 1.0, weights [-9.84, 12.40, 8.37] (with max_iter=10000, ten times more iterations).

Swapping expit for tanh (same class redefined, Python just lets that run live in a notebook session), the problem disappears. Makes sense: tanh ranges from -1 to +1, exactly the labels' range. Checked it by hand again: tanh(-1000) = -1.0, so -1 - tanh(-1000) = 0.0, the error genuinely reaches zero this time. Aggarwal states this relationship directly in chapter 1: $\tanh(v) = 2 \cdot \text{sigmoid}(2v) - 1$, and it's exactly that range shift, from $(0,1)$ to $(-1,1)$, that fixes the mismatch with ±1 labels.

class HingeLoss(CostFunction):
  @staticmethod
  def get_gradient(X, y, y_pred):
    marginal_errors = (y * y_pred) < 1
    marginal_ys = np.copy(y)
    marginal_ys[~marginal_errors] = 0
    return X.T @ marginal_ys

Output: accuracy 1.0, weights [-8.70, 11.59, 7.71].

Notice marginal_ys[~marginal_errors] = 0: points that are already well classified, with room to spare (y · ŷ ≥ 1), get zeroed out and contribute nothing to the gradient. Only points inside the margin (or misclassified) participate in the update. That's literally SVM's central idea: only the points near the boundary (the "support vectors") matter for deciding where it sits. The rest of the dataset gets ignored once it's already well separated.

All four curves, side by side

The professor has a reference image saved in the notebook comparing the four penalty curves as a function of the "margin" ($y \cdot \hat{y}$: positive and large means a confident correct call, negative means a mistake). I recreated the same idea here, interactively:

Hover over any point on the x-axis and compare all four. Notice the shapes: Widrow-Hoff is a parabola, it keeps penalizing even a point that's already correctly classified with room to spare (margin > 1), because it doesn't know "correct is correct", it only knows how to measure distance to a continuous target. Perceptron and Hinge are the only two that fully zero out once a point is well classified (perceptron zeros as soon as it crosses 0, hinge requires crossing 1, with room to spare). Logistic never truly zeroes out, it only approaches zero, which is the price it pays for returning a smooth probability instead of a binary decision.

All four boundaries, side by side

A different penalty curve is abstract. A different boundary is concrete. I took the real weights each cost function found (skipping the broken expit version, that one gets its own section next) and drew all four over the same batch of points (mine, generated with the same rule as the notebook, since once again only the image got saved):

The four lines land close to each other, all cutting through the middle of the gap between the two point clouds, even though they came from completely different computations. That tracks: for this dataset (nicely separable, no ambiguous point near the boundary), any reasonable cost function converges to pretty much the same answer, the real difference between them only shows up in weights that run off to infinity (Widrow-Hoff, if you let it train too long) or in which points "matter" for the gradient (hinge ignoring the ones with room to spare), not in the boundary's final position in this easy case.

Wrapping up

Practical application

I ran all four cost functions (Widrow-Hoff, perceptron criterion, hinge, and the log-likelihood version with tanh) on Iris (setosa vs. versicolor), plus the expit version on purpose, to confirm the sigmoid problem isn't exclusive to the notebook's toy synthetic dataset.

Four out of five hit 100% (Iris has a generous enough margin for any of them to find a perfect boundary), and sigmoid falls behind again, this time on real data, not just the notebook's synthetic dataset. Confirms it wasn't a one-run coincidence: the range mismatch between expit and ±1 labels genuinely hurts convergence, on every dataset I tried it on.

Notice the call doesn't even change shape, it's the same NeuralNetwork + SGD as always, just pointing at HingeLoss (the winning line in this comparison, alongside perceptron and Widrow-Hoff) instead of any new class:

model = NeuralNetwork(training_algorithm=SGD(max_iter=1000, learning_rate=0.01, cost_function=HingeLoss()))
model.fit(X_train, y_train)

Learned weights: bias = -3.35, w = [0.954, 1.355]. Plotting the 30 test points, never seen during training:

Comparing with the HingeLoss/classification-boundary-comparison-chart from above, line by line: hinge's get_gradient is the same, marginal_ys[~marginal_errors] = 0 zeros out anyone already well classified with room to spare, X.T @ marginal_ys sums only the contribution of whoever's left, and none of that knows (or needs to know) whether X/y came from the synthetic createDataset or the real load_iris. The only thing that actually changes between the two uses is the data going in. Zero test points on the wrong side again, Iris's generous margin leaving little room for any reasonable cost function to miss.