The Sigmoid Got Fixed, the Hinge Lost Its Margin

Lecture 4a, and it picks up exactly where last post left off. Same dataset, same pluggable CostFunction framework, but the professor went back and touched two of the implementations. Worth checking line by line what changed, because one change fixes a real bug, and the other, without meaning to, creates a new one.

The scaffolding barely changed, just a cosmetic detail

def createDataset(n=20):
  X = np.random.rand(n,2)
  coefs = np.array([1, 1])
  intercept = 1
  labels = X @ coefs - intercept
  y = np.sign(labels)
  return X, y

The only real difference from last post's dataset: y = np.sign(labels) instead of np.array(labels>0, dtype=int)*2-1. Two different computations, the exact same result (-1 or +1), just a more direct way to write the same thing. TrainingAlgorithm, SGD, and WidrowHoff came over unchanged, not a comma different, so I won't repeat them here.

Output (WidrowHoff): accuracy 1.0, weights [-1.98, 1.20, 2.36].

SmoothedSurrogate: same essence, get_cost even more decorative

class SmoothedSurrogate(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    y_pred = (y_pred >= 0).astype(int)
    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))
  @staticmethod
  def get_gradient(X, y, y_pred):
    y_pred = np.sign(y_pred)
    return X.T @ (y - y_pred)

get_gradient stays identical in spirit to the perceptron criterion I already covered: measures error after sign(). get_cost changed (y_pred >= 0 becomes 0/1, no longer -1/+1), but as I already noted, get_cost never enters training, only get_gradient gets called by SGD. It gets even more crooked than before, but it still doesn't matter for anything.

Output: accuracy 1.0, weights [-0.51, 0.56, 0.47].

The real fix: expit finally matches the label

class LogLikehood(CostFunction):
  @staticmethod
  def get_gradient(X, y, y_pred):
    y = (y>0).astype(int)
    return X.T @ (y - expit(y_pred))

Compare with the broken version I investigated last post:

# aula03b, the original bug
def get_gradient(X, y, y_pred):
  return X.T @ (y - expit(y_pred))

The difference is a single line: y = (y>0).astype(int). That converts the -1/+1 label to 0/1 before subtracting from expit(y_pred), which only ever returns values in (0,1). That's exactly the fix I predicted last post: line up the label's range with the sigmoid's range, instead of comparing -1 (impossible for expit to reach) against a number that never goes negative.

Output: accuracy 1.0, weights [-5.44, 3.33, 5.82]. Compared with the 0.65 I found before, the difference speaks for itself.

The "fixed" hinge loss that actually turned into something else

class HingeLoss(CostFunction):
  @staticmethod
  def get_gradient(X, y, y_pred):
    y_pred = np.sign(y_pred)
    marginal_errors = (y * y_pred) < 1
    marginal_ys = np.copy(y)
    marginal_ys[~marginal_errors] = 0
    return X.T @ marginal_ys

Again, compare with the earlier version:

# aula03b
def get_gradient(X, y, y_pred):
  marginal_errors = (y * y_pred) < 1
  marginal_ys = np.copy(y)
  marginal_ys[~marginal_errors] = 0
  return X.T @ marginal_ys

The only change is the line y_pred = np.sign(y_pred), added right at the top. It looks harmless, the same sign() that's shown up throughout this playlist, but it changes the entire meaning of marginal_errors. Before, y * y_pred used the continuous pre-activation, so y * ŷ < 1 covered both a misclassified point (y * ŷ < 0) and a correctly classified one with no room to spare (0 ≤ y * ŷ < 1), those latter ones are the "support vectors" I explained last post: hinge loss keeps pushing the boundary until they get some breathing room. Now, with y_pred already passed through sign(), y * y_pred can only be -1 (wrong) or +1 (right), never anything in between. < 1 becomes a synonym for "just wrong", which is exactly the perceptron criterion again, just written with more lines.

Output: accuracy 1.0, weights [-0.58, 0.53, 0.59].

To confirm this isn't me nitpicking one line of code, I built a concrete example: took an already-trained boundary (w = [-2.5, 3.0, 2.75], which separates the 20 points with 1.0 accuracy) and checked, point by point, who each rule considers "marginal" (i.e., who still contributes to the gradient).

Point 6 is the most telling: its pre-activation is -0.001, practically sitting on the boundary, exactly the kind of point that should keep "bothering" training. The old rule (continuous margin) recognizes that and keeps adjusting its weight. The new rule, having already applied sign(), only sees that it landed (by the barest margin) on the correct side, and gives up on it. In total, 9 of the 20 points fall into that band (0 ≤ y·ŷ < 1, correct but with no room to spare): the old rule keeps adjusting the weight on them, the new rule considers all 9 already settled.

The four boundaries from this lecture, side by side

Same visual exercise as last post, now with the real weights this fixed version found:

Once again all four converge to pretty much the same line, on this easy dataset. The difference the analysis above uncovered isn't in where the final boundary lands, it's in which points keep getting adjusted throughout training, something this chart (just the final result) can't show on its own.

Wrapping up

Practical application

I repeated the comparison, real hinge (continuous margin) vs. aula04a hinge (margin via sign()), on two real datasets already used in this playlist.

On Iris (setosa vs. versicolor, again), I ran both variants with the same learning_rate=0.01 and the same seed:

A technical tie, and it makes sense: Iris has too generous a margin for any point to be left in the 0 ≤ y·ŷ < 1 band once it converges, so the two rules end up agreeing on who's "marginal": nobody.

I switched to the two-moons dataset (make_moons, already used in the XOR post, noise=0.25 this time to guarantee real overlap between the classes, no perfect linear separation) and ran 20 different seeds for each version:

With data where the linear boundary isn't perfect, the difference shows up: real hinge beats the aula04a version on test accuracy across the 20 seeds, and the weight norm is almost 4 times bigger, exactly because it keeps adjusting the boundary on points that land correctly but with no room to spare, pushing toward a more confident separation instead of stopping as soon as the sign already matches. That's the practical proof that a continuous margin isn't just a theoretical detail, it changes the outcome once the dataset isn't trivially separable.

Taking one specific seed out of those 20 (test accuracy 0.878 for real hinge, 0.789 for the sign() version, close to the average of both), plotting both boundaries over the 90 test points, the same ones already used in the ELM post:

You can see the "confidence" difference between the two lines: real hinge (bigger weight) cuts more perpendicular to the overlap band between the moons, taking on more risk to separate better; the aula04a version, with a small weight, stays more "lukewarm", stopping adjustment as soon as clearly correct points fall out of the sum, even with no room to spare.