Squared Error Doesn't Work Here: the Logistic Loss

Every bit of linear regression so far used squared error as the cost function, a smooth, well-behaved bowl where gradient descent always finds the bottom. The obvious temptation is to reuse the same idea for logistic regression: take the sigmoid's output and measure its squared error against the label. This lab shows, with real numbers, why that temptation is a trap.

Reusing squared error (and a hint that it's going to go wrong)

$J(w,b) = \frac{1}{2m}\sum_{i=0}^{m-1}\left(f_{w,b}(x^{(i)})-y^{(i)}\right)^2, \quad f_{w,b}(x^{(i)}) = \text{sigmoid}(wx^{(i)}+b)$

Same formula as always, except f now already went through the sigmoid before entering the math. I fixed b=-3 (close to where a decent fit for this data ends up) and varied w:

Look at the last rows: from w=10 to w=20, the cost barely moves at all (0.1667 → 0.1669). That's not a coincidence, it's the sigmoid saturating: for large w, sigmoid(wx+b) is already so close to 0 or 1 that pushing w even higher barely moves the output, and by extension barely moves the cost. The gradient in that region is tiny. If gradient descent lands there, it barely budges.

Rotate the chart below and compare it to linear regression's smooth bowl: it's no longer a symmetric surface with one clean, well-defined bottom, it's got folds and a long plateau where the slope nearly vanishes.

Squared error isn't a bad idea by itself, it just implicitly assumes the noise in the data is well-behaved (gaussian), which makes sense for predicting a continuous number. Glued to a sigmoid, that assumption breaks, and the cost surface loses its guarantee of being a single clean bowl. Classification needs a loss function actually designed for it.

The logistic loss: punishes confident mistakes hard

The idea behind the logistic loss is simple to explain with no formula at all first: if the correct label is 1, I want to punish hard when the model said "almost certainly 0", and barely punish at all when it said "almost certainly 1". And the mirror image for label 0. The formula, in two parts:

Look at the blue curve (y=1): when f is close to 1 (the model got it right, confidently), the loss drops to nearly 0. When f is close to 0 (the model got it badly wrong, said "almost certainly not" for a case that was actually yes), the loss explodes, heading toward infinity. The red curve (y=0) is the exact mirror, flipped around. Unlike squared error's parabola, these curves never flatten out: they always have slope, so gradient descent always has a clear direction to follow, no matter how bad the prediction is.

One single line, ready to code

Two formulas split by an if are annoying to program. Since y only ever takes 0 or 1, both can be combined into one line, using y and (1-y) as switches that turn each term on and off:

$loss = -y^{(i)}\log\left(f_{w,b}(x^{(i)})\right) - (1-y^{(i)})\log\left(1-f_{w,b}(x^{(i)})\right)$

When y=1, the second term zeroes out (1-1=0) and only -log(f) survives. When y=0, the first term zeroes out and only -log(1-f) survives. The same line of code covers both cases, no if needed.

Wrapping up

Practical Application

Using the logistic model already fit in the sigmoid lab (w=3.9565, b=-1.1326, worst concave points variable), I compute the loss for three real test-set patients, chosen to show both extremes and the middle:

for name, x, y_real in patients:
    f = sigmoid(w*x + b)
    loss = -y_real*np.log(f) - (1-y_real)*np.log(1-f)
    print(f"{name}: f={f:.4f} loss={loss:.4f}")

Output:

Patient	worst concave points	true y	f(x)	loss
Clearly malignant	0.2867	1	0.9999	0.0001
Clearly benign	0.0000	0	0.0003	0.0003
Ambiguous	0.1335	0	0.5097	0.7127

The model gets the first two patients right with total confidence, their loss is practically zero. The third is the ambiguous case, sitting right on the decision boundary (the model gave it a 51% chance of being malignant, but the patient was benign), and that single patient's loss (0.7127) is already bigger than the other two combined. It's exactly this kind of point, near the boundary and misclassified, that pushes the fit hardest during training, because it's the one carrying the high loss.