Classification: Why Regression Isn't Enough

Week 3 kicks off a new topic: classification. And before anything else, a heads up: this material gets denser fast (logistic regression, decision boundaries, a whole new cost function), so from here on I'm going to lean harder on intuition and line-by-line code than on heavy math. If you want the full derivation, it exists and I link to it when it makes sense, but the goal here is understanding what the code does and why, not memorizing formulas.

Regression predicts a number, classification decides a category

Regression predicts a continuous value: house price, tumor size in millimeters, anything that can be 152.37. Classification decides between categories: an email is spam or not, a tumor is benign or malignant. The simplest case, binary classification, has just two options, usually coded as 0 (negative) and 1 (positive).

x_train = np.array([0., 1, 2, 3, 4, 5])
y_train = np.array([0,  0, 0, 1, 1, 1])

Here x_train is a tumor's size (on some scale) and y_train is 0 for benign, 1 for malignant. Notice y only ever takes two values, never 0.3 or 2.7, that's what sets this problem apart from regression.

The temptation: I already know regression, why not just use it here?

Fair question: if I already know how to fit a line to a set of points, why not fit a line to this 0/1 y and use a cutoff in the middle (> 0.5 becomes 1, < 0.5 becomes 0)? This question already came up, in quite some depth, back in the Pattern Recognition playlist: that course's professor reused the exact same linear regression class to classify breast cancer, and it worked reasonably well, until he showed why least squares carries a structural bias for this kind of problem (Bishop explains why: least squares assumes gaussian noise, and a binary label doesn't have gaussian noise). That post ended with an explicit hook toward logistic regression, which that course hadn't reached yet. Well, it's arrived, just in a different playlist: this one.

Andrew Ng's course shows the same problem in a more visual way, no Bishop required: add one extreme point to the data and watch the line distort completely.

Testing it live: the line warps around one extreme point

The original notebook uses a clickable Jupyter widget (plt_one_addpt_onclick) that only runs inside the course's own environment, so I rebuilt the same idea as a browser-native interactive component: it actually fits least squares live, on every click.

Without the extra points, the line it finds (w≈0.257, b≈-0.143) already separates all six patients perfectly, the 0.5 cutoff lands right in the middle. Click "Add large tumors": two new malignant patients show up far out, at x=10 and x=11. The line, which has to pass close to all the points, flattens out trying to reach the new ones, and the patient at x=3 (still malignant) ends up on the wrong side of the boundary. That patient didn't move, the line moved underneath it. That's exactly the same structural problem Bishop explains: points that are "too correct", far from the boundary, still pull on the fit, because they contribute squared error even though they don't need to.

What this lecture leaves you with

The course sums it up plainly: linear regression is insufficient for classification. The next lab jumps straight into the fix.

Wrapping up

Practical Application

I reuse the real breast cancer dataset (load_breast_cancer, 569 patients, the same one already used in the Pattern Recognition playlist), with that post's own most-correlated variable, worst concave points, z-scored using training-set statistics only.

w, b = fit_linear_1d(X_train_z, y_train)  # closed-form least squares
print(f"w={w:.4f} b={b:.4f}")

Output: w=0.3812 b=0.3714, training accuracy 0.9099, test accuracy 0.9298.

The accuracy isn't bad at all (this dataset is fairly separable on this variable, with no dramatic outlier to break everything the way the toy example above did). But look at what the regression's raw output does for the patients with the most extreme tumors:

scores = w * X_train_z.flatten() + b
print(scores.min(), scores.max())

Output: -0.2956 to 1.4043.

1.4043 doesn't mean anything as a probability of cancer, probability never goes above 1. The line has no ceiling: it extrapolates freely to any number, positive or negative, no matter how absurd that is as an answer to "what's the chance this tumor is malignant?". That missing ceiling is exactly what the next lab fixes.