Memorized the Test, Didn't Learn the Subject

The original lab is a clickable Jupyter widget that only runs inside the course's own environment. Without it, I rebuilt the same intuition as two interactive components that run right here on the page, one for regression (you already used this piece in the feature engineering lab) and a new one, for classification, today's actual topic.

A word that already showed up, now with a name

Back in the feature engineering post you already saw this: a high-degree polynomial stops learning the data's trend and starts memorizing every training point instead, zigzagging between them. That has a name: overfitting. The opposite, a model too simple to even catch the obvious pattern, is underfitting. Today that same pair of problems shows up again, this time on the classification side, and it looks different.

The same trick as always: mapping features

For a decision boundary to curve, I need the same polynomial-features trick, just now over two variables at once: turn x0 and x1 into a batch of new terms, x0², x0·x1, x1², and so on up to whatever degree I pick. Run logistic regression again on top of those terms, same as always. The boundary is still "linear" in those new terms, it's just that once I go back to the original x0, x1 space, that line became a curve.

Testing on a dataset no straight line can separate

A ring of patients from one class, surrounding a central cluster from the other. No line can cut through a ring while leaving the center entirely on one side, so this dataset is the perfect test for watching low degree fail and high degree overreach:

At degree 1 (leave lambda at 0), the boundary can only be a straight line, and a line can't separate a ring from a center at all: accuracy is stuck at 50%, the textbook underfitting look. Bump it to degree 2 and the magic happens: a circle already solves it perfectly, accuracy jumps to 100%. So far, so good.

Now push it up to degree 8 or 10. Notice: accuracy stays at 100%, it doesn't get worse. But look at the shape of the colored region: instead of a clean circle, it's full of folds and tongues stretching out to reach each point individually. That's overfitting in classification: the model didn't learn "there's a ring around a center", it memorized the exact position of each of the 20 patients I showed it. A new patient landing in one of those odd folds would probably get misclassified, even though the model gets 100% of the ones it already saw.

Training accuracy can lie to you

This is the lab's most important point: 100% training accuracy doesn't mean a good model. It only says the model is flexible enough to memorize what it's already seen. To know if it actually learned anything, the test has to be on people it never saw, exactly the test accuracy I've been reporting since the first decision boundary lab.

What to do about it

The course lists three fixes, before touching any code yet:

• More training data: with more examples spread across the ring and the center, it gets harder for a crazy boundary to "fit" all of them perfectly.
• Fewer features: a lower degree (or fewer input variables) takes flexibility away from the model, it just can't bend as much anymore.
• Regularization: instead of cutting features, you keep all of them but force the weights to stay small, without zeroing any out. It's the most commonly used fix in practice, and it's the next lab's topic.

Wrapping up

Practical Application

Back to the real breast cancer dataset (worst concave points + worst perimeter, full train/test split, 455 patients to train on, 114 to test). I train mapped logistic regression at increasing degrees and compare training cost against cost on patients never seen:

Training cost keeps dropping as the degree climbs (makes sense, more flexibility always helps explain what's already been seen). Test cost does the opposite: it starts rising already at degree 2 and never stops getting worse. Test accuracy confirms it: 97.4% at degree 1, dropping to 94.7% at degree 8, exactly while training accuracy climbs from 93.2% to 94.5%. The model that's "more accurate" on paper (training) is the worse one on patients it never saw.

To see this same phenomenon on the real data itself, not just the cost plot:

At low degree the boundary is basically the same line from earlier labs. Raise the degree and watch the sample's edges: the colored region starts making tight curls around isolated patients, trying to "get" each one, exactly the pattern the cost plot already called out as getting worse, not better.