Regularization: Keep Every Feature, Just Don't Let Any of Them Boss Around

The last lab closed with three fixes for overfitting: more data, fewer features, or regularization. The first two have a catch: more data doesn't always exist, and cutting features throws away information that might actually help. Regularization solves it differently: keep every feature, just discourage any weight from growing too large. b stays out of that penalty, only the w_j feel the brake.

Cost with regularization

For linear regression, the new term is just the sum of squared weights, scaled by lambda and divided by 2m:

$J(\mathbf{w},b) = \frac{1}{2m} \sum\limits_{i = 0}^{m-1} (f_{\mathbf{w},b}(\mathbf{x}^{(i)}) - y^{(i)})^2 + \frac{\lambda}{2m} \sum_{j=0}^{n-1} w_j^2$

def compute_cost_linear_reg(X, y, w, b, lambda_ = 1):
    m  = X.shape[0]
    n  = len(w)
    cost = 0.
    for i in range(m):
        f_wb_i = np.dot(X[i], w) + b
        cost = cost + (f_wb_i - y[i])**2
    cost = cost / (2 * m)

    reg_cost = 0
    for j in range(n):
        reg_cost += (w[j]**2)
    reg_cost = (lambda_/(2*m)) * reg_cost

    return cost + reg_cost

The function's first half is the usual cost, untouched. The second half (reg_cost) just sums each weight squared, scales by lambda_/(2m), and adds it at the end. Testing with fixed random data (np.random.seed(1), six features, lambda=0.7):

Output: Regularized cost: 0.07917239320214275

For logistic regression, the change is identical, the regularization term is copy-pasted:

$J(\mathbf{w},b) = \frac{1}{m} \sum_{i=0}^{m-1} \left[ -y^{(i)} \log\left(f_{\mathbf{w},b}(\mathbf{x}^{(i)})\right) - \left( 1 - y^{(i)}\right) \log \left( 1 - f_{\mathbf{w},b}(\mathbf{x}^{(i)}) \right) \right] + \frac{\lambda}{2m} \sum_{j=0}^{n-1} w_j^2$

def compute_cost_logistic_reg(X, y, w, b, lambda_ = 1):
    m, n = X.shape
    cost = 0.
    for i in range(m):
        z_i = np.dot(X[i], w) + b
        f_wb_i = sigmoid(z_i)
        cost += -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i)
    cost = cost/m

    reg_cost = 0
    for j in range(n):
        reg_cost += (w[j]**2)
    reg_cost = (lambda_/(2*m)) * reg_cost

    return cost + reg_cost

The usual cost in the first half, the same reg_cost from above pasted into the second. Testing with the same random data:

Output: Regularized cost: 0.6850849138741673

Gradient with regularization

The gradient also picks up an extra term, only on w_j (b still isn't penalized at all):

$\frac{\partial J(\mathbf{w},b)}{\partial w_j} = \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{\mathbf{w},b}(\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)} + \frac{\lambda}{m} w_j$

The (lambda/m) * w_j term is the only difference between this gradient and the usual one, for either linear or logistic regression, the only thing that changes between the two is how f gets computed:

def compute_gradient_logistic_reg(X, y, w, b, lambda_):
    m, n = X.shape
    dj_dw = np.zeros((n,))
    dj_db = 0.0

    for i in range(m):
        f_wb_i = sigmoid(np.dot(X[i], w) + b)
        err_i  = f_wb_i - y[i]
        for j in range(n):
            dj_dw[j] = dj_dw[j] + err_i * X[i, j]
        dj_db = dj_db + err_i
    dj_dw = dj_dw / m
    dj_db = dj_db / m

    for j in range(n):
        dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]

    return dj_db, dj_dw

The for i loop is the usual logistic gradient, untouched. The extra for j at the end is the only new line: it adds (lambda_/m) * w[j] to each position of dj_dw, after the average has already been taken. Testing (linear and logistic, same random data, lambda=0.7):

Output (linear): dj_db: 0.6648774569425726 · dj_dw: [0.2965, 0.4912, 0.2165]

Output (logistic): dj_db: 0.341798994972791 · dj_dw: [0.1738, 0.3201, 0.1078]

Rerunning the ring example

The same component from the last lab, now with the lambda slider taking center stage. Keep the degree fixed at 10 (already well overfit, as we saw) and raise lambda gradually:

Accuracy stays at 100% across most of the lambda range, because this toy dataset is too easy to actually confuse the model. But watch the color intensity: at lambda=0, the central region is heavily saturated, the model is nearly certain (above 99.9%) even close to the ring's edge. At lambda=5, that same edge gets a lot paler, confidence drops to around 81.6% at the same spot. Regularization didn't change the final decision here, but it made the model less "loud", less confident beyond what the data actually supports. It's still possible to overdo it: push lambda up to 10 and watch the central region start shrinking, a sign the model is now over-regularized, pulling weights toward zero to the point of hurting even the easy decision it had already nailed.

The hook to this week's graded assignment

This week's graded assignment (the part I don't convert here because its raw data didn't ship with the notebook) takes a two-exam-score dataset to decide admission, then a two-quality-test dataset to decide whether a microchip passes. The second part asks for exactly what this post already built: mapping two features into a high-degree polynomial (the course's map_feature function is the same mapping the component above already does under the hood) and then regularizing the logistic regression for that boundary, free to curve without going crazy. You already watched the end result working above, just on a different dataset.

Wrapping up

Practical Application

On the real breast cancer dataset (degree fixed at 8, already seen in the last lab as mildly overfit), sweeping lambda:

A small lambda (1 or 2) genuinely recovers a bit of test accuracy over lambda=0 (94.7% → 95.6%), but the effect here is modest, much subtler than on the toy ring. Makes sense: with only two features, this real dataset's boundary is already close to a straight line by nature, overfitting was never that severe here to begin with. And the opposite shows up clearly too: lambda=10 already makes both accuracies worse than lambda=1, over-regularization hurting a problem that didn't need that much of a brake.

Raise lambda slowly and watch the folds along the edges from the last lab smooth out. Regularization isn't an automatic win, it's a tool: it helps a lot when overfitting is severe, helps little (and can even hurt) when the problem never called for that much flexibility in the first place.