Gradient Descent Didn't Need to Learn Anything New

Back in Week 1, gradient descent for linear regression was: compute the gradient, take a step, repeat. The code by now already has sigmoid, decision boundary, logistic loss and cost ready to go. One thing's missing: running gradient descent on top of that new cost to find the w and b that minimize it. And here's the satisfying part of this chapter: the loop that takes the steps is literally the same one. Only the math inside the loop (the gradient) changes, because f swapped from a line to a sigmoid.

The loop doesn't change, only what it derives

$w_j = w_j - \alpha \frac{\partial J(\mathbf{w},b)}{\partial w_j} \qquad b = b - \alpha \frac{\partial J(\mathbf{w},b)}{\partial b}$

Same recipe as always: take the gradient, step in the opposite direction by alpha (updating w and b at the same time), repeat until it stops improving. And the partial derivatives have the same familiar shape too:

$\frac{\partial J(\mathbf{w},b)}{\partial w_j} = \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{\mathbf{w},b}(\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)}, \qquad \frac{\partial J(\mathbf{w},b)}{\partial b} = \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{\mathbf{w},b}(\mathbf{x}^{(i)}) - y^{(i)})$

Glance at it quickly and this formula is indistinguishable from linear regression's. The good kind of gotcha: it is identical, byte for byte. What changes is what f means: in linear regression, f = w·x + b; here, f = sigmoid(w·x + b). That swap, hidden inside f, is what makes the rest of the math behave differently without rewriting a single line of the gradient formula.

The code, line by line

def compute_gradient_logistic(X, y, w, b):
    m, n = X.shape
    dj_dw = np.zeros((n,))
    dj_db = 0.

    for i in range(m):
        f_wb_i = sigmoid(np.dot(X[i], w) + b)
        err_i = f_wb_i - y[i]
        for j in range(n):
            dj_dw[j] = dj_dw[j] + err_i * X[i, j]
        dj_db = dj_db + err_i
    dj_dw = dj_dw / m
    dj_db = dj_db / m

    return dj_db, dj_dw

• f_wb_i = sigmoid(...): the same linear combination as always, squashed by the sigmoid into a probability.
• err_i = f_wb_i - y[i]: this patient's error, prediction minus real label.
• the inner for j multiplies that error by each variable X[i,j] and accumulates it into dj_dw[j], the same "error times input" mechanic linear regression already used.
• dj_db accumulates just the raw error, no multiplication, also identical to linear regression.
• at the end, both become an average (/ m).

Checking against w=[2,3], b=1 on the usual dataset:

dj_db_tmp, dj_dw_tmp = compute_gradient_logistic(X_tmp, y_tmp, w_tmp, b_tmp)
print(f"dj_db: {dj_db_tmp}")
print(f"dj_dw: {dj_dw_tmp.tolist()}")

Output: dj_db: 0.49861806546328574 · dj_dw: [0.498333393278696, 0.49883942983996693]

The outer loop really is copy-paste

def gradient_descent(X, y, w_in, b_in, alpha, num_iters):
    w = copy.deepcopy(w_in)
    b = b_in

    for i in range(num_iters):
        dj_db, dj_dw = compute_gradient_logistic(X, y, w, b)
        w = w - alpha * dj_dw
        b = b - alpha * dj_db

        if i % math.ceil(num_iters / 10) == 0:
            print(f"Iteration {i:4d}: Cost {compute_cost_logistic(X, y, w, b)}")

    return w, b

Compare it to Week 1's gradient_descent: the only real difference is which cost function feeds the progress print. The for loop, the w = w - alpha * dj_dw update, all identical.

Running it on the usual six-patient dataset, alpha=0.1, 10,000 iterations:

Output:

Iteration    0: Cost 0.6846
Iteration 1000: Cost 0.1591
Iteration 2000: Cost 0.0846
Iteration 3000: Cost 0.0571
Iteration 4000: Cost 0.0429
Iteration 5000: Cost 0.0343
Iteration 6000: Cost 0.0286
Iteration 7000: Cost 0.0245
Iteration 8000: Cost 0.0214
Iteration 9000: Cost 0.0190

w: [5.2812, 5.0782], b: -14.2224

Cost plummets early (from 0.68 to 0.16 in just the first thousand rounds) and then keeps refining slowly, the same "learns fast at first, polishes later" curve that's shown up on every convex cost surface trained so far in this playlist.

The boundary separates all six patients with no mistakes, exactly like the hand-trained version shown earlier, except this time I watched gradient descent get there step by step, instead of receiving the weights ready-made.

Walking downhill, live

The dataset above has two variables, so its cost surface lives in a 4-dimensional space (w0, w1, b, cost) that can't be drawn. To actually watch gradient descent walk, I go back to a single-variable dataset, the same six patients from the week's first lab:

x_train = np.array([0., 1, 2, 3, 4, 5])
y_train = np.array([0,  0, 0, 1, 1, 1])

With just w and b, the logistic cost becomes a real 3D surface, the same convex bowl the last lab already compared against the "wrong cost" attempt (that one, riddled with fake valleys). Drag alpha, click "Run" and watch the red dot descend to the bottom, always along the path the convex shape guarantees:

Notice the controls are the same ones you already used on the linear regression posts: alpha, step-by-step, run in batches, switch between 2D/3D/parabola. Gradient descent doesn't get any smarter or dumber working on a different cost, it just descends whatever surface you hand it.

Wrapping up

Practical Application

I reuse the same real variable this playlist keeps coming back to (worst concave points, z-score normalized), now with the 60-patient sample from the decision boundary lab, and run gradient descent myself instead of receiving the weights ready-made:

Running with alpha=0.5, cost drops from 0.6931 (the worst possible guess, -log(0.5)) to 0.2372 and converges around w=4.175, b=-1.094, close to the w=3.9565, b=-1.1326 earlier labs cite for the full training set (this sample here is only 60 patients, the full model saw a lot more people). Accuracy on this sample: 0.90, in the same ballpark as the 0.91-0.93 earlier labs already reported. The same loop that ran on six toy points now runs on real patients, no code change needed, just a different dataset going in.