From Loss to Cost: Summing Up Everyone's Error

The logistic loss measures one example's error. The course already draws this distinction: loss is about a single example, cost is the loss summed (averaged) over the entire training set. Cost is what gradient descent minimizes.

The full cost function

$J(\mathbf{w},b) = \frac{1}{m}\sum_{i=0}^{m-1}\left[-y^{(i)}\log\left(f_{\mathbf{w},b}(\mathbf{x}^{(i)})\right) - (1-y^{(i)})\log\left(1-f_{\mathbf{w},b}(\mathbf{x}^{(i)})\right)\right]$

Looks complicated, but it's just one example's loss, the same single-line formula from the last lab, sitting inside a sum divided by m. The average loss across everyone.

The code, line by line

def compute_cost_logistic(X, y, w, b):
    m = X.shape[0]
    cost = 0.0
    for i in range(m):
        z_i = np.dot(X[i],w) + b
        f_wb_i = sigmoid(z_i)
        cost +=  -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i)
    cost = cost / m
    return cost

• m = X.shape[0]: how many examples are in the set.
• the for loop walks through each patient i.
• z_i = np.dot(X[i],w) + b: the linear part, the same combination as always, except now X[i] can hold several variables, so it's a dot product instead of a plain multiplication.
• f_wb_i = sigmoid(z_i): squash it into a probability.
• the middle line adds that patient's loss to the running total cost, the same single-line formula from the last lab.
• at the end, cost / m takes the average.

Testing with two different values of b

Same dataset as always, two candidate boundaries: b=-3 and b=-4 (keeping w=[1,1] fixed both times).

print("Cost with b = -3:", compute_cost_logistic(X_train, y_train, np.array([1,1]), -3))
print("Cost with b = -4:", compute_cost_logistic(X_train, y_train, np.array([1,1]), -4))

Output: Cost with b = -3: 0.3669 · Cost with b = -4: 0.5037

The cost already gives away which boundary is worse, before I need to look at any plot. But the plot confirms it:

The b=-4 line (dashed) shifts up and to the right relative to b=-3, and in that shift it ends up putting two y=1 patients on the wrong side (below the line, predicted as 0). Makes sense its cost is higher: a high cost is literally the model saying "this boundary explains the data I saw worse than the other one".

Wrapping up

Practical Application

I use the real two-variable model from the decision boundary lab (worst concave points + worst perimeter, both normalized), with weights w=[2.5494, 4.0739] learned via gradient descent, and compare the cost of the trained b against a deliberately worse b (shifted by +2):

print("Cost with trained b:", compute_cost_logistic(X_train_z, y_train, w, b))
print("Cost with b + 2:", compute_cost_logistic(X_train_z, y_train, w, b + 2))

Output: Cost with trained b: 0.1461 · Cost with b + 2: 0.2391

The cost jumps by more than a third just from shifting the boundary by two units. And accuracy feels it in practice: with the trained b, training accuracy is 0.9319; with the shifted b, it drops to 0.8989. A "close enough" b (the boundary's direction, given by w, stays the same) already costs 3 percentage points of accuracy, and the cost calls out exactly that worsening, in the exact form gradient descent sees it: as a bigger number, one it'll try to shrink again on the next steps.