Decision Boundary: Where the Sigmoid Changes Its Mind

I've already seen the sigmoid squash any number into (0, 1), with the 0.5 cutoff deciding the final class. This lab takes an example with two input variables and shows exactly where, geometrically, that cutoff happens.

The data: two patients, two variables

X = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])
y = np.array([0, 0, 0, 1, 1, 1])

Six patients, each with two variables (x0, x1). The first three are y=0, the last three are y=1.

Where the 0.5 cutoff sits

Suppose the model has already been trained (the "how to train it" part is the next two labs' job) and landed on these parameters: b=-3, w0=1, w1=1. The model is

$f(\mathbf{x}) = g(x_0+x_1-3)$

I already know that sigmoid(z) >= 0.5 exactly when z >= 0. So:

• predicts y=1 when $x_0+x_1-3 \geq 0$
• predicts y=0 when $x_0+x_1-3 < 0$

The boundary between the two decisions is the line where the inequality becomes an equality: $x_0+x_1-3=0$, that is, $x_1 = 3-x_0$. That line is the decision boundary: everything the sigmoid does afterward just squashes the distance to that line into a probability, but the final binary decision (y=0 or y=1) only depends on which side of the line a point falls on.

All three y=0 patients fall below the line, all three y=1 patients fall above or on it. The model separates all six perfectly, just by looking at which side of the line each one lands on.

Boundaries don't have to be straight lines

All of this came from $z = w_0x_0+w_1x_1+b$, a linear combination of the inputs, which is why the boundary comes out straight. But nothing stops you from feeding the sigmoid polynomial terms instead, say $z = x_0^2+x_1-1$. The boundary becomes a curve (a parabola, in this case), but the logic is identical: the boundary is always wherever z=0, except now z isn't linear in x anymore. That's the same feature engineering trick I already saw in regression, applied to classification.

Wrapping up

Practical Application

I fit real logistic regression (gradient descent) on the real breast cancer dataset, now with two variables: worst concave points and worst perimeter, the two most correlated with the diagnosis, both z-scored:

w, b = fit_logistic(X_train_z, y_train)  # two columns
print(f"w={w}, b={b:.4f}")

Output: w=[2.5494, 4.0739], b=-1.0038, training accuracy 0.9319, test accuracy 0.9737.

Decision boundary over 60 real training patients (coordinates already normalized):

Most malignant patients (blue) fall above and to the right of the boundary, most benign ones (orange) fall below and to the left, exactly the separation pattern that 97% test accuracy already suggested. The handful of points on the "wrong" side of the line are exactly the model's mistakes, cases where these two variables alone aren't enough to decide confidently, the same kind of ambiguity I already saw using just one variable.