The Sigmoid Function: Putting a Ceiling on Regression

In the last lab, linear regression's raw output reached 1.4043 for one real patient, a number that means nothing as a "probability of cancer". It was missing a ceiling. That's exactly what the sigmoid function (also called the logistic function) fixes: it takes any number, positive, negative, huge, tiny, and always returns something between 0 and 1.

The formula, no drama

$g(z) = \frac{1}{1+e^{-z}}$

You don't need to memorize this to understand what it does: when z is very negative, e^(-z) blows up and the fraction goes to nearly 0. When z is very positive, e^(-z) shrinks to nearly 0 and the fraction goes to nearly 1. Right in the middle, z=0, it's exactly 0.5. It never actually reaches 0 or 1, only gets closer and closer.

def sigmoid(z):
    g = 1/(1+np.exp(-z))
    return g

Four lines. np.exp computes $e^z$ element by element, so this same function works whether z is a single number or a whole array at once (vectorization).

Output (z from -10 to 10, a few values):

z	sigmoid(z)
-10	0.0000
-2	0.1192
-1	0.2689
0	0.5000
1	0.7311
2	0.8808
10	1.0000

Logistic regression: linear regression with a built-in ceiling

Logistic regression is just that: take the usual linear model, $z = \mathbf{w}\cdot\mathbf{x}+b$, and run the result through the sigmoid.

$f_{\mathbf{w},b}(\mathbf{x}) = g(\mathbf{w}\cdot\mathbf{x}+b) = \frac{1}{1+e^{-(\mathbf{w}\cdot\mathbf{x}+b)}}$

The output $f_{\mathbf{w},b}(\mathbf{x})$ is now interpreted as a probability: the estimated chance of y=1 given x. To turn it into a final decision, the same 0.5 cutoff as before: f >= 0.5 predicts 1, otherwise 0.

The same test from the last lab, now with a ceiling

Back to the same tumor example, now with a toggle to switch between linear and logistic regression:

Add the large tumors again (same as last lab): with linear regression, x=3 keeps getting classified wrong. Check "use logistic regression": the sigmoid curve, unlike a straight line, finds a way to accommodate the new extreme points without warping in the middle, because it already saturates near 0 and 1 instead of climbing forever. x=3 goes back to being classified correctly. No point moved, only the shape of the model learning the data changed.

Wrapping up

Practical Application

In the last lab, raw linear regression on the real breast cancer dataset (worst concave points, 569 patients) returned 1.4043 for one patient, a number that makes no sense as a probability. I run the sigmoid on that exact number:

print(sigmoid(1.4043))

Output: 0.8028.

0.8028 is already an answer that makes sense: "80% chance of being malignant". I fit real logistic regression on this same dataset (gradient descent, the same worst concave points variable, z-scored):

w, b = fit_logistic_1d(X_train_z, y_train)
print(f"w={w:.4f} b={b:.4f}")

Output: w=3.9565 b=-1.1326, training accuracy 0.9055, test accuracy 0.9211.

On this particular dataset, with no dramatic outlier, the logistic version's accuracy ends up very close to the linear version from the last lab (0.9099/0.9298 there, 0.9055/0.9211 here), because real patients don't have some absurd "tumor size of 10" dragging the line around. The difference that matters isn't the accuracy on this specific dataset, it's the output: every prediction from the logistic version is already born between 0 and 1, ready to be read as a probability, without needing to trust that no future patient will ever have an extreme enough value to blow up the math.