Final Evaluation, ROC Curve, and Hyperparameter Tuning

In chapter 2 I defined the criterion before seeing any numbers (F1 first, AUC as tiebreaker) and Random Forest won the comparison across four model families. This closing chapter trains it for real, on the entire official training set, measures it on the official test set (the 175,341 sessions the model never saw), and doesn't stop at accuracy: I open up the decision cutoff, the ROC curve, which variables weighed most, exactly where the model gets it wrong, and whether tuning hyperparameters can improve it. The full, executed notebook, covering all three parts of this project, is linked at the top of chapter 1.

Final evaluation: Random Forest, on the real official test set

I trained the Random Forest (200 trees, with class_weight="balanced", a way of telling the model that missing an attack, the rare class, costs more than missing a normal session, without needing to duplicate or drop a single row the way resampling tried to do back with credit card fraud) on the entire official training set and measured it on the official test set, the 175,341 records the model never saw:

modelo_final = Pipeline([
    ("prep", preprocessador),
    ("clf", RandomForestClassifier(n_estimators=200, random_state=42, class_weight="balanced", n_jobs=-1)),
])
modelo_final.fit(X_train, y_train)
y_pred = modelo_final.predict(X_test)

print(classification_report(y_test, y_pred, digits=4))

The same Pipeline as always, now with the entire preprocessador (the 39 StandardScaler plus 3 OneHotEncoder) glued in front of the final Random Forest, a single .fit() training both stages together. classification_report is what computes precision, recall, and F1 per class in one shot, the same table right below.

Precision here is: of the sessions the model flagged as attack, how many really were. Recall is the opposite: of the real attacks, how many the model caught. F1 folds both into one score. Overall accuracy is 90.43%, but the number I actually trust is the attack class's recall: 87.07%, meaning the model catches 103,912 of the 119,341 real attacks in the test set, and lets 15,429 slip through. In security, this kind of false negative (an attack classified as normal) is the expensive mistake, it's literally the intruder who walks past unnoticed. On the other side, 1,349 normal sessions turned into false alarms, annoying for whoever investigates them, but much less serious than letting a real attack through. AUC on the official test set lands at 0.9830, close to the 1.0 ceiling, a sign the model knows how to rank sessions by risk well even before any decision threshold enters the picture, the same ranking metric I already saw with credit card fraud.

Dialing the cutoff: precision against recall

All that evaluation used scikit-learn's default cutoff: .predict() decides "attack" whenever the risk score (the fraction of trees that voted attack, the same number you can get from .predict_proba()) crosses 0.5. But that 0.5 is just one arbitrary point on a whole line of options, the same threshold idea I already saw in the classification-with-regression lesson, except now you can actually turn the dial.

For that, I wrote a second notebook (datasets/unsw-nb15/02_dados_extras_scatter_e_threshold.ipynb), only because the original notebook never saved the continuous y_score, just the y_pred already thresholded at 0.5. This notebook's first cells rebuild, cell by cell, exactly the same pipeline from chapters 2 and 3:

df_train = pd.read_csv("/content/data/UNSW_NB15_training-set.csv")
df_test = pd.read_csv("/content/data/UNSW_NB15_testing-set.csv")

DESCARTAR = ["id", "attack_cat", "label"]
X_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])
y_train = df_train["label"]
X_test = df_test.drop(columns=[c for c in DESCARTAR if c in df_test.columns])
y_test = df_test["label"]

cat_cols = X_train.select_dtypes(include=["object"]).columns.tolist()
num_cols = X_train.select_dtypes(include=["number"]).columns.tolist()

transformadores = [
    ("num", StandardScaler(), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), cat_cols),
]
preprocessador = ColumnTransformer(transformers=transformadores)

modelo_final = Pipeline([
    ("prep", preprocessador),
    ("clf", RandomForestClassifier(n_estimators=200, random_state=42, class_weight="balanced", n_jobs=-1)),
])
modelo_final.fit(X_train, y_train)

y_pred = modelo_final.predict(X_test)
y_score = modelo_final.predict_proba(X_test)[:, 1]

Cell by cell, this is the same code from chapter 2 (the official partition, the ColumnTransformer) and from the start of this chapter (the Pipeline with the 200-tree Random Forest), no new line anywhere, just rebuilt here because this notebook runs isolated, in a Colab runtime that doesn't carry over the other two notebooks' state. The only difference is the last line: besides y_pred, I also keep y_score, the continuous probability, so I can test cutoffs other than 0.5 in the next block. With the pipeline retrained, the new block sweeps 19 different cutoffs:

thresholds = np.round(np.arange(0.05, 1.0, 0.05), 2)
threshold_sweep = []
for t in thresholds:
    y_pred_t = (y_score >= t).astype(int)
    threshold_sweep.append({
        "threshold": float(t),
        "precision": round(float(precision_score(y_test, y_pred_t, zero_division=0)), 4),
        "recall": round(float(recall_score(y_test, y_pred_t, zero_division=0)), 4),
        "f1": round(float(f1_score(y_test, y_pred_t, zero_division=0)), 4),
    })

np.arange(0.05, 1.0, 0.05) generates the 19 cutoffs (0.05, 0.10, ..., 0.95), and the loop tests each one: y_score >= t turns the continuous score into a binary label at that specific cutoff (the same comparison .predict() does internally, except fixing t instead of always using 0.5), and all three metrics get recomputed from scratch on every pass. zero_division=0 avoids an error in case some extreme cutoff zeroes out an entire class. The result, 19 rows of precision/recall/F1, is what feeds the interactive chart below:

Drag the slider and watch the trade. Dropping the cutoff to 0.05 (flag "attack" on any suspicion at all, however small) pushes recall to 0.9781, catching almost everyone, but drags precision down to 0.9234, more false alarms for the security team to chase for nothing. Pushing the cutoff up to 0.95 (only flag "attack" when the model is nearly certain) flips it: precision of 0.999, almost no false alarms, but recall collapses to 0.6951, letting nearly a third of real attacks through. There's no single right cutoff, separate from the business question. If chasing a false alarm is cheap, it's worth pulling the cutoff down and eating more false alarms in exchange for catching more real attacks. If every investigation is expensive, the cutoff goes up.

At the cutoff I used through the whole post (0.5), pairing class_weight="balanced" with that default cutoff already delivers 0.8715 recall without me touching anything by hand (the numbers match, off by a thousandth, the table further up, since I rebuilt the pipeline from scratch in this separate notebook). That's why I never went chasing a better cutoff for the final evaluation: the class imbalance was already being compensated for during training itself, not just at decision time.

The ROC curve: the same cutoff, seen from another angle

The confusion matrix and the precision/recall table above evaluate the model at a single decision cutoff (the default, 0.5). But the slider I just showed already gives away the rest of the picture: the Random Forest doesn't just return a label, it estimates a probability P(attack | x) for every session, and the cutoff is an operational security decision, not a fixed mathematical truth.

The ROC curve (Receiver Operating Characteristic) turns that trade into a picture: sweeping across every possible cutoff, it plots the true positive rate (the same recall as always) against the false positive rate (the fraction of normal sessions that turned into false alarms). AUC (Area Under the Curve, the area under that plot) condenses the whole chart into a single number, with a direct probabilistic reading: it's the chance the model gives a higher score to a randomly picked attack than to a randomly picked normal session. AUC = 1.0 is the perfect classifier, AUC = 0.5 is the same as a coin flip (the chart's dotted diagonal). In the original notebook, that computation is just two lines:

y_score = modelo_final.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_score)

predict_proba returns two columns, the probability of each class, [:, 1] grabs just class 1's column (attack). roc_auc_score takes the true label and that continuous score, no fixed cutoff involved at all, and computes AUC directly. The original notebook stopped there and only drew the chart with scikit-learn's own RocCurveDisplay.from_predictions(...), a static image, never saving the curve's points anywhere.

To get a real, interactive chart here on the blog, I wrote a third notebook (datasets/unsw-nb15/03_curva_roc.ipynb), with the same rebuild as always at the start (third time this block shows up in this project by now, so I'll just repeat it, no re-explaining):

df_train = pd.read_csv("/content/data/UNSW_NB15_training-set.csv")
df_test = pd.read_csv("/content/data/UNSW_NB15_testing-set.csv")

DESCARTAR = ["id", "attack_cat", "label"]
X_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])
y_train = df_train["label"]
X_test = df_test.drop(columns=[c for c in DESCARTAR if c in df_test.columns])
y_test = df_test["label"]

cat_cols = X_train.select_dtypes(include=["object"]).columns.tolist()
num_cols = X_train.select_dtypes(include=["number"]).columns.tolist()

transformadores = [
    ("num", StandardScaler(), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), cat_cols),
]
preprocessador = ColumnTransformer(transformers=transformadores)

modelo_final = Pipeline([
    ("prep", preprocessador),
    ("clf", RandomForestClassifier(n_estimators=200, random_state=42, class_weight="balanced", n_jobs=-1)),
])
modelo_final.fit(X_train, y_train)

y_score = modelo_final.predict_proba(X_test)[:, 1]

And, on top of that y_score, extracts the curve point by point:

fpr, tpr, thresholds = roc_curve(y_test, y_score)
auc = roc_auc_score(y_test, y_score)
print(f"AUC (conferencia, deve bater com 0.9830 ja publicado): {auc:.4f}")
print(f"Curva bruta tem {len(fpr)} pontos, vamos reduzir pra 50 pra plotar no blog")

fpr_grid = np.linspace(0, 1, 50)
tpr_grid = np.interp(fpr_grid, fpr, tpr)

roc_curve_points = [
    {"fpr": round(float(f), 4), "tpr": round(float(t), 4)}
    for f, t in zip(fpr_grid, tpr_grid)
]

roc_curve(y_test, y_score) internally sweeps every unique score present in y_score as a candidate cutoff, and returns three same-length arrays: fpr, tpr, and the thresholds that produced each pair. On this dataset the raw curve came out with 282 points, more granularity than a blog chart needs and a bigger JSON than necessary. np.linspace(0, 1, 50) creates 50 evenly spaced positions on the false-positive axis, and np.interp(fpr_grid, fpr, tpr) estimates the real true-positive rate at each of those 50 positions, interpolating between the nearest real points, nothing invented: it's the same curve, just sampled on a grid that's lighter to draw. The confirmation print lands on AUC = 0.9830, matching the number computed in the original notebook exactly.

Loading real data...

The curve hugs the top-left corner, far from the dotted diagonal of chance: at a false-positive cutoff of just 2%, recall already jumps to 86%, and the curve stays nearly glued to the ceiling from there on. That shows the model separates the two classes well independent of any specific cutoff, and that's exactly the slack that lets me pull the cutoff down, like I did in the previous section, without precision collapsing along with it.

Who decides: the 15 variables that matter most

Random Forest lets you peek at which variables most reduced impurity across the trees during training, the same Gini yardstick I already saw with decision trees, just now summed across 200 trees instead of just one:

ohe = modelo_final.named_steps["prep"].named_transformers_["cat"]
nomes_cat = ohe.get_feature_names_out(cat_cols)
nomes_features = np.concatenate([num_cols, nomes_cat])

importancias = modelo_final.named_steps["clf"].feature_importances_
imp = pd.Series(importancias, index=nomes_features).sort_values(ascending=False)

This block exists to solve an annoying problem: feature_importances_ returns just a raw array of numbers, no name attached, and after the ColumnTransformer the three original categorical columns (proto, service, state) don't even exist anymore, they turned into dozens of new binary columns, like proto_tcp, proto_udp, service_dns. To reconstruct "which number in the list belongs to which variable," I need to go fetch those names back from inside the already-trained pipeline itself: modelo_final.named_steps["prep"] grabs the preprocessing stage, .named_transformers_["cat"] grabs specifically the OneHotEncoder that ran on the categorical columns, and .get_feature_names_out(cat_cols) asks it "what were the names of the new columns you created?" np.concatenate([num_cols, nomes_cat]) glues that list back together with the 39 numeric ones up front (which kept their original names, StandardScaler never creates a new column), respecting the same order ColumnTransformer uses internally. Only after that does pd.Series(importancias, index=nomes_features) manage to pair every importance number with the right name.

The winner, ct_dst_src_ltm, counts how many of the last 100 connections share the same source-destination pair, a direct way to flag scanning or flooding behavior: normal traffic rarely hits the exact same source-destination pair repeatedly, hundreds of times in a row, automated attacks do. Notice sttl (the TTL that already showed up in the correlation table) appears again, in second place, confirming that 0.91 correlation with the label wasn't empty statistical coincidence, it's a real signal. And a good chunk of the rest of the list (ct_dst_sport_ltm, ct_srv_dst, ct_state_ttl, ct_srv_src) belongs to the same "how many recent connections look like this one" family, the kind of variable specifically designed to catch a burst of repeated connections, the most common signature of automated attack versus human traffic.

Where the model gets it wrong: 400 real sessions across the 3 most important variables

The top two spots in the importance ranking, ct_dst_src_ltm and sttl, plus the third, sbytes, fit together in one space. It's the same notebook 02 that generated the threshold sweep, with the same retrained pipeline, that also extracts this sample:

FEATURES_3D = ["sttl", "ct_dst_src_ltm", "sbytes"]
N_SAMPLE = 400

amostra_idx = X_test.sample(n=N_SAMPLE, random_state=42).index
y_pred_series = pd.Series(y_pred, index=X_test.index)

sample_points = []
for i in amostra_idx:
    sample_points.append({
        "x": float(X_test.loc[i, FEATURES_3D[0]]),
        "y": float(X_test.loc[i, FEATURES_3D[1]]),
        "z": float(X_test.loc[i, FEATURES_3D[2]]),
        "label_real": int(y_test.loc[i]),
        "label_previsto": int(y_pred_series.loc[i]),
    })

X_test.sample(n=400, random_state=42) draws 400 indices from the official test set, the fixed seed is just so I can reproduce the same sample later. y_pred_series repackages the raw y_pred array (which only has position, no index) into a Series sharing X_test's index, so a specific session's prediction can be looked up by .loc[i] inside the loop. For each of the 400 sampled sessions, the loop keeps the three features of interest (sttl, ct_dst_src_ltm, sbytes) plus the true label and the predicted one, nothing invented, it's exactly the official X_test/y_test, just narrowed down to a sample small enough to plot. I plotted them on those 3 axes:

Loading real data...

In this sample of 400: 247 correct attack calls, 112 correct normal calls, 36 false negatives, and only 5 false positives, a ratio close to the full test set's. One thing that jumps out while spinning the cloud around: sttl isn't really continuous, it clusters into a handful of round values (31, 62, 254...), because TTL is literally a router-hop counter each operating system initializes at a default value (64, 128, or 255) and decrements by one at every hop. Among the 283 attack sessions in this sample, sttl=254 shows up in 238 of them (84%), but it also shows up in 37 of the 117 normal sessions (32%), the most common value on both sides, far from a clean separator on its own. sttl=31, on the other hand, is almost exclusively normal traffic: 74 of the 117 normal sessions, zero attacks. That's exactly why no single variable decides anything here. You can see the 36 red points (false negatives) scattered through the middle of the cloud, never forming an isolated island, and that's where the other 39 variables the chart doesn't show come in, working together to close the 87% recall the final model delivers.

Trying to improve with GridSearch, and hitting my own ceiling

I ran GridSearchCV varying the number of trees (200 or 400), max depth (unlimited or capped at 30), and minimum leaf size (1 or 2), 8 combinations, 3-fold cross-validation, on a sample of 30 thousand sessions to keep the cost manageable:

param_grid = {
    "clf__n_estimators": [200, 400],
    "clf__max_depth": [None, 30],
    "clf__min_samples_leaf": [1, 2],
}

grid = GridSearchCV(pipe_rf, param_grid, scoring="f1", cv=cv, n_jobs=-1, verbose=1)
grid.fit(Xg, yg)

The clf__ prefix in front of each key isn't decoration: pipe_rf is a Pipeline with a stage named "clf" (the Random Forest), and the __ is scikit-learn's way of saying "the n_estimators parameter belongs to the clf stage, not any other one in the pipeline," the same model__ syntax that already showed up in the pipeline and GridSearch post. Without that prefix, GridSearchCV wouldn't know which stage to touch.

Output: best combination found, n_estimators=200, max_depth=None, min_samples_leaf=1, with a validation F1 of 0.9716. Exactly the default parameters of the Random Forest I'd already trained before.

To avoid blindly trusting that validation F1 (computed on a sample of 30 thousand, remember), I retrained the winning configuration on the full official training set and evaluated it on the official test set, the same rigorous way as the final evaluation:

melhores = {k.replace("clf__", ""): v for k, v in grid.best_params_.items()}
modelo_tunado = Pipeline([("prep", preprocessador),
                          ("clf", RandomForestClassifier(random_state=42, class_weight="balanced",
                                                         n_jobs=-1, **melhores))])
modelo_tunado.fit(X_train, y_train)
y_pred_t = modelo_tunado.predict(X_test)
y_score_t = modelo_tunado.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred_t, digits=4))
print(f"AUC-ROC (ajustada): {roc_auc_score(y_test, y_score_t):.4f}")

grid.best_params_ keeps its keys with the clf__ prefix (the same prefix explained above), so the first line strips that prefix so the values can be passed straight into RandomForestClassifier's constructor via **melhores. The rest is the same Pipeline as always, just with the winning hyperparameters in place of the defaults, trained on the entire official training set (not the 30-thousand search sample anymore) and evaluated on the official test set.

Output: precision, recall, and F1 per class identical to the original Random Forest, decimal place for decimal place, and AUC-ROC (ajustada): 0.9830.

The result matches, decimal place for decimal place, the original Random Forest: same precision, recall, and F1 per class, same 0.9830 AUC. It's not a search failure, it's the search confirming something worth saying plainly: for this dataset, with this set of candidates tested, scikit-learn's default guess (100 or 200 trees, unlimited depth) was already at the ceiling this hyperparameter family reaches. Not every optimization run finds something better to optimize toward, the same kind of "boring" result I already saw in the ensembles post, when plain logistic regression beat every ensemble no amount of tuning would have changed.

Wrapping up the project (and the whole playlist)

This project closes a loop for me: I started this playlist fitting a line with the normal equation, went through trees, clustering, dimensionality reduction, text turning into vectors, and ended up applying all of it on my own, no lecture to follow, on a real network security problem, with a real result (0.98 AUC on the official test set) I can defend line by line, from the raw CSV all the way to the final decision tree. Thanks, Professor Boldt, I've got the rest from here.