Switching Datasets and Picking a Candidate

In chapter 1 I explored the 2.5 million raw UNSW-NB15 sessions: class imbalance, missing data, a hidden quality bug, redundancy between variables, and the decision not to prune any feature. I closed by flagging a problem I hadn't solved yet: srcip, dstip, sport, and dsport separate too well, in a way that doesn't generalize. This chapter starts by solving exactly that, switching datasets, and ends by choosing, with a real criterion, which model family to train.

Why switch datasets: data leakage

UNSW-NB15 comes in two formats. One is what I used the entire previous chapter: the four raw CSVs, with all 49 fields, including srcip, dstip, sport, dsport, and the Stime/Ltime timestamps. The other is the official partition (UNSW_NB15_training-set.csv and UNSW_NB15_testing-set.csv), a subset the dataset's own authors (Moustafa & Slay, 2015) already split into train and test, without those identifiers.

I choose the official partition for modeling, and the central reason is data leakage: an IP or a port identifies a specific machine in this lab, not a pattern of attack behavior. If I let srcip into the model, it would learn something like "59.166.0.4 is always suspicious" instead of learning what makes an attack look like an attack (session duration, byte volume, TTL, and so on). That artificially inflates performance inside this specific lab and doesn't generalize to any network outside it. The Stime/Ltime timestamps have the same problem: flows close together in time tend to be correlated with each other (the same attack session generating several rows in a row), and using the raw timestamp would let that correlation leak into the model.

There's a second reason, less about leakage and more about scientific method: the official partition is the standard benchmark in the literature for this dataset. Training and evaluating on it makes my result comparable to other published work using the same dataset, something I'd lose if I invented my own random split over the raw files, and I'd run the additional risk of leakage between train and test from correlated flows (same host, same session) landing on both sides of the split.

The switch has a real cost: I'm giving up 2.5 million rows for a much smaller set.

df_train = pd.read_csv("/content/data/UNSW_NB15_training-set.csv")
df_test  = pd.read_csv("/content/data/UNSW_NB15_testing-set.csv")

Output: training with 82,332 rows, testing with 175,341, each with 45 columns.

Notice the test set is larger than the training set here, the opposite of what I'd expect from a common 80/20 split. UNSW-NB15's authors deliberately designed a generous test set, precisely to give a more trustworthy measure of generalization. And there's another inversion hiding in there: in the official partition, the attack class becomes the majority in the test set (further along, in chapter 3, this shows up in the numbers: 119,341 attack sessions against 56,000 normal ones), the opposite proportion I saw in the raw-file EDA. It's a reminder that EDA and modeling, here, don't run over the same dataframe, and one's proportions don't carry over to the other.

The official partition's columns

The partition's 45 columns aren't the same 49 from the raw files. The authors already removed the identifiers and the timestamps, and organized what's left into groups that make sense to read together:

• Basic flow: dur (duration), proto, service, state.
• Volume and rate: spkts/dpkts (packets), sbytes/dbytes (bytes), rate, sload/dload (load).
• TCP and timing: sttl/dttl (the TTL that already showed up in chapter 1), swin/dwin (TCP window), tcprtt, synack, ackdat.
• Connection counters (ct_...): how many recent connections share a service, host, or port, the variable family specifically designed to catch scanning and repetition.
• Targets: label (0/1) and attack_cat.

Of the 42 input columns (dropping id and the two targets), 39 are numeric and 3 are categorical (proto, service, state):

DESCARTAR = ["id", "attack_cat", "label"]
X_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])
y_train = df_train["label"]

cat_cols = X_train.select_dtypes(include=["object"]).columns.tolist()
num_cols = X_train.select_dtypes(include=["number"]).columns.tolist()

Output: categoricals ['proto', 'service', 'state'], 39 numeric.

select_dtypes picks out the 3 categoricals and the 39 numeric columns on its own, no need to type a single column name by hand.

Building the preprocessing pipeline

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

transformadores = [
    ("num", StandardScaler(), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), cat_cols),
]
preprocessador = ColumnTransformer(transformers=transformadores)

ColumnTransformer applies, in parallel, a different transformer to each column group: StandardScaler on the 39 numeric ones (centers at mean 0, std 1, essential for Logistic Regression, harmless for a tree) and OneHotEncoder on the 3 categorical ones, the same pair that already showed up in Titanic, now solving network protocol instead of boarding port. One detail worth flagging: handle_unknown="ignore". If the test set brings a service value the training set never saw, instead of throwing an error, OneHotEncoder simply zeroes out that entire row for that variable, the same spirit as the [UNK] token I already saw handling an unknown word.

This cell doesn't return a chart or a number, it just assembles the transformer. And that's exactly where its most important point lives: I don't call .fit() on this preprocessador by itself, it goes whole into a Pipeline, together with the classifier. That matters because, in the cross-validation coming up next, each fold trains its own StandardScaler using only that fold's training slice, the mean and standard deviation never see the validation slice. If I'd run preprocessador.fit_transform(X_train) before splitting into folds, the mean and std computed over the whole training set would leak statistical information from the validation fold into training, a subtle leak (much smaller than the IP one, but real) that Pipeline avoids for free.

Comparing four candidates, on a quick sample

Before choosing a model, I defined the criterion before seeing any numbers: priority on F1-score (balance between not letting attacks through and not generating false alarms), with AUC as the tiebreaker. Since classes are imbalanced, I use class_weight="balanced" wherever the model accepts it, and stratified cross-validation (keeps the class proportion in every fold).

I tested four quite different families, each carrying a different hypothesis about the data:

• Logistic Regression: assumes a linear decision boundary, a simple, interpretable baseline.
• Decision Tree: captures nonlinear relationships, but tends to overfit on its own.
• Random Forest: aggregates many trees, reducing variance via bagging.
• Gaussian Naive Bayes: assumes independence between variables given the class, fast, a baseline.

modelos = {
    "Regressao Logistica": LogisticRegression(max_iter=1000, class_weight="balanced"),
    "Arvore de Decisao": DecisionTreeClassifier(random_state=42, class_weight="balanced"),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42, class_weight="balanced", n_jobs=-1),
    "Naive Bayes": GaussianNB(),
}

score = lambda clf, m: cross_val_score(Pipeline([("prep", preprocessador), ("clf", clf)]), Xs, ys, cv=cv, scoring=m, n_jobs=-1).mean()
linhas = [{"Modelo": n, "F1": score(c, "f1"), "AUC": score(c, "roc_auc")} for n, c in modelos.items()]

Worth opening this block up in slow motion, it packs quite a bit into two lines. modelos is just a dictionary, name → classifier instance, the four candidates I want to compare. score is a one-line function (a lambda): it takes a classifier clf and a metric name m, builds a brand-new Pipeline gluing the same preprocessador in front of that specific classifier (each candidate needs its own pipeline, you can't reuse a pipeline already fit for a different model), runs 3-fold cross-validation on that pipeline (on a sample of 20 thousand training sessions, running this on the full 82 thousand for every candidate would burn time needlessly at this screening stage), and averages the 3 scores. The line below calls that score function twice for each of the 4 models, once asking for "f1", once for "roc_auc". 4 models × 2 metrics × 3 folds: 24 complete training runs, hiding behind two lines of code.

Random Forest wins on both metrics (F1 0.97, AUC 1.00 on this screening sample), the Decision Tree alone comes right behind it, and Gaussian Naive Bayes falls well behind the other three (F1 0.54). That tracks for the same reason I already saw in the breast cancer PCA post: Naive Bayes assumes the variables are independent of each other given the class, and chapter 1 already showed a good chunk of these 39 numeric variables are anything but independent, several of them telling the same story in different words. The theoretical expectation also holds up: Random Forest combines the trees' low bias with bagging's low variance, exactly the kind of robustness I'd expect to handle well the 39 variables, many of them redundant with each other, that chapter 1 mapped out.

Closing this chapter

With the criterion defined before seeing the numbers (F1 first, AUC as tiebreaker) and the result matching the theoretical expectation, Random Forest is the one chosen for the final training run. In chapter 3 I train it on the entire official training set, evaluate on the official test set (never seen), open up the ROC curve, the decision cutoff, which variables weighed most on the decision, and whether tuning hyperparameters can improve it.