[{"data":1,"prerenderedAt":639},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo":3,"chapter-en-network-intrusion-detection-02-metodologia-selecao-de-modelo":4},null,{"id":5,"title":6,"body":7,"cover":3,"date":624,"description":625,"extension":626,"meta":627,"navigation":294,"order":121,"path":628,"project":629,"seo":630,"status":631,"stem":632,"tags":633,"__hash__":638},"projectChapters\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo.md","Switching Datasets and Picking a Candidate",{"type":8,"value":9,"toc":617},"minimark",[10,35,40,73,92,99,102,125,134,145,149,152,255,268,308,319,325,329,375,413,442,446,469,472,498,548,592,598,606,610,613],[11,12,13,18,19,23,24,23,27,30,31,34],"p",{},[14,15,17],"a",{"href":16},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes","In chapter 1"," I explored the 2.5 million raw UNSW-NB15 sessions: class imbalance, missing data, a hidden quality bug, redundancy between variables, and the decision not to prune any feature. I closed by flagging a problem I hadn't solved yet: ",[20,21,22],"code",{},"srcip",", ",[20,25,26],{},"dstip",[20,28,29],{},"sport",", and ",[20,32,33],{},"dsport"," separate too well, in a way that doesn't generalize. This chapter starts by solving exactly that, switching datasets, and ends by choosing, with a real criterion, which model family to train.",[36,37,39],"h2",{"id":38},"why-switch-datasets-data-leakage","Why switch datasets: data leakage",[11,41,42,43,23,45,23,47,23,49,51,52,55,56,59,60,64,65,68,69,72],{},"UNSW-NB15 comes in two formats. One is what I used the entire previous chapter: the four raw CSVs, with all 49 fields, including ",[20,44,22],{},[20,46,26],{},[20,48,29],{},[20,50,33],{},", and the ",[20,53,54],{},"Stime","\u002F",[20,57,58],{},"Ltime"," timestamps. The other is the ",[61,62,63],"strong",{},"official partition"," (",[20,66,67],{},"UNSW_NB15_training-set.csv"," and ",[20,70,71],{},"UNSW_NB15_testing-set.csv","), a subset the dataset's own authors (Moustafa & Slay, 2015) already split into train and test, without those identifiers.",[11,74,75,76,79,80,82,83,86,87,55,89,91],{},"I choose the official partition for modeling, and the central reason is ",[61,77,78],{},"data leakage",": an IP or a port identifies a specific machine in this lab, not a pattern of attack behavior. If I let ",[20,81,22],{}," into the model, it would learn something like \"",[20,84,85],{},"59.166.0.4"," is always suspicious\" instead of learning what makes an attack look like an attack (session duration, byte volume, TTL, and so on). That artificially inflates performance inside this specific lab and doesn't generalize to any network outside it. The ",[20,88,54],{},[20,90,58],{}," timestamps have the same problem: flows close together in time tend to be correlated with each other (the same attack session generating several rows in a row), and using the raw timestamp would let that correlation leak into the model.",[11,93,94,95,98],{},"There's a second reason, less about leakage and more about scientific method: the official partition is the ",[61,96,97],{},"standard benchmark in the literature"," for this dataset. Training and evaluating on it makes my result comparable to other published work using the same dataset, something I'd lose if I invented my own random split over the raw files, and I'd run the additional risk of leakage between train and test from correlated flows (same host, same session) landing on both sides of the split.",[11,100,101],{},"The switch has a real cost: I'm giving up 2.5 million rows for a much smaller set.",[103,104,109],"pre",{"className":105,"code":106,"language":107,"meta":108,"style":108},"language-python shiki shiki-themes github-light github-dark","df_train = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_training-set.csv\")\ndf_test  = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n","python","",[20,110,111,119],{"__ignoreMap":108},[112,113,116],"span",{"class":114,"line":115},"line",1,[112,117,118],{},"df_train = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_training-set.csv\")\n",[112,120,122],{"class":114,"line":121},2,[112,123,124],{},"df_test  = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n",[126,127,128],"blockquote",{},[11,129,130,133],{},[61,131,132],{},"Output:"," training with 82,332 rows, testing with 175,341, each with 45 columns.",[11,135,136,137,140,141,144],{},"Notice the test set is ",[61,138,139],{},"larger"," than the training set here, the opposite of what I'd expect from a common 80\u002F20 split. UNSW-NB15's authors deliberately designed a generous test set, precisely to give a more trustworthy measure of generalization. And there's another inversion hiding in there: in the official partition, the attack class becomes the ",[61,142,143],{},"majority"," in the test set (further along, in chapter 3, this shows up in the numbers: 119,341 attack sessions against 56,000 normal ones), the opposite proportion I saw in the raw-file EDA. It's a reminder that EDA and modeling, here, don't run over the same dataframe, and one's proportions don't carry over to the other.",[36,146,148],{"id":147},"the-official-partitions-columns","The official partition's columns",[11,150,151],{},"The partition's 45 columns aren't the same 49 from the raw files. The authors already removed the identifiers and the timestamps, and organized what's left into groups that make sense to read together:",[153,154,155,176,205,233,243],"ul",{},[156,157,158,161,162,165,166,23,169,23,172,175],"li",{},[61,159,160],{},"Basic flow:"," ",[20,163,164],{},"dur"," (duration), ",[20,167,168],{},"proto",[20,170,171],{},"service",[20,173,174],{},"state",".",[156,177,178,161,181,55,184,187,188,55,191,194,195,23,198,55,201,204],{},[61,179,180],{},"Volume and rate:",[20,182,183],{},"spkts",[20,185,186],{},"dpkts"," (packets), ",[20,189,190],{},"sbytes",[20,192,193],{},"dbytes"," (bytes), ",[20,196,197],{},"rate",[20,199,200],{},"sload",[20,202,203],{},"dload"," (load).",[156,206,207,161,210,55,213,216,217,55,220,223,224,23,227,23,230,175],{},[61,208,209],{},"TCP and timing:",[20,211,212],{},"sttl",[20,214,215],{},"dttl"," (the TTL that already showed up in chapter 1), ",[20,218,219],{},"swin",[20,221,222],{},"dwin"," (TCP window), ",[20,225,226],{},"tcprtt",[20,228,229],{},"synack",[20,231,232],{},"ackdat",[156,234,235,242],{},[61,236,237,238,241],{},"Connection counters (",[20,239,240],{},"ct_...","):"," how many recent connections share a service, host, or port, the variable family specifically designed to catch scanning and repetition.",[156,244,245,161,248,251,252,175],{},[61,246,247],{},"Targets:",[20,249,250],{},"label"," (0\u002F1) and ",[20,253,254],{},"attack_cat",[11,256,257,258,261,262,23,264,23,266,241],{},"Of the 42 input columns (dropping ",[20,259,260],{},"id"," and the two targets), 39 are numeric and 3 are categorical (",[20,263,168],{},[20,265,171],{},[20,267,174],{},[103,269,271],{"className":105,"code":270,"language":107,"meta":108,"style":108},"DESCARTAR = [\"id\", \"attack_cat\", \"label\"]\nX_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])\ny_train = df_train[\"label\"]\n\ncat_cols = X_train.select_dtypes(include=[\"object\"]).columns.tolist()\nnum_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n",[20,272,273,278,283,289,296,302],{"__ignoreMap":108},[112,274,275],{"class":114,"line":115},[112,276,277],{},"DESCARTAR = [\"id\", \"attack_cat\", \"label\"]\n",[112,279,280],{"class":114,"line":121},[112,281,282],{},"X_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])\n",[112,284,286],{"class":114,"line":285},3,[112,287,288],{},"y_train = df_train[\"label\"]\n",[112,290,292],{"class":114,"line":291},4,[112,293,295],{"emptyLinePlaceholder":294},true,"\n",[112,297,299],{"class":114,"line":298},5,[112,300,301],{},"cat_cols = X_train.select_dtypes(include=[\"object\"]).columns.tolist()\n",[112,303,305],{"class":114,"line":304},6,[112,306,307],{},"num_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n",[126,309,310],{},[11,311,312,314,315,318],{},[61,313,132],{}," categoricals ",[20,316,317],{},"['proto', 'service', 'state']",", 39 numeric.",[11,320,321,324],{},[20,322,323],{},"select_dtypes"," picks out the 3 categoricals and the 39 numeric columns on its own, no need to type a single column name by hand.",[36,326,328],{"id":327},"building-the-preprocessing-pipeline","Building the preprocessing pipeline",[103,330,332],{"className":105,"code":331,"language":107,"meta":108,"style":108},"from sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\n\ntransformadores = [\n    (\"num\", StandardScaler(), num_cols),\n    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False), cat_cols),\n]\npreprocessador = ColumnTransformer(transformers=transformadores)\n",[20,333,334,339,344,348,353,358,363,369],{"__ignoreMap":108},[112,335,336],{"class":114,"line":115},[112,337,338],{},"from sklearn.compose import ColumnTransformer\n",[112,340,341],{"class":114,"line":121},[112,342,343],{},"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",[112,345,346],{"class":114,"line":285},[112,347,295],{"emptyLinePlaceholder":294},[112,349,350],{"class":114,"line":291},[112,351,352],{},"transformadores = [\n",[112,354,355],{"class":114,"line":298},[112,356,357],{},"    (\"num\", StandardScaler(), num_cols),\n",[112,359,360],{"class":114,"line":304},[112,361,362],{},"    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False), cat_cols),\n",[112,364,366],{"class":114,"line":365},7,[112,367,368],{},"]\n",[112,370,372],{"class":114,"line":371},8,[112,373,374],{},"preprocessador = ColumnTransformer(transformers=transformadores)\n",[11,376,377,380,381,384,385,388,389,393,394,397,398,400,401,403,404,412],{},[20,378,379],{},"ColumnTransformer"," applies, in parallel, a different transformer to each column group: ",[20,382,383],{},"StandardScaler"," on the 39 numeric ones (centers at mean 0, std 1, essential for Logistic Regression, harmless for a tree) and ",[20,386,387],{},"OneHotEncoder"," on the 3 categorical ones, ",[14,390,392],{"href":391},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic","the same pair that already showed up in Titanic",", now solving network protocol instead of boarding port. One detail worth flagging: ",[20,395,396],{},"handle_unknown=\"ignore\"",". If the test set brings a ",[20,399,171],{}," value the training set never saw, instead of throwing an error, ",[20,402,387],{}," simply zeroes out that entire row for that variable, ",[14,405,407,408,411],{"href":406},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnlp-intro","the same spirit as the ",[20,409,410],{},"[UNK]"," token"," I already saw handling an unknown word.",[11,414,415,416,419,420,423,424,427,428,431,432,434,435,438,439,441],{},"This cell doesn't return a chart or a number, it just ",[61,417,418],{},"assembles"," the transformer. And that's exactly where its most important point lives: I don't call ",[20,421,422],{},".fit()"," on this ",[20,425,426],{},"preprocessador"," by itself, it goes whole into a ",[20,429,430],{},"Pipeline",", together with the classifier. That matters because, in the cross-validation coming up next, each fold trains its own ",[20,433,383],{}," using only that fold's training slice, the mean and standard deviation never see the validation slice. If I'd run ",[20,436,437],{},"preprocessador.fit_transform(X_train)"," before splitting into folds, the mean and std computed over the whole training set would leak statistical information from the validation fold into training, a subtle leak (much smaller than the IP one, but real) that ",[20,440,430],{}," avoids for free.",[36,443,445],{"id":444},"comparing-four-candidates-on-a-quick-sample","Comparing four candidates, on a quick sample",[11,447,448,449,452,453,456,457,460,461,464,465,468],{},"Before choosing a model, I defined the criterion ",[61,450,451],{},"before seeing any numbers",": priority on ",[61,454,455],{},"F1-score"," (balance between not letting attacks through and not generating false alarms), with ",[61,458,459],{},"AUC"," as the tiebreaker. Since classes are imbalanced, I use ",[20,462,463],{},"class_weight=\"balanced\""," wherever the model accepts it, and ",[61,466,467],{},"stratified"," cross-validation (keeps the class proportion in every fold).",[11,470,471],{},"I tested four quite different families, each carrying a different hypothesis about the data:",[153,473,474,480,486,492],{},[156,475,476,479],{},[61,477,478],{},"Logistic Regression:"," assumes a linear decision boundary, a simple, interpretable baseline.",[156,481,482,485],{},[61,483,484],{},"Decision Tree:"," captures nonlinear relationships, but tends to overfit on its own.",[156,487,488,491],{},[61,489,490],{},"Random Forest:"," aggregates many trees, reducing variance via bagging.",[156,493,494,497],{},[61,495,496],{},"Gaussian Naive Bayes:"," assumes independence between variables given the class, fast, a baseline.",[103,499,501],{"className":105,"code":500,"language":107,"meta":108,"style":108},"modelos = {\n    \"Regressao Logistica\": LogisticRegression(max_iter=1000, class_weight=\"balanced\"),\n    \"Arvore de Decisao\": DecisionTreeClassifier(random_state=42, class_weight=\"balanced\"),\n    \"Random Forest\": RandomForestClassifier(n_estimators=100, random_state=42, class_weight=\"balanced\", n_jobs=-1),\n    \"Naive Bayes\": GaussianNB(),\n}\n\nscore = lambda clf, m: cross_val_score(Pipeline([(\"prep\", preprocessador), (\"clf\", clf)]), Xs, ys, cv=cv, scoring=m, n_jobs=-1).mean()\nlinhas = [{\"Modelo\": n, \"F1\": score(c, \"f1\"), \"AUC\": score(c, \"roc_auc\")} for n, c in modelos.items()]\n",[20,502,503,508,513,518,523,528,533,537,542],{"__ignoreMap":108},[112,504,505],{"class":114,"line":115},[112,506,507],{},"modelos = {\n",[112,509,510],{"class":114,"line":121},[112,511,512],{},"    \"Regressao Logistica\": LogisticRegression(max_iter=1000, class_weight=\"balanced\"),\n",[112,514,515],{"class":114,"line":285},[112,516,517],{},"    \"Arvore de Decisao\": DecisionTreeClassifier(random_state=42, class_weight=\"balanced\"),\n",[112,519,520],{"class":114,"line":291},[112,521,522],{},"    \"Random Forest\": RandomForestClassifier(n_estimators=100, random_state=42, class_weight=\"balanced\", n_jobs=-1),\n",[112,524,525],{"class":114,"line":298},[112,526,527],{},"    \"Naive Bayes\": GaussianNB(),\n",[112,529,530],{"class":114,"line":304},[112,531,532],{},"}\n",[112,534,535],{"class":114,"line":365},[112,536,295],{"emptyLinePlaceholder":294},[112,538,539],{"class":114,"line":371},[112,540,541],{},"score = lambda clf, m: cross_val_score(Pipeline([(\"prep\", preprocessador), (\"clf\", clf)]), Xs, ys, cv=cv, scoring=m, n_jobs=-1).mean()\n",[112,543,545],{"class":114,"line":544},9,[112,546,547],{},"linhas = [{\"Modelo\": n, \"F1\": score(c, \"f1\"), \"AUC\": score(c, \"roc_auc\")} for n, c in modelos.items()]\n",[11,549,550,551,554,555,558,559,562,563,566,567,570,571,573,574,576,577,580,581,583,584,587,588,591],{},"Worth opening this block up in slow motion, it packs quite a bit into two lines. ",[20,552,553],{},"modelos"," is just a dictionary, name → classifier instance, the four candidates I want to compare. ",[20,556,557],{},"score"," is a one-line function (a ",[20,560,561],{},"lambda","): it takes a classifier ",[20,564,565],{},"clf"," and a metric name ",[20,568,569],{},"m",", builds a brand-new ",[20,572,430],{}," gluing the same ",[20,575,426],{}," in front of ",[61,578,579],{},"that specific"," classifier (each candidate needs its own pipeline, you can't reuse a pipeline already fit for a different model), runs 3-fold cross-validation on that pipeline (on a sample of 20 thousand training sessions, running this on the full 82 thousand for every candidate would burn time needlessly at this screening stage), and averages the 3 scores. The line below calls that ",[20,582,557],{}," function twice for each of the 4 models, once asking for ",[20,585,586],{},"\"f1\"",", once for ",[20,589,590],{},"\"roc_auc\"",". 4 models × 2 metrics × 3 folds: 24 complete training runs, hiding behind two lines of code.",[593,594],"grouped-bar-chart",{":categories":595,":series":596,"y-label":597},"[\"Logistic Regression\",\"Decision Tree\",\"Random Forest\",\"Naive Bayes\"]","[{\"name\":\"F1 (attack class)\",\"values\":[0.92,0.95,0.97,0.54],\"color\":\"#0033cc\"},{\"name\":\"AUC\",\"values\":[0.98,0.95,1.00,0.72],\"color\":\"#cc3300\"}]","metric",[11,599,600,601,605],{},"Random Forest wins on both metrics (F1 0.97, AUC 1.00 on this screening sample), the Decision Tree alone comes right behind it, and Gaussian Naive Bayes falls well behind the other three (F1 0.54). That tracks for the same reason ",[14,602,604],{"href":603},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpca","I already saw in the breast cancer PCA post",": Naive Bayes assumes the variables are independent of each other given the class, and chapter 1 already showed a good chunk of these 39 numeric variables are anything but independent, several of them telling the same story in different words. The theoretical expectation also holds up: Random Forest combines the trees' low bias with bagging's low variance, exactly the kind of robustness I'd expect to handle well the 39 variables, many of them redundant with each other, that chapter 1 mapped out.",[36,607,609],{"id":608},"closing-this-chapter","Closing this chapter",[11,611,612],{},"With the criterion defined before seeing the numbers (F1 first, AUC as tiebreaker) and the result matching the theoretical expectation, Random Forest is the one chosen for the final training run. In chapter 3 I train it on the entire official training set, evaluate on the official test set (never seen), open up the ROC curve, the decision cutoff, which variables weighed most on the decision, and whether tuning hyperparameters can improve it.",[614,615,616],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":108,"searchDepth":121,"depth":121,"links":618},[619,620,621,622,623],{"id":38,"depth":121,"text":39},{"id":147,"depth":121,"text":148},{"id":327,"depth":121,"text":328},{"id":444,"depth":121,"text":445},{"id":608,"depth":121,"text":609},"2026-08-26","Chapter 2 of my final project for Pattern Recognition: why I give up 2.5 million raw sessions for a smaller, curated partition, the preprocessing pipeline that avoids leakage inside cross-validation, and a fair comparison across four model families before choosing which one to actually train.","md",{},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo","network-intrusion-detection",{"title":6,"description":625},"published","en\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo",[634,635,636,637],"security","scikit-learn","unsw-nb15","pipeline","LruE7gK-T8j5XziZPVB2sr222QXOTis6zrWA4lMNFNE",1788965102308]