[{"data":1,"prerenderedAt":3354},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection":3,"project-en-network-intrusion-detection":4,"project-chapters-en-network-intrusion-detection":72},null,{"id":5,"title":6,"body":7,"cover":3,"description":58,"extension":59,"meta":60,"navigation":61,"order":56,"path":62,"seo":63,"status":64,"stem":65,"tags":66,"__hash__":71},"projects\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002Findex.md","Intrusion Detection: Hunting Attacks in 2.5 Million Network Sessions",{"type":8,"value":9,"toc":54},"minimark",[10,24,43],[11,12,13,14,18,19,23],"p",{},"This project is different even from Olist. I built Olist chapter by chapter, with Professor Boldt around only in the shape of the course he taught. This one is literally my ",[15,16,17],"strong",{},"final project"," for the Pattern Recognition course, the closing exam for everything the rest of this playlist taught: EDA on messy, imbalanced data, preprocessing with a ",[20,21,22],"code",{},"ColumnTransformer",", honest model comparison, evaluation beyond accuracy, GridSearch. Except this time there's no professor showing the way, it was me alone with the dataset and Bishop by my side.",[11,25,26,27,34,35,38,39,42],{},"The dataset is ",[28,29,33],"a",{"href":30,"rel":31},"https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fmrwellsdavid\u002Funsw-nb15",[32],"nofollow","UNSW-NB15",", created by the Australian Centre for Cyber Security (ACCS) by simulating real network traffic mixed with nine different attack families (from brute force to worms), captured from real packets in a lab environment. The task is the most classic one in network security: looking at a network session's characteristics (how long it lasted, how many bytes it exchanged, which protocol it used, and so on), decide whether it's ",[15,36,37],{},"normal"," traffic or an ",[15,40,41],{},"attack",".",[11,44,45,46,50,51,42],{},"Unlike Olist, this isn't independent angles over the same master dataframe, it's an end-to-end flow across three chapters, each one assuming what the last already showed: ",[28,47,49],{"href":48},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes","chapter 1"," is the full exploratory analysis on the raw files, chapter 2 switches datasets and picks the model, chapter 3 closes with the final evaluation, the ROC curve, and hyperparameter tuning. Start with ",[28,52,53],{"href":48},"chapter 1 here",{"title":55,"searchDepth":56,"depth":56,"links":57},"",2,[],"My final project for the Pattern Recognition course: a network traffic classifier trained on UNSW-NB15, from the raw CSV (2.5 million sessions) to a Random Forest hitting 98.3% AUC on the official test set.","md",{},true,"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection",{"title":6,"description":58},"published","en\u002Fprojects\u002Fnetwork-intrusion-detection\u002Findex",[67,68,69,70],"scikit-learn","random-forest","security","anomaly-detection","uMaSR56Q8Nu3f5E7RJCOQqii3xDXXw6rxfZgKcb5lJo",[73,1520,2088],{"id":74,"title":75,"body":76,"cover":3,"date":1509,"description":1510,"extension":59,"meta":1511,"navigation":61,"order":129,"path":48,"project":1512,"seo":1513,"status":64,"stem":1514,"tags":1515,"__hash__":1519},"projectChapters\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes.md","Exploring 2.5 Million Network Sessions",{"type":8,"value":77,"toc":1498},[78,95,98,103,117,168,185,208,305,354,369,390,400,439,461,470,477,485,489,496,501,550,554,557,591,630,795,817,821,824,941,961,968,972,1006,1031,1078,1119,1123,1143,1148,1151,1165,1169,1175,1210,1259,1282,1286,1293,1315,1322,1378,1389,1400,1404,1407,1491,1494],[11,79,80,83,84,38,86,88,89,94],{},[28,81,82],{"href":62},"As I said in the project's opening page",", this is my final project for the Pattern Recognition course, no professor showing the way this time. Think of a network session like a phone call between two computers: it has a start and an end, a duration, an amount of information exchanged on each side (the bytes), and a shared \"language\" the two sides agreed on (the protocol, like TCP or UDP). The mission for the whole project: given that small packet of information about the call, decide whether it's ",[15,85,37],{},[15,87,41],{},". Full, executed notebook ",[28,90,93],{"href":91,"rel":92},"https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1tDmkJZOjd4yia-iPxcajnsjOSeaJJGGs?usp=sharing",[32],"here",", all three chapters come from it.",[11,96,97],{},"This first chapter is just the exploratory analysis (EDA). Before training any model I need to understand what I'm working with: how imbalanced the dataset is, where data is missing and why, whether any column is dirty, which variables tell the same story (redundancy), and how to decide, with a real criterion, which ones go into the model. Chapter 2 switches datasets (I explain why) and trains the candidates, chapter 3 closes with the final evaluation, the ROC curve, and hyperparameter tuning.",[99,100,102],"h2",{"id":101},"the-dataset-25-million-sessions-almost-9-in-10-are-normal-traffic","The dataset: 2.5 million sessions, almost 9 in 10 are normal traffic",[11,104,105,108,109,112,113,116],{},[28,106,33],{"href":30,"rel":107},[32]," was born from a real lab: the Australian Centre for Cyber Security generated network traffic mixing normal browsing with nine different attack families running on purpose, and captured everything at the raw packet level. The first practical problem: the four raw CSV files (",[20,110,111],{},"UNSW-NB15_1.csv"," through ",[20,114,115],{},"_4.csv",") carry no column names at all, just raw numbers. The names live separately, in a fifth file, the feature \"catalog\":",[118,119,123],"pre",{"className":120,"code":121,"language":122,"meta":55,"style":55},"language-python shiki shiki-themes github-light github-dark","features = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FNUSW-NB15_features.csv\", encoding=\"latin-1\")\nfeatures.columns = features.columns.str.strip()\npd.set_option(\"display.max_colwidth\", None)\npd.set_option(\"display.max_rows\", None)\n\nprint(f\"Total de features descritas: {len(features)}\")\ndisplay(features[[\"No.\", \"Name\", \"Type\", \"Description\"]])\n","python",[20,124,125,133,138,144,150,156,162],{"__ignoreMap":55},[126,127,130],"span",{"class":128,"line":129},"line",1,[126,131,132],{},"features = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FNUSW-NB15_features.csv\", encoding=\"latin-1\")\n",[126,134,135],{"class":128,"line":56},[126,136,137],{},"features.columns = features.columns.str.strip()\n",[126,139,141],{"class":128,"line":140},3,[126,142,143],{},"pd.set_option(\"display.max_colwidth\", None)\n",[126,145,147],{"class":128,"line":146},4,[126,148,149],{},"pd.set_option(\"display.max_rows\", None)\n",[126,151,153],{"class":128,"line":152},5,[126,154,155],{"emptyLinePlaceholder":61},"\n",[126,157,159],{"class":128,"line":158},6,[126,160,161],{},"print(f\"Total de features descritas: {len(features)}\")\n",[126,163,165],{"class":128,"line":164},7,[126,166,167],{},"display(features[[\"No.\", \"Name\", \"Type\", \"Description\"]])\n",[169,170,171],"blockquote",{},[11,172,173,176,177,180,181,184],{},[15,174,175],{},"Output:"," ",[20,178,179],{},"Total de features descritas: 49",", followed by a table with the number, name, declared type, and description of each of the 49 columns (for instance, ",[20,182,183],{},"sttl"," shows up described as \"Source to destination time to live value\").",[11,186,187,190,191,194,195,198,199,202,203,207],{},[20,188,189],{},"encoding=\"latin-1\""," isn't decoration: this catalog's CSV has characters outside standard UTF-8 (accents and symbols from whoever documented the dataset), and without the right encoding ",[20,192,193],{},"read_csv"," breaks. The two ",[20,196,197],{},"pd.set_option"," lines exist only for this specific display: without them, pandas truncates the ",[20,200,201],{},"Description"," column (which has long sentences) and cuts the table off with \"...\" partway through the 49 rows, the opposite of what I want when the whole point is to ",[204,205,206],"em",{},"read"," each feature's description. This catalog is what I use next to actually name the columns:",[118,209,211],{"className":120,"code":210,"language":122,"meta":55,"style":55},"import re\n\n# Filtragem apenas dos arquivos de dados -> UNSW-NB15_1.csv até UNSW-NB15_4.csv\npartes = sorted([\n    f for f in os.listdir(\"\u002Fcontent\u002Fdata\")\n    if re.match(r\"UNSW-NB15_\\d+\\.csv\", f)\n])\n\n# Extraindo os nomes das colunas do arquivo de features\ncolunas = features[\"Name\"].tolist()\n\n# Carregando e concatenando todas as partes em um único DataFrame\ndf = pd.concat(\n    [pd.read_csv(f\"\u002Fcontent\u002Fdata\u002F{p}\", header=None, names=colunas, low_memory=False)\n     for p in partes],\n    ignore_index=True\n)\n",[20,212,213,218,222,227,232,237,242,247,252,258,264,269,275,281,287,293,299],{"__ignoreMap":55},[126,214,215],{"class":128,"line":129},[126,216,217],{},"import re\n",[126,219,220],{"class":128,"line":56},[126,221,155],{"emptyLinePlaceholder":61},[126,223,224],{"class":128,"line":140},[126,225,226],{},"# Filtragem apenas dos arquivos de dados -> UNSW-NB15_1.csv até UNSW-NB15_4.csv\n",[126,228,229],{"class":128,"line":146},[126,230,231],{},"partes = sorted([\n",[126,233,234],{"class":128,"line":152},[126,235,236],{},"    f for f in os.listdir(\"\u002Fcontent\u002Fdata\")\n",[126,238,239],{"class":128,"line":158},[126,240,241],{},"    if re.match(r\"UNSW-NB15_\\d+\\.csv\", f)\n",[126,243,244],{"class":128,"line":164},[126,245,246],{},"])\n",[126,248,250],{"class":128,"line":249},8,[126,251,155],{"emptyLinePlaceholder":61},[126,253,255],{"class":128,"line":254},9,[126,256,257],{},"# Extraindo os nomes das colunas do arquivo de features\n",[126,259,261],{"class":128,"line":260},10,[126,262,263],{},"colunas = features[\"Name\"].tolist()\n",[126,265,267],{"class":128,"line":266},11,[126,268,155],{"emptyLinePlaceholder":61},[126,270,272],{"class":128,"line":271},12,[126,273,274],{},"# Carregando e concatenando todas as partes em um único DataFrame\n",[126,276,278],{"class":128,"line":277},13,[126,279,280],{},"df = pd.concat(\n",[126,282,284],{"class":128,"line":283},14,[126,285,286],{},"    [pd.read_csv(f\"\u002Fcontent\u002Fdata\u002F{p}\", header=None, names=colunas, low_memory=False)\n",[126,288,290],{"class":128,"line":289},15,[126,291,292],{},"     for p in partes],\n",[126,294,296],{"class":128,"line":295},16,[126,297,298],{},"    ignore_index=True\n",[126,300,302],{"class":128,"line":301},17,[126,303,304],{},")\n",[11,306,307,308,311,312,314,315,318,319,322,323,112,325,327,328,331,332,335,336,339,340,339,343,339,346,349,350,353],{},"Worth opening this block up, there's a gotcha hiding in it: without that ",[20,309,310],{},"names=colunas"," inside ",[20,313,193],{},", pandas would number the columns ",[20,316,317],{},"0, 1, 2..."," and I'd lose every column's label. Before that, ",[20,320,321],{},"re.match(r\"UNSW-NB15_\\d+\\.csv\", f)"," filters, out of everything the download brought in, only the four files that matter here (",[20,324,111],{},[20,326,115],{},"), deliberately dropping the feature catalog itself and the official partition files (",[20,329,330],{},"UNSW_NB15_training-set.csv"," and so on, which I only use starting in chapter 2) that live in the same folder. ",[20,333,334],{},"sorted(...)"," guarantees the four pieces land in the right order (",[20,337,338],{},"_1",", ",[20,341,342],{},"_2",[20,344,345],{},"_3",[20,347,348],{},"_4","), not whatever order the filesystem happens to list them in. The list comprehension in brackets reads the four files, one at a time, already gluing on the right names as it reads, and ",[20,351,352],{},"pd.concat(..., ignore_index=True)"," stacks the four pieces on top of each other, resetting the row count from zero (without that, each piece would restart at 0, and I'd have the same index repeated four times).",[118,355,357],{"className":120,"code":356,"language":122,"meta":55,"style":55},"print(f\"shape do dataset: \\n {df.shape}\\n\\n\")\ndf.head(5)\n",[20,358,359,364],{"__ignoreMap":55},[126,360,361],{"class":128,"line":129},[126,362,363],{},"print(f\"shape do dataset: \\n {df.shape}\\n\\n\")\n",[126,365,366],{"class":128,"line":56},[126,367,368],{},"df.head(5)\n",[169,370,371],{},[11,372,373,176,375,378,379,339,382,385,386,389],{},[15,374,175],{},[20,376,377],{},"(2540047, 49)",", followed by the first 5 real rows, a UDP session with ",[20,380,381],{},"sport=1390",[20,383,384],{},"dsport=53"," (the DNS port), and ",[20,387,388],{},"dur=0.001055"," seconds, among others.",[11,391,392,393,395,396,399],{},"A cheap, simple sanity check that concatenating the four files and matching up the names actually worked: if ",[20,394,310],{}," had misaligned a column somewhere, ",[20,397,398],{},"df.head()"," would show text where I expected a number, or the other way around, and I'd catch that right away instead of much later in a more serious analysis.",[118,401,403],{"className":120,"code":402,"language":122,"meta":55,"style":55},"print(f\"Linhas: {df.shape[0]:,}\")\nprint(f\"Colunas: {df.shape[1]}\")\nprint(\"Tipagem dos dados:\")\nprint(df.dtypes.value_counts())\n\npd.options.display.float_format = \"{:,.2f}\".format\ndisplay(df.describe().T)\n",[20,404,405,410,415,420,425,429,434],{"__ignoreMap":55},[126,406,407],{"class":128,"line":129},[126,408,409],{},"print(f\"Linhas: {df.shape[0]:,}\")\n",[126,411,412],{"class":128,"line":56},[126,413,414],{},"print(f\"Colunas: {df.shape[1]}\")\n",[126,416,417],{"class":128,"line":140},[126,418,419],{},"print(\"Tipagem dos dados:\")\n",[126,421,422],{"class":128,"line":146},[126,423,424],{},"print(df.dtypes.value_counts())\n",[126,426,427],{"class":128,"line":152},[126,428,155],{"emptyLinePlaceholder":61},[126,430,431],{"class":128,"line":158},[126,432,433],{},"pd.options.display.float_format = \"{:,.2f}\".format\n",[126,435,436],{"class":128,"line":164},[126,437,438],{},"display(df.describe().T)\n",[169,440,441],{},[11,442,443,176,445,448,449,452,453,456,457,460],{},[15,444,175],{},[20,446,447],{},"int64"," for 28 columns, ",[20,450,451],{},"float64"," for 12, ",[20,454,455],{},"object"," for 9. ",[20,458,459],{},"df.describe().T"," (transposed, one row per numeric column) returns count, mean, std, min, quartiles, and max for all 40 numeric columns at once.",[11,462,463,464,466,467,469],{},"That third group, the 9 ",[20,465,455],{}," columns, deserves suspicion, because a column landing as ",[20,468,455],{}," doesn't necessarily mean \"this column is really text.\" Sometimes it's pandas saying \"I found something in here that doesn't fit a clean number,\" and later in this chapter that's exactly the clue that leads to a bug.",[471,472],"category-bar-chart",{":points":473,"color":474,"x-label":475,"y-label":476},"[{\"label\":\"Normal\",\"value\":2218764},{\"label\":\"Attack\",\"value\":321283}]","#0033cc","class","number of sessions",[11,478,479,480,484],{},"2,218,764 normal sessions against 321,283 attack ones, a ratio of nearly 7 to 1. I've seen this kind of imbalance before ",[28,481,483],{"href":482},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","in credit card fraud detection",", there it was 0.17% fraud, here it's 12.6% attack, much less extreme, but still imbalanced enough for accuracy alone to lie: just guessing \"normal\" every time, a lazy model already gets 87.4% right without learning anything.",[99,486,488],{"id":487},"what-kind-of-attack-is-in-there","What kind of attack is in there",[11,490,491,492,495],{},"The ",[20,493,494],{},"attack_cat"," column (only filled in when the session is an attack, which is why 87.35% of it is empty, exactly the fraction of normal traffic) says which attack family each malicious session belongs to:",[471,497],{":points":498,"color":499,"x-label":500,"y-label":476},"[{\"label\":\"Generic\",\"value\":215481},{\"label\":\"Exploits\",\"value\":44525},{\"label\":\"Fuzzers\",\"value\":24246},{\"label\":\"DoS\",\"value\":16353},{\"label\":\"Reconnaissance\",\"value\":13987},{\"label\":\"Analysis\",\"value\":2677},{\"label\":\"Backdoor\",\"value\":2329},{\"label\":\"Shellcode\",\"value\":1511},{\"label\":\"Worms\",\"value\":174}]","#cc3300","attack category",[11,502,503,504,507,508,511,512,515,516,339,519,515,522,525,526,515,529,532,533,537,538,541,542,545,546,549],{},"Before I merged these into 9 categories, the raw data actually had 13 different labels, because ",[20,505,506],{},"\" Fuzzers\""," (with a leading space) and ",[20,509,510],{},"\"Fuzzers\""," counted as two separate categories to pandas, same silliness with ",[20,513,514],{},"\"Reconnaissance\"","\u002F",[20,517,518],{},"\" Reconnaissance\"",[20,520,521],{},"\"Backdoor\"",[20,523,524],{},"\"Backdoors\"",", and ",[20,527,528],{},"\"Shellcode\"",[20,530,531],{},"\" Shellcode\"",". The same kind of typo mess ",[28,534,536],{"href":535},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic","I already saw in the Titanic dataset",", except here instead of a missing column it's a category duplicated by a stray space, a ",[20,539,540],{},".strip()"," would fix it. ",[20,543,544],{},"Generic"," dominates by far (215 thousand sessions, two thirds of all attacks), and it's techniques that forge a hash collision to fool encryption, much easier to automate at scale than, say, a ",[20,547,548],{},"Worms"," attack (only 174 sessions, an attack that needs to replicate itself from machine to machine, rarer to happen inside a controlled lab).",[99,551,553],{"id":552},"redundancy-half-the-columns-tell-the-same-story","Redundancy: half the columns tell the same story",[11,555,556],{},"With 40 numeric variables, a natural question is: how many of them actually carry genuinely different information? I ran the Pearson correlation between pairs of numeric variables on a sample of 50 thousand sessions (computing this on the full 2.5 million would be needlessly expensive):",[118,558,560],{"className":120,"code":559,"language":122,"meta":55,"style":55},"sample = df[colunas_numericas].sample(n=50000, random_state=42)\ncorr_matrix = sample.corr()\n\nmask_pairs = np.triu(np.ones_like(corr_matrix, dtype=bool), k=1)\ncorr_pairs = corr_matrix.where(mask_pairs).stack()\nfortes = corr_pairs[corr_pairs.abs() >= 0.8].sort_values(key=abs, ascending=False)\n",[20,561,562,567,572,576,581,586],{"__ignoreMap":55},[126,563,564],{"class":128,"line":129},[126,565,566],{},"sample = df[colunas_numericas].sample(n=50000, random_state=42)\n",[126,568,569],{"class":128,"line":56},[126,570,571],{},"corr_matrix = sample.corr()\n",[126,573,574],{"class":128,"line":140},[126,575,155],{"emptyLinePlaceholder":61},[126,577,578],{"class":128,"line":146},[126,579,580],{},"mask_pairs = np.triu(np.ones_like(corr_matrix, dtype=bool), k=1)\n",[126,582,583],{"class":128,"line":152},[126,584,585],{},"corr_pairs = corr_matrix.where(mask_pairs).stack()\n",[126,587,588],{"class":128,"line":158},[126,589,590],{},"fortes = corr_pairs[corr_pairs.abs() >= 0.8].sort_values(key=abs, ascending=False)\n",[11,592,593,594,597,598,339,601,604,605,608,609,604,611,613,614,617,618,621,622,625,626,629],{},"That last block is worth taking apart, because ",[20,595,596],{},"corr_matrix"," on its own is a duplicate trap: it's a 40×40 table, every variable crossed against every other, but it's ",[15,599,600],{},"symmetric",[20,602,603],{},"sbytes"," crossed with ",[20,606,607],{},"sloss"," and ",[20,610,607],{},[20,612,603],{}," are the same number showing up twice. Counting that twice inflates the list for nothing. ",[20,615,616],{},"np.triu(..., k=1)"," draws a \"grid\" that marks only the upper triangle of the table as true, skipping the diagonal (the ",[20,619,620],{},"k=1",", otherwise I'd count every variable correlated with itself, always 1.0). ",[20,623,624],{},"corr_matrix.where(mask_pairs)"," erases (turns blank) everything that fell outside that grid, leaving only the unique half of the table. ",[20,627,628],{},".stack()"," finds that half and \"unfolds\" it into a long list of pairs, dropping the blanks on its own. All that's left is filtering for strong correlation (|r| ≥ 0.8) and sorting strongest to weakest, exactly what the last line does:",[631,632,633,648],"table",{},[634,635,636],"thead",{},[637,638,639,644],"tr",{},[640,641,643],"th",{"align":642},"left","Variable pair",[640,645,647],{"align":646},"right","Correlation",[649,650,651,666,678,691,702,713,725,737,750,761,773,784],"tbody",{},[637,652,653,663],{},[654,655,656,659,660],"td",{"align":642},[20,657,658],{},"Stime"," × ",[20,661,662],{},"Ltime",[654,664,665],{"align":646},"1.00",[637,667,668,676],{},[654,669,670,659,673],{"align":642},[20,671,672],{},"swin",[20,674,675],{},"dwin",[654,677,665],{"align":646},[637,679,680,688],{},[654,681,682,659,685],{"align":642},[20,683,684],{},"dloss",[20,686,687],{},"Dpkts",[654,689,690],{"align":646},"0.99",[637,692,693,700],{},[654,694,695,659,698],{"align":642},[20,696,697],{},"dbytes",[20,699,684],{},[654,701,690],{"align":646},[637,703,704,710],{},[654,705,706,659,708],{"align":642},[20,707,603],{},[20,709,607],{},[654,711,712],{"align":646},"0.96",[637,714,715,723],{},[654,716,717,659,720],{"align":642},[20,718,719],{},"ct_srv_src",[20,721,722],{},"ct_srv_dst",[654,724,712],{"align":646},[637,726,727,735],{},[654,728,729,659,732],{"align":642},[20,730,731],{},"ct_dst_ltm",[20,733,734],{},"ct_src_dport_ltm",[654,736,712],{"align":646},[637,738,739,747],{},[654,740,741,659,744],{"align":642},[20,742,743],{},"tcprtt",[20,745,746],{},"synack",[654,748,749],{"align":646},"0.92",[637,751,752,759],{},[654,753,754,659,756],{"align":642},[20,755,743],{},[20,757,758],{},"ackdat",[654,760,749],{"align":646},[637,762,763,770],{},[654,764,765,659,767],{"align":642},[20,766,183],{},[20,768,769],{},"ct_state_ttl",[654,771,772],{"align":646},"0.91",[637,774,775,782],{},[654,776,777,659,779],{"align":642},[20,778,183],{},[20,780,781],{},"Label",[654,783,772],{"align":646},[637,785,786,792],{},[654,787,788,659,790],{"align":642},[20,789,769],{},[20,791,781],{},[654,793,794],{"align":646},"0.88",[11,796,797,798,608,800,802,803,806,807,809,810,813,814,816],{},"Some of these pairs are redundant by construction, not coincidence: ",[20,799,658],{},[20,801,662],{}," are the start and end instants of the same session, so a 1.00 correlation is almost automatic (the difference between the two is literally ",[20,804,805],{},"dur",", another column). But ",[20,808,183],{}," (the TTL, \"time to live,\" the counter every router along the path decrements until the packet expires) correlating ",[15,811,812],{},"0.91 directly with the attack label"," is the most interesting finding of the whole EDA: automated attack tooling tends to generate packets with a TTL noticeably different from normal browser traffic, a technical \"accent\" that by itself already carries a lot of information about who's on the other end of the connection. I'm keeping that name, ",[20,815,183],{},", in mind, it comes back in the next two chapters.",[99,818,820],{"id":819},"boxplots-by-class-a-visual-check-before-the-models-own-importance","Boxplots by class: a visual check before the model's own importance",[11,822,823],{},"Before any model enters the picture, a simpler question already helps anticipate which variables separate normal traffic from attacks well: does a feature's distribution visibly shift between the two classes?",[118,825,827],{"className":120,"code":826,"language":122,"meta":55,"style":55},"features_box = [\"dur\", \"sbytes\", \"dbytes\", \"Sload\", \"Dload\"]\n\nfig, axes = plt.subplots(1, len(features_box), figsize=(18, 5))\n\nfor i, feat in enumerate(features_box):\n    sns.boxplot(\n        data=df,\n        x=\"Label\",\n        y=feat,\n        ax=axes[i],\n        palette=[\"steelblue\", \"tomato\"],\n        showfliers=False,   # tirando outliers extremos para melhor visualização\n        hue=\"Label\",\n        legend=False\n    )\n    axes[i].set_title(feat)\n    axes[i].set_xticks([0, 1])\n    axes[i].set_xticklabels([\"Normal\", \"Ataque\"])\n\nplt.suptitle(\"Distribuição de Features por Classe\", y=1.02)\nplt.tight_layout()\nplt.show()\n",[20,828,829,834,838,843,847,852,857,862,867,872,877,882,887,892,897,902,907,912,918,923,929,935],{"__ignoreMap":55},[126,830,831],{"class":128,"line":129},[126,832,833],{},"features_box = [\"dur\", \"sbytes\", \"dbytes\", \"Sload\", \"Dload\"]\n",[126,835,836],{"class":128,"line":56},[126,837,155],{"emptyLinePlaceholder":61},[126,839,840],{"class":128,"line":140},[126,841,842],{},"fig, axes = plt.subplots(1, len(features_box), figsize=(18, 5))\n",[126,844,845],{"class":128,"line":146},[126,846,155],{"emptyLinePlaceholder":61},[126,848,849],{"class":128,"line":152},[126,850,851],{},"for i, feat in enumerate(features_box):\n",[126,853,854],{"class":128,"line":158},[126,855,856],{},"    sns.boxplot(\n",[126,858,859],{"class":128,"line":164},[126,860,861],{},"        data=df,\n",[126,863,864],{"class":128,"line":249},[126,865,866],{},"        x=\"Label\",\n",[126,868,869],{"class":128,"line":254},[126,870,871],{},"        y=feat,\n",[126,873,874],{"class":128,"line":260},[126,875,876],{},"        ax=axes[i],\n",[126,878,879],{"class":128,"line":266},[126,880,881],{},"        palette=[\"steelblue\", \"tomato\"],\n",[126,883,884],{"class":128,"line":271},[126,885,886],{},"        showfliers=False,   # tirando outliers extremos para melhor visualização\n",[126,888,889],{"class":128,"line":277},[126,890,891],{},"        hue=\"Label\",\n",[126,893,894],{"class":128,"line":283},[126,895,896],{},"        legend=False\n",[126,898,899],{"class":128,"line":289},[126,900,901],{},"    )\n",[126,903,904],{"class":128,"line":295},[126,905,906],{},"    axes[i].set_title(feat)\n",[126,908,909],{"class":128,"line":301},[126,910,911],{},"    axes[i].set_xticks([0, 1])\n",[126,913,915],{"class":128,"line":914},18,[126,916,917],{},"    axes[i].set_xticklabels([\"Normal\", \"Ataque\"])\n",[126,919,921],{"class":128,"line":920},19,[126,922,155],{"emptyLinePlaceholder":61},[126,924,926],{"class":128,"line":925},20,[126,927,928],{},"plt.suptitle(\"Distribuição de Features por Classe\", y=1.02)\n",[126,930,932],{"class":128,"line":931},21,[126,933,934],{},"plt.tight_layout()\n",[126,936,938],{"class":128,"line":937},22,[126,939,940],{},"plt.show()\n",[11,942,943,944,339,946,339,948,339,950,339,953,956,957,960],{},"Each subplot compares one feature's distribution (",[20,945,805],{},[20,947,603],{},[20,949,697],{},[20,951,952],{},"Sload",[20,954,955],{},"Dload",") between normal and attack sessions, side by side. The box's middle line is the median, the box itself spans the 1st to 3rd quartile (the middle 50% of the data), and the \"whiskers\" show the typical spread. ",[20,958,959],{},"showfliers=False"," just drops the most extreme outliers from the picture so it doesn't get squashed flat, they're still in the data, they just don't show up in this particular drawing.",[11,962,963,964,967],{},"The reading that matters: when the \"Normal\" box and the \"Attack\" box sit in clearly different value ranges, with little overlap, that variable alone already helps separate the classes. When the two boxes overlap almost entirely, the variable alone discriminates poorly. It's a quick visual check that ",[15,965,966],{},"anticipates"," what the Random Forest's importance confirms quantitatively later in chapter 3, without waiting for the model to finish training to get a first hint.",[99,969,971],{"id":970},"who-the-categorical-columns-are-and-why-thats-already-a-warning-sign","Who the categorical columns are, and why that's already a warning sign",[11,973,974,975,978,979,339,982,339,985,339,988,339,991,339,994,339,997,339,1000,525,1003,1005],{},"Pulling out the columns pandas read as text (",[20,976,977],{},"select_dtypes(include=[\"object\"])","), nine show up: ",[20,980,981],{},"srcip",[20,983,984],{},"sport",[20,986,987],{},"dstip",[20,989,990],{},"dsport",[20,992,993],{},"proto",[20,995,996],{},"state",[20,998,999],{},"service",[20,1001,1002],{},"ct_ftp_cmd",[20,1004,494],{},". Counting the most frequent values of each, two quite different groups appear:",[118,1007,1009],{"className":120,"code":1008,"language":122,"meta":55,"style":55},"categoricas = df.select_dtypes(include=[\"object\"]).columns.tolist()\nfor col in categoricas:\n    print(f\"\\n{col}:\")\n    print(df[col].value_counts().head(10))\n",[20,1010,1011,1016,1021,1026],{"__ignoreMap":55},[126,1012,1013],{"class":128,"line":129},[126,1014,1015],{},"categoricas = df.select_dtypes(include=[\"object\"]).columns.tolist()\n",[126,1017,1018],{"class":128,"line":56},[126,1019,1020],{},"for col in categoricas:\n",[126,1022,1023],{"class":128,"line":140},[126,1024,1025],{},"    print(f\"\\n{col}:\")\n",[126,1027,1028],{"class":128,"line":146},[126,1029,1030],{},"    print(df[col].value_counts().head(10))\n",[11,1032,1033,1035,1036,1039,1040,1043,1044,1046,1047,1050,1051,1054,1055,1058,1059,1061,1062,1065,1066,1069,1070,1073,1074,1077],{},[20,1034,993],{}," is dominated by ",[20,1037,1038],{},"tcp"," (1,495,074 sessions) and ",[20,1041,1042],{},"udp"," (990,435), ",[20,1045,996],{}," by ",[20,1048,1049],{},"FIN"," (1,478,689), ",[20,1052,1053],{},"CON"," (560,588), and ",[20,1056,1057],{},"INT"," (490,471), and ",[20,1060,999],{}," has a huge ",[20,1063,1064],{},"-"," (1,246,397, service not identified by the sniffer), followed by ",[20,1067,1068],{},"dns"," (781,668) and ",[20,1071,1072],{},"http"," (206,273). These three have few categories and no order between them, good candidates for ",[15,1075,1076],{},"one-hot encoding"," further along.",[11,1079,1080,339,1082,339,1084,525,1086,1088,1089,1091,1092,112,1095,1098,1099,608,1101,1103,1104,1107,1108,1110,1111,1114,1115,1118],{},[20,1081,981],{},[20,1083,984],{},[20,1085,987],{},[20,1087,990],{}," are a different species: extremely high-cardinality identifiers. ",[20,1090,981],{}," alone has only 10 unique values in this lab (",[20,1093,1094],{},"59.166.0.0",[20,1096,1097],{},"59.166.0.9",", the attacking\u002Ftraffic-generating machines), but ",[20,1100,984],{},[20,1102,990],{}," have thousands. The reason I don't use any of these four in the model isn't lack of signal, source IP probably separates attack machines very well in this specific lab, it's the opposite: it separates ",[15,1105,1106],{},"too well",", in a way that doesn't generalize. A model trained on ",[20,1109,981],{}," would memorize \"",[20,1112,1113],{},"59.166.0.4"," is always suspicious\" instead of learning the ",[204,1116,1117],{},"behavior"," of an attack, and outside this lab, with completely different IPs, that memorized rule is worthless. I come back to this point with more care in chapter 2, when I explain the dataset switch.",[99,1120,1122],{"id":1121},"missing-data-structural-not-a-collection-failure","Missing data: structural, not a collection failure",[118,1124,1126],{"className":120,"code":1125,"language":122,"meta":55,"style":55},"nulos = df.isnull().sum()\nnulos_percentual = (nulos \u002F len(df) * 100)\nresumo_nulos = pd.DataFrame({\"nulos\": nulos, \"percentual\": nulos_percentual}).sort_values(\"percentual\", ascending=False)\n",[20,1127,1128,1133,1138],{"__ignoreMap":55},[126,1129,1130],{"class":128,"line":129},[126,1131,1132],{},"nulos = df.isnull().sum()\n",[126,1134,1135],{"class":128,"line":56},[126,1136,1137],{},"nulos_percentual = (nulos \u002F len(df) * 100)\n",[126,1139,1140],{"class":128,"line":140},[126,1141,1142],{},"resumo_nulos = pd.DataFrame({\"nulos\": nulos, \"percentual\": nulos_percentual}).sort_values(\"percentual\", ascending=False)\n",[471,1144],{":points":1145,"color":499,"x-label":1146,"y-label":1147},"[{\"label\":\"attack_cat\",\"value\":87.35},{\"label\":\"is_ftp_login\",\"value\":56.29},{\"label\":\"ct_flw_http_mthd\",\"value\":53.08}]","column","% missing",[11,1149,1150],{},"Only three columns out of 49 have real missing values (the other 46 sit at 0.00%). My first version of this chart was a heatmap over 5,000 sampled rows, a classic way to visualize missingness patterns, but with only 3 columns affected out of 49 it turned into a thin strip with basically no information, time wasted staring at a mostly-blank image. I swapped it for a straight bar chart of the percentage missing per column, which answers exactly the question I actually wanted to ask: \"how much is missing in each column?\" That was my first practical lesson of this chapter: the \"correct\" chart in theory isn't always the right chart for your specific data.",[11,1152,1153,1154,1156,1157,1160,1161,1164],{},"And the three real gaps tell a structural story, not a collection problem. ",[20,1155,494],{}," (87.35%) is only filled in when the session is an attack, the 87.35% blank is exactly the normal sessions, the same thing I already explained above. ",[20,1158,1159],{},"is_ftp_login"," (56.29%) and ",[20,1162,1163],{},"ct_flw_http_mthd"," (53.08%) only make sense for FTP and HTTP sessions respectively, a session of another protocol has nothing to fill in there. There's no blind imputation to do here, because the absence itself is already information (this session isn't FTP, for instance), erasing or filling those columns with any value would destroy that signal.",[99,1166,1168],{"id":1167},"a-subtle-bug-when-empty-doesnt-count-as-missing","A subtle bug: when \"empty\" doesn't count as missing",[11,1170,1171,1172,1174],{},"I said only three columns have missing values. But I noticed something odd looking at the values of ",[20,1173,1002],{}," (the number of FTP commands in the session, which should be a number) in the list of categoricals:",[118,1176,1178],{"className":120,"code":1177,"language":122,"meta":55,"style":55},"ct_ftp_cmd:\n     1429879\n0     686192\n0     380306\n1      29605\n1      10472\n",[20,1179,1180,1185,1190,1195,1200,1205],{"__ignoreMap":55},[126,1181,1182],{"class":128,"line":129},[126,1183,1184],{},"ct_ftp_cmd:\n",[126,1186,1187],{"class":128,"line":56},[126,1188,1189],{},"     1429879\n",[126,1191,1192],{"class":128,"line":140},[126,1193,1194],{},"0     686192\n",[126,1196,1197],{"class":128,"line":146},[126,1198,1199],{},"0     380306\n",[126,1201,1202],{"class":128,"line":152},[126,1203,1204],{},"1      29605\n",[126,1206,1207],{"class":128,"line":158},[126,1208,1209],{},"1      10472\n",[11,1211,1212,1213,1216,1217,1220,1221,1224,1225,1227,1228,1230,1231,1233,1234,1237,1238,1241,1242,1245,1246,1249,1250,1252,1253,1255,1256,1258],{},"Two clues here. First: the most common category, with 1,429,879 sessions, has no label at all, it's an ",[15,1214,1215],{},"empty string"," (",[20,1218,1219],{},"\"\"","), not a ",[20,1222,1223],{},"NaN",". And 1,429,879 is exactly the number of missing values in ",[20,1226,1159],{}," from above, meaning every session without an FTP login also has ",[20,1229,1002],{}," unfilled, which makes sense, but pandas never counted it as missing because technically it isn't ",[20,1232,1223],{},", it's an empty string disguised as a valid category. ",[20,1235,1236],{},"df.isnull().sum()"," simply never saw this problem. Second clue: ",[20,1239,1240],{},"\"0\""," shows up ",[15,1243,1244],{},"twice"," in the list, once with 686,192 sessions and once with 380,306, because one has a trailing space glued on (",[20,1247,1248],{},"\"0 \"",") and the other doesn't, the same kind of typo mess I already saw in ",[20,1251,494],{},". Those two things combined (an empty string instead of a null, plus whitespace duplicating a category) are why ",[20,1254,1002],{}," turned into ",[20,1257,455],{}," back in the dtype table: a column that should be purely numeric got \"contaminated\" enough for pandas to give up reading it as a number.",[11,1260,1261,1262,1264,1265,1268,1269,1271,1272,1275,1276,1279,1280,42],{},"The lesson that sticks: ",[20,1263,1236],{}," is a good first filter, but it isn't the final word on what's actually missing. An empty string, a lone space, a generic ",[20,1266,1267],{},"\"-\""," (like the one I already saw in ",[20,1270,999],{},"), all of that sails right past a naive ",[20,1273,1274],{},".isnull()",". Always worth cross-checking with ",[20,1277,1278],{},"df.dtypes"," and being suspicious of any naturally-numeric column that landed as ",[20,1281,455],{},[99,1283,1285],{"id":1284},"selecting-or-not-selecting-features","Selecting (or not selecting) features",[11,1287,1288,1289,1292],{},"With the numeric and categorical EDA in hand, the central question of any project with lots of variables shows up: which ones do I actually use? The course treats that choice as ",[15,1290,1291],{},"feature selection",", and there are three classic families:",[1294,1295,1296,1303,1309],"ul",{},[1297,1298,1299,1302],"li",{},[15,1300,1301],{},"Filter:"," a statistical criterion, independent of any model. The redundancy analysis by correlation above is exactly that: two variables with r ≈ 1.00 carry almost the same information, one would be a candidate to drop.",[1297,1304,1305,1308],{},[15,1306,1307],{},"Wrapper:"," testing subsets of variables by actually running the model (recursive feature elimination, for instance). Powerful, but expensive, because every subset tested is a full training run.",[1297,1310,1311,1314],{},[15,1312,1313],{},"Embedded:"," the model itself scores each variable's importance while it trains. That's what the Random Forest's importance does, and I get there in chapter 3.",[11,1316,1317,1318,1321],{},"Before deciding anything, I ran a histogram on eight numeric features chosen only to inspect the ",[204,1319,1320],{},"shape"," of the distributions, not to rank importance:",[118,1323,1325],{"className":120,"code":1324,"language":122,"meta":55,"style":55},"features_ilustrativas = [\"dur\", \"sbytes\", \"dbytes\", \"sttl\", \"dttl\", \"sloss\", \"dloss\", \"Sload\"]\n\naxes = df[features_ilustrativas].hist(figsize=(14, 8), bins=50, color=\"steelblue\",\n                                      edgecolor=\"black\", log=True)\nfor ax in axes.ravel():\n    ax.set_xlabel(\"Valor da feature\")\n    ax.set_ylabel(\"Frequencia (escala log)\")\n\nplt.suptitle(\"Distribuicao de Features Numericas (amostra ilustrativa)\", y=1.02)\nplt.tight_layout()\nplt.show()\n",[20,1326,1327,1332,1336,1341,1346,1351,1356,1361,1365,1370,1374],{"__ignoreMap":55},[126,1328,1329],{"class":128,"line":129},[126,1330,1331],{},"features_ilustrativas = [\"dur\", \"sbytes\", \"dbytes\", \"sttl\", \"dttl\", \"sloss\", \"dloss\", \"Sload\"]\n",[126,1333,1334],{"class":128,"line":56},[126,1335,155],{"emptyLinePlaceholder":61},[126,1337,1338],{"class":128,"line":140},[126,1339,1340],{},"axes = df[features_ilustrativas].hist(figsize=(14, 8), bins=50, color=\"steelblue\",\n",[126,1342,1343],{"class":128,"line":146},[126,1344,1345],{},"                                      edgecolor=\"black\", log=True)\n",[126,1347,1348],{"class":128,"line":152},[126,1349,1350],{},"for ax in axes.ravel():\n",[126,1352,1353],{"class":128,"line":158},[126,1354,1355],{},"    ax.set_xlabel(\"Valor da feature\")\n",[126,1357,1358],{"class":128,"line":164},[126,1359,1360],{},"    ax.set_ylabel(\"Frequencia (escala log)\")\n",[126,1362,1363],{"class":128,"line":249},[126,1364,155],{"emptyLinePlaceholder":61},[126,1366,1367],{"class":128,"line":254},[126,1368,1369],{},"plt.suptitle(\"Distribuicao de Features Numericas (amostra ilustrativa)\", y=1.02)\n",[126,1371,1372],{"class":128,"line":260},[126,1373,934],{},[126,1375,1376],{"class":128,"line":266},[126,1377,940],{},[11,1379,1380,1381,1384,1385,1388],{},"The comment I left next to this cell in the original notebook is worth repeating here, because it heads off a wrong reading: these eight features are ",[15,1382,1383],{},"illustrative",", a sample to inspect the shape of the distribution, not the result of any importance-based selection (the real importance comes from the correlation I already covered, and from the Random Forest's own importance in chapter 3). The Y axis uses a log scale (",[20,1386,1387],{},"log=True",") on purpose: bytes transferred and session duration have extremely skewed distributions, most sessions are short and light, a minority are far heavier, and on a linear scale that minority would flatten the majority into a line at zero.",[11,1390,1391,1392,1395,1396,1399],{},"With that visual check, plus the boxplots by class and the correlation already covered above, it's worth writing down the decision I made: ",[15,1393,1394],{},"I'm not pruning any feature in the pipeline."," The Random Forest I use in chapter 3 is reasonably robust to redundant variables and to differences in scale between them, so I treat this selection as ",[15,1397,1398],{},"interpretive analysis"," (understanding what weighs on the decision) rather than a step that actually changes what enters the model. That's a deliberate choice, not laziness: a Logistic Regression or a KNN would probably benefit from cutting the redundancy found here, but for the model I choose in the next chapter, the gain wouldn't be worth the extra complexity of deciding \"which of these two nearly-identical columns do I cut.\"",[99,1401,1403],{"id":1402},"main-takeaways-before-modeling","Main takeaways, before modeling",[11,1405,1406],{},"Closing this first chapter, the points I'm carrying with me for the rest of the project:",[631,1408,1409,1419],{},[634,1410,1411],{},[637,1412,1413,1416],{},[640,1414,1415],{"align":642},"What the EDA showed",[640,1417,1418],{"align":642},"Why it matters later",[649,1420,1421,1429,1443,1455,1467,1475],{},[637,1422,1423,1426],{},[654,1424,1425],{"align":642},"12.6% attack class imbalance",[654,1427,1428],{"align":642},"Accuracy alone lies, F1 and AUC are the metrics I'll actually trust",[637,1430,1431,1440],{},[654,1432,1433,339,1435,525,1437,1439],{"align":642},[20,1434,494],{},[20,1436,1159],{},[20,1438,1163],{}," have structural nulls",[654,1441,1442],{"align":642},"I won't impute blindly, the absence itself is already signal",[637,1444,1445,1450],{},[654,1446,1447,1449],{"align":642},[20,1448,1002],{}," was hiding missing data as an empty string",[654,1451,1452,1454],{"align":642},[20,1453,1274],{}," alone isn't enough, always cross-check with dtype",[637,1456,1457,1464],{},[654,1458,1459,608,1461,1463],{"align":642},[20,1460,494],{},[20,1462,1002],{}," had a category duplicated by a stray space",[654,1465,1466],{"align":642},"Fixed before any count was worth trusting",[637,1468,1469,1472],{},[654,1470,1471],{"align":642},"Half of the 40 numeric variables are redundant with each other (correlation ≥ 0.8)",[654,1473,1474],{"align":642},"Random Forest tolerates this, but recording the redundancy matters for interpreting later",[637,1476,1477,1488],{},[654,1478,1479,515,1481,515,1483,515,1485,1487],{"align":642},[20,1480,981],{},[20,1482,987],{},[20,1484,984],{},[20,1486,990],{}," separate too well, in a way that doesn't generalize",[654,1489,1490],{"align":642},"Dropped in chapter 2, along with the full explanation why",[11,1492,1493],{},"In chapter 2 I swap the raw dataset I used this whole chapter for the official partition the UNSW-NB15 authors themselves curated and published, explain in detail why I'm making that switch (it's about data leakage, not a whim), build the preprocessing pipeline, and compare four model families on a fair cross-validation before choosing which one to actually train.",[1495,1496,1497],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":55,"searchDepth":56,"depth":56,"links":1499},[1500,1501,1502,1503,1504,1505,1506,1507,1508],{"id":101,"depth":56,"text":102},{"id":487,"depth":56,"text":488},{"id":552,"depth":56,"text":553},{"id":819,"depth":56,"text":820},{"id":970,"depth":56,"text":971},{"id":1121,"depth":56,"text":1122},{"id":1167,"depth":56,"text":1168},{"id":1284,"depth":56,"text":1285},{"id":1402,"depth":56,"text":1403},"2026-08-26","Chapter 1 of my final project for Pattern Recognition: the full exploratory analysis of UNSW-NB15 on the raw files, class imbalance, missing data, a quality bug I only caught because I doubted the wrong dtype, redundancy between variables, and the deliberate choice not to prune a single feature.",{},"network-intrusion-detection",{"title":75,"description":1510},"en\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes",[69,1516,1517,1518],"eda","unsw-nb15","feature-selection","25YG0ook_ie6Jt5189imcX4PrOnhEFFS8pvlVCTXZ-w",{"id":1521,"title":1522,"body":1523,"cover":3,"date":1509,"description":2080,"extension":59,"meta":2081,"navigation":61,"order":56,"path":2082,"project":1512,"seo":2083,"status":64,"stem":2084,"tags":2085,"__hash__":2087},"projectChapters\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo.md","Switching Datasets and Picking a Candidate",{"type":8,"value":1524,"toc":2073},[1525,1540,1544,1570,1588,1595,1598,1613,1620,1631,1635,1638,1724,1737,1771,1782,1788,1792,1836,1872,1901,1905,1928,1931,1957,2006,2050,2056,2064,2068,2071],[11,1526,1527,1530,1531,339,1533,339,1535,525,1537,1539],{},[28,1528,1529],{"href":48},"In chapter 1"," I explored the 2.5 million raw UNSW-NB15 sessions: class imbalance, missing data, a hidden quality bug, redundancy between variables, and the decision not to prune any feature. I closed by flagging a problem I hadn't solved yet: ",[20,1532,981],{},[20,1534,987],{},[20,1536,984],{},[20,1538,990],{}," separate too well, in a way that doesn't generalize. This chapter starts by solving exactly that, switching datasets, and ends by choosing, with a real criterion, which model family to train.",[99,1541,1543],{"id":1542},"why-switch-datasets-data-leakage","Why switch datasets: data leakage",[11,1545,1546,1547,339,1549,339,1551,339,1553,1555,1556,515,1558,1560,1561,1216,1564,608,1566,1569],{},"UNSW-NB15 comes in two formats. One is what I used the entire previous chapter: the four raw CSVs, with all 49 fields, including ",[20,1548,981],{},[20,1550,987],{},[20,1552,984],{},[20,1554,990],{},", and the ",[20,1557,658],{},[20,1559,662],{}," timestamps. The other is the ",[15,1562,1563],{},"official partition",[20,1565,330],{},[20,1567,1568],{},"UNSW_NB15_testing-set.csv","), a subset the dataset's own authors (Moustafa & Slay, 2015) already split into train and test, without those identifiers.",[11,1571,1572,1573,1576,1577,1579,1580,1582,1583,515,1585,1587],{},"I choose the official partition for modeling, and the central reason is ",[15,1574,1575],{},"data leakage",": an IP or a port identifies a specific machine in this lab, not a pattern of attack behavior. If I let ",[20,1578,981],{}," into the model, it would learn something like \"",[20,1581,1113],{}," is always suspicious\" instead of learning what makes an attack look like an attack (session duration, byte volume, TTL, and so on). That artificially inflates performance inside this specific lab and doesn't generalize to any network outside it. The ",[20,1584,658],{},[20,1586,662],{}," timestamps have the same problem: flows close together in time tend to be correlated with each other (the same attack session generating several rows in a row), and using the raw timestamp would let that correlation leak into the model.",[11,1589,1590,1591,1594],{},"There's a second reason, less about leakage and more about scientific method: the official partition is the ",[15,1592,1593],{},"standard benchmark in the literature"," for this dataset. Training and evaluating on it makes my result comparable to other published work using the same dataset, something I'd lose if I invented my own random split over the raw files, and I'd run the additional risk of leakage between train and test from correlated flows (same host, same session) landing on both sides of the split.",[11,1596,1597],{},"The switch has a real cost: I'm giving up 2.5 million rows for a much smaller set.",[118,1599,1601],{"className":120,"code":1600,"language":122,"meta":55,"style":55},"df_train = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_training-set.csv\")\ndf_test  = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n",[20,1602,1603,1608],{"__ignoreMap":55},[126,1604,1605],{"class":128,"line":129},[126,1606,1607],{},"df_train = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_training-set.csv\")\n",[126,1609,1610],{"class":128,"line":56},[126,1611,1612],{},"df_test  = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n",[169,1614,1615],{},[11,1616,1617,1619],{},[15,1618,175],{}," training with 82,332 rows, testing with 175,341, each with 45 columns.",[11,1621,1622,1623,1626,1627,1630],{},"Notice the test set is ",[15,1624,1625],{},"larger"," than the training set here, the opposite of what I'd expect from a common 80\u002F20 split. UNSW-NB15's authors deliberately designed a generous test set, precisely to give a more trustworthy measure of generalization. And there's another inversion hiding in there: in the official partition, the attack class becomes the ",[15,1628,1629],{},"majority"," in the test set (further along, in chapter 3, this shows up in the numbers: 119,341 attack sessions against 56,000 normal ones), the opposite proportion I saw in the raw-file EDA. It's a reminder that EDA and modeling, here, don't run over the same dataframe, and one's proportions don't carry over to the other.",[99,1632,1634],{"id":1633},"the-official-partitions-columns","The official partition's columns",[11,1636,1637],{},"The partition's 45 columns aren't the same 49 from the raw files. The authors already removed the identifiers and the timestamps, and organized what's left into groups that make sense to read together:",[1294,1639,1640,1654,1681,1703,1713],{},[1297,1641,1642,176,1645,1647,1648,339,1650,339,1652,42],{},[15,1643,1644],{},"Basic flow:",[20,1646,805],{}," (duration), ",[20,1649,993],{},[20,1651,999],{},[20,1653,996],{},[1297,1655,1656,176,1659,515,1662,1665,1666,515,1668,1670,1671,339,1674,515,1677,1680],{},[15,1657,1658],{},"Volume and rate:",[20,1660,1661],{},"spkts",[20,1663,1664],{},"dpkts"," (packets), ",[20,1667,603],{},[20,1669,697],{}," (bytes), ",[20,1672,1673],{},"rate",[20,1675,1676],{},"sload",[20,1678,1679],{},"dload"," (load).",[1297,1682,1683,176,1686,515,1688,1691,1692,515,1694,1696,1697,339,1699,339,1701,42],{},[15,1684,1685],{},"TCP and timing:",[20,1687,183],{},[20,1689,1690],{},"dttl"," (the TTL that already showed up in chapter 1), ",[20,1693,672],{},[20,1695,675],{}," (TCP window), ",[20,1698,743],{},[20,1700,746],{},[20,1702,758],{},[1297,1704,1705,1712],{},[15,1706,1707,1708,1711],{},"Connection counters (",[20,1709,1710],{},"ct_...","):"," how many recent connections share a service, host, or port, the variable family specifically designed to catch scanning and repetition.",[1297,1714,1715,176,1718,1721,1722,42],{},[15,1716,1717],{},"Targets:",[20,1719,1720],{},"label"," (0\u002F1) and ",[20,1723,494],{},[11,1725,1726,1727,1730,1731,339,1733,339,1735,1711],{},"Of the 42 input columns (dropping ",[20,1728,1729],{},"id"," and the two targets), 39 are numeric and 3 are categorical (",[20,1732,993],{},[20,1734,999],{},[20,1736,996],{},[118,1738,1740],{"className":120,"code":1739,"language":122,"meta":55,"style":55},"DESCARTAR = [\"id\", \"attack_cat\", \"label\"]\nX_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])\ny_train = df_train[\"label\"]\n\ncat_cols = X_train.select_dtypes(include=[\"object\"]).columns.tolist()\nnum_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n",[20,1741,1742,1747,1752,1757,1761,1766],{"__ignoreMap":55},[126,1743,1744],{"class":128,"line":129},[126,1745,1746],{},"DESCARTAR = [\"id\", \"attack_cat\", \"label\"]\n",[126,1748,1749],{"class":128,"line":56},[126,1750,1751],{},"X_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])\n",[126,1753,1754],{"class":128,"line":140},[126,1755,1756],{},"y_train = df_train[\"label\"]\n",[126,1758,1759],{"class":128,"line":146},[126,1760,155],{"emptyLinePlaceholder":61},[126,1762,1763],{"class":128,"line":152},[126,1764,1765],{},"cat_cols = X_train.select_dtypes(include=[\"object\"]).columns.tolist()\n",[126,1767,1768],{"class":128,"line":158},[126,1769,1770],{},"num_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n",[169,1772,1773],{},[11,1774,1775,1777,1778,1781],{},[15,1776,175],{}," categoricals ",[20,1779,1780],{},"['proto', 'service', 'state']",", 39 numeric.",[11,1783,1784,1787],{},[20,1785,1786],{},"select_dtypes"," picks out the 3 categoricals and the 39 numeric columns on its own, no need to type a single column name by hand.",[99,1789,1791],{"id":1790},"building-the-preprocessing-pipeline","Building the preprocessing pipeline",[118,1793,1795],{"className":120,"code":1794,"language":122,"meta":55,"style":55},"from sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\n\ntransformadores = [\n    (\"num\", StandardScaler(), num_cols),\n    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False), cat_cols),\n]\npreprocessador = ColumnTransformer(transformers=transformadores)\n",[20,1796,1797,1802,1807,1811,1816,1821,1826,1831],{"__ignoreMap":55},[126,1798,1799],{"class":128,"line":129},[126,1800,1801],{},"from sklearn.compose import ColumnTransformer\n",[126,1803,1804],{"class":128,"line":56},[126,1805,1806],{},"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",[126,1808,1809],{"class":128,"line":140},[126,1810,155],{"emptyLinePlaceholder":61},[126,1812,1813],{"class":128,"line":146},[126,1814,1815],{},"transformadores = [\n",[126,1817,1818],{"class":128,"line":152},[126,1819,1820],{},"    (\"num\", StandardScaler(), num_cols),\n",[126,1822,1823],{"class":128,"line":158},[126,1824,1825],{},"    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False), cat_cols),\n",[126,1827,1828],{"class":128,"line":164},[126,1829,1830],{},"]\n",[126,1832,1833],{"class":128,"line":249},[126,1834,1835],{},"preprocessador = ColumnTransformer(transformers=transformadores)\n",[11,1837,1838,1840,1841,1844,1845,1848,1849,1852,1853,1856,1857,1859,1860,1862,1863,1871],{},[20,1839,22],{}," applies, in parallel, a different transformer to each column group: ",[20,1842,1843],{},"StandardScaler"," on the 39 numeric ones (centers at mean 0, std 1, essential for Logistic Regression, harmless for a tree) and ",[20,1846,1847],{},"OneHotEncoder"," on the 3 categorical ones, ",[28,1850,1851],{"href":535},"the same pair that already showed up in Titanic",", now solving network protocol instead of boarding port. One detail worth flagging: ",[20,1854,1855],{},"handle_unknown=\"ignore\"",". If the test set brings a ",[20,1858,999],{}," value the training set never saw, instead of throwing an error, ",[20,1861,1847],{}," simply zeroes out that entire row for that variable, ",[28,1864,1866,1867,1870],{"href":1865},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnlp-intro","the same spirit as the ",[20,1868,1869],{},"[UNK]"," token"," I already saw handling an unknown word.",[11,1873,1874,1875,1878,1879,1882,1883,1886,1887,1890,1891,1893,1894,1897,1898,1900],{},"This cell doesn't return a chart or a number, it just ",[15,1876,1877],{},"assembles"," the transformer. And that's exactly where its most important point lives: I don't call ",[20,1880,1881],{},".fit()"," on this ",[20,1884,1885],{},"preprocessador"," by itself, it goes whole into a ",[20,1888,1889],{},"Pipeline",", together with the classifier. That matters because, in the cross-validation coming up next, each fold trains its own ",[20,1892,1843],{}," using only that fold's training slice, the mean and standard deviation never see the validation slice. If I'd run ",[20,1895,1896],{},"preprocessador.fit_transform(X_train)"," before splitting into folds, the mean and std computed over the whole training set would leak statistical information from the validation fold into training, a subtle leak (much smaller than the IP one, but real) that ",[20,1899,1889],{}," avoids for free.",[99,1902,1904],{"id":1903},"comparing-four-candidates-on-a-quick-sample","Comparing four candidates, on a quick sample",[11,1906,1907,1908,1911,1912,1915,1916,1919,1920,1923,1924,1927],{},"Before choosing a model, I defined the criterion ",[15,1909,1910],{},"before seeing any numbers",": priority on ",[15,1913,1914],{},"F1-score"," (balance between not letting attacks through and not generating false alarms), with ",[15,1917,1918],{},"AUC"," as the tiebreaker. Since classes are imbalanced, I use ",[20,1921,1922],{},"class_weight=\"balanced\""," wherever the model accepts it, and ",[15,1925,1926],{},"stratified"," cross-validation (keeps the class proportion in every fold).",[11,1929,1930],{},"I tested four quite different families, each carrying a different hypothesis about the data:",[1294,1932,1933,1939,1945,1951],{},[1297,1934,1935,1938],{},[15,1936,1937],{},"Logistic Regression:"," assumes a linear decision boundary, a simple, interpretable baseline.",[1297,1940,1941,1944],{},[15,1942,1943],{},"Decision Tree:"," captures nonlinear relationships, but tends to overfit on its own.",[1297,1946,1947,1950],{},[15,1948,1949],{},"Random Forest:"," aggregates many trees, reducing variance via bagging.",[1297,1952,1953,1956],{},[15,1954,1955],{},"Gaussian Naive Bayes:"," assumes independence between variables given the class, fast, a baseline.",[118,1958,1960],{"className":120,"code":1959,"language":122,"meta":55,"style":55},"modelos = {\n    \"Regressao Logistica\": LogisticRegression(max_iter=1000, class_weight=\"balanced\"),\n    \"Arvore de Decisao\": DecisionTreeClassifier(random_state=42, class_weight=\"balanced\"),\n    \"Random Forest\": RandomForestClassifier(n_estimators=100, random_state=42, class_weight=\"balanced\", n_jobs=-1),\n    \"Naive Bayes\": GaussianNB(),\n}\n\nscore = lambda clf, m: cross_val_score(Pipeline([(\"prep\", preprocessador), (\"clf\", clf)]), Xs, ys, cv=cv, scoring=m, n_jobs=-1).mean()\nlinhas = [{\"Modelo\": n, \"F1\": score(c, \"f1\"), \"AUC\": score(c, \"roc_auc\")} for n, c in modelos.items()]\n",[20,1961,1962,1967,1972,1977,1982,1987,1992,1996,2001],{"__ignoreMap":55},[126,1963,1964],{"class":128,"line":129},[126,1965,1966],{},"modelos = {\n",[126,1968,1969],{"class":128,"line":56},[126,1970,1971],{},"    \"Regressao Logistica\": LogisticRegression(max_iter=1000, class_weight=\"balanced\"),\n",[126,1973,1974],{"class":128,"line":140},[126,1975,1976],{},"    \"Arvore de Decisao\": DecisionTreeClassifier(random_state=42, class_weight=\"balanced\"),\n",[126,1978,1979],{"class":128,"line":146},[126,1980,1981],{},"    \"Random Forest\": RandomForestClassifier(n_estimators=100, random_state=42, class_weight=\"balanced\", n_jobs=-1),\n",[126,1983,1984],{"class":128,"line":152},[126,1985,1986],{},"    \"Naive Bayes\": GaussianNB(),\n",[126,1988,1989],{"class":128,"line":158},[126,1990,1991],{},"}\n",[126,1993,1994],{"class":128,"line":164},[126,1995,155],{"emptyLinePlaceholder":61},[126,1997,1998],{"class":128,"line":249},[126,1999,2000],{},"score = lambda clf, m: cross_val_score(Pipeline([(\"prep\", preprocessador), (\"clf\", clf)]), Xs, ys, cv=cv, scoring=m, n_jobs=-1).mean()\n",[126,2002,2003],{"class":128,"line":254},[126,2004,2005],{},"linhas = [{\"Modelo\": n, \"F1\": score(c, \"f1\"), \"AUC\": score(c, \"roc_auc\")} for n, c in modelos.items()]\n",[11,2007,2008,2009,2012,2013,2016,2017,2020,2021,2024,2025,2028,2029,2031,2032,2034,2035,2038,2039,2041,2042,2045,2046,2049],{},"Worth opening this block up in slow motion, it packs quite a bit into two lines. ",[20,2010,2011],{},"modelos"," is just a dictionary, name → classifier instance, the four candidates I want to compare. ",[20,2014,2015],{},"score"," is a one-line function (a ",[20,2018,2019],{},"lambda","): it takes a classifier ",[20,2022,2023],{},"clf"," and a metric name ",[20,2026,2027],{},"m",", builds a brand-new ",[20,2030,1889],{}," gluing the same ",[20,2033,1885],{}," in front of ",[15,2036,2037],{},"that specific"," classifier (each candidate needs its own pipeline, you can't reuse a pipeline already fit for a different model), runs 3-fold cross-validation on that pipeline (on a sample of 20 thousand training sessions, running this on the full 82 thousand for every candidate would burn time needlessly at this screening stage), and averages the 3 scores. The line below calls that ",[20,2040,2015],{}," function twice for each of the 4 models, once asking for ",[20,2043,2044],{},"\"f1\"",", once for ",[20,2047,2048],{},"\"roc_auc\"",". 4 models × 2 metrics × 3 folds: 24 complete training runs, hiding behind two lines of code.",[2051,2052],"grouped-bar-chart",{":categories":2053,":series":2054,"y-label":2055},"[\"Logistic Regression\",\"Decision Tree\",\"Random Forest\",\"Naive Bayes\"]","[{\"name\":\"F1 (attack class)\",\"values\":[0.92,0.95,0.97,0.54],\"color\":\"#0033cc\"},{\"name\":\"AUC\",\"values\":[0.98,0.95,1.00,0.72],\"color\":\"#cc3300\"}]","metric",[11,2057,2058,2059,2063],{},"Random Forest wins on both metrics (F1 0.97, AUC 1.00 on this screening sample), the Decision Tree alone comes right behind it, and Gaussian Naive Bayes falls well behind the other three (F1 0.54). That tracks for the same reason ",[28,2060,2062],{"href":2061},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpca","I already saw in the breast cancer PCA post",": Naive Bayes assumes the variables are independent of each other given the class, and chapter 1 already showed a good chunk of these 39 numeric variables are anything but independent, several of them telling the same story in different words. The theoretical expectation also holds up: Random Forest combines the trees' low bias with bagging's low variance, exactly the kind of robustness I'd expect to handle well the 39 variables, many of them redundant with each other, that chapter 1 mapped out.",[99,2065,2067],{"id":2066},"closing-this-chapter","Closing this chapter",[11,2069,2070],{},"With the criterion defined before seeing the numbers (F1 first, AUC as tiebreaker) and the result matching the theoretical expectation, Random Forest is the one chosen for the final training run. In chapter 3 I train it on the entire official training set, evaluate on the official test set (never seen), open up the ROC curve, the decision cutoff, which variables weighed most on the decision, and whether tuning hyperparameters can improve it.",[1495,2072,1497],{},{"title":55,"searchDepth":56,"depth":56,"links":2074},[2075,2076,2077,2078,2079],{"id":1542,"depth":56,"text":1543},{"id":1633,"depth":56,"text":1634},{"id":1790,"depth":56,"text":1791},{"id":1903,"depth":56,"text":1904},{"id":2066,"depth":56,"text":2067},"Chapter 2 of my final project for Pattern Recognition: why I give up 2.5 million raw sessions for a smaller, curated partition, the preprocessing pipeline that avoids leakage inside cross-validation, and a fair comparison across four model families before choosing which one to actually train.",{},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo",{"title":1522,"description":2080},"en\u002Fprojects\u002Fnetwork-intrusion-detection\u002F02-metodologia-selecao-de-modelo",[69,67,1517,2086],"pipeline","LruE7gK-T8j5XziZPVB2sr222QXOTis6zrWA4lMNFNE",{"id":2089,"title":2090,"body":2091,"cover":3,"date":1509,"description":3347,"extension":59,"meta":3348,"navigation":61,"order":140,"path":3349,"project":1512,"seo":3350,"status":64,"stem":3351,"tags":3352,"__hash__":3353},"projectChapters\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F03-avaliacao-final-e-ajuste.md","Final Evaluation, ROC Curve, and Hyperparameter Tuning",{"type":8,"value":2092,"toc":3338},[2093,2102,2106,2116,2159,2181,2188,2245,2259,2263,2279,2294,2411,2426,2481,2502,2508,2511,2517,2521,2528,2537,2551,2569,2576,2681,2687,2748,2775,2782,2789,2793,2801,2835,2891,2897,2917,2921,2932,3009,3047,3055,3069,3073,3080,3123,3163,3178,3181,3235,3255,3265,3273,3277,3328,3336],[11,2094,2095,2098,2099,42],{},[28,2096,2097],{"href":2082},"In chapter 2"," I defined the criterion before seeing any numbers (F1 first, AUC as tiebreaker) and Random Forest won the comparison across four model families. This closing chapter trains it for real, on the entire official training set, measures it on the official test set (the 175,341 sessions the model never saw), and doesn't stop at accuracy: I open up the decision cutoff, the ROC curve, which variables weighed most, exactly where the model gets it wrong, and whether tuning hyperparameters can improve it. The full, executed notebook, covering all three parts of this project, is linked ",[28,2100,2101],{"href":48},"at the top of chapter 1",[99,2103,2105],{"id":2104},"final-evaluation-random-forest-on-the-real-official-test-set","Final evaluation: Random Forest, on the real official test set",[11,2107,2108,2109,2111,2112,2115],{},"I trained the Random Forest (200 trees, with ",[20,2110,1922],{},", a way of telling the model that missing an attack, the rare class, costs more than missing a normal session, without needing to duplicate or drop a single row ",[28,2113,2114],{"href":482},"the way resampling tried to do back with credit card fraud",") on the entire official training set and measured it on the official test set, the 175,341 records the model never saw:",[118,2117,2119],{"className":120,"code":2118,"language":122,"meta":55,"style":55},"modelo_final = Pipeline([\n    (\"prep\", preprocessador),\n    (\"clf\", RandomForestClassifier(n_estimators=200, random_state=42, class_weight=\"balanced\", n_jobs=-1)),\n])\nmodelo_final.fit(X_train, y_train)\ny_pred = modelo_final.predict(X_test)\n\nprint(classification_report(y_test, y_pred, digits=4))\n",[20,2120,2121,2126,2131,2136,2140,2145,2150,2154],{"__ignoreMap":55},[126,2122,2123],{"class":128,"line":129},[126,2124,2125],{},"modelo_final = Pipeline([\n",[126,2127,2128],{"class":128,"line":56},[126,2129,2130],{},"    (\"prep\", preprocessador),\n",[126,2132,2133],{"class":128,"line":140},[126,2134,2135],{},"    (\"clf\", RandomForestClassifier(n_estimators=200, random_state=42, class_weight=\"balanced\", n_jobs=-1)),\n",[126,2137,2138],{"class":128,"line":146},[126,2139,246],{},[126,2141,2142],{"class":128,"line":152},[126,2143,2144],{},"modelo_final.fit(X_train, y_train)\n",[126,2146,2147],{"class":128,"line":158},[126,2148,2149],{},"y_pred = modelo_final.predict(X_test)\n",[126,2151,2152],{"class":128,"line":164},[126,2153,155],{"emptyLinePlaceholder":61},[126,2155,2156],{"class":128,"line":249},[126,2157,2158],{},"print(classification_report(y_test, y_pred, digits=4))\n",[11,2160,2161,2162,2164,2165,2167,2168,2170,2171,2173,2174,2176,2177,2180],{},"The same ",[20,2163,1889],{}," as always, now with the entire ",[20,2166,1885],{}," (the 39 ",[20,2169,1843],{}," plus 3 ",[20,2172,1847],{},") glued in front of the final Random Forest, a single ",[20,2175,1881],{}," training both stages together. ",[20,2178,2179],{},"classification_report"," is what computes precision, recall, and F1 per class in one shot, the same table right below.",[2182,2183],"heatmap-chart",{":cells":2184,":x-categories":2185,":y-categories":2185,"x-label":2186,"y-label":2187},"[{\"x\":\"Normal\",\"y\":\"Normal\",\"value\":54651},{\"x\":\"Attack\",\"y\":\"Normal\",\"value\":1349},{\"x\":\"Normal\",\"y\":\"Attack\",\"value\":15429},{\"x\":\"Attack\",\"y\":\"Attack\",\"value\":103912}]","[\"Normal\",\"Attack\"]","predicted","actual",[631,2189,2190,2209],{},[634,2191,2192],{},[637,2193,2194,2197,2200,2203,2206],{},[640,2195,2196],{"align":642},"Class",[640,2198,2199],{"align":646},"Precision",[640,2201,2202],{"align":646},"Recall",[640,2204,2205],{"align":646},"F1",[640,2207,2208],{"align":646},"Sessions in test",[649,2210,2211,2228],{},[637,2212,2213,2216,2219,2222,2225],{},[654,2214,2215],{"align":642},"Normal",[654,2217,2218],{"align":646},"0.7798",[654,2220,2221],{"align":646},"0.9759",[654,2223,2224],{"align":646},"0.8669",[654,2226,2227],{"align":646},"56,000",[637,2229,2230,2233,2236,2239,2242],{},[654,2231,2232],{"align":642},"Attack",[654,2234,2235],{"align":646},"0.9872",[654,2237,2238],{"align":646},"0.8707",[654,2240,2241],{"align":646},"0.9253",[654,2243,2244],{"align":646},"119,341",[11,2246,2247,2248,2251,2252,2255,2256,42],{},"Precision here is: of the sessions the model flagged as attack, how many really were. Recall is the opposite: of the real attacks, how many the model caught. F1 folds both into one score. Overall accuracy is 90.43%, but the number I actually trust is the attack class's recall: ",[15,2249,2250],{},"87.07%",", meaning the model catches 103,912 of the 119,341 real attacks in the test set, and lets 15,429 slip through. In security, this kind of false negative (an attack classified as normal) is the expensive mistake, it's literally the intruder who walks past unnoticed. On the other side, 1,349 normal sessions turned into false alarms, annoying for whoever investigates them, but much less serious than letting a real attack through. ",[15,2253,2254],{},"AUC on the official test set lands at 0.9830",", close to the 1.0 ceiling, a sign the model knows how to rank sessions by risk well even before any decision threshold enters the picture, ",[28,2257,2258],{"href":482},"the same ranking metric I already saw with credit card fraud",[99,2260,2262],{"id":2261},"dialing-the-cutoff-precision-against-recall","Dialing the cutoff: precision against recall",[11,2264,2265,2266,2269,2270,2273,2274,2278],{},"All that evaluation used scikit-learn's default cutoff: ",[20,2267,2268],{},".predict()"," decides \"attack\" whenever the risk score (the fraction of trees that voted attack, the same number you can get from ",[20,2271,2272],{},".predict_proba()",") crosses 0.5. But that 0.5 is just one arbitrary point on a whole line of options, the same threshold idea ",[28,2275,2277],{"href":2276},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fclassification-threshold","I already saw in the classification-with-regression lesson",", except now you can actually turn the dial.",[11,2280,2281,2282,2285,2286,2289,2290,2293],{},"For that, I wrote a second notebook (",[20,2283,2284],{},"datasets\u002Funsw-nb15\u002F02_dados_extras_scatter_e_threshold.ipynb","), only because the original notebook never saved the continuous ",[20,2287,2288],{},"y_score",", just the ",[20,2291,2292],{},"y_pred"," already thresholded at 0.5. This notebook's first cells rebuild, cell by cell, exactly the same pipeline from chapters 2 and 3:",[118,2295,2297],{"className":120,"code":2296,"language":122,"meta":55,"style":55},"df_train = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_training-set.csv\")\ndf_test = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n\nDESCARTAR = [\"id\", \"attack_cat\", \"label\"]\nX_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])\ny_train = df_train[\"label\"]\nX_test = df_test.drop(columns=[c for c in DESCARTAR if c in df_test.columns])\ny_test = df_test[\"label\"]\n\ncat_cols = X_train.select_dtypes(include=[\"object\"]).columns.tolist()\nnum_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n\ntransformadores = [\n    (\"num\", StandardScaler(), num_cols),\n    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False), cat_cols),\n]\npreprocessador = ColumnTransformer(transformers=transformadores)\n\nmodelo_final = Pipeline([\n    (\"prep\", preprocessador),\n    (\"clf\", RandomForestClassifier(n_estimators=200, random_state=42, class_weight=\"balanced\", n_jobs=-1)),\n])\nmodelo_final.fit(X_train, y_train)\n\ny_pred = modelo_final.predict(X_test)\ny_score = modelo_final.predict_proba(X_test)[:, 1]\n",[20,2298,2299,2303,2308,2312,2316,2320,2324,2329,2334,2338,2342,2346,2350,2354,2358,2362,2366,2370,2374,2378,2382,2386,2390,2395,2400,2405],{"__ignoreMap":55},[126,2300,2301],{"class":128,"line":129},[126,2302,1607],{},[126,2304,2305],{"class":128,"line":56},[126,2306,2307],{},"df_test = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n",[126,2309,2310],{"class":128,"line":140},[126,2311,155],{"emptyLinePlaceholder":61},[126,2313,2314],{"class":128,"line":146},[126,2315,1746],{},[126,2317,2318],{"class":128,"line":152},[126,2319,1751],{},[126,2321,2322],{"class":128,"line":158},[126,2323,1756],{},[126,2325,2326],{"class":128,"line":164},[126,2327,2328],{},"X_test = df_test.drop(columns=[c for c in DESCARTAR if c in df_test.columns])\n",[126,2330,2331],{"class":128,"line":249},[126,2332,2333],{},"y_test = df_test[\"label\"]\n",[126,2335,2336],{"class":128,"line":254},[126,2337,155],{"emptyLinePlaceholder":61},[126,2339,2340],{"class":128,"line":260},[126,2341,1765],{},[126,2343,2344],{"class":128,"line":266},[126,2345,1770],{},[126,2347,2348],{"class":128,"line":271},[126,2349,155],{"emptyLinePlaceholder":61},[126,2351,2352],{"class":128,"line":277},[126,2353,1815],{},[126,2355,2356],{"class":128,"line":283},[126,2357,1820],{},[126,2359,2360],{"class":128,"line":289},[126,2361,1825],{},[126,2363,2364],{"class":128,"line":295},[126,2365,1830],{},[126,2367,2368],{"class":128,"line":301},[126,2369,1835],{},[126,2371,2372],{"class":128,"line":914},[126,2373,155],{"emptyLinePlaceholder":61},[126,2375,2376],{"class":128,"line":920},[126,2377,2125],{},[126,2379,2380],{"class":128,"line":925},[126,2381,2130],{},[126,2383,2384],{"class":128,"line":931},[126,2385,2135],{},[126,2387,2388],{"class":128,"line":937},[126,2389,246],{},[126,2391,2393],{"class":128,"line":2392},23,[126,2394,2144],{},[126,2396,2398],{"class":128,"line":2397},24,[126,2399,155],{"emptyLinePlaceholder":61},[126,2401,2403],{"class":128,"line":2402},25,[126,2404,2149],{},[126,2406,2408],{"class":128,"line":2407},26,[126,2409,2410],{},"y_score = modelo_final.predict_proba(X_test)[:, 1]\n",[11,2412,2413,2414,2416,2417,2419,2420,2422,2423,2425],{},"Cell by cell, this is the same code from chapter 2 (the official partition, the ",[20,2415,22],{},") and from the start of this chapter (the ",[20,2418,1889],{}," with the 200-tree Random Forest), no new line anywhere, just rebuilt here because this notebook runs isolated, in a Colab runtime that doesn't carry over the other two notebooks' state. The only difference is the last line: besides ",[20,2421,2292],{},", I also keep ",[20,2424,2288],{},", the continuous probability, so I can test cutoffs other than 0.5 in the next block. With the pipeline retrained, the new block sweeps 19 different cutoffs:",[118,2427,2429],{"className":120,"code":2428,"language":122,"meta":55,"style":55},"thresholds = np.round(np.arange(0.05, 1.0, 0.05), 2)\nthreshold_sweep = []\nfor t in thresholds:\n    y_pred_t = (y_score >= t).astype(int)\n    threshold_sweep.append({\n        \"threshold\": float(t),\n        \"precision\": round(float(precision_score(y_test, y_pred_t, zero_division=0)), 4),\n        \"recall\": round(float(recall_score(y_test, y_pred_t, zero_division=0)), 4),\n        \"f1\": round(float(f1_score(y_test, y_pred_t, zero_division=0)), 4),\n    })\n",[20,2430,2431,2436,2441,2446,2451,2456,2461,2466,2471,2476],{"__ignoreMap":55},[126,2432,2433],{"class":128,"line":129},[126,2434,2435],{},"thresholds = np.round(np.arange(0.05, 1.0, 0.05), 2)\n",[126,2437,2438],{"class":128,"line":56},[126,2439,2440],{},"threshold_sweep = []\n",[126,2442,2443],{"class":128,"line":140},[126,2444,2445],{},"for t in thresholds:\n",[126,2447,2448],{"class":128,"line":146},[126,2449,2450],{},"    y_pred_t = (y_score >= t).astype(int)\n",[126,2452,2453],{"class":128,"line":152},[126,2454,2455],{},"    threshold_sweep.append({\n",[126,2457,2458],{"class":128,"line":158},[126,2459,2460],{},"        \"threshold\": float(t),\n",[126,2462,2463],{"class":128,"line":164},[126,2464,2465],{},"        \"precision\": round(float(precision_score(y_test, y_pred_t, zero_division=0)), 4),\n",[126,2467,2468],{"class":128,"line":249},[126,2469,2470],{},"        \"recall\": round(float(recall_score(y_test, y_pred_t, zero_division=0)), 4),\n",[126,2472,2473],{"class":128,"line":254},[126,2474,2475],{},"        \"f1\": round(float(f1_score(y_test, y_pred_t, zero_division=0)), 4),\n",[126,2477,2478],{"class":128,"line":260},[126,2479,2480],{},"    })\n",[11,2482,2483,2486,2487,2490,2491,2493,2494,2497,2498,2501],{},[20,2484,2485],{},"np.arange(0.05, 1.0, 0.05)"," generates the 19 cutoffs (0.05, 0.10, ..., 0.95), and the loop tests each one: ",[20,2488,2489],{},"y_score >= t"," turns the continuous score into a binary label at that specific cutoff (the same comparison ",[20,2492,2268],{}," does internally, except fixing ",[20,2495,2496],{},"t"," instead of always using 0.5), and all three metrics get recomputed from scratch on every pass. ",[20,2499,2500],{},"zero_division=0"," avoids an error in case some extreme cutoff zeroes out an entire class. The result, 19 rows of precision\u002Frecall\u002FF1, is what feeds the interactive chart below:",[2503,2504],"intrusion-threshold-explorer",{"precision-label":2505,"recall-label":2506,"threshold-label":2507},"precision","recall","model cutoff",[11,2509,2510],{},"Drag the slider and watch the trade. Dropping the cutoff to 0.05 (flag \"attack\" on any suspicion at all, however small) pushes recall to 0.9781, catching almost everyone, but drags precision down to 0.9234, more false alarms for the security team to chase for nothing. Pushing the cutoff up to 0.95 (only flag \"attack\" when the model is nearly certain) flips it: precision of 0.999, almost no false alarms, but recall collapses to 0.6951, letting nearly a third of real attacks through. There's no single right cutoff, separate from the business question. If chasing a false alarm is cheap, it's worth pulling the cutoff down and eating more false alarms in exchange for catching more real attacks. If every investigation is expensive, the cutoff goes up.",[11,2512,2513,2514,2516],{},"At the cutoff I used through the whole post (0.5), pairing ",[20,2515,1922],{}," with that default cutoff already delivers 0.8715 recall without me touching anything by hand (the numbers match, off by a thousandth, the table further up, since I rebuilt the pipeline from scratch in this separate notebook). That's why I never went chasing a better cutoff for the final evaluation: the class imbalance was already being compensated for during training itself, not just at decision time.",[99,2518,2520],{"id":2519},"the-roc-curve-the-same-cutoff-seen-from-another-angle","The ROC curve: the same cutoff, seen from another angle",[11,2522,2523,2524,2527],{},"The confusion matrix and the precision\u002Frecall table above evaluate the model at a ",[15,2525,2526],{},"single"," decision cutoff (the default, 0.5). But the slider I just showed already gives away the rest of the picture: the Random Forest doesn't just return a label, it estimates a probability P(attack | x) for every session, and the cutoff is an operational security decision, not a fixed mathematical truth.",[11,2529,491,2530,2533,2534,2536],{},[15,2531,2532],{},"ROC curve"," (Receiver Operating Characteristic) turns that trade into a picture: sweeping across every possible cutoff, it plots the true positive rate (the same recall as always) against the false positive rate (the fraction of normal sessions that turned into false alarms). ",[15,2535,1918],{}," (Area Under the Curve, the area under that plot) condenses the whole chart into a single number, with a direct probabilistic reading: it's the chance the model gives a higher score to a randomly picked attack than to a randomly picked normal session. AUC = 1.0 is the perfect classifier, AUC = 0.5 is the same as a coin flip (the chart's dotted diagonal). In the original notebook, that computation is just two lines:",[118,2538,2540],{"className":120,"code":2539,"language":122,"meta":55,"style":55},"y_score = modelo_final.predict_proba(X_test)[:, 1]\nauc = roc_auc_score(y_test, y_score)\n",[20,2541,2542,2546],{"__ignoreMap":55},[126,2543,2544],{"class":128,"line":129},[126,2545,2410],{},[126,2547,2548],{"class":128,"line":56},[126,2549,2550],{},"auc = roc_auc_score(y_test, y_score)\n",[11,2552,2553,2556,2557,2560,2561,2564,2565,2568],{},[20,2554,2555],{},"predict_proba"," returns two columns, the probability of each class, ",[20,2558,2559],{},"[:, 1]"," grabs just class 1's column (attack). ",[20,2562,2563],{},"roc_auc_score"," takes the true label and that continuous score, no fixed cutoff involved at all, and computes AUC directly. The original notebook stopped there and only drew the chart with scikit-learn's own ",[20,2566,2567],{},"RocCurveDisplay.from_predictions(...)",", a static image, never saving the curve's points anywhere.",[11,2570,2571,2572,2575],{},"To get a real, interactive chart here on the blog, I wrote a third notebook (",[20,2573,2574],{},"datasets\u002Funsw-nb15\u002F03_curva_roc.ipynb","), with the same rebuild as always at the start (third time this block shows up in this project by now, so I'll just repeat it, no re-explaining):",[118,2577,2579],{"className":120,"code":2578,"language":122,"meta":55,"style":55},"df_train = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_training-set.csv\")\ndf_test = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FUNSW_NB15_testing-set.csv\")\n\nDESCARTAR = [\"id\", \"attack_cat\", \"label\"]\nX_train = df_train.drop(columns=[c for c in DESCARTAR if c in df_train.columns])\ny_train = df_train[\"label\"]\nX_test = df_test.drop(columns=[c for c in DESCARTAR if c in df_test.columns])\ny_test = df_test[\"label\"]\n\ncat_cols = X_train.select_dtypes(include=[\"object\"]).columns.tolist()\nnum_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n\ntransformadores = [\n    (\"num\", StandardScaler(), num_cols),\n    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False), cat_cols),\n]\npreprocessador = ColumnTransformer(transformers=transformadores)\n\nmodelo_final = Pipeline([\n    (\"prep\", preprocessador),\n    (\"clf\", RandomForestClassifier(n_estimators=200, random_state=42, class_weight=\"balanced\", n_jobs=-1)),\n])\nmodelo_final.fit(X_train, y_train)\n\ny_score = modelo_final.predict_proba(X_test)[:, 1]\n",[20,2580,2581,2585,2589,2593,2597,2601,2605,2609,2613,2617,2621,2625,2629,2633,2637,2641,2645,2649,2653,2657,2661,2665,2669,2673,2677],{"__ignoreMap":55},[126,2582,2583],{"class":128,"line":129},[126,2584,1607],{},[126,2586,2587],{"class":128,"line":56},[126,2588,2307],{},[126,2590,2591],{"class":128,"line":140},[126,2592,155],{"emptyLinePlaceholder":61},[126,2594,2595],{"class":128,"line":146},[126,2596,1746],{},[126,2598,2599],{"class":128,"line":152},[126,2600,1751],{},[126,2602,2603],{"class":128,"line":158},[126,2604,1756],{},[126,2606,2607],{"class":128,"line":164},[126,2608,2328],{},[126,2610,2611],{"class":128,"line":249},[126,2612,2333],{},[126,2614,2615],{"class":128,"line":254},[126,2616,155],{"emptyLinePlaceholder":61},[126,2618,2619],{"class":128,"line":260},[126,2620,1765],{},[126,2622,2623],{"class":128,"line":266},[126,2624,1770],{},[126,2626,2627],{"class":128,"line":271},[126,2628,155],{"emptyLinePlaceholder":61},[126,2630,2631],{"class":128,"line":277},[126,2632,1815],{},[126,2634,2635],{"class":128,"line":283},[126,2636,1820],{},[126,2638,2639],{"class":128,"line":289},[126,2640,1825],{},[126,2642,2643],{"class":128,"line":295},[126,2644,1830],{},[126,2646,2647],{"class":128,"line":301},[126,2648,1835],{},[126,2650,2651],{"class":128,"line":914},[126,2652,155],{"emptyLinePlaceholder":61},[126,2654,2655],{"class":128,"line":920},[126,2656,2125],{},[126,2658,2659],{"class":128,"line":925},[126,2660,2130],{},[126,2662,2663],{"class":128,"line":931},[126,2664,2135],{},[126,2666,2667],{"class":128,"line":937},[126,2668,246],{},[126,2670,2671],{"class":128,"line":2392},[126,2672,2144],{},[126,2674,2675],{"class":128,"line":2397},[126,2676,155],{"emptyLinePlaceholder":61},[126,2678,2679],{"class":128,"line":2402},[126,2680,2410],{},[11,2682,2683,2684,2686],{},"And, on top of that ",[20,2685,2288],{},", extracts the curve point by point:",[118,2688,2690],{"className":120,"code":2689,"language":122,"meta":55,"style":55},"fpr, tpr, thresholds = roc_curve(y_test, y_score)\nauc = roc_auc_score(y_test, y_score)\nprint(f\"AUC (conferencia, deve bater com 0.9830 ja publicado): {auc:.4f}\")\nprint(f\"Curva bruta tem {len(fpr)} pontos, vamos reduzir pra 50 pra plotar no blog\")\n\nfpr_grid = np.linspace(0, 1, 50)\ntpr_grid = np.interp(fpr_grid, fpr, tpr)\n\nroc_curve_points = [\n    {\"fpr\": round(float(f), 4), \"tpr\": round(float(t), 4)}\n    for f, t in zip(fpr_grid, tpr_grid)\n]\n",[20,2691,2692,2697,2701,2706,2711,2715,2720,2725,2729,2734,2739,2744],{"__ignoreMap":55},[126,2693,2694],{"class":128,"line":129},[126,2695,2696],{},"fpr, tpr, thresholds = roc_curve(y_test, y_score)\n",[126,2698,2699],{"class":128,"line":56},[126,2700,2550],{},[126,2702,2703],{"class":128,"line":140},[126,2704,2705],{},"print(f\"AUC (conferencia, deve bater com 0.9830 ja publicado): {auc:.4f}\")\n",[126,2707,2708],{"class":128,"line":146},[126,2709,2710],{},"print(f\"Curva bruta tem {len(fpr)} pontos, vamos reduzir pra 50 pra plotar no blog\")\n",[126,2712,2713],{"class":128,"line":152},[126,2714,155],{"emptyLinePlaceholder":61},[126,2716,2717],{"class":128,"line":158},[126,2718,2719],{},"fpr_grid = np.linspace(0, 1, 50)\n",[126,2721,2722],{"class":128,"line":164},[126,2723,2724],{},"tpr_grid = np.interp(fpr_grid, fpr, tpr)\n",[126,2726,2727],{"class":128,"line":249},[126,2728,155],{"emptyLinePlaceholder":61},[126,2730,2731],{"class":128,"line":254},[126,2732,2733],{},"roc_curve_points = [\n",[126,2735,2736],{"class":128,"line":260},[126,2737,2738],{},"    {\"fpr\": round(float(f), 4), \"tpr\": round(float(t), 4)}\n",[126,2740,2741],{"class":128,"line":266},[126,2742,2743],{},"    for f, t in zip(fpr_grid, tpr_grid)\n",[126,2745,2746],{"class":128,"line":271},[126,2747,1830],{},[11,2749,2750,2753,2754,2756,2757,339,2760,1555,2763,2766,2767,2770,2771,2774],{},[20,2751,2752],{},"roc_curve(y_test, y_score)"," internally sweeps every unique score present in ",[20,2755,2288],{}," as a candidate cutoff, and returns three same-length arrays: ",[20,2758,2759],{},"fpr",[20,2761,2762],{},"tpr",[20,2764,2765],{},"thresholds"," that produced each pair. On this dataset the raw curve came out with 282 points, more granularity than a blog chart needs and a bigger JSON than necessary. ",[20,2768,2769],{},"np.linspace(0, 1, 50)"," creates 50 evenly spaced positions on the false-positive axis, and ",[20,2772,2773],{},"np.interp(fpr_grid, fpr, tpr)"," estimates the real true-positive rate at each of those 50 positions, interpolating between the nearest real points, nothing invented: it's the same curve, just sampled on a grid that's lighter to draw. The confirmation print lands on AUC = 0.9830, matching the number computed in the original notebook exactly.",[2776,2777],"intrusion-roc-chart",{"model-label":2778,"random-label":2779,"x-label":2780,"y-label":2781},"Random Forest (AUC = 0.983)","random (AUC = 0.5)","false positive rate","true positive rate",[11,2783,2784,2785,2788],{},"The curve hugs the top-left corner, far from the dotted diagonal of chance: at a false-positive cutoff of just 2%, recall already jumps to 86%, and the curve stays nearly glued to the ceiling from there on. That shows the model separates the two classes well ",[15,2786,2787],{},"independent of any specific cutoff",", and that's exactly the slack that lets me pull the cutoff down, like I did in the previous section, without precision collapsing along with it.",[99,2790,2792],{"id":2791},"who-decides-the-15-variables-that-matter-most","Who decides: the 15 variables that matter most",[11,2794,2795,2796,2800],{},"Random Forest lets you peek at which variables most reduced impurity across the trees during training, ",[28,2797,2799],{"href":2798},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees","the same Gini yardstick I already saw with decision trees",", just now summed across 200 trees instead of just one:",[118,2802,2804],{"className":120,"code":2803,"language":122,"meta":55,"style":55},"ohe = modelo_final.named_steps[\"prep\"].named_transformers_[\"cat\"]\nnomes_cat = ohe.get_feature_names_out(cat_cols)\nnomes_features = np.concatenate([num_cols, nomes_cat])\n\nimportancias = modelo_final.named_steps[\"clf\"].feature_importances_\nimp = pd.Series(importancias, index=nomes_features).sort_values(ascending=False)\n",[20,2805,2806,2811,2816,2821,2825,2830],{"__ignoreMap":55},[126,2807,2808],{"class":128,"line":129},[126,2809,2810],{},"ohe = modelo_final.named_steps[\"prep\"].named_transformers_[\"cat\"]\n",[126,2812,2813],{"class":128,"line":56},[126,2814,2815],{},"nomes_cat = ohe.get_feature_names_out(cat_cols)\n",[126,2817,2818],{"class":128,"line":140},[126,2819,2820],{},"nomes_features = np.concatenate([num_cols, nomes_cat])\n",[126,2822,2823],{"class":128,"line":146},[126,2824,155],{"emptyLinePlaceholder":61},[126,2826,2827],{"class":128,"line":152},[126,2828,2829],{},"importancias = modelo_final.named_steps[\"clf\"].feature_importances_\n",[126,2831,2832],{"class":128,"line":158},[126,2833,2834],{},"imp = pd.Series(importancias, index=nomes_features).sort_values(ascending=False)\n",[11,2836,2837,2838,2841,2842,2844,2845,339,2847,339,2849,2851,2852,339,2855,339,2858,2861,2862,2865,2866,2869,2870,2872,2873,2876,2877,2880,2881,2883,2884,2886,2887,2890],{},"This block exists to solve an annoying problem: ",[20,2839,2840],{},"feature_importances_"," returns just a raw array of numbers, no name attached, and after the ",[20,2843,22],{}," the three original categorical columns (",[20,2846,993],{},[20,2848,999],{},[20,2850,996],{},") don't even exist anymore, they turned into dozens of new binary columns, like ",[20,2853,2854],{},"proto_tcp",[20,2856,2857],{},"proto_udp",[20,2859,2860],{},"service_dns",". To reconstruct \"which number in the list belongs to which variable,\" I need to go fetch those names back from inside the already-trained pipeline itself: ",[20,2863,2864],{},"modelo_final.named_steps[\"prep\"]"," grabs the preprocessing stage, ",[20,2867,2868],{},".named_transformers_[\"cat\"]"," grabs specifically the ",[20,2871,1847],{}," that ran on the categorical columns, and ",[20,2874,2875],{},".get_feature_names_out(cat_cols)"," asks it \"what were the names of the new columns you created?\" ",[20,2878,2879],{},"np.concatenate([num_cols, nomes_cat])"," glues that list back together with the 39 numeric ones up front (which kept their original names, ",[20,2882,1843],{}," never creates a new column), respecting the same order ",[20,2885,22],{}," uses internally. Only after that does ",[20,2888,2889],{},"pd.Series(importancias, index=nomes_features)"," manage to pair every importance number with the right name.",[471,2892],{":points":2893,"color":2894,"x-label":2895,"y-label":2896},"[{\"label\":\"ct_dst_src_ltm\",\"value\":0.074},{\"label\":\"sttl\",\"value\":0.059},{\"label\":\"sbytes\",\"value\":0.054},{\"label\":\"sload\",\"value\":0.052},{\"label\":\"ct_dst_sport_ltm\",\"value\":0.051},{\"label\":\"smean\",\"value\":0.047},{\"label\":\"ct_srv_dst\",\"value\":0.044},{\"label\":\"ct_state_ttl\",\"value\":0.042},{\"label\":\"rate\",\"value\":0.033},{\"label\":\"dbytes\",\"value\":0.033},{\"label\":\"dload\",\"value\":0.031},{\"label\":\"dur\",\"value\":0.029},{\"label\":\"synack\",\"value\":0.028},{\"label\":\"ct_srv_src\",\"value\":0.027},{\"label\":\"dmean\",\"value\":0.026}]","#33aa55","variable","importance (impurity reduction)",[11,2898,2899,2900,2903,2904,2906,2907,339,2910,339,2912,339,2914,2916],{},"The winner, ",[20,2901,2902],{},"ct_dst_src_ltm",", counts how many of the last 100 connections share the same source-destination pair, a direct way to flag scanning or flooding behavior: normal traffic rarely hits the exact same source-destination pair repeatedly, hundreds of times in a row, automated attacks do. Notice ",[20,2905,183],{}," (the TTL that already showed up in the correlation table) appears again, in second place, confirming that 0.91 correlation with the label wasn't empty statistical coincidence, it's a real signal. And a good chunk of the rest of the list (",[20,2908,2909],{},"ct_dst_sport_ltm",[20,2911,722],{},[20,2913,769],{},[20,2915,719],{},") belongs to the same \"how many recent connections look like this one\" family, the kind of variable specifically designed to catch a burst of repeated connections, the most common signature of automated attack versus human traffic.",[99,2918,2920],{"id":2919},"where-the-model-gets-it-wrong-400-real-sessions-across-the-3-most-important-variables","Where the model gets it wrong: 400 real sessions across the 3 most important variables",[11,2922,2923,2924,608,2926,2928,2929,2931],{},"The top two spots in the importance ranking, ",[20,2925,2902],{},[20,2927,183],{},", plus the third, ",[20,2930,603],{},", fit together in one space. It's the same notebook 02 that generated the threshold sweep, with the same retrained pipeline, that also extracts this sample:",[118,2933,2935],{"className":120,"code":2934,"language":122,"meta":55,"style":55},"FEATURES_3D = [\"sttl\", \"ct_dst_src_ltm\", \"sbytes\"]\nN_SAMPLE = 400\n\namostra_idx = X_test.sample(n=N_SAMPLE, random_state=42).index\ny_pred_series = pd.Series(y_pred, index=X_test.index)\n\nsample_points = []\nfor i in amostra_idx:\n    sample_points.append({\n        \"x\": float(X_test.loc[i, FEATURES_3D[0]]),\n        \"y\": float(X_test.loc[i, FEATURES_3D[1]]),\n        \"z\": float(X_test.loc[i, FEATURES_3D[2]]),\n        \"label_real\": int(y_test.loc[i]),\n        \"label_previsto\": int(y_pred_series.loc[i]),\n    })\n",[20,2936,2937,2942,2947,2951,2956,2961,2965,2970,2975,2980,2985,2990,2995,3000,3005],{"__ignoreMap":55},[126,2938,2939],{"class":128,"line":129},[126,2940,2941],{},"FEATURES_3D = [\"sttl\", \"ct_dst_src_ltm\", \"sbytes\"]\n",[126,2943,2944],{"class":128,"line":56},[126,2945,2946],{},"N_SAMPLE = 400\n",[126,2948,2949],{"class":128,"line":140},[126,2950,155],{"emptyLinePlaceholder":61},[126,2952,2953],{"class":128,"line":146},[126,2954,2955],{},"amostra_idx = X_test.sample(n=N_SAMPLE, random_state=42).index\n",[126,2957,2958],{"class":128,"line":152},[126,2959,2960],{},"y_pred_series = pd.Series(y_pred, index=X_test.index)\n",[126,2962,2963],{"class":128,"line":158},[126,2964,155],{"emptyLinePlaceholder":61},[126,2966,2967],{"class":128,"line":164},[126,2968,2969],{},"sample_points = []\n",[126,2971,2972],{"class":128,"line":249},[126,2973,2974],{},"for i in amostra_idx:\n",[126,2976,2977],{"class":128,"line":254},[126,2978,2979],{},"    sample_points.append({\n",[126,2981,2982],{"class":128,"line":260},[126,2983,2984],{},"        \"x\": float(X_test.loc[i, FEATURES_3D[0]]),\n",[126,2986,2987],{"class":128,"line":266},[126,2988,2989],{},"        \"y\": float(X_test.loc[i, FEATURES_3D[1]]),\n",[126,2991,2992],{"class":128,"line":271},[126,2993,2994],{},"        \"z\": float(X_test.loc[i, FEATURES_3D[2]]),\n",[126,2996,2997],{"class":128,"line":277},[126,2998,2999],{},"        \"label_real\": int(y_test.loc[i]),\n",[126,3001,3002],{"class":128,"line":283},[126,3003,3004],{},"        \"label_previsto\": int(y_pred_series.loc[i]),\n",[126,3006,3007],{"class":128,"line":289},[126,3008,2480],{},[11,3010,3011,3014,3015,3018,3019,3021,3022,3025,3026,3029,3030,3033,3034,339,3036,339,3038,3040,3041,515,3043,3046],{},[20,3012,3013],{},"X_test.sample(n=400, random_state=42)"," draws 400 indices from the official test set, the fixed seed is just so I can reproduce the same sample later. ",[20,3016,3017],{},"y_pred_series"," repackages the raw ",[20,3020,2292],{}," array (which only has position, no index) into a ",[20,3023,3024],{},"Series"," sharing ",[20,3027,3028],{},"X_test","'s index, so a specific session's prediction can be looked up by ",[20,3031,3032],{},".loc[i]"," inside the loop. For each of the 400 sampled sessions, the loop keeps the three features of interest (",[20,3035,183],{},[20,3037,2902],{},[20,3039,603],{},") plus the true label and the predicted one, nothing invented, it's exactly the official ",[20,3042,3028],{},[20,3044,3045],{},"y_test",", just narrowed down to a sample small enough to plot. I plotted them on those 3 axes:",[3048,3049],"intrusion-sample-scatter",{"false-negative-label":3050,"false-positive-label":3051,"true-negative-label":3052,"true-positive-label":3053,"x-label":3054,"y-label":2902,"z-label":603},"false negative (missed attack)","false positive (false alarm)","correct: normal","correct: attack","sttl (source TTL)",[11,3056,3057,3058,3060,3061,3064,3065,3068],{},"In this sample of 400: 247 correct attack calls, 112 correct normal calls, 36 false negatives, and only 5 false positives, a ratio close to the full test set's. One thing that jumps out while spinning the cloud around: ",[20,3059,183],{}," isn't really continuous, it clusters into a handful of round values (31, 62, 254...), because TTL is literally a router-hop counter each operating system initializes at a default value (64, 128, or 255) and decrements by one at every hop. Among the 283 attack sessions in this sample, ",[20,3062,3063],{},"sttl=254"," shows up in 238 of them (84%), but it also shows up in 37 of the 117 normal sessions (32%), the most common value on both sides, far from a clean separator on its own. ",[20,3066,3067],{},"sttl=31",", on the other hand, is almost exclusively normal traffic: 74 of the 117 normal sessions, zero attacks. That's exactly why no single variable decides anything here. You can see the 36 red points (false negatives) scattered through the middle of the cloud, never forming an isolated island, and that's where the other 39 variables the chart doesn't show come in, working together to close the 87% recall the final model delivers.",[99,3070,3072],{"id":3071},"trying-to-improve-with-gridsearch-and-hitting-my-own-ceiling","Trying to improve with GridSearch, and hitting my own ceiling",[11,3074,3075,3076,3079],{},"I ran ",[20,3077,3078],{},"GridSearchCV"," varying the number of trees (200 or 400), max depth (unlimited or capped at 30), and minimum leaf size (1 or 2), 8 combinations, 3-fold cross-validation, on a sample of 30 thousand sessions to keep the cost manageable:",[118,3081,3083],{"className":120,"code":3082,"language":122,"meta":55,"style":55},"param_grid = {\n    \"clf__n_estimators\": [200, 400],\n    \"clf__max_depth\": [None, 30],\n    \"clf__min_samples_leaf\": [1, 2],\n}\n\ngrid = GridSearchCV(pipe_rf, param_grid, scoring=\"f1\", cv=cv, n_jobs=-1, verbose=1)\ngrid.fit(Xg, yg)\n",[20,3084,3085,3090,3095,3100,3105,3109,3113,3118],{"__ignoreMap":55},[126,3086,3087],{"class":128,"line":129},[126,3088,3089],{},"param_grid = {\n",[126,3091,3092],{"class":128,"line":56},[126,3093,3094],{},"    \"clf__n_estimators\": [200, 400],\n",[126,3096,3097],{"class":128,"line":140},[126,3098,3099],{},"    \"clf__max_depth\": [None, 30],\n",[126,3101,3102],{"class":128,"line":146},[126,3103,3104],{},"    \"clf__min_samples_leaf\": [1, 2],\n",[126,3106,3107],{"class":128,"line":152},[126,3108,1991],{},[126,3110,3111],{"class":128,"line":158},[126,3112,155],{"emptyLinePlaceholder":61},[126,3114,3115],{"class":128,"line":164},[126,3116,3117],{},"grid = GridSearchCV(pipe_rf, param_grid, scoring=\"f1\", cv=cv, n_jobs=-1, verbose=1)\n",[126,3119,3120],{"class":128,"line":249},[126,3121,3122],{},"grid.fit(Xg, yg)\n",[11,3124,491,3125,3128,3129,3132,3133,3135,3136,3139,3140,3143,3144,3147,3148,3150,3151,3159,3160,3162],{},[20,3126,3127],{},"clf__"," prefix in front of each key isn't decoration: ",[20,3130,3131],{},"pipe_rf"," is a ",[20,3134,1889],{}," with a stage named ",[20,3137,3138],{},"\"clf\""," (the Random Forest), and the ",[20,3141,3142],{},"__"," is scikit-learn's way of saying \"the ",[20,3145,3146],{},"n_estimators"," parameter belongs to the ",[20,3149,2023],{}," stage, not any other one in the pipeline,\" ",[28,3152,3154,3155,3158],{"href":3153},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation","the same ",[20,3156,3157],{},"model__"," syntax that already showed up in the pipeline and GridSearch post",". Without that prefix, ",[20,3161,3078],{}," wouldn't know which stage to touch.",[169,3164,3165],{},[11,3166,3167,3169,3170,3173,3174,3177],{},[15,3168,175],{}," best combination found, ",[20,3171,3172],{},"n_estimators=200, max_depth=None, min_samples_leaf=1",", with a validation F1 of 0.9716. Exactly the ",[15,3175,3176],{},"default"," parameters of the Random Forest I'd already trained before.",[11,3179,3180],{},"To avoid blindly trusting that validation F1 (computed on a sample of 30 thousand, remember), I retrained the winning configuration on the full official training set and evaluated it on the official test set, the same rigorous way as the final evaluation:",[118,3182,3184],{"className":120,"code":3183,"language":122,"meta":55,"style":55},"melhores = {k.replace(\"clf__\", \"\"): v for k, v in grid.best_params_.items()}\nmodelo_tunado = Pipeline([(\"prep\", preprocessador),\n                          (\"clf\", RandomForestClassifier(random_state=42, class_weight=\"balanced\",\n                                                         n_jobs=-1, **melhores))])\nmodelo_tunado.fit(X_train, y_train)\ny_pred_t = modelo_tunado.predict(X_test)\ny_score_t = modelo_tunado.predict_proba(X_test)[:, 1]\n\nprint(classification_report(y_test, y_pred_t, digits=4))\nprint(f\"AUC-ROC (ajustada): {roc_auc_score(y_test, y_score_t):.4f}\")\n",[20,3185,3186,3191,3196,3201,3206,3211,3216,3221,3225,3230],{"__ignoreMap":55},[126,3187,3188],{"class":128,"line":129},[126,3189,3190],{},"melhores = {k.replace(\"clf__\", \"\"): v for k, v in grid.best_params_.items()}\n",[126,3192,3193],{"class":128,"line":56},[126,3194,3195],{},"modelo_tunado = Pipeline([(\"prep\", preprocessador),\n",[126,3197,3198],{"class":128,"line":140},[126,3199,3200],{},"                          (\"clf\", RandomForestClassifier(random_state=42, class_weight=\"balanced\",\n",[126,3202,3203],{"class":128,"line":146},[126,3204,3205],{},"                                                         n_jobs=-1, **melhores))])\n",[126,3207,3208],{"class":128,"line":152},[126,3209,3210],{},"modelo_tunado.fit(X_train, y_train)\n",[126,3212,3213],{"class":128,"line":158},[126,3214,3215],{},"y_pred_t = modelo_tunado.predict(X_test)\n",[126,3217,3218],{"class":128,"line":164},[126,3219,3220],{},"y_score_t = modelo_tunado.predict_proba(X_test)[:, 1]\n",[126,3222,3223],{"class":128,"line":249},[126,3224,155],{"emptyLinePlaceholder":61},[126,3226,3227],{"class":128,"line":254},[126,3228,3229],{},"print(classification_report(y_test, y_pred_t, digits=4))\n",[126,3231,3232],{"class":128,"line":260},[126,3233,3234],{},"print(f\"AUC-ROC (ajustada): {roc_auc_score(y_test, y_score_t):.4f}\")\n",[11,3236,3237,3240,3241,3243,3244,3247,3248,3251,3252,3254],{},[20,3238,3239],{},"grid.best_params_"," keeps its keys with the ",[20,3242,3127],{}," prefix (the same prefix explained above), so the first line strips that prefix so the values can be passed straight into ",[20,3245,3246],{},"RandomForestClassifier","'s constructor via ",[20,3249,3250],{},"**melhores",". The rest is the same ",[20,3253,1889],{}," as always, just with the winning hyperparameters in place of the defaults, trained on the entire official training set (not the 30-thousand search sample anymore) and evaluated on the official test set.",[169,3256,3257],{},[11,3258,3259,3261,3262,42],{},[15,3260,175],{}," precision, recall, and F1 per class identical to the original Random Forest, decimal place for decimal place, and ",[20,3263,3264],{},"AUC-ROC (ajustada): 0.9830",[11,3266,3267,3268,3272],{},"The result matches, decimal place for decimal place, the original Random Forest: same precision, recall, and F1 per class, same 0.9830 AUC. It's not a search failure, it's the search confirming something worth saying plainly: for this dataset, with this set of candidates tested, scikit-learn's default guess (100 or 200 trees, unlimited depth) was already at the ceiling this hyperparameter family reaches. Not every optimization run finds something better to optimize toward, ",[28,3269,3271],{"href":3270},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fensembles","the same kind of \"boring\" result I already saw in the ensembles post",", when plain logistic regression beat every ensemble no amount of tuning would have changed.",[99,3274,3276],{"id":3275},"wrapping-up-the-project-and-the-whole-playlist","Wrapping up the project (and the whole playlist)",[631,3278,3279,3289],{},[634,3280,3281],{},[637,3282,3283,3286],{},[640,3284,3285],{"align":642},"What I already knew, from an earlier post",[640,3287,3288],{"align":642},"What this final project tied together",[649,3290,3291,3299,3312,3320],{},[637,3292,3293,3296],{},[654,3294,3295],{"align":642},"Imbalanced data fools accuracy",[654,3297,3298],{"align":642},"Here, with 12.6% attack, the minority class's recall matters more than the overall number",[637,3300,3301,3304],{},[654,3302,3303],{"align":642},"High correlation signals redundancy",[654,3305,3306,3307,659,3309,3311],{"align":642},"Correlation between a feature and the label (",[20,3308,183],{},[20,3310,781],{},", 0.91) can signal real signal, not just redundancy between two variables",[637,3313,3314,3317],{},[654,3315,3316],{"align":642},"Random Forest handles nonlinear patterns well",[654,3318,3319],{"align":642},"Here it beat Logistic Regression, a lone Decision Tree, and Naive Bayes on both axes (F1 and AUC)",[637,3321,3322,3325],{},[654,3323,3324],{"align":642},"GridSearch finds the best point in the search space",[654,3326,3327],{"align":642},"Sometimes the best point already is the default, and it's worth finding that out by measuring, not assuming",[11,3329,3330,3331,3335],{},"This project closes a loop for me: I started this playlist ",[28,3332,3334],{"href":3333},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","fitting a line with the normal equation",", went through trees, clustering, dimensionality reduction, text turning into vectors, and ended up applying all of it on my own, no lecture to follow, on a real network security problem, with a real result (0.98 AUC on the official test set) I can defend line by line, from the raw CSV all the way to the final decision tree. Thanks, Professor Boldt, I've got the rest from here.",[1495,3337,1497],{},{"title":55,"searchDepth":56,"depth":56,"links":3339},[3340,3341,3342,3343,3344,3345,3346],{"id":2104,"depth":56,"text":2105},{"id":2261,"depth":56,"text":2262},{"id":2519,"depth":56,"text":2520},{"id":2791,"depth":56,"text":2792},{"id":2919,"depth":56,"text":2920},{"id":3071,"depth":56,"text":3072},{"id":3275,"depth":56,"text":3276},"Chapter 3, the closing chapter of my final project for Pattern Recognition: the Random Forest chosen in chapter 2 evaluated on the official test set (98.3% AUC), the decision cutoff explored end to end, the ROC curve, which variables weighed most, where the model gets it wrong, and finding out that hyperparameter tuning doesn't always beat the default guess.",{},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F03-avaliacao-final-e-ajuste",{"title":2090,"description":3347},"en\u002Fprojects\u002Fnetwork-intrusion-detection\u002F03-avaliacao-final-e-ajuste",[69,68,1517,70],"cM7iP6Lojuo65771MsH9v17M-W_ORqpIi09OM6z00tw",1788965099071]