[{"data":1,"prerenderedAt":1471},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes":3,"chapter-en-network-intrusion-detection-01-explorando-25-milhoes-de-sessoes":4},null,{"id":5,"title":6,"body":7,"cover":3,"date":1456,"description":1457,"extension":1458,"meta":1459,"navigation":100,"order":73,"path":1460,"project":1461,"seo":1462,"status":1463,"stem":1464,"tags":1465,"__hash__":1470},"projectChapters\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes.md","Exploring 2.5 Million Network Sessions",{"type":8,"value":9,"toc":1445},"minimark",[10,35,38,43,60,114,131,154,251,300,315,336,346,385,407,416,423,431,435,442,447,496,500,503,537,576,741,763,767,770,887,907,914,918,952,977,1024,1065,1069,1089,1094,1097,1111,1115,1121,1156,1205,1229,1233,1240,1262,1269,1325,1336,1347,1351,1354,1438,1441],[11,12,13,18,19,23,24,27,28,34],"p",{},[14,15,17],"a",{"href":16},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection","As I said in the project's opening page",", this is my final project for the Pattern Recognition course, no professor showing the way this time. Think of a network session like a phone call between two computers: it has a start and an end, a duration, an amount of information exchanged on each side (the bytes), and a shared \"language\" the two sides agreed on (the protocol, like TCP or UDP). The mission for the whole project: given that small packet of information about the call, decide whether it's ",[20,21,22],"strong",{},"normal"," traffic or an ",[20,25,26],{},"attack",". Full, executed notebook ",[14,29,33],{"href":30,"rel":31},"https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1tDmkJZOjd4yia-iPxcajnsjOSeaJJGGs?usp=sharing",[32],"nofollow","here",", all three chapters come from it.",[11,36,37],{},"This first chapter is just the exploratory analysis (EDA). Before training any model I need to understand what I'm working with: how imbalanced the dataset is, where data is missing and why, whether any column is dirty, which variables tell the same story (redundancy), and how to decide, with a real criterion, which ones go into the model. Chapter 2 switches datasets (I explain why) and trains the candidates, chapter 3 closes with the final evaluation, the ROC curve, and hyperparameter tuning.",[39,40,42],"h2",{"id":41},"the-dataset-25-million-sessions-almost-9-in-10-are-normal-traffic","The dataset: 2.5 million sessions, almost 9 in 10 are normal traffic",[11,44,45,50,51,55,56,59],{},[14,46,49],{"href":47,"rel":48},"https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fmrwellsdavid\u002Funsw-nb15",[32],"UNSW-NB15"," was born from a real lab: the Australian Centre for Cyber Security generated network traffic mixing normal browsing with nine different attack families running on purpose, and captured everything at the raw packet level. The first practical problem: the four raw CSV files (",[52,53,54],"code",{},"UNSW-NB15_1.csv"," through ",[52,57,58],{},"_4.csv",") carry no column names at all, just raw numbers. The names live separately, in a fifth file, the feature \"catalog\":",[61,62,67],"pre",{"className":63,"code":64,"language":65,"meta":66,"style":66},"language-python shiki shiki-themes github-light github-dark","features = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FNUSW-NB15_features.csv\", encoding=\"latin-1\")\nfeatures.columns = features.columns.str.strip()\npd.set_option(\"display.max_colwidth\", None)\npd.set_option(\"display.max_rows\", None)\n\nprint(f\"Total de features descritas: {len(features)}\")\ndisplay(features[[\"No.\", \"Name\", \"Type\", \"Description\"]])\n","python","",[52,68,69,77,83,89,95,102,108],{"__ignoreMap":66},[70,71,74],"span",{"class":72,"line":73},"line",1,[70,75,76],{},"features = pd.read_csv(\"\u002Fcontent\u002Fdata\u002FNUSW-NB15_features.csv\", encoding=\"latin-1\")\n",[70,78,80],{"class":72,"line":79},2,[70,81,82],{},"features.columns = features.columns.str.strip()\n",[70,84,86],{"class":72,"line":85},3,[70,87,88],{},"pd.set_option(\"display.max_colwidth\", None)\n",[70,90,92],{"class":72,"line":91},4,[70,93,94],{},"pd.set_option(\"display.max_rows\", None)\n",[70,96,98],{"class":72,"line":97},5,[70,99,101],{"emptyLinePlaceholder":100},true,"\n",[70,103,105],{"class":72,"line":104},6,[70,106,107],{},"print(f\"Total de features descritas: {len(features)}\")\n",[70,109,111],{"class":72,"line":110},7,[70,112,113],{},"display(features[[\"No.\", \"Name\", \"Type\", \"Description\"]])\n",[115,116,117],"blockquote",{},[11,118,119,122,123,126,127,130],{},[20,120,121],{},"Output:"," ",[52,124,125],{},"Total de features descritas: 49",", followed by a table with the number, name, declared type, and description of each of the 49 columns (for instance, ",[52,128,129],{},"sttl"," shows up described as \"Source to destination time to live value\").",[11,132,133,136,137,140,141,144,145,148,149,153],{},[52,134,135],{},"encoding=\"latin-1\""," isn't decoration: this catalog's CSV has characters outside standard UTF-8 (accents and symbols from whoever documented the dataset), and without the right encoding ",[52,138,139],{},"read_csv"," breaks. The two ",[52,142,143],{},"pd.set_option"," lines exist only for this specific display: without them, pandas truncates the ",[52,146,147],{},"Description"," column (which has long sentences) and cuts the table off with \"...\" partway through the 49 rows, the opposite of what I want when the whole point is to ",[150,151,152],"em",{},"read"," each feature's description. This catalog is what I use next to actually name the columns:",[61,155,157],{"className":63,"code":156,"language":65,"meta":66,"style":66},"import re\n\n# Filtragem apenas dos arquivos de dados -> UNSW-NB15_1.csv até UNSW-NB15_4.csv\npartes = sorted([\n    f for f in os.listdir(\"\u002Fcontent\u002Fdata\")\n    if re.match(r\"UNSW-NB15_\\d+\\.csv\", f)\n])\n\n# Extraindo os nomes das colunas do arquivo de features\ncolunas = features[\"Name\"].tolist()\n\n# Carregando e concatenando todas as partes em um único DataFrame\ndf = pd.concat(\n    [pd.read_csv(f\"\u002Fcontent\u002Fdata\u002F{p}\", header=None, names=colunas, low_memory=False)\n     for p in partes],\n    ignore_index=True\n)\n",[52,158,159,164,168,173,178,183,188,193,198,204,210,215,221,227,233,239,245],{"__ignoreMap":66},[70,160,161],{"class":72,"line":73},[70,162,163],{},"import re\n",[70,165,166],{"class":72,"line":79},[70,167,101],{"emptyLinePlaceholder":100},[70,169,170],{"class":72,"line":85},[70,171,172],{},"# Filtragem apenas dos arquivos de dados -> UNSW-NB15_1.csv até UNSW-NB15_4.csv\n",[70,174,175],{"class":72,"line":91},[70,176,177],{},"partes = sorted([\n",[70,179,180],{"class":72,"line":97},[70,181,182],{},"    f for f in os.listdir(\"\u002Fcontent\u002Fdata\")\n",[70,184,185],{"class":72,"line":104},[70,186,187],{},"    if re.match(r\"UNSW-NB15_\\d+\\.csv\", f)\n",[70,189,190],{"class":72,"line":110},[70,191,192],{},"])\n",[70,194,196],{"class":72,"line":195},8,[70,197,101],{"emptyLinePlaceholder":100},[70,199,201],{"class":72,"line":200},9,[70,202,203],{},"# Extraindo os nomes das colunas do arquivo de features\n",[70,205,207],{"class":72,"line":206},10,[70,208,209],{},"colunas = features[\"Name\"].tolist()\n",[70,211,213],{"class":72,"line":212},11,[70,214,101],{"emptyLinePlaceholder":100},[70,216,218],{"class":72,"line":217},12,[70,219,220],{},"# Carregando e concatenando todas as partes em um único DataFrame\n",[70,222,224],{"class":72,"line":223},13,[70,225,226],{},"df = pd.concat(\n",[70,228,230],{"class":72,"line":229},14,[70,231,232],{},"    [pd.read_csv(f\"\u002Fcontent\u002Fdata\u002F{p}\", header=None, names=colunas, low_memory=False)\n",[70,234,236],{"class":72,"line":235},15,[70,237,238],{},"     for p in partes],\n",[70,240,242],{"class":72,"line":241},16,[70,243,244],{},"    ignore_index=True\n",[70,246,248],{"class":72,"line":247},17,[70,249,250],{},")\n",[11,252,253,254,257,258,260,261,264,265,268,269,55,271,273,274,277,278,281,282,285,286,285,289,285,292,295,296,299],{},"Worth opening this block up, there's a gotcha hiding in it: without that ",[52,255,256],{},"names=colunas"," inside ",[52,259,139],{},", pandas would number the columns ",[52,262,263],{},"0, 1, 2..."," and I'd lose every column's label. Before that, ",[52,266,267],{},"re.match(r\"UNSW-NB15_\\d+\\.csv\", f)"," filters, out of everything the download brought in, only the four files that matter here (",[52,270,54],{},[52,272,58],{},"), deliberately dropping the feature catalog itself and the official partition files (",[52,275,276],{},"UNSW_NB15_training-set.csv"," and so on, which I only use starting in chapter 2) that live in the same folder. ",[52,279,280],{},"sorted(...)"," guarantees the four pieces land in the right order (",[52,283,284],{},"_1",", ",[52,287,288],{},"_2",[52,290,291],{},"_3",[52,293,294],{},"_4","), not whatever order the filesystem happens to list them in. The list comprehension in brackets reads the four files, one at a time, already gluing on the right names as it reads, and ",[52,297,298],{},"pd.concat(..., ignore_index=True)"," stacks the four pieces on top of each other, resetting the row count from zero (without that, each piece would restart at 0, and I'd have the same index repeated four times).",[61,301,303],{"className":63,"code":302,"language":65,"meta":66,"style":66},"print(f\"shape do dataset: \\n {df.shape}\\n\\n\")\ndf.head(5)\n",[52,304,305,310],{"__ignoreMap":66},[70,306,307],{"class":72,"line":73},[70,308,309],{},"print(f\"shape do dataset: \\n {df.shape}\\n\\n\")\n",[70,311,312],{"class":72,"line":79},[70,313,314],{},"df.head(5)\n",[115,316,317],{},[11,318,319,122,321,324,325,285,328,331,332,335],{},[20,320,121],{},[52,322,323],{},"(2540047, 49)",", followed by the first 5 real rows, a UDP session with ",[52,326,327],{},"sport=1390",[52,329,330],{},"dsport=53"," (the DNS port), and ",[52,333,334],{},"dur=0.001055"," seconds, among others.",[11,337,338,339,341,342,345],{},"A cheap, simple sanity check that concatenating the four files and matching up the names actually worked: if ",[52,340,256],{}," had misaligned a column somewhere, ",[52,343,344],{},"df.head()"," would show text where I expected a number, or the other way around, and I'd catch that right away instead of much later in a more serious analysis.",[61,347,349],{"className":63,"code":348,"language":65,"meta":66,"style":66},"print(f\"Linhas: {df.shape[0]:,}\")\nprint(f\"Colunas: {df.shape[1]}\")\nprint(\"Tipagem dos dados:\")\nprint(df.dtypes.value_counts())\n\npd.options.display.float_format = \"{:,.2f}\".format\ndisplay(df.describe().T)\n",[52,350,351,356,361,366,371,375,380],{"__ignoreMap":66},[70,352,353],{"class":72,"line":73},[70,354,355],{},"print(f\"Linhas: {df.shape[0]:,}\")\n",[70,357,358],{"class":72,"line":79},[70,359,360],{},"print(f\"Colunas: {df.shape[1]}\")\n",[70,362,363],{"class":72,"line":85},[70,364,365],{},"print(\"Tipagem dos dados:\")\n",[70,367,368],{"class":72,"line":91},[70,369,370],{},"print(df.dtypes.value_counts())\n",[70,372,373],{"class":72,"line":97},[70,374,101],{"emptyLinePlaceholder":100},[70,376,377],{"class":72,"line":104},[70,378,379],{},"pd.options.display.float_format = \"{:,.2f}\".format\n",[70,381,382],{"class":72,"line":110},[70,383,384],{},"display(df.describe().T)\n",[115,386,387],{},[11,388,389,122,391,394,395,398,399,402,403,406],{},[20,390,121],{},[52,392,393],{},"int64"," for 28 columns, ",[52,396,397],{},"float64"," for 12, ",[52,400,401],{},"object"," for 9. ",[52,404,405],{},"df.describe().T"," (transposed, one row per numeric column) returns count, mean, std, min, quartiles, and max for all 40 numeric columns at once.",[11,408,409,410,412,413,415],{},"That third group, the 9 ",[52,411,401],{}," columns, deserves suspicion, because a column landing as ",[52,414,401],{}," doesn't necessarily mean \"this column is really text.\" Sometimes it's pandas saying \"I found something in here that doesn't fit a clean number,\" and later in this chapter that's exactly the clue that leads to a bug.",[417,418],"category-bar-chart",{":points":419,"color":420,"x-label":421,"y-label":422},"[{\"label\":\"Normal\",\"value\":2218764},{\"label\":\"Attack\",\"value\":321283}]","#0033cc","class","number of sessions",[11,424,425,426,430],{},"2,218,764 normal sessions against 321,283 attack ones, a ratio of nearly 7 to 1. I've seen this kind of imbalance before ",[14,427,429],{"href":428},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","in credit card fraud detection",", there it was 0.17% fraud, here it's 12.6% attack, much less extreme, but still imbalanced enough for accuracy alone to lie: just guessing \"normal\" every time, a lazy model already gets 87.4% right without learning anything.",[39,432,434],{"id":433},"what-kind-of-attack-is-in-there","What kind of attack is in there",[11,436,437,438,441],{},"The ",[52,439,440],{},"attack_cat"," column (only filled in when the session is an attack, which is why 87.35% of it is empty, exactly the fraction of normal traffic) says which attack family each malicious session belongs to:",[417,443],{":points":444,"color":445,"x-label":446,"y-label":422},"[{\"label\":\"Generic\",\"value\":215481},{\"label\":\"Exploits\",\"value\":44525},{\"label\":\"Fuzzers\",\"value\":24246},{\"label\":\"DoS\",\"value\":16353},{\"label\":\"Reconnaissance\",\"value\":13987},{\"label\":\"Analysis\",\"value\":2677},{\"label\":\"Backdoor\",\"value\":2329},{\"label\":\"Shellcode\",\"value\":1511},{\"label\":\"Worms\",\"value\":174}]","#cc3300","attack category",[11,448,449,450,453,454,457,458,461,462,285,465,461,468,471,472,461,475,478,479,483,484,487,488,491,492,495],{},"Before I merged these into 9 categories, the raw data actually had 13 different labels, because ",[52,451,452],{},"\" Fuzzers\""," (with a leading space) and ",[52,455,456],{},"\"Fuzzers\""," counted as two separate categories to pandas, same silliness with ",[52,459,460],{},"\"Reconnaissance\"","\u002F",[52,463,464],{},"\" Reconnaissance\"",[52,466,467],{},"\"Backdoor\"",[52,469,470],{},"\"Backdoors\"",", and ",[52,473,474],{},"\"Shellcode\"",[52,476,477],{},"\" Shellcode\"",". The same kind of typo mess ",[14,480,482],{"href":481},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic","I already saw in the Titanic dataset",", except here instead of a missing column it's a category duplicated by a stray space, a ",[52,485,486],{},".strip()"," would fix it. ",[52,489,490],{},"Generic"," dominates by far (215 thousand sessions, two thirds of all attacks), and it's techniques that forge a hash collision to fool encryption, much easier to automate at scale than, say, a ",[52,493,494],{},"Worms"," attack (only 174 sessions, an attack that needs to replicate itself from machine to machine, rarer to happen inside a controlled lab).",[39,497,499],{"id":498},"redundancy-half-the-columns-tell-the-same-story","Redundancy: half the columns tell the same story",[11,501,502],{},"With 40 numeric variables, a natural question is: how many of them actually carry genuinely different information? I ran the Pearson correlation between pairs of numeric variables on a sample of 50 thousand sessions (computing this on the full 2.5 million would be needlessly expensive):",[61,504,506],{"className":63,"code":505,"language":65,"meta":66,"style":66},"sample = df[colunas_numericas].sample(n=50000, random_state=42)\ncorr_matrix = sample.corr()\n\nmask_pairs = np.triu(np.ones_like(corr_matrix, dtype=bool), k=1)\ncorr_pairs = corr_matrix.where(mask_pairs).stack()\nfortes = corr_pairs[corr_pairs.abs() >= 0.8].sort_values(key=abs, ascending=False)\n",[52,507,508,513,518,522,527,532],{"__ignoreMap":66},[70,509,510],{"class":72,"line":73},[70,511,512],{},"sample = df[colunas_numericas].sample(n=50000, random_state=42)\n",[70,514,515],{"class":72,"line":79},[70,516,517],{},"corr_matrix = sample.corr()\n",[70,519,520],{"class":72,"line":85},[70,521,101],{"emptyLinePlaceholder":100},[70,523,524],{"class":72,"line":91},[70,525,526],{},"mask_pairs = np.triu(np.ones_like(corr_matrix, dtype=bool), k=1)\n",[70,528,529],{"class":72,"line":97},[70,530,531],{},"corr_pairs = corr_matrix.where(mask_pairs).stack()\n",[70,533,534],{"class":72,"line":104},[70,535,536],{},"fortes = corr_pairs[corr_pairs.abs() >= 0.8].sort_values(key=abs, ascending=False)\n",[11,538,539,540,543,544,285,547,550,551,554,555,550,557,559,560,563,564,567,568,571,572,575],{},"That last block is worth taking apart, because ",[52,541,542],{},"corr_matrix"," on its own is a duplicate trap: it's a 40×40 table, every variable crossed against every other, but it's ",[20,545,546],{},"symmetric",[52,548,549],{},"sbytes"," crossed with ",[52,552,553],{},"sloss"," and ",[52,556,553],{},[52,558,549],{}," are the same number showing up twice. Counting that twice inflates the list for nothing. ",[52,561,562],{},"np.triu(..., k=1)"," draws a \"grid\" that marks only the upper triangle of the table as true, skipping the diagonal (the ",[52,565,566],{},"k=1",", otherwise I'd count every variable correlated with itself, always 1.0). ",[52,569,570],{},"corr_matrix.where(mask_pairs)"," erases (turns blank) everything that fell outside that grid, leaving only the unique half of the table. ",[52,573,574],{},".stack()"," finds that half and \"unfolds\" it into a long list of pairs, dropping the blanks on its own. All that's left is filtering for strong correlation (|r| ≥ 0.8) and sorting strongest to weakest, exactly what the last line does:",[577,578,579,594],"table",{},[580,581,582],"thead",{},[583,584,585,590],"tr",{},[586,587,589],"th",{"align":588},"left","Variable pair",[586,591,593],{"align":592},"right","Correlation",[595,596,597,612,624,637,648,659,671,683,696,707,719,730],"tbody",{},[583,598,599,609],{},[600,601,602,605,606],"td",{"align":588},[52,603,604],{},"Stime"," × ",[52,607,608],{},"Ltime",[600,610,611],{"align":592},"1.00",[583,613,614,622],{},[600,615,616,605,619],{"align":588},[52,617,618],{},"swin",[52,620,621],{},"dwin",[600,623,611],{"align":592},[583,625,626,634],{},[600,627,628,605,631],{"align":588},[52,629,630],{},"dloss",[52,632,633],{},"Dpkts",[600,635,636],{"align":592},"0.99",[583,638,639,646],{},[600,640,641,605,644],{"align":588},[52,642,643],{},"dbytes",[52,645,630],{},[600,647,636],{"align":592},[583,649,650,656],{},[600,651,652,605,654],{"align":588},[52,653,549],{},[52,655,553],{},[600,657,658],{"align":592},"0.96",[583,660,661,669],{},[600,662,663,605,666],{"align":588},[52,664,665],{},"ct_srv_src",[52,667,668],{},"ct_srv_dst",[600,670,658],{"align":592},[583,672,673,681],{},[600,674,675,605,678],{"align":588},[52,676,677],{},"ct_dst_ltm",[52,679,680],{},"ct_src_dport_ltm",[600,682,658],{"align":592},[583,684,685,693],{},[600,686,687,605,690],{"align":588},[52,688,689],{},"tcprtt",[52,691,692],{},"synack",[600,694,695],{"align":592},"0.92",[583,697,698,705],{},[600,699,700,605,702],{"align":588},[52,701,689],{},[52,703,704],{},"ackdat",[600,706,695],{"align":592},[583,708,709,716],{},[600,710,711,605,713],{"align":588},[52,712,129],{},[52,714,715],{},"ct_state_ttl",[600,717,718],{"align":592},"0.91",[583,720,721,728],{},[600,722,723,605,725],{"align":588},[52,724,129],{},[52,726,727],{},"Label",[600,729,718],{"align":592},[583,731,732,738],{},[600,733,734,605,736],{"align":588},[52,735,715],{},[52,737,727],{},[600,739,740],{"align":592},"0.88",[11,742,743,744,554,746,748,749,752,753,755,756,759,760,762],{},"Some of these pairs are redundant by construction, not coincidence: ",[52,745,604],{},[52,747,608],{}," are the start and end instants of the same session, so a 1.00 correlation is almost automatic (the difference between the two is literally ",[52,750,751],{},"dur",", another column). But ",[52,754,129],{}," (the TTL, \"time to live,\" the counter every router along the path decrements until the packet expires) correlating ",[20,757,758],{},"0.91 directly with the attack label"," is the most interesting finding of the whole EDA: automated attack tooling tends to generate packets with a TTL noticeably different from normal browser traffic, a technical \"accent\" that by itself already carries a lot of information about who's on the other end of the connection. I'm keeping that name, ",[52,761,129],{},", in mind, it comes back in the next two chapters.",[39,764,766],{"id":765},"boxplots-by-class-a-visual-check-before-the-models-own-importance","Boxplots by class: a visual check before the model's own importance",[11,768,769],{},"Before any model enters the picture, a simpler question already helps anticipate which variables separate normal traffic from attacks well: does a feature's distribution visibly shift between the two classes?",[61,771,773],{"className":63,"code":772,"language":65,"meta":66,"style":66},"features_box = [\"dur\", \"sbytes\", \"dbytes\", \"Sload\", \"Dload\"]\n\nfig, axes = plt.subplots(1, len(features_box), figsize=(18, 5))\n\nfor i, feat in enumerate(features_box):\n    sns.boxplot(\n        data=df,\n        x=\"Label\",\n        y=feat,\n        ax=axes[i],\n        palette=[\"steelblue\", \"tomato\"],\n        showfliers=False,   # tirando outliers extremos para melhor visualização\n        hue=\"Label\",\n        legend=False\n    )\n    axes[i].set_title(feat)\n    axes[i].set_xticks([0, 1])\n    axes[i].set_xticklabels([\"Normal\", \"Ataque\"])\n\nplt.suptitle(\"Distribuição de Features por Classe\", y=1.02)\nplt.tight_layout()\nplt.show()\n",[52,774,775,780,784,789,793,798,803,808,813,818,823,828,833,838,843,848,853,858,864,869,875,881],{"__ignoreMap":66},[70,776,777],{"class":72,"line":73},[70,778,779],{},"features_box = [\"dur\", \"sbytes\", \"dbytes\", \"Sload\", \"Dload\"]\n",[70,781,782],{"class":72,"line":79},[70,783,101],{"emptyLinePlaceholder":100},[70,785,786],{"class":72,"line":85},[70,787,788],{},"fig, axes = plt.subplots(1, len(features_box), figsize=(18, 5))\n",[70,790,791],{"class":72,"line":91},[70,792,101],{"emptyLinePlaceholder":100},[70,794,795],{"class":72,"line":97},[70,796,797],{},"for i, feat in enumerate(features_box):\n",[70,799,800],{"class":72,"line":104},[70,801,802],{},"    sns.boxplot(\n",[70,804,805],{"class":72,"line":110},[70,806,807],{},"        data=df,\n",[70,809,810],{"class":72,"line":195},[70,811,812],{},"        x=\"Label\",\n",[70,814,815],{"class":72,"line":200},[70,816,817],{},"        y=feat,\n",[70,819,820],{"class":72,"line":206},[70,821,822],{},"        ax=axes[i],\n",[70,824,825],{"class":72,"line":212},[70,826,827],{},"        palette=[\"steelblue\", \"tomato\"],\n",[70,829,830],{"class":72,"line":217},[70,831,832],{},"        showfliers=False,   # tirando outliers extremos para melhor visualização\n",[70,834,835],{"class":72,"line":223},[70,836,837],{},"        hue=\"Label\",\n",[70,839,840],{"class":72,"line":229},[70,841,842],{},"        legend=False\n",[70,844,845],{"class":72,"line":235},[70,846,847],{},"    )\n",[70,849,850],{"class":72,"line":241},[70,851,852],{},"    axes[i].set_title(feat)\n",[70,854,855],{"class":72,"line":247},[70,856,857],{},"    axes[i].set_xticks([0, 1])\n",[70,859,861],{"class":72,"line":860},18,[70,862,863],{},"    axes[i].set_xticklabels([\"Normal\", \"Ataque\"])\n",[70,865,867],{"class":72,"line":866},19,[70,868,101],{"emptyLinePlaceholder":100},[70,870,872],{"class":72,"line":871},20,[70,873,874],{},"plt.suptitle(\"Distribuição de Features por Classe\", y=1.02)\n",[70,876,878],{"class":72,"line":877},21,[70,879,880],{},"plt.tight_layout()\n",[70,882,884],{"class":72,"line":883},22,[70,885,886],{},"plt.show()\n",[11,888,889,890,285,892,285,894,285,896,285,899,902,903,906],{},"Each subplot compares one feature's distribution (",[52,891,751],{},[52,893,549],{},[52,895,643],{},[52,897,898],{},"Sload",[52,900,901],{},"Dload",") between normal and attack sessions, side by side. The box's middle line is the median, the box itself spans the 1st to 3rd quartile (the middle 50% of the data), and the \"whiskers\" show the typical spread. ",[52,904,905],{},"showfliers=False"," just drops the most extreme outliers from the picture so it doesn't get squashed flat, they're still in the data, they just don't show up in this particular drawing.",[11,908,909,910,913],{},"The reading that matters: when the \"Normal\" box and the \"Attack\" box sit in clearly different value ranges, with little overlap, that variable alone already helps separate the classes. When the two boxes overlap almost entirely, the variable alone discriminates poorly. It's a quick visual check that ",[20,911,912],{},"anticipates"," what the Random Forest's importance confirms quantitatively later in chapter 3, without waiting for the model to finish training to get a first hint.",[39,915,917],{"id":916},"who-the-categorical-columns-are-and-why-thats-already-a-warning-sign","Who the categorical columns are, and why that's already a warning sign",[11,919,920,921,924,925,285,928,285,931,285,934,285,937,285,940,285,943,285,946,471,949,951],{},"Pulling out the columns pandas read as text (",[52,922,923],{},"select_dtypes(include=[\"object\"])","), nine show up: ",[52,926,927],{},"srcip",[52,929,930],{},"sport",[52,932,933],{},"dstip",[52,935,936],{},"dsport",[52,938,939],{},"proto",[52,941,942],{},"state",[52,944,945],{},"service",[52,947,948],{},"ct_ftp_cmd",[52,950,440],{},". Counting the most frequent values of each, two quite different groups appear:",[61,953,955],{"className":63,"code":954,"language":65,"meta":66,"style":66},"categoricas = df.select_dtypes(include=[\"object\"]).columns.tolist()\nfor col in categoricas:\n    print(f\"\\n{col}:\")\n    print(df[col].value_counts().head(10))\n",[52,956,957,962,967,972],{"__ignoreMap":66},[70,958,959],{"class":72,"line":73},[70,960,961],{},"categoricas = df.select_dtypes(include=[\"object\"]).columns.tolist()\n",[70,963,964],{"class":72,"line":79},[70,965,966],{},"for col in categoricas:\n",[70,968,969],{"class":72,"line":85},[70,970,971],{},"    print(f\"\\n{col}:\")\n",[70,973,974],{"class":72,"line":91},[70,975,976],{},"    print(df[col].value_counts().head(10))\n",[11,978,979,981,982,985,986,989,990,992,993,996,997,1000,1001,1004,1005,1007,1008,1011,1012,1015,1016,1019,1020,1023],{},[52,980,939],{}," is dominated by ",[52,983,984],{},"tcp"," (1,495,074 sessions) and ",[52,987,988],{},"udp"," (990,435), ",[52,991,942],{}," by ",[52,994,995],{},"FIN"," (1,478,689), ",[52,998,999],{},"CON"," (560,588), and ",[52,1002,1003],{},"INT"," (490,471), and ",[52,1006,945],{}," has a huge ",[52,1009,1010],{},"-"," (1,246,397, service not identified by the sniffer), followed by ",[52,1013,1014],{},"dns"," (781,668) and ",[52,1017,1018],{},"http"," (206,273). These three have few categories and no order between them, good candidates for ",[20,1021,1022],{},"one-hot encoding"," further along.",[11,1025,1026,285,1028,285,1030,471,1032,1034,1035,1037,1038,55,1041,1044,1045,554,1047,1049,1050,1053,1054,1056,1057,1060,1061,1064],{},[52,1027,927],{},[52,1029,930],{},[52,1031,933],{},[52,1033,936],{}," are a different species: extremely high-cardinality identifiers. ",[52,1036,927],{}," alone has only 10 unique values in this lab (",[52,1039,1040],{},"59.166.0.0",[52,1042,1043],{},"59.166.0.9",", the attacking\u002Ftraffic-generating machines), but ",[52,1046,930],{},[52,1048,936],{}," have thousands. The reason I don't use any of these four in the model isn't lack of signal, source IP probably separates attack machines very well in this specific lab, it's the opposite: it separates ",[20,1051,1052],{},"too well",", in a way that doesn't generalize. A model trained on ",[52,1055,927],{}," would memorize \"",[52,1058,1059],{},"59.166.0.4"," is always suspicious\" instead of learning the ",[150,1062,1063],{},"behavior"," of an attack, and outside this lab, with completely different IPs, that memorized rule is worthless. I come back to this point with more care in chapter 2, when I explain the dataset switch.",[39,1066,1068],{"id":1067},"missing-data-structural-not-a-collection-failure","Missing data: structural, not a collection failure",[61,1070,1072],{"className":63,"code":1071,"language":65,"meta":66,"style":66},"nulos = df.isnull().sum()\nnulos_percentual = (nulos \u002F len(df) * 100)\nresumo_nulos = pd.DataFrame({\"nulos\": nulos, \"percentual\": nulos_percentual}).sort_values(\"percentual\", ascending=False)\n",[52,1073,1074,1079,1084],{"__ignoreMap":66},[70,1075,1076],{"class":72,"line":73},[70,1077,1078],{},"nulos = df.isnull().sum()\n",[70,1080,1081],{"class":72,"line":79},[70,1082,1083],{},"nulos_percentual = (nulos \u002F len(df) * 100)\n",[70,1085,1086],{"class":72,"line":85},[70,1087,1088],{},"resumo_nulos = pd.DataFrame({\"nulos\": nulos, \"percentual\": nulos_percentual}).sort_values(\"percentual\", ascending=False)\n",[417,1090],{":points":1091,"color":445,"x-label":1092,"y-label":1093},"[{\"label\":\"attack_cat\",\"value\":87.35},{\"label\":\"is_ftp_login\",\"value\":56.29},{\"label\":\"ct_flw_http_mthd\",\"value\":53.08}]","column","% missing",[11,1095,1096],{},"Only three columns out of 49 have real missing values (the other 46 sit at 0.00%). My first version of this chart was a heatmap over 5,000 sampled rows, a classic way to visualize missingness patterns, but with only 3 columns affected out of 49 it turned into a thin strip with basically no information, time wasted staring at a mostly-blank image. I swapped it for a straight bar chart of the percentage missing per column, which answers exactly the question I actually wanted to ask: \"how much is missing in each column?\" That was my first practical lesson of this chapter: the \"correct\" chart in theory isn't always the right chart for your specific data.",[11,1098,1099,1100,1102,1103,1106,1107,1110],{},"And the three real gaps tell a structural story, not a collection problem. ",[52,1101,440],{}," (87.35%) is only filled in when the session is an attack, the 87.35% blank is exactly the normal sessions, the same thing I already explained above. ",[52,1104,1105],{},"is_ftp_login"," (56.29%) and ",[52,1108,1109],{},"ct_flw_http_mthd"," (53.08%) only make sense for FTP and HTTP sessions respectively, a session of another protocol has nothing to fill in there. There's no blind imputation to do here, because the absence itself is already information (this session isn't FTP, for instance), erasing or filling those columns with any value would destroy that signal.",[39,1112,1114],{"id":1113},"a-subtle-bug-when-empty-doesnt-count-as-missing","A subtle bug: when \"empty\" doesn't count as missing",[11,1116,1117,1118,1120],{},"I said only three columns have missing values. But I noticed something odd looking at the values of ",[52,1119,948],{}," (the number of FTP commands in the session, which should be a number) in the list of categoricals:",[61,1122,1124],{"className":63,"code":1123,"language":65,"meta":66,"style":66},"ct_ftp_cmd:\n     1429879\n0     686192\n0     380306\n1      29605\n1      10472\n",[52,1125,1126,1131,1136,1141,1146,1151],{"__ignoreMap":66},[70,1127,1128],{"class":72,"line":73},[70,1129,1130],{},"ct_ftp_cmd:\n",[70,1132,1133],{"class":72,"line":79},[70,1134,1135],{},"     1429879\n",[70,1137,1138],{"class":72,"line":85},[70,1139,1140],{},"0     686192\n",[70,1142,1143],{"class":72,"line":91},[70,1144,1145],{},"0     380306\n",[70,1147,1148],{"class":72,"line":97},[70,1149,1150],{},"1      29605\n",[70,1152,1153],{"class":72,"line":104},[70,1154,1155],{},"1      10472\n",[11,1157,1158,1159,1162,1163,1166,1167,1170,1171,1173,1174,1176,1177,1179,1180,1183,1184,1187,1188,1191,1192,1195,1196,1198,1199,1201,1202,1204],{},"Two clues here. First: the most common category, with 1,429,879 sessions, has no label at all, it's an ",[20,1160,1161],{},"empty string"," (",[52,1164,1165],{},"\"\"","), not a ",[52,1168,1169],{},"NaN",". And 1,429,879 is exactly the number of missing values in ",[52,1172,1105],{}," from above, meaning every session without an FTP login also has ",[52,1175,948],{}," unfilled, which makes sense, but pandas never counted it as missing because technically it isn't ",[52,1178,1169],{},", it's an empty string disguised as a valid category. ",[52,1181,1182],{},"df.isnull().sum()"," simply never saw this problem. Second clue: ",[52,1185,1186],{},"\"0\""," shows up ",[20,1189,1190],{},"twice"," in the list, once with 686,192 sessions and once with 380,306, because one has a trailing space glued on (",[52,1193,1194],{},"\"0 \"",") and the other doesn't, the same kind of typo mess I already saw in ",[52,1197,440],{},". Those two things combined (an empty string instead of a null, plus whitespace duplicating a category) are why ",[52,1200,948],{}," turned into ",[52,1203,401],{}," back in the dtype table: a column that should be purely numeric got \"contaminated\" enough for pandas to give up reading it as a number.",[11,1206,1207,1208,1210,1211,1214,1215,1217,1218,1221,1222,1225,1226,1228],{},"The lesson that sticks: ",[52,1209,1182],{}," is a good first filter, but it isn't the final word on what's actually missing. An empty string, a lone space, a generic ",[52,1212,1213],{},"\"-\""," (like the one I already saw in ",[52,1216,945],{},"), all of that sails right past a naive ",[52,1219,1220],{},".isnull()",". Always worth cross-checking with ",[52,1223,1224],{},"df.dtypes"," and being suspicious of any naturally-numeric column that landed as ",[52,1227,401],{},".",[39,1230,1232],{"id":1231},"selecting-or-not-selecting-features","Selecting (or not selecting) features",[11,1234,1235,1236,1239],{},"With the numeric and categorical EDA in hand, the central question of any project with lots of variables shows up: which ones do I actually use? The course treats that choice as ",[20,1237,1238],{},"feature selection",", and there are three classic families:",[1241,1242,1243,1250,1256],"ul",{},[1244,1245,1246,1249],"li",{},[20,1247,1248],{},"Filter:"," a statistical criterion, independent of any model. The redundancy analysis by correlation above is exactly that: two variables with r ≈ 1.00 carry almost the same information, one would be a candidate to drop.",[1244,1251,1252,1255],{},[20,1253,1254],{},"Wrapper:"," testing subsets of variables by actually running the model (recursive feature elimination, for instance). Powerful, but expensive, because every subset tested is a full training run.",[1244,1257,1258,1261],{},[20,1259,1260],{},"Embedded:"," the model itself scores each variable's importance while it trains. That's what the Random Forest's importance does, and I get there in chapter 3.",[11,1263,1264,1265,1268],{},"Before deciding anything, I ran a histogram on eight numeric features chosen only to inspect the ",[150,1266,1267],{},"shape"," of the distributions, not to rank importance:",[61,1270,1272],{"className":63,"code":1271,"language":65,"meta":66,"style":66},"features_ilustrativas = [\"dur\", \"sbytes\", \"dbytes\", \"sttl\", \"dttl\", \"sloss\", \"dloss\", \"Sload\"]\n\naxes = df[features_ilustrativas].hist(figsize=(14, 8), bins=50, color=\"steelblue\",\n                                      edgecolor=\"black\", log=True)\nfor ax in axes.ravel():\n    ax.set_xlabel(\"Valor da feature\")\n    ax.set_ylabel(\"Frequencia (escala log)\")\n\nplt.suptitle(\"Distribuicao de Features Numericas (amostra ilustrativa)\", y=1.02)\nplt.tight_layout()\nplt.show()\n",[52,1273,1274,1279,1283,1288,1293,1298,1303,1308,1312,1317,1321],{"__ignoreMap":66},[70,1275,1276],{"class":72,"line":73},[70,1277,1278],{},"features_ilustrativas = [\"dur\", \"sbytes\", \"dbytes\", \"sttl\", \"dttl\", \"sloss\", \"dloss\", \"Sload\"]\n",[70,1280,1281],{"class":72,"line":79},[70,1282,101],{"emptyLinePlaceholder":100},[70,1284,1285],{"class":72,"line":85},[70,1286,1287],{},"axes = df[features_ilustrativas].hist(figsize=(14, 8), bins=50, color=\"steelblue\",\n",[70,1289,1290],{"class":72,"line":91},[70,1291,1292],{},"                                      edgecolor=\"black\", log=True)\n",[70,1294,1295],{"class":72,"line":97},[70,1296,1297],{},"for ax in axes.ravel():\n",[70,1299,1300],{"class":72,"line":104},[70,1301,1302],{},"    ax.set_xlabel(\"Valor da feature\")\n",[70,1304,1305],{"class":72,"line":110},[70,1306,1307],{},"    ax.set_ylabel(\"Frequencia (escala log)\")\n",[70,1309,1310],{"class":72,"line":195},[70,1311,101],{"emptyLinePlaceholder":100},[70,1313,1314],{"class":72,"line":200},[70,1315,1316],{},"plt.suptitle(\"Distribuicao de Features Numericas (amostra ilustrativa)\", y=1.02)\n",[70,1318,1319],{"class":72,"line":206},[70,1320,880],{},[70,1322,1323],{"class":72,"line":212},[70,1324,886],{},[11,1326,1327,1328,1331,1332,1335],{},"The comment I left next to this cell in the original notebook is worth repeating here, because it heads off a wrong reading: these eight features are ",[20,1329,1330],{},"illustrative",", a sample to inspect the shape of the distribution, not the result of any importance-based selection (the real importance comes from the correlation I already covered, and from the Random Forest's own importance in chapter 3). The Y axis uses a log scale (",[52,1333,1334],{},"log=True",") on purpose: bytes transferred and session duration have extremely skewed distributions, most sessions are short and light, a minority are far heavier, and on a linear scale that minority would flatten the majority into a line at zero.",[11,1337,1338,1339,1342,1343,1346],{},"With that visual check, plus the boxplots by class and the correlation already covered above, it's worth writing down the decision I made: ",[20,1340,1341],{},"I'm not pruning any feature in the pipeline."," The Random Forest I use in chapter 3 is reasonably robust to redundant variables and to differences in scale between them, so I treat this selection as ",[20,1344,1345],{},"interpretive analysis"," (understanding what weighs on the decision) rather than a step that actually changes what enters the model. That's a deliberate choice, not laziness: a Logistic Regression or a KNN would probably benefit from cutting the redundancy found here, but for the model I choose in the next chapter, the gain wouldn't be worth the extra complexity of deciding \"which of these two nearly-identical columns do I cut.\"",[39,1348,1350],{"id":1349},"main-takeaways-before-modeling","Main takeaways, before modeling",[11,1352,1353],{},"Closing this first chapter, the points I'm carrying with me for the rest of the project:",[577,1355,1356,1366],{},[580,1357,1358],{},[583,1359,1360,1363],{},[586,1361,1362],{"align":588},"What the EDA showed",[586,1364,1365],{"align":588},"Why it matters later",[595,1367,1368,1376,1390,1402,1414,1422],{},[583,1369,1370,1373],{},[600,1371,1372],{"align":588},"12.6% attack class imbalance",[600,1374,1375],{"align":588},"Accuracy alone lies, F1 and AUC are the metrics I'll actually trust",[583,1377,1378,1387],{},[600,1379,1380,285,1382,471,1384,1386],{"align":588},[52,1381,440],{},[52,1383,1105],{},[52,1385,1109],{}," have structural nulls",[600,1388,1389],{"align":588},"I won't impute blindly, the absence itself is already signal",[583,1391,1392,1397],{},[600,1393,1394,1396],{"align":588},[52,1395,948],{}," was hiding missing data as an empty string",[600,1398,1399,1401],{"align":588},[52,1400,1220],{}," alone isn't enough, always cross-check with dtype",[583,1403,1404,1411],{},[600,1405,1406,554,1408,1410],{"align":588},[52,1407,440],{},[52,1409,948],{}," had a category duplicated by a stray space",[600,1412,1413],{"align":588},"Fixed before any count was worth trusting",[583,1415,1416,1419],{},[600,1417,1418],{"align":588},"Half of the 40 numeric variables are redundant with each other (correlation ≥ 0.8)",[600,1420,1421],{"align":588},"Random Forest tolerates this, but recording the redundancy matters for interpreting later",[583,1423,1424,1435],{},[600,1425,1426,461,1428,461,1430,461,1432,1434],{"align":588},[52,1427,927],{},[52,1429,933],{},[52,1431,930],{},[52,1433,936],{}," separate too well, in a way that doesn't generalize",[600,1436,1437],{"align":588},"Dropped in chapter 2, along with the full explanation why",[11,1439,1440],{},"In chapter 2 I swap the raw dataset I used this whole chapter for the official partition the UNSW-NB15 authors themselves curated and published, explain in detail why I'm making that switch (it's about data leakage, not a whim), build the preprocessing pipeline, and compare four model families on a fair cross-validation before choosing which one to actually train.",[1442,1443,1444],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":66,"searchDepth":79,"depth":79,"links":1446},[1447,1448,1449,1450,1451,1452,1453,1454,1455],{"id":41,"depth":79,"text":42},{"id":433,"depth":79,"text":434},{"id":498,"depth":79,"text":499},{"id":765,"depth":79,"text":766},{"id":916,"depth":79,"text":917},{"id":1067,"depth":79,"text":1068},{"id":1113,"depth":79,"text":1114},{"id":1231,"depth":79,"text":1232},{"id":1349,"depth":79,"text":1350},"2026-08-26","Chapter 1 of my final project for Pattern Recognition: the full exploratory analysis of UNSW-NB15 on the raw files, class imbalance, missing data, a quality bug I only caught because I doubted the wrong dtype, redundancy between variables, and the deliberate choice not to prune a single feature.","md",{},"\u002Fen\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes","network-intrusion-detection",{"title":6,"description":1457},"published","en\u002Fprojects\u002Fnetwork-intrusion-detection\u002F01-explorando-25-milhoes-de-sessoes",[1466,1467,1468,1469],"security","eda","unsw-nb15","feature-selection","25YG0ook_ie6Jt5189imcX4PrOnhEFFS8pvlVCTXZ-w",1788965102296]