Exploring 2.5 Million Network Sessions

As I said in the project's opening page, this is my final project for the Pattern Recognition course, no professor showing the way this time. Think of a network session like a phone call between two computers: it has a start and an end, a duration, an amount of information exchanged on each side (the bytes), and a shared "language" the two sides agreed on (the protocol, like TCP or UDP). The mission for the whole project: given that small packet of information about the call, decide whether it's normal traffic or an attack. Full, executed notebook here, all three chapters come from it.

This first chapter is just the exploratory analysis (EDA). Before training any model I need to understand what I'm working with: how imbalanced the dataset is, where data is missing and why, whether any column is dirty, which variables tell the same story (redundancy), and how to decide, with a real criterion, which ones go into the model. Chapter 2 switches datasets (I explain why) and trains the candidates, chapter 3 closes with the final evaluation, the ROC curve, and hyperparameter tuning.

The dataset: 2.5 million sessions, almost 9 in 10 are normal traffic

UNSW-NB15 was born from a real lab: the Australian Centre for Cyber Security generated network traffic mixing normal browsing with nine different attack families running on purpose, and captured everything at the raw packet level. The first practical problem: the four raw CSV files (UNSW-NB15_1.csv through _4.csv) carry no column names at all, just raw numbers. The names live separately, in a fifth file, the feature "catalog":

features = pd.read_csv("/content/data/NUSW-NB15_features.csv", encoding="latin-1")
features.columns = features.columns.str.strip()
pd.set_option("display.max_colwidth", None)
pd.set_option("display.max_rows", None)

print(f"Total de features descritas: {len(features)}")
display(features[["No.", "Name", "Type", "Description"]])

Output: Total de features descritas: 49, followed by a table with the number, name, declared type, and description of each of the 49 columns (for instance, sttl shows up described as "Source to destination time to live value").

encoding="latin-1" isn't decoration: this catalog's CSV has characters outside standard UTF-8 (accents and symbols from whoever documented the dataset), and without the right encoding read_csv breaks. The two pd.set_option lines exist only for this specific display: without them, pandas truncates the Description column (which has long sentences) and cuts the table off with "..." partway through the 49 rows, the opposite of what I want when the whole point is to read each feature's description. This catalog is what I use next to actually name the columns:

import re

# Filtragem apenas dos arquivos de dados -> UNSW-NB15_1.csv até UNSW-NB15_4.csv
partes = sorted([
    f for f in os.listdir("/content/data")
    if re.match(r"UNSW-NB15_\d+\.csv", f)
])

# Extraindo os nomes das colunas do arquivo de features
colunas = features["Name"].tolist()

# Carregando e concatenando todas as partes em um único DataFrame
df = pd.concat(
    [pd.read_csv(f"/content/data/{p}", header=None, names=colunas, low_memory=False)
     for p in partes],
    ignore_index=True
)

Worth opening this block up, there's a gotcha hiding in it: without that names=colunas inside read_csv, pandas would number the columns 0, 1, 2... and I'd lose every column's label. Before that, re.match(r"UNSW-NB15_\d+\.csv", f) filters, out of everything the download brought in, only the four files that matter here (UNSW-NB15_1.csv through _4.csv), deliberately dropping the feature catalog itself and the official partition files (UNSW_NB15_training-set.csv and so on, which I only use starting in chapter 2) that live in the same folder. sorted(...) guarantees the four pieces land in the right order (_1, _2, _3, _4), not whatever order the filesystem happens to list them in. The list comprehension in brackets reads the four files, one at a time, already gluing on the right names as it reads, and pd.concat(..., ignore_index=True) stacks the four pieces on top of each other, resetting the row count from zero (without that, each piece would restart at 0, and I'd have the same index repeated four times).

print(f"shape do dataset: \n {df.shape}\n\n")
df.head(5)

Output: (2540047, 49), followed by the first 5 real rows, a UDP session with sport=1390, dsport=53 (the DNS port), and dur=0.001055 seconds, among others.

A cheap, simple sanity check that concatenating the four files and matching up the names actually worked: if names=colunas had misaligned a column somewhere, df.head() would show text where I expected a number, or the other way around, and I'd catch that right away instead of much later in a more serious analysis.

print(f"Linhas: {df.shape[0]:,}")
print(f"Colunas: {df.shape[1]}")
print("Tipagem dos dados:")
print(df.dtypes.value_counts())

pd.options.display.float_format = "{:,.2f}".format
display(df.describe().T)

Output: int64 for 28 columns, float64 for 12, object for 9. df.describe().T (transposed, one row per numeric column) returns count, mean, std, min, quartiles, and max for all 40 numeric columns at once.

That third group, the 9 object columns, deserves suspicion, because a column landing as object doesn't necessarily mean "this column is really text." Sometimes it's pandas saying "I found something in here that doesn't fit a clean number," and later in this chapter that's exactly the clue that leads to a bug.

2,218,764 normal sessions against 321,283 attack ones, a ratio of nearly 7 to 1. I've seen this kind of imbalance before in credit card fraud detection, there it was 0.17% fraud, here it's 12.6% attack, much less extreme, but still imbalanced enough for accuracy alone to lie: just guessing "normal" every time, a lazy model already gets 87.4% right without learning anything.

What kind of attack is in there

The attack_cat column (only filled in when the session is an attack, which is why 87.35% of it is empty, exactly the fraction of normal traffic) says which attack family each malicious session belongs to:

Before I merged these into 9 categories, the raw data actually had 13 different labels, because " Fuzzers" (with a leading space) and "Fuzzers" counted as two separate categories to pandas, same silliness with "Reconnaissance"/" Reconnaissance", "Backdoor"/"Backdoors", and "Shellcode"/" Shellcode". The same kind of typo mess I already saw in the Titanic dataset, except here instead of a missing column it's a category duplicated by a stray space, a .strip() would fix it. Generic dominates by far (215 thousand sessions, two thirds of all attacks), and it's techniques that forge a hash collision to fool encryption, much easier to automate at scale than, say, a Worms attack (only 174 sessions, an attack that needs to replicate itself from machine to machine, rarer to happen inside a controlled lab).

Redundancy: half the columns tell the same story

With 40 numeric variables, a natural question is: how many of them actually carry genuinely different information? I ran the Pearson correlation between pairs of numeric variables on a sample of 50 thousand sessions (computing this on the full 2.5 million would be needlessly expensive):

sample = df[colunas_numericas].sample(n=50000, random_state=42)
corr_matrix = sample.corr()

mask_pairs = np.triu(np.ones_like(corr_matrix, dtype=bool), k=1)
corr_pairs = corr_matrix.where(mask_pairs).stack()
fortes = corr_pairs[corr_pairs.abs() >= 0.8].sort_values(key=abs, ascending=False)

That last block is worth taking apart, because corr_matrix on its own is a duplicate trap: it's a 40×40 table, every variable crossed against every other, but it's symmetric, sbytes crossed with sloss and sloss crossed with sbytes are the same number showing up twice. Counting that twice inflates the list for nothing. np.triu(..., k=1) draws a "grid" that marks only the upper triangle of the table as true, skipping the diagonal (the k=1, otherwise I'd count every variable correlated with itself, always 1.0). corr_matrix.where(mask_pairs) erases (turns blank) everything that fell outside that grid, leaving only the unique half of the table. .stack() finds that half and "unfolds" it into a long list of pairs, dropping the blanks on its own. All that's left is filtering for strong correlation (|r| ≥ 0.8) and sorting strongest to weakest, exactly what the last line does:

Some of these pairs are redundant by construction, not coincidence: Stime and Ltime are the start and end instants of the same session, so a 1.00 correlation is almost automatic (the difference between the two is literally dur, another column). But sttl (the TTL, "time to live," the counter every router along the path decrements until the packet expires) correlating 0.91 directly with the attack label is the most interesting finding of the whole EDA: automated attack tooling tends to generate packets with a TTL noticeably different from normal browser traffic, a technical "accent" that by itself already carries a lot of information about who's on the other end of the connection. I'm keeping that name, sttl, in mind, it comes back in the next two chapters.

Boxplots by class: a visual check before the model's own importance

Before any model enters the picture, a simpler question already helps anticipate which variables separate normal traffic from attacks well: does a feature's distribution visibly shift between the two classes?

features_box = ["dur", "sbytes", "dbytes", "Sload", "Dload"]

fig, axes = plt.subplots(1, len(features_box), figsize=(18, 5))

for i, feat in enumerate(features_box):
    sns.boxplot(
        data=df,
        x="Label",
        y=feat,
        ax=axes[i],
        palette=["steelblue", "tomato"],
        showfliers=False,   # tirando outliers extremos para melhor visualização
        hue="Label",
        legend=False
    )
    axes[i].set_title(feat)
    axes[i].set_xticks([0, 1])
    axes[i].set_xticklabels(["Normal", "Ataque"])

plt.suptitle("Distribuição de Features por Classe", y=1.02)
plt.tight_layout()
plt.show()

Each subplot compares one feature's distribution (dur, sbytes, dbytes, Sload, Dload) between normal and attack sessions, side by side. The box's middle line is the median, the box itself spans the 1st to 3rd quartile (the middle 50% of the data), and the "whiskers" show the typical spread. showfliers=False just drops the most extreme outliers from the picture so it doesn't get squashed flat, they're still in the data, they just don't show up in this particular drawing.

The reading that matters: when the "Normal" box and the "Attack" box sit in clearly different value ranges, with little overlap, that variable alone already helps separate the classes. When the two boxes overlap almost entirely, the variable alone discriminates poorly. It's a quick visual check that anticipates what the Random Forest's importance confirms quantitatively later in chapter 3, without waiting for the model to finish training to get a first hint.

Who the categorical columns are, and why that's already a warning sign

Pulling out the columns pandas read as text (select_dtypes(include=["object"])), nine show up: srcip, sport, dstip, dsport, proto, state, service, ct_ftp_cmd, and attack_cat. Counting the most frequent values of each, two quite different groups appear:

categoricas = df.select_dtypes(include=["object"]).columns.tolist()
for col in categoricas:
    print(f"\n{col}:")
    print(df[col].value_counts().head(10))

proto is dominated by tcp (1,495,074 sessions) and udp (990,435), state by FIN (1,478,689), CON (560,588), and INT (490,471), and service has a huge - (1,246,397, service not identified by the sniffer), followed by dns (781,668) and http (206,273). These three have few categories and no order between them, good candidates for one-hot encoding further along.

srcip, sport, dstip, and dsport are a different species: extremely high-cardinality identifiers. srcip alone has only 10 unique values in this lab (59.166.0.0 through 59.166.0.9, the attacking/traffic-generating machines), but sport and dsport have thousands. The reason I don't use any of these four in the model isn't lack of signal, source IP probably separates attack machines very well in this specific lab, it's the opposite: it separates too well, in a way that doesn't generalize. A model trained on srcip would memorize "59.166.0.4 is always suspicious" instead of learning the behavior of an attack, and outside this lab, with completely different IPs, that memorized rule is worthless. I come back to this point with more care in chapter 2, when I explain the dataset switch.

Missing data: structural, not a collection failure

nulos = df.isnull().sum()
nulos_percentual = (nulos / len(df) * 100)
resumo_nulos = pd.DataFrame({"nulos": nulos, "percentual": nulos_percentual}).sort_values("percentual", ascending=False)

Only three columns out of 49 have real missing values (the other 46 sit at 0.00%). My first version of this chart was a heatmap over 5,000 sampled rows, a classic way to visualize missingness patterns, but with only 3 columns affected out of 49 it turned into a thin strip with basically no information, time wasted staring at a mostly-blank image. I swapped it for a straight bar chart of the percentage missing per column, which answers exactly the question I actually wanted to ask: "how much is missing in each column?" That was my first practical lesson of this chapter: the "correct" chart in theory isn't always the right chart for your specific data.

And the three real gaps tell a structural story, not a collection problem. attack_cat (87.35%) is only filled in when the session is an attack, the 87.35% blank is exactly the normal sessions, the same thing I already explained above. is_ftp_login (56.29%) and ct_flw_http_mthd (53.08%) only make sense for FTP and HTTP sessions respectively, a session of another protocol has nothing to fill in there. There's no blind imputation to do here, because the absence itself is already information (this session isn't FTP, for instance), erasing or filling those columns with any value would destroy that signal.

A subtle bug: when "empty" doesn't count as missing

I said only three columns have missing values. But I noticed something odd looking at the values of ct_ftp_cmd (the number of FTP commands in the session, which should be a number) in the list of categoricals:

ct_ftp_cmd:
     1429879
0     686192
0     380306
1      29605
1      10472

Two clues here. First: the most common category, with 1,429,879 sessions, has no label at all, it's an empty string (""), not a NaN. And 1,429,879 is exactly the number of missing values in is_ftp_login from above, meaning every session without an FTP login also has ct_ftp_cmd unfilled, which makes sense, but pandas never counted it as missing because technically it isn't NaN, it's an empty string disguised as a valid category. df.isnull().sum() simply never saw this problem. Second clue: "0" shows up twice in the list, once with 686,192 sessions and once with 380,306, because one has a trailing space glued on ("0 ") and the other doesn't, the same kind of typo mess I already saw in attack_cat. Those two things combined (an empty string instead of a null, plus whitespace duplicating a category) are why ct_ftp_cmd turned into object back in the dtype table: a column that should be purely numeric got "contaminated" enough for pandas to give up reading it as a number.

The lesson that sticks: df.isnull().sum() is a good first filter, but it isn't the final word on what's actually missing. An empty string, a lone space, a generic "-" (like the one I already saw in service), all of that sails right past a naive .isnull(). Always worth cross-checking with df.dtypes and being suspicious of any naturally-numeric column that landed as object.

Selecting (or not selecting) features

With the numeric and categorical EDA in hand, the central question of any project with lots of variables shows up: which ones do I actually use? The course treats that choice as feature selection, and there are three classic families:

• Filter: a statistical criterion, independent of any model. The redundancy analysis by correlation above is exactly that: two variables with r ≈ 1.00 carry almost the same information, one would be a candidate to drop.
• Wrapper: testing subsets of variables by actually running the model (recursive feature elimination, for instance). Powerful, but expensive, because every subset tested is a full training run.
• Embedded: the model itself scores each variable's importance while it trains. That's what the Random Forest's importance does, and I get there in chapter 3.

Before deciding anything, I ran a histogram on eight numeric features chosen only to inspect the shape of the distributions, not to rank importance:

features_ilustrativas = ["dur", "sbytes", "dbytes", "sttl", "dttl", "sloss", "dloss", "Sload"]

axes = df[features_ilustrativas].hist(figsize=(14, 8), bins=50, color="steelblue",
                                      edgecolor="black", log=True)
for ax in axes.ravel():
    ax.set_xlabel("Valor da feature")
    ax.set_ylabel("Frequencia (escala log)")

plt.suptitle("Distribuicao de Features Numericas (amostra ilustrativa)", y=1.02)
plt.tight_layout()
plt.show()

The comment I left next to this cell in the original notebook is worth repeating here, because it heads off a wrong reading: these eight features are illustrative, a sample to inspect the shape of the distribution, not the result of any importance-based selection (the real importance comes from the correlation I already covered, and from the Random Forest's own importance in chapter 3). The Y axis uses a log scale (log=True) on purpose: bytes transferred and session duration have extremely skewed distributions, most sessions are short and light, a minority are far heavier, and on a linear scale that minority would flatten the majority into a line at zero.

With that visual check, plus the boxplots by class and the correlation already covered above, it's worth writing down the decision I made: I'm not pruning any feature in the pipeline. The Random Forest I use in chapter 3 is reasonably robust to redundant variables and to differences in scale between them, so I treat this selection as interpretive analysis (understanding what weighs on the decision) rather than a step that actually changes what enters the model. That's a deliberate choice, not laziness: a Logistic Regression or a KNN would probably benefit from cutting the redundancy found here, but for the model I choose in the next chapter, the gain wouldn't be worth the extra complexity of deciding "which of these two nearly-identical columns do I cut."

Main takeaways, before modeling

Closing this first chapter, the points I'm carrying with me for the rest of the project:

In chapter 2 I swap the raw dataset I used this whole chapter for the official partition the UNSW-NB15 authors themselves curated and published, explain in detail why I'm making that switch (it's about data leakage, not a whim), build the preprocessing pipeline, and compare four model families on a fair cross-validation before choosing which one to actually train.