Good AUC, Zero F1: the Default Threshold Trap

Chapter 3 picked the features for each of the four scenarios. Now it's time to actually train the models, compare them against a baseline, and log every attempt with MLflow (parameters and metrics per run, kept local only, mlruns/, never uploaded anywhere, it's a traceability good practice, not something that turns into a chart here on the blog). Full executed notebook, public on Colab.

Scenario 1: delivery delay, and the default threshold trap

Baseline (always guesses "on time") → Logistic Regression → Random Forest → XGBoost, on Chapter 3's honest features (95,968 orders, real delay rate of 6.76%):

Loading real data...

Something odd jumps out of that table: Random Forest has the best AUC of the bunch (0.7388), but F1, precision and recall are all zero. Not a bug, it's just the mechanics of it. AUC measures whether the model can rank risky orders above safe ones, comparing pairs of examples without ever committing to a decision, and Random Forest is genuinely good at that. F1, precision and recall, on the other hand, depend on a binary call: above 0.5 probability the model shouts "this one's late!", below it stays quiet. With only 6.76% of orders actually late, Random Forest learned the majority class so well it never crosses 0.5 probability for any order, not even the ones that really are late. It knows how to rank risk, it just never "bets" on the positive class.

Precision here means: of the orders the model flagged as "will be late", how many actually were. Recall is the flip side: of the orders that actually were late, how many did the model catch. F1 is the harmonic mean of the two, a way to summarize "the model is good on both ends" into one number. Same trio I already used back in the Pattern Recognition playlist, on credit card fraud detection, another badly imbalanced classification problem, and the symptom looks the same: a ranking metric (AUC) looks great, a decision metric (F1) exposes that the model, as it stands, can't decide anything on its own.

XGBoost, despite a slightly lower AUC (0.7261), is the only one that actually "bets": F1 0.0268, precision 0.40 (when it flags "will be late", it's right 4 out of 10 times), recall 0.0139 (but it only catches 1.4% of real delays). None of the four is production-ready as published here. The real next step would be tuning the decision threshold instead of blindly using 0.5, or balancing the class during training, but that's a topic for another day, this chapter is about comparing models, not tuning decision thresholds.

Scenario 2: review score (regression)

Baseline (always guesses the mean) → Linear Regression → Random Forest → XGBoost, on Chapter 3's features (95,829 orders):

Loading real data...

R² measures how much of the score's variation the model explains (1.0 would be a perfect prediction, 0 would be "always guess the mean"). MAE is the average error in stars, how far off the model lands, on average, from the real score. Random Forest edges out XGBoost here (0.2028 vs. 0.1861), the opposite of what you'd usually expect from those two. But the more honest number is how low the ceiling is for both: not even the best model clears 20% of explained variance. That matches what Chapter 3 already found using mutual information, atraso_dias and delivery_days dominate the score by a wide margin, leaving little signal in the remaining features (price, freight, installments) for any model to exploit. The customer's rating depends on a lot of things that aren't in this dataframe (actual product quality, packaging, support), it's not a lack of a better model, it's a lack of feature.

Scenario 3: freight value (regression)

Baseline → Linear Regression → Random Forest → XGBoost, on the product's physical features plus Chapter 3's distance_km (98,650 orders):

Loading real data...

This is where XGBoost clearly wins over the other three, R² 0.6385 and an average error of R$ 4.31. Makes physical sense: carriers compute freight with a formula that blends weight, dimensions and distance in a non-linear way (volumetric weight, distance price brackets), and tree-based models (Random Forest, XGBoost) capture that kind of non-linearity far better than a Linear Regression's straight line, which still comes in a respectable second (R² 0.53). Compared to Scenario 2's low ceiling, this is the scenario where "more relevant numeric data" genuinely buys real model performance.

Scenario 4: customer segmentation, first pass at RFM

Unlike the previous three, there's no "correct model" to compare here, it's K-means running across several values of K (from 2 to 8) on Chapter 3's three scaled RFM variables (recency, frequency, monetary value). To pick K, I used a different metric than the elbow method the professor used back in the Pattern Recognition playlist: the silhouette coefficient. It measures, for each customer, how similar they are to their own cluster compared to the nearest neighboring cluster, and the average across everyone becomes that K's score. It ranges from -1 (customer in the wrong cluster) to 1 (tight, well-separated clusters).

Loading real data...

K=2 leads by a wide margin (0.743, nearly double any other K), and inertia drops smoothly and continuously with no obvious elbow, two signals agreeing: the statistically "cleanest" cut is splitting the base into just two groups. Except I already know, from Chapter 3, what that cut probably IS: the 3.06% of customers who bought more than once against the 96.94% one-time buyers. Statistically it's the "purest" K, but from a business angle it's almost the same information I already had without running any clustering at all. From K=3 onward the silhouette bounces around without a clear pattern (between 0.42 and 0.49), with no clear second-best K standing out. I'll carry that tension (statistically optimal K vs. a segmentation rich enough to act on) into Chapter 5, which is where I actually pick a final K, visualize the clusters, and interpret what each one means.

Wrapping up the chapter

All four models trained and compared against a baseline, with MLflow logging every attempt. The most important finding wasn't "which model won", it was the demonstration that a good AUC doesn't guarantee a good decision: Scenario 1's Random Forest has the best ranking metric and zero practical use at the default threshold, a real trap for anyone who checks one metric and calls it a day. Freight is the scenario where the model "buys" the most real performance (XGBoost, R² 0.64), review score runs into a low ceiling of available information, and RFM raises an open question about what "better" clustering even means, which carries into Chapter 5, along with SHAP to open up these models' black box and the project's final conclusions.