Good AUC, Zero F1: the Default Threshold Trap

Chapter 3 picked the features for each of the four scenarios. Now it's time to actually train the models, compare them against a baseline, and log every attempt with MLflow (parameters and metrics per run, kept local only, mlruns/, never uploaded anywhere, it's a traceability good practice, not something that turns into a chart here on the blog). Full executed notebook, public on Colab. This notebook also starts from scratch (reloads the 9 CSVs, rebuilds the master dataframe) and rebuilds Chapter 3's features:

geo_media = geolocation.groupby('geolocation_zip_code_prefix')[['geolocation_lat', 'geolocation_lng']].mean().reset_index()

def haversine(lat1, lon1, lat2, lon2):
    R = 6371
    phi1, phi2 = np.radians(lat1), np.radians(lat2)
    dphi = np.radians(lat2 - lat1)
    dlambda = np.radians(lon2 - lon1)
    a = np.sin(dphi / 2) ** 2 + np.cos(phi1) * np.cos(phi2) * np.sin(dlambda / 2) ** 2
    return 2 * R * np.arcsin(np.sqrt(a))

m = mestre.copy()
m['customer_zip_code_prefix'] = m['customer_zip_code_prefix'].astype('int64')
m['seller_zip_code_prefix'] = m['seller_zip_code_prefix'].astype('int64')
geo_media['geolocation_zip_code_prefix'] = geo_media['geolocation_zip_code_prefix'].astype('int64')
m = m.merge(geo_media.rename(columns={'geolocation_zip_code_prefix': 'customer_zip_code_prefix', 'geolocation_lat': 'cust_lat', 'geolocation_lng': 'cust_lng'}), on='customer_zip_code_prefix', how='left')
m = m.merge(geo_media.rename(columns={'geolocation_zip_code_prefix': 'seller_zip_code_prefix', 'geolocation_lat': 'sell_lat', 'geolocation_lng': 'sell_lng'}), on='seller_zip_code_prefix', how='left')
m['distance_km'] = haversine(m['cust_lat'], m['cust_lng'], m['sell_lat'], m['sell_lng'])
m['approval_hours'] = (m['order_approved_at'] - m['order_purchase_timestamp']).dt.total_seconds() / 3600
m['month'] = m['order_purchase_timestamp'].dt.month
m['atraso_dias'] = (m['order_delivered_customer_date'] - m['order_estimated_delivery_date']).dt.days
m['delivery_days'] = (m['order_delivered_customer_date'] - m['order_purchase_timestamp']).dt.days

Same code as always: distance_km (Haversine, with Chapter 3's int64 fix), approval_hours, month, atraso_dias, and delivery_days, all already covered carefully back there.

Scenario 1: delivery delay, and the default threshold trap

Baseline (always guesses "on time") → Logistic Regression → Random Forest → XGBoost, on Chapter 3's honest features:

delay_df = (
    m.drop_duplicates(subset='order_id')
    [['order_id', 'distance_km', 'product_weight_g', 'price', 'freight_value', 'approval_hours', 'month', 'atraso_dias']]
    .dropna()
)
delay_df['atrasado'] = (delay_df['atraso_dias'] > 0).astype(int)

features_1 = ['distance_km', 'product_weight_g', 'price', 'freight_value', 'approval_hours', 'month']
X1, y1 = delay_df[features_1], delay_df['atrasado']
X1tr, X1te, y1tr, y1te = train_test_split(X1, y1, test_size=0.2, random_state=42, stratify=y1)

modelos_1 = {
    'Baseline': DummyClassifier(strategy='most_frequent'),
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'Random Forest': RandomForestClassifier(n_estimators=100, max_depth=8, random_state=42),
    'XGBoost': xgb.XGBClassifier(n_estimators=100, max_depth=6, random_state=42, eval_metric='logloss'),
}

resultados_1 = []
for nome, modelo in modelos_1.items():
    with mlflow.start_run(run_name=f"atraso_{nome}"):
        modelo.fit(X1tr, y1tr)
        proba = modelo.predict_proba(X1te)[:, 1]
        pred = modelo.predict(X1te)
        auc = roc_auc_score(y1te, proba)
        f1 = f1_score(y1te, pred)
        prec = precision_score(y1te, pred, zero_division=0)
        rec = recall_score(y1te, pred)
        mlflow.log_params({'modelo': nome, 'cenario': 'atraso'})
        mlflow.log_metrics({'auc': auc, 'f1': f1, 'precision': prec, 'recall': rec})
        resultados_1.append({'model': nome, 'value': round(float(auc), 4)})

DummyClassifier(strategy='most_frequent') is the most honest baseline there is: always guesses the majority class ("on time"), without looking at a single feature, any real model has to beat this to justify its own existence (95,968 orders used, real delay rate of 6.76%). The four models go into a name → instance dictionary, and the for loop trains, predicts, and scores all four the exact same way, inside a with mlflow.start_run(run_name=...) block: each iteration becomes a logged "run", mlflow.log_params records which model and which scenario produced that result, mlflow.log_metrics records the four metrics, all saved locally under mlruns/, queryable later (mlflow ui) without recomputing anything. predict_proba(...)[:, 1] grabs the continuous probability for the "late" class (for AUC), predict(...) already returns the binary decision thresholded at 0.5 by default (for F1/precision/recall), the same continuous-score-vs-binary-decision distinction that already showed up in the intrusion detection project.

Loading real data...

Something odd jumps out of that table: Random Forest has the best AUC of the bunch (0.7388), but F1, precision and recall are all zero. Not a bug, it's just the mechanics of it. AUC measures whether the model can rank risky orders above safe ones, comparing pairs of examples without ever committing to a decision, and Random Forest is genuinely good at that. F1, precision and recall, on the other hand, depend on a binary call: above 0.5 probability the model shouts "this one's late!", below it stays quiet. With only 6.76% of orders actually late, Random Forest learned the majority class so well it never crosses 0.5 probability for any order, not even the ones that really are late. It knows how to rank risk, it just never "bets" on the positive class.

Precision here means: of the orders the model flagged as "will be late", how many actually were. Recall is the flip side: of the orders that actually were late, how many did the model catch. F1 is the harmonic mean of the two, a way to summarize "the model is good on both ends" into one number. Same trio I already used back in the Pattern Recognition playlist, on credit card fraud detection, another badly imbalanced classification problem, and the symptom looks the same: a ranking metric (AUC) looks great, a decision metric (F1) exposes that the model, as it stands, can't decide anything on its own.

XGBoost, despite a slightly lower AUC (0.7261), is the only one that actually "bets": F1 0.0268, precision 0.40 (when it flags "will be late", it's right 4 out of 10 times), recall 0.0139 (but it only catches 1.4% of real delays). None of the four is production-ready as published here. The real next step would be tuning the decision threshold instead of blindly using 0.5, or balancing the class during training, but that's a topic for another day, this chapter is about comparing models, not tuning decision thresholds.

Scenario 2: review score (regression)

Baseline (always guesses the mean) → Linear Regression → Random Forest → XGBoost, on Chapter 3's features:

s2 = (
    m.drop_duplicates(subset='order_id')
    [['order_id', 'atraso_dias', 'delivery_days', 'price', 'freight_value', 'payment_installments', 'review_score']]
    .dropna()
)

features_2 = ['atraso_dias', 'delivery_days', 'price', 'freight_value', 'payment_installments']
X2, y2 = s2[features_2], s2['review_score']
X2tr, X2te, y2tr, y2te = train_test_split(X2, y2, test_size=0.2, random_state=42)

modelos_2 = {
    'Baseline': DummyRegressor(strategy='mean'),
    'Linear Regression': LinearRegression(),
    'Random Forest': RandomForestRegressor(n_estimators=100, max_depth=8, random_state=42),
    'XGBoost': xgb.XGBRegressor(n_estimators=100, max_depth=6, random_state=42),
}

resultados_2 = []
for nome, modelo in modelos_2.items():
    with mlflow.start_run(run_name=f"nota_{nome}"):
        modelo.fit(X2tr, y2tr)
        pred = modelo.predict(X2te)
        r2 = r2_score(y2te, pred)
        mae = mean_absolute_error(y2te, pred)
        mlflow.log_params({'modelo': nome, 'cenario': 'nota'})
        mlflow.log_metrics({'r2': r2, 'mae': mae})
        resultados_2.append({'model': nome, 'value': round(float(r2), 4)})

Same structure as Scenario 1 (a dictionary of models, a for loop with mlflow.start_run), except this is regression now: DummyRegressor(strategy='mean') is the baseline that always guesses the training set's average score, and the metrics switch to R² (how much of the score's variance the model explains) and MAE (mean absolute error, in stars). The review score, 1 through 5, becomes a continuous target here instead of a categorical one, a choice Chapter 3 already justified (95,829 orders used).

Loading real data...

R² measures how much of the score's variation the model explains (1.0 would be a perfect prediction, 0 would be "always guess the mean"). MAE is the average error in stars, how far off the model lands, on average, from the real score. Random Forest edges out XGBoost here (0.2028 vs. 0.1861), the opposite of what you'd usually expect from those two. But the more honest number is how low the ceiling is for both: not even the best model clears 20% of explained variance. That matches what Chapter 3 already found using mutual information, atraso_dias and delivery_days dominate the score by a wide margin, leaving little signal in the remaining features (price, freight, installments) for any model to exploit. The customer's rating depends on a lot of things that aren't in this dataframe (actual product quality, packaging, support), it's not a lack of a better model, it's a lack of feature.

Scenario 3: freight value (regression)

Baseline → Linear Regression → Random Forest → XGBoost, on the product's physical features plus Chapter 3's distance_km:

s3 = (
    m.drop_duplicates(subset='order_id')
    [['order_id', 'freight_value', 'product_weight_g', 'product_length_cm', 'product_height_cm', 'product_width_cm', 'distance_km']]
    .dropna()
)

features_3 = ['product_weight_g', 'product_length_cm', 'product_height_cm', 'product_width_cm', 'distance_km']
X3, y3 = s3[features_3], s3['freight_value']
X3tr, X3te, y3tr, y3te = train_test_split(X3, y3, test_size=0.2, random_state=42)

modelos_3 = {
    'Baseline': DummyRegressor(strategy='mean'),
    'Linear Regression': LinearRegression(),
    'Random Forest': RandomForestRegressor(n_estimators=100, max_depth=8, random_state=42),
    'XGBoost': xgb.XGBRegressor(n_estimators=100, max_depth=6, random_state=42),
}

resultados_3 = []
for nome, modelo in modelos_3.items():
    with mlflow.start_run(run_name=f"frete_{nome}"):
        modelo.fit(X3tr, y3tr)
        pred = modelo.predict(X3te)
        r2 = r2_score(y3te, pred)
        mae = mean_absolute_error(y3te, pred)
        mlflow.log_params({'modelo': nome, 'cenario': 'frete'})
        mlflow.log_metrics({'r2': r2, 'mae': mae})
        resultados_3.append({'model': nome, 'value': round(float(r2), 4)})

Same skeleton again (dictionary of models, for loop with mlflow.start_run, R²/MAE), this time with distance_km added to the four physical product features Chapter 3 had already isolated by correlation (98,650 orders used).

Loading real data...

This is where XGBoost clearly wins over the other three, R² 0.6385 and an average error of R$ 4.31. Makes physical sense: carriers compute freight with a formula that blends weight, dimensions and distance in a non-linear way (volumetric weight, distance price brackets), and tree-based models (Random Forest, XGBoost) capture that kind of non-linearity far better than a Linear Regression's straight line, which still comes in a respectable second (R² 0.53). Compared to Scenario 2's low ceiling, this is the scenario where "more relevant numeric data" genuinely buys real model performance.

Scenario 4: customer segmentation, first pass at RFM

Unlike the previous three, there's no "correct model" to compare here, it's K-means running across several values of K (from 2 to 8) on Chapter 3's three RFM variables (recency, frequency, monetary value).

data_max = orders['order_purchase_timestamp'].max()
orders_validos = orders[orders['order_status'] != 'canceled']
oc = orders_validos.merge(customers, on='customer_id', how='left')
pagamento_por_pedido = payments.groupby('order_id')['payment_value'].sum().reset_index()
oc = oc.merge(pagamento_por_pedido, on='order_id', how='left')

rfm = oc.groupby('customer_unique_id').agg(
    recencia_dias=('order_purchase_timestamp', lambda x: (data_max - x.max()).days),
    frequencia=('order_id', 'nunique'),
    valor_monetario=('payment_value', 'sum'),
).reset_index().dropna()

scaler = StandardScaler()
rfm_escalado = scaler.fit_transform(rfm[['recencia_dias', 'frequencia', 'valor_monetario']])

resultados_4 = []
for k in range(2, 9):
    with mlflow.start_run(run_name=f"rfm_k{k}"):
        kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
        labels = kmeans.fit_predict(rfm_escalado)
        sil = silhouette_score(rfm_escalado, labels, sample_size=5000, random_state=42)
        mlflow.log_params({'k': k, 'cenario': 'rfm'})
        mlflow.log_metrics({'silhouette': sil, 'inertia': kmeans.inertia_})
        resultados_4.append({'model': f'k={k}', 'value': round(float(sil), 4)})

Rebuilding RFM is identical to Chapter 3 (data_max, valid orders, recency/frequency/value). What's new here is StandardScaler: the three RFM variables live on very different scales (recency in days, frequency as a small count, monetary value in reais reaching into the thousands), and K-means decides clusters by Euclidean distance, so without scaling, whichever variable has the largest numeric range would dominate the distance on its own, the same reason for normalizing before any distance-based method that already showed up back in the Pattern Recognition playlist. The for k in range(2, 9) trains a different K-means for each candidate K. To pick which K is best, I used a different metric than the elbow method the professor used back in the playlist: the silhouette coefficient, computed with silhouette_score(..., sample_size=5000, random_state=42). It measures, for each customer, how similar they are to their own cluster compared to the nearest neighboring cluster, and the average across everyone becomes that K's score, ranging from -1 (customer in the wrong cluster) to 1 (tight, well-separated clusters). The sample_size=5000 exists because this metric compares every pair of points against each other, a cost that grows quadratically with the number of customers and would stall computing over the full nearly-95,000 customers, a random sample of 5,000 already estimates the score without stalling.

Loading real data...

K=2 leads by a wide margin (0.743, nearly double any other K), and inertia drops smoothly and continuously with no obvious elbow, two signals agreeing: the statistically "cleanest" cut is splitting the base into just two groups. Except I already know, from Chapter 3, what that cut probably IS: the 3.06% of customers who bought more than once against the 96.94% one-time buyers. Statistically it's the "purest" K, but from a business angle it's almost the same information I already had without running any clustering at all. From K=3 onward the silhouette bounces around without a clear pattern (between 0.42 and 0.49), with no clear second-best K standing out. I'll carry that tension (statistically optimal K vs. a segmentation rich enough to act on) into Chapter 5, which is where I actually pick a final K, visualize the clusters, and interpret what each one means.

Wrapping up the chapter

All four models trained and compared against a baseline, with MLflow logging every attempt. The most important finding wasn't "which model won", it was the demonstration that a good AUC doesn't guarantee a good decision: Scenario 1's Random Forest has the best ranking metric and zero practical use at the default threshold, a real trap for anyone who checks one metric and calls it a day. Freight is the scenario where the model "buys" the most real performance (XGBoost, R² 0.64), review score runs into a low ceiling of available information, and RFM raises an open question about what "better" clustering even means, which carries into Chapter 5, along with SHAP to open up these models' black box and the project's final conclusions.