Data Leakage: the Easiest Way to Fool Yourself

Chapters 1 and 2 built the master dataframe and showed six exploratory angles. Now I need to pick, for each of the four scenarios defined back in Chapter 1, which features actually make sense to use. Full executed notebook, public on Colab.

A new feature first: customer-seller distance

Chapter 2 showed that customer state correlates with delivery time. Before diving into the scenarios, I computed a real distance, not just "same state or not": average latitude/longitude per zip code prefix (olist_geolocation_dataset, over a million rows, hence averaging first), and the Haversine formula between customer and seller.

geo_media = geolocation.groupby('geolocation_zip_code_prefix')[['geolocation_lat', 'geolocation_lng']].mean().reset_index()

def haversine(lat1, lon1, lat2, lon2):
    R = 6371
    phi1, phi2 = np.radians(lat1), np.radians(lat2)
    dphi = np.radians(lat2 - lat1)
    dlambda = np.radians(lon2 - lon1)
    a = np.sin(dphi / 2) ** 2 + np.cos(phi1) * np.cos(phi2) * np.sin(dlambda / 2) ** 2
    return 2 * R * np.arcsin(np.sqrt(a))

Found a real bug at this step before publishing: on the first run, the merge between customer_zip_code_prefix/seller_zip_code_prefix and the geolocation table lost 70% of the data (only 30% coverage), because the zip-code column's data type wasn't guaranteed to match on both sides across every environment. Forced explicit int64 on all three before merging, and coverage went back to what it should be: only 581 orders out of 118,310 ended up without a distance (99.5% coverage). Average customer-seller distance across all of Brazil is 597 km, with a max of 8,678 km, a number that proves just how huge the country is.

Scenario 1: delivery delay, and the data leakage demonstration

Candidates: distance_km, product_weight_g, price, freight_value, approval time (approval_hours), purchase month. The atrasado target is defined by comparing order_delivered_customer_date against order_estimated_delivery_date. And this is where the classic trap of every ML course lives: if I include atraso_dias (the actual difference between those two dates) as a feature, the model isn't learning to predict delay, it's reading the answer straight off the exam.

features_honestas = ['distance_km', 'product_weight_g', 'price', 'freight_value', 'approval_hours', 'month']
features_vazadas = features_honestas + ['atraso_dias']

Trained both, a plain RandomForest, same seed, same train/test split:

Loading real data...

The honest model lands at AUC 0.7388 (95,968 orders used, real delay rate of 6.76%). The leaky model lands at AUC 1.0, perfect, because atraso_dias > 0 is practically the target's own definition. One detail worth calling out: the honest model's raw accuracy (93.24%) looks great at first glance, but since only 6.76% of orders are late, a dumb model that always guesses "on time" already scores 93.24% without learning anything. Accuracy alone is misleading with classes this imbalanced, AUC is the metric that tells the real story here.

Also worth noting, the honest model's feature importance:

Loading real data...

month leads by a wide margin, at 0.36 importance, practically double the runner-up (distance_km, 0.18). This ties directly back to Chapter 2's finding: November (Black Friday) strains logistics in a way that physical distance alone can't explain. Seasonality matters more for predicting delay than how far the seller is from the customer.

Scenario 2: review score

Candidates: atraso_dias, delivery time, price, freight, number of installments. Since review_score is a bimodal ordinal scale (Chapter 2 already showed this), I used mutual information instead of plain Pearson correlation:

mi = mutual_info_classif(X2, y2, random_state=42)

atraso_dias and delivery_days dominate, almost 10 times more informative than price or freight (95,829 orders used). Confirms, with a different technique, exactly what Chapter 2's correlation already pointed at.

Scenario 3: freight value

Candidates: weight and the product's three dimensions. Plain Pearson correlation is enough here (98,650 orders used):

Weight moderately dominates over any single dimension, which makes physical sense: carriers charge by volumetric weight, but actual weight pulls the bill harder than any one side of the box alone.

Scenario 4: customer segmentation (RFM)

This isn't feature selection in the supervised sense, it's engineering the three variables that will feed Chapter 5's clustering: Recency, Frequency, and Monetary value per customer.

rfm = oc.groupby('customer_unique_id').agg(
    recencia_dias=('order_purchase_timestamp', lambda x: (data_max - x.max()).days),
    frequencia=('order_id', 'nunique'),
    valor_monetario=('payment_value', 'sum'),
).reset_index()

95,560 unique customers. And the number that stood out the most: only 2,924 customers (3.06% of the total) placed more than one order. Average frequency is 1.034, meaning the entire marketplace is dominated by one-time purchases. That completely changes how to think about segmentation in Chapter 5: there won't be a large "loyal customer" category, RFM here will likely separate more by amount spent and recency than by purchase frequency.

Wrapping up the chapter

Customer-seller distance computed (and a real merge bug fixed along the way), data leakage demonstrated in practice (the gap between honest AUC 0.74 and leaky AUC 1.0 is the most concrete lesson I can give on the subject), and each scenario's features picked with a technique suited to its problem type. Seasonality turned out to matter more than distance for delay, delay dominates the review score, weight dominates freight, and the customer base is overwhelmingly one-time buyers. Next chapter I train the real models for each scenario, with experiment tracking and metric comparison.