[{"data":1,"prerenderedAt":477},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fprojects\u002Folist-ecommerce\u002F01-intro-relational-model-eda":3,"chapter-en-olist-ecommerce-01-intro-relational-model-eda":4},null,{"id":5,"title":6,"body":7,"cover":3,"date":464,"description":465,"extension":466,"meta":467,"navigation":306,"order":45,"path":468,"project":469,"seo":470,"status":471,"stem":472,"tags":473,"__hash__":476},"projectChapters\u002Fen\u002Fprojects\u002Folist-ecommerce\u002F01-intro-relational-model-eda.md","Nine Tables, One Business: Olist's Relational Model",{"type":8,"value":9,"toc":456},"minimark",[10,27,32,97,100,140,144,147,267,272,276,292,355,374,378,381,427,431,434,438,445,449,452],[11,12,13,14,18,19,26],"p",{},"Before training any model, I need to understand the shape of the data. And the Olist dataset isn't a single table, it's a genuine mini relational database: nine CSVs, each representing a business entity (order, item, payment, review, customer, seller, product), tied together by keys. I ran all of this in the ",[15,16,17],"code",{},"01_intro_eda.ipynb"," notebook, right inside the dataset's own directory, and left it public on Colab: ",[20,21,25],"a",{"href":22,"rel":23},"https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1kQvo5YjlLPiODqGhlAGZNk84ikg-PZg7?usp=sharing",[24],"nofollow","check out the full notebook here",".",[28,29,31],"h2",{"id":30},"the-relational-model","The relational model",[33,34,39],"pre",{"className":35,"code":36,"language":37,"meta":38,"style":38},"language-python shiki shiki-themes github-light github-dark","orders = pd.read_csv('olist_orders_dataset.csv')\norder_items = pd.read_csv('olist_order_items_dataset.csv')\npayments = pd.read_csv('olist_order_payments_dataset.csv')\nreviews = pd.read_csv('olist_order_reviews_dataset.csv')\ncustomers = pd.read_csv('olist_customers_dataset.csv')\nsellers = pd.read_csv('olist_sellers_dataset.csv')\nproducts = pd.read_csv('olist_products_dataset.csv')\ngeolocation = pd.read_csv('olist_geolocation_dataset.csv')\ncategory_translation = pd.read_csv('product_category_name_translation.csv')\n","python","",[15,40,41,49,55,61,67,73,79,85,91],{"__ignoreMap":38},[42,43,46],"span",{"class":44,"line":45},"line",1,[42,47,48],{},"orders = pd.read_csv('olist_orders_dataset.csv')\n",[42,50,52],{"class":44,"line":51},2,[42,53,54],{},"order_items = pd.read_csv('olist_order_items_dataset.csv')\n",[42,56,58],{"class":44,"line":57},3,[42,59,60],{},"payments = pd.read_csv('olist_order_payments_dataset.csv')\n",[42,62,64],{"class":44,"line":63},4,[42,65,66],{},"reviews = pd.read_csv('olist_order_reviews_dataset.csv')\n",[42,68,70],{"class":44,"line":69},5,[42,71,72],{},"customers = pd.read_csv('olist_customers_dataset.csv')\n",[42,74,76],{"class":44,"line":75},6,[42,77,78],{},"sellers = pd.read_csv('olist_sellers_dataset.csv')\n",[42,80,82],{"class":44,"line":81},7,[42,83,84],{},"products = pd.read_csv('olist_products_dataset.csv')\n",[42,86,88],{"class":44,"line":87},8,[42,89,90],{},"geolocation = pd.read_csv('olist_geolocation_dataset.csv')\n",[42,92,94],{"class":44,"line":93},9,[42,95,96],{},"category_translation = pd.read_csv('product_category_name_translation.csv')\n",[98,99],"olist-relational-model",{},[11,101,102,105,106,109,110,109,113,116,117,120,121,109,124,109,127,130,131,134,135,139],{},[15,103,104],{},"order_id"," is the key stitching almost everything together: it shows up in ",[15,107,108],{},"orders",", ",[15,111,112],{},"order_items",[15,114,115],{},"payments",", and ",[15,118,119],{},"reviews",". ",[15,122,123],{},"customer_id",[15,125,126],{},"product_id",[15,128,129],{},"seller_id",", and the zip code prefix close out the rest of the links. One detail worth flagging already: ",[15,132,133],{},"olist_order_reviews_dataset.csv"," has 104,719 raw lines of text in the file, but pandas only recognizes ",[136,137,138],"strong",{},"99,224 real records"," when reading the CSV. The difference is because a lot of review comments contain literal line breaks inside the text field (the customer wrote across several paragraphs), and pandas' parser correctly counts that as one single record, while counting raw file lines overestimates. A good reminder that \"number of lines in the file\" and \"number of records\" aren't always the same thing in a CSV with free-text fields.",[28,141,143],{"id":142},"data-quality-nulls-have-a-story","Data quality: nulls have a story",[11,145,146],{},"Not every null is a problem, sometimes it's information. Across the tables where nulls show up:",[148,149,150,171],"table",{},[151,152,153],"thead",{},[154,155,156,161,164,168],"tr",{},[157,158,160],"th",{"align":159},"left","Table",[157,162,163],{"align":159},"Column",[157,165,167],{"align":166},"right","Nulls",[157,169,170],{"align":166},"%",[172,173,174,188,201,214,227,240,254],"tbody",{},[154,175,176,179,182,185],{},[177,178,108],"td",{"align":159},[177,180,181],{"align":159},"order_approved_at",[177,183,184],{"align":166},"160",[177,186,187],{"align":166},"0.2%",[154,189,190,192,195,198],{},[177,191,108],{"align":159},[177,193,194],{"align":159},"order_delivered_carrier_date",[177,196,197],{"align":166},"1,783",[177,199,200],{"align":166},"1.8%",[154,202,203,205,208,211],{},[177,204,108],{"align":159},[177,206,207],{"align":159},"order_delivered_customer_date",[177,209,210],{"align":166},"2,965",[177,212,213],{"align":166},"3.0%",[154,215,216,218,221,224],{},[177,217,119],{"align":159},[177,219,220],{"align":159},"review_comment_title",[177,222,223],{"align":166},"87,656",[177,225,226],{"align":166},"88.3%",[154,228,229,231,234,237],{},[177,230,119],{"align":159},[177,232,233],{"align":159},"review_comment_message",[177,235,236],{"align":166},"58,247",[177,238,239],{"align":166},"58.7%",[154,241,242,245,248,251],{},[177,243,244],{"align":159},"products",[177,246,247],{"align":159},"product_category_name (+ 3 other product columns)",[177,249,250],{"align":166},"610",[177,252,253],{"align":166},"1.9%",[154,255,256,258,261,264],{},[177,257,244],{"align":159},[177,259,260],{"align":159},"weight\u002Fdimensions (4 columns)",[177,262,263],{"align":166},"2",[177,265,266],{"align":166},"0.0%",[11,268,269,271],{},[15,270,207],{}," being null on 3% of orders isn't a capture error, it's an order that never arrived (canceled, lost, or still in transit when the dataset was frozen). That becomes an important feature down the line, when I build the delay scenario: an order with no delivery date has no way to compute a delay, so those 2,965 orders need explicit handling (excluded from the delay analysis, or treated as their own category), I can't just fill them with zero or the mean. Reviews missing a title or comment (88% and 59% of cases) aren't a problem either, most customers just leave the star rating and write nothing, completely normal e-commerce review behavior.",[28,273,275],{"id":274},"building-the-master-dataframe","Building the master dataframe",[11,277,278,279,282,283,285,286,288,289,291],{},"The dataset's natural granularity is ",[136,280,281],{},"order item",", not whole order: one ",[15,284,104],{}," can have several items, from different sellers, each with its own price and freight. That's why the merge starts from ",[15,287,112],{},", not ",[15,290,108],{},":",[33,293,295],{"className":35,"code":294,"language":37,"meta":38,"style":38},"produtos_com_categoria_en = products.merge(category_translation, on='product_category_name', how='left')\n\nmestre = (\n    order_items\n    .merge(orders, on='order_id', how='left')\n    .merge(customers, on='customer_id', how='left')\n    .merge(produtos_com_categoria_en, on='product_id', how='left')\n    .merge(sellers, on='seller_id', how='left')\n    .merge(payments, on='order_id', how='left')\n    .merge(reviews, on='order_id', how='left')\n)\n",[15,296,297,302,308,313,318,323,328,333,338,343,349],{"__ignoreMap":38},[42,298,299],{"class":44,"line":45},[42,300,301],{},"produtos_com_categoria_en = products.merge(category_translation, on='product_category_name', how='left')\n",[42,303,304],{"class":44,"line":51},[42,305,307],{"emptyLinePlaceholder":306},true,"\n",[42,309,310],{"class":44,"line":57},[42,311,312],{},"mestre = (\n",[42,314,315],{"class":44,"line":63},[42,316,317],{},"    order_items\n",[42,319,320],{"class":44,"line":69},[42,321,322],{},"    .merge(orders, on='order_id', how='left')\n",[42,324,325],{"class":44,"line":75},[42,326,327],{},"    .merge(customers, on='customer_id', how='left')\n",[42,329,330],{"class":44,"line":81},[42,331,332],{},"    .merge(produtos_com_categoria_en, on='product_id', how='left')\n",[42,334,335],{"class":44,"line":87},[42,336,337],{},"    .merge(sellers, on='seller_id', how='left')\n",[42,339,340],{"class":44,"line":93},[42,341,342],{},"    .merge(payments, on='order_id', how='left')\n",[42,344,346],{"class":44,"line":345},10,[42,347,348],{},"    .merge(reviews, on='order_id', how='left')\n",[42,350,352],{"class":44,"line":351},11,[42,353,354],{},")\n",[11,356,357,359,360,363,364,366,367,369,370,373],{},[15,358,112],{}," alone has 112,650 rows. After all the merges, the master dataframe has ",[136,361,362],{},"118,310 rows",", more than the starting point. That's not a bug: when an order is paid across several installments recorded as separate rows in ",[15,365,115],{},", or gets more than one entry in ",[15,368,119],{},", the merge multiplies that order-item row for each combination. It's expected relational-merge behavior, but it's exactly the kind of thing that, if I don't check the ",[15,371,372],{},"shape"," before and after, slips by unnoticed and inflates counts in any later aggregation.",[28,375,377],{"id":376},"the-four-ml-scenarios","The four ML scenarios",[11,379,380],{},"With the master dataframe in hand, defined for the chapters ahead:",[382,383,384,397,407,421],"ol",{},[385,386,387,390,391,393,394,26],"li",{},[136,388,389],{},"Delivery delay"," (binary classification): compare ",[15,392,207],{}," against ",[15,395,396],{},"order_estimated_delivery_date",[385,398,399,402,403,406],{},[136,400,401],{},"Review score"," (multiclass classification or regression): ",[15,404,405],{},"review_score",", from 1 to 5.",[385,408,409,412,413,416,417,420],{},[136,410,411],{},"Freight or order value"," (regression): ",[15,414,415],{},"freight_value",", or the sum of ",[15,418,419],{},"price"," per order.",[385,422,423,426],{},[136,424,425],{},"Customer segmentation"," (clustering, no target): Recency, Frequency, and Monetary value per customer, the RFM technique.",[28,428,430],{"id":429},"orders-per-month-the-black-friday-spike","Orders per month: the Black Friday spike",[11,432,433],{},"A first temporal look, counting unique orders by purchase month:",[435,436],"olist-orders-per-month-chart",{"x-label":437,"y-label":108},"month",[11,439,440,441,444],{},"September 2016 starts with only 4 orders (Olist had barely launched), volume grows month over month through 2017, and November 2017 jumps sharply to ",[136,442,443],{},"7,544 orders",", against 4,631 in October and 5,673 in December of the same year. That's Black Friday, an isolated spike that breaks the smooth growth trend, and it's going to matter when I get to seasonality in the feature-engineering chapters. September and October 2018 show up with only 16 and 4 orders, a sign the dataset was frozen mid-month, not that sales collapsed.",[28,446,448],{"id":447},"wrapping-up-the-chapter","Wrapping up the chapter",[11,450,451],{},"Relational model mapped, master dataframe assembled (118,310 rows), data quality checked (the delivery and review nulls have real explanations, not errors), and the four scenarios defined. Next chapter I get into the real visualization work: a geographic map of orders and delay by state, category distribution, payment method, and the relationship between delay and review score, which already gives an \"aha moment\" before training a single model.",[453,454,455],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":38,"searchDepth":51,"depth":51,"links":457},[458,459,460,461,462,463],{"id":30,"depth":51,"text":31},{"id":142,"depth":51,"text":143},{"id":274,"depth":51,"text":275},{"id":376,"depth":51,"text":377},{"id":429,"depth":51,"text":430},{"id":447,"depth":51,"text":448},"2026-08-24","First chapter of the Olist case study: I load the nine CSVs, build the master dataframe, check real data quality, and define the four ML scenarios the rest of the project will cover.","md",{},"\u002Fen\u002Fprojects\u002Folist-ecommerce\u002F01-intro-relational-model-eda","olist-ecommerce",{"title":6,"description":465},"published","en\u002Fprojects\u002Folist-ecommerce\u002F01-intro-relational-model-eda",[474,475],"pandas","eda","M6y2XePQUxjrKgwzJwkI_8pqI8zOp2HhATq5VmgyFiU",1787605216016]