[{"data":1,"prerenderedAt":4760},["ShallowReactive",2],{"lang-switch-post-\u002Fen":3,"home-playlists-en":4,"home-latest-posts-en":234,"home-latest-timeline-en":4685},null,[5,40,96,165,193],{"id":6,"title":7,"body":8,"cover":3,"description":30,"extension":31,"meta":32,"navigation":33,"order":34,"path":35,"seo":36,"status":37,"stem":38,"__hash__":39},"playlists\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Findex.md","Machine Learning Specialization (Andrew Ng)",{"type":9,"value":10,"toc":26},"minimark",[11,20,23],[12,13,14,15,19],"p",{},"This playlist is my public study notebook while going through the ",[16,17,18],"strong",{},"Machine Learning Specialization",", by Andrew Ng (DeepLearning.AI \u002F Stanford), probably the most recommended course for anyone getting started in ML.",[12,21,22],{},"The idea here isn't just \"solve the notebook and move on.\" Every lab in the course becomes a post where I retell what I understood, with an everyday-life metaphor, an example, and of course, the correct technical term, because you'll need it when you go looking for more on the topic later.",[12,24,25],{},"We start at the start: representing the simplest model there is, linear regression with one variable.",{"title":27,"searchDepth":28,"depth":28,"links":29},"",2,[],"My step-by-step notes going through Andrew Ng's Machine Learning Specialization (DeepLearning.AI \u002F Stanford), course by course, lab by lab.","md",{},true,1,"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization",{"title":7,"description":30},"published","en\u002Fplaylists\u002Fmachine-learning-specialization\u002Findex","doASMuy-kKTfF67w2hQII9iS08pq0gbnf-FwE2AmImQ",{"id":41,"title":42,"body":43,"cover":3,"description":90,"extension":31,"meta":91,"navigation":33,"order":28,"path":92,"seo":93,"status":37,"stem":94,"__hash__":95},"playlists\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Findex.md","Pattern Recognition",{"type":9,"value":44,"toc":88},[45,58,70,80],[12,46,47,48,51,52,57],{},"This playlist is my study notebook from the Pattern Recognition course I took, taught by ",[16,49,50],{},"Dr. Francisco Boldt",". The guy is excellent, he genuinely codes the models by hand, live, in class, no pre-baked formula slides, and he's teaching me ",[53,54,56],"a",{"href":55},"\u002Fen\u002Fplaylists\u002Fneural-networks","neural networks now too",". If you landed here coming from one of his classes, you already know what I mean.",[12,59,60,61,64,65,69],{},"Unlike the ",[53,62,63],{"href":35},"Andrew Ng specialization playlist",", the lecture notebooks here are a lot leaner: barely any markdown cells, it's the professor live-coding and everyone following along. So the work of digging into the \"why\" behind each line of code is heavier here, and for that I lean on ",[66,67,68],"em",{},"Pattern Recognition and Machine Learning",", by Christopher Bishop (2006), pretty much a bible in the field, as the theoretical reference.",[12,71,72,73,79],{},"The notebooks come from the course repository, ",[53,74,78],{"href":75,"rel":76},"https:\u002F\u002Fgithub.com\u002Fpablobelmiro\u002Faulasml\u002Ftree\u002F2026-1",[77],"nofollow","pablobelmiro\u002Faulasml",", a fork of Dr. Boldt's own repository, where he publishes each lecture's code. In this playlist's posts, whoever \"wrote\" the code is always him, the professor; the foundational explanation, with the metaphor, the slightly-off analogy, and the buddy-sitting-next-to-you tone, that part is mine.",[12,81,82,83,87],{},"Every lecture becomes a post here. We start at the start: the same ",[53,84,86],{"href":85},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation","linear regression problem from the other playlist",", except this time the code is being written live on the whiteboard.",{"title":27,"searchDepth":28,"depth":28,"links":89},[],"My notes from my Pattern Recognition course, lecture by lecture, with Bishop's book as the theoretical backbone.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition",{"title":42,"description":90},"en\u002Fplaylists\u002Fpattern-recognition\u002Findex","y_rvvoQ-pc7yj5eUqMw_dVQddrlu3I5aEN4sROdZOdk",{"id":97,"title":98,"body":99,"cover":3,"description":159,"extension":31,"meta":160,"navigation":33,"order":161,"path":55,"seo":162,"status":37,"stem":163,"__hash__":164},"playlists\u002Fen\u002Fplaylists\u002Fneural-networks\u002Findex.md","Neural Networks",{"type":9,"value":100,"toc":157},[101,110,127,141],[12,102,103,104,106,107,109],{},"Second playlist with ",[16,105,50],{}," (the first was ",[53,108,42],{"href":92},"), now in his Neural Networks course. Same style as always: code by hand, live, in class, no pre-baked formula slides. If you've already read the previous playlist, you know exactly what to expect.",[12,111,112,113,116,117,121,122,126],{},"The reference book changes: here I use ",[66,114,115],{},"Neural Networks and Deep Learning: A Textbook",", by Charu Aggarwal (2018), to play the role Bishop played in the previous playlist, filling in the foundation the notebook only shows in code. A lot of the material also connects straight back to what I've already covered: ",[53,118,120],{"href":119},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","the delta rule and linear regression already showed up in Pattern Recognition",", and ",[53,123,125],{"href":124},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Floss-functions","loss functions already showed up in Andrew Ng's specialization",", so whenever it fits I'll pull those threads instead of reteaching from scratch.",[12,128,72,129,134,135,140],{},[53,130,133],{"href":131,"rel":132},"https:\u002F\u002Fgithub.com\u002Fpablobelmiro\u002Faulasann",[77],"pablobelmiro\u002Faulasann",", a fork of Dr. Boldt's own repository, ",[53,136,139],{"href":137,"rel":138},"https:\u002F\u002Fgithub.com\u002Ffboldt\u002Faulasann",[77],"fboldt\u002Faulasann",", where he publishes each lecture's code. Same convention as the previous playlist: whoever \"wrote\" the code is always him, the professor; the foundational explanation is mine.",[12,142,143,144,147,148,151,152,156],{},"One important detail this time: this course is being taught ",[16,145,146],{},"right now",", live, and the repository only has the beginning of the course as of this moment (perceptron, Adaline, cost functions, and a cliffhanger right at the edge of what a single neuron can solve). Unlike the Pattern Recognition playlist, which I only started once the whole course had already ended, this one is a ",[16,149,150],{},"living playlist",": it grows every time the professor publishes a new lecture, and I come back to keep going. We start at the very beginning: ",[53,153,155],{"href":154},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","the simplest neuron there is",".",{"title":27,"searchDepth":28,"depth":28,"links":158},[],"My notes from my Neural Networks course, lecture by lecture, with Aggarwal's book as the theoretical backbone. A living playlist, growing along with the course.",{},3,{"title":98,"description":159},"en\u002Fplaylists\u002Fneural-networks\u002Findex","fD14SmK9Haa_ztrp627bgiWBVDETZgSpdrWnwlC_LnU",{"id":166,"title":167,"body":168,"cover":3,"description":186,"extension":31,"meta":187,"navigation":33,"order":188,"path":189,"seo":190,"status":37,"stem":191,"__hash__":192},"playlists\u002Fen\u002Fplaylists\u002Fpapers\u002Findex.md","Papers",{"type":9,"value":169,"toc":184},[170,173,176],[12,171,172],{},"The other three playlists here are real study: lecture notebook, textbook on the side, every line of code picked apart until it hurts. This one is a different vibe. Every once in a while I read a paper or survey that gets me excited, usually skimmed diagonally, not read page by page with a magnifying glass, and I wanted a place to talk about it without all the ceremony of a lecture.",[12,174,175],{},"That's what this playlist is: I tell you what I found coolest about the paper, with a crooked metaphor, some joking around here and there, and whenever it fits, some simple Python code and a chart just to give you that view. Don't expect notebook-level rigor or section-by-section coverage, the goal here is a straight conversation about an interesting idea, not an academic summary.",[12,177,178,179,183],{},"First in the series is about a topic there's no escaping these days: ",[53,180,182],{"href":181},"\u002Fen\u002Fplaylists\u002Fpapers\u002Fagentic-reasoning-for-large-language-models","agentic reasoning in language models",", a giant survey trying to organize everything that became trendy to call an \"AI agent\".",{"title":27,"searchDepth":28,"depth":28,"links":185},[],"A diagonal read of articles and surveys I found cool, with no pretense of turning into a lecture. Simple code, a chart to give you a view, and the same conversation as always with you.",{},4,"\u002Fen\u002Fplaylists\u002Fpapers",{"title":167,"description":186},"en\u002Fplaylists\u002Fpapers\u002Findex","TwtxQvLC6nBRqD_S4n46i-MnkOFuDQKRx5oRX-hKPTo",{"id":194,"title":195,"body":196,"cover":3,"description":227,"extension":31,"meta":228,"navigation":33,"order":229,"path":230,"seo":231,"status":37,"stem":232,"__hash__":233},"playlists\u002Fen\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Findex.md","Small Language Models",{"type":9,"value":197,"toc":225},[198,209,222],[12,199,200,201,208],{},"This playlist comes from a book, not a live class or a course notebook: ",[53,202,205],{"href":203,"rel":204},"https:\u002F\u002Fleanpub.com\u002Fhowtobuildandfine-tuneasmalllanguagemodel",[77],[66,206,207],{},"How to Build and Fine-Tune a Small Language Model",", by J. Paul Liu. It's a lean book, more of a practical project script with code than a theory brick, and that's a compliment: every chapter comes with a dollar cost estimate, a training time, a minimum hardware spec attached. Seeing an ML book tell you \"this costs 50 dollars and takes an hour\" before throwing you into a sea of math is rare, and I wanted to pull that thread with you.",[12,210,211,212,216,217,221],{},"The book's full plan is ambitious, 12 chapters going from \"why train your own model\" all the way to production deployment with ethics and everything. For now I'm covering the first two: ",[53,213,215],{"href":214},"\u002Fen\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Fsmall-vs-giant-language-models","Chapter 1",", the case for why a small specialized model beats a generalist giant in plenty of real scenarios, and ",[53,218,220],{"href":219},"\u002Fen\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Fbuilding-gpt-from-scratch-dom-casmurro","Chapter 2",", which builds a character-level GPT from scratch, the same classic exercise from Andrej Karpathy's \"Let's Build GPT\" video (the book credits this openly). If more posts on the following chapters happen, I'll come back here, but for now that's it.",[12,223,224],{},"A funny detail before we start: the book's table of contents opens with a section called \"AI Use Disclaimer\". So the author used AI to help write a book teaching you to build AI, and I use uncle Claude to write this blog about the book. It's AI all the way down, but at least nobody here is pretending otherwise.",{"title":27,"searchDepth":28,"depth":28,"links":226},[],"Chapter by chapter through \"How to Build and Fine-Tune a Small Language Model\", by J. Paul Liu. Why train your own model instead of outsourcing everything to an API, and how to build a GPT from scratch until it writes like a 19th-century Brazilian novelist.",{},5,"\u002Fen\u002Fplaylists\u002Fhow-to-build-and-finetune-slm",{"title":195,"description":227},"en\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Findex","eTa95D4OCbBPJ0M1Phqi3IbxK0pTLR9Kxy507jmCikc",[235,3863,4019],{"id":236,"title":237,"body":238,"cover":3,"date":3851,"description":3852,"extension":31,"meta":3853,"navigation":33,"order":28,"path":219,"playlist":3854,"seo":3855,"status":37,"stem":3856,"tags":3857,"__hash__":3862},"posts\u002Fen\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Fbuilding-gpt-from-scratch-dom-casmurro.md","Teaching a GPT From Scratch to Write Like a 19th-Century Brazilian Novelist",{"type":9,"value":239,"toc":3831},[240,268,271,276,290,326,329,337,344,348,373,391,394,439,468,471,532,553,557,582,585,644,661,665,735,765,776,780,925,950,975,982,986,1249,1276,1310,1338,1342,1357,1420,1439,1448,1455,1459,1462,1551,1579,1582,1611,1646,1655,1690,1709,1749,1767,1780,1784,1904,1953,1962,1965,1971,1978,1982,2002,2011,2016,2025,2030,2039,2044,2081,2090,2096,2105,2111,2125,2129,2253,2284,2304,2314,2334,2344,2352,3125,3166,3181,3222,3455,3492,3496,3623,3637,3641,3663,3740,3743,3747,3750,3758,3761,3765,3809,3813,3816,3824,3827],[12,241,242,243,248,249,253,254,259,260,267],{},"Chapter 2 of the book builds a tiny GPT, about 200 thousand parameters, character-level, from scratch. It's literally Andrej Karpathy's \"Let's Build GPT from Scratch\" video turned into prose, the book admits the source upfront. I ran the exercise with two notebooks side by side: ",[53,244,247],{"href":245,"rel":246},"https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002Fkarpathy\u002Fng-video-lecture\u002Fblob\u002Fmaster\u002Fgpt_dev.ipynb",[77],"the original notebook Karpathy himself cites as the video's source"," (",[250,251,252],"code",{},"gpt_dev.ipynb",", trained on Shakespeare), and ",[53,255,258],{"href":256,"rel":257},"https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1NIpd8W15jVf8oFwNKkOrB0Z3B4qaRZtW?usp=sharing",[77],"my own",", where I swapped Shakespeare for ",[53,261,264],{"href":262,"rel":263},"https:\u002F\u002Fwww.gutenberg.org\u002Febooks\u002F55752",[77],[66,265,266],{},"Dom Casmurro",", Machado de Assis's classic Brazilian novel, public domain text, to see whether the same tiny architecture picks up Portuguese too.",[12,269,270],{},"This post is the whole notebook cleaned up with you, cell by cell, full code, nothing skipped. It's not a summary, it's a dissection.",[272,273,275],"h2",{"id":274},"downloading-and-checking-the-dataset","Downloading and checking the dataset",[277,278,282],"pre",{"className":279,"code":280,"language":281,"meta":27,"style":27},"language-python shiki shiki-themes github-light github-dark","!wget https:\u002F\u002Fwww.gutenberg.org\u002Fcache\u002Fepub\u002F55752\u002Fpg55752.txt\n","python",[250,283,284],{"__ignoreMap":27},[285,286,288],"span",{"class":287,"line":34},"line",[285,289,280],{},[277,291,293],{"className":279,"code":292,"language":281,"meta":27,"style":27},"with open('pg55752.txt', 'r') as file:\n    texto = file.read()\n\nprint(\"Tamanho do dataset\", len(texto))\nprint(\"===== Trecho do texto =====\")\nprint(texto[:1000])\n",[250,294,295,300,305,310,315,320],{"__ignoreMap":27},[285,296,297],{"class":287,"line":34},[285,298,299],{},"with open('pg55752.txt', 'r') as file:\n",[285,301,302],{"class":287,"line":28},[285,303,304],{},"    texto = file.read()\n",[285,306,307],{"class":287,"line":161},[285,308,309],{"emptyLinePlaceholder":33},"\n",[285,311,312],{"class":287,"line":188},[285,313,314],{},"print(\"Tamanho do dataset\", len(texto))\n",[285,316,317],{"class":287,"line":229},[285,318,319],{},"print(\"===== Trecho do texto =====\")\n",[285,321,323],{"class":287,"line":322},6,[285,324,325],{},"print(texto[:1000])\n",[12,327,328],{},"Real output:",[277,330,335],{"className":331,"code":333,"language":334},[332],"language-text","Tamanho do dataset 400944\n===== Trecho do texto =====\nThe Project Gutenberg eBook of Dom Casmurro\n...\nTitle: Dom Casmurro\nAuthor: Machado de Assis\n","text",[250,336,333],{"__ignoreMap":27},[12,338,339,340,343],{},"400,944 characters, including the standard Gutenberg header (license, title, author) that ships with the raw ",[250,341,342],{},".txt"," file, that wasn't cleaned up, it's the text exactly as the site delivers it.",[272,345,347],{"id":346},"vocabulary-every-character-becomes-a-number","Vocabulary: every character becomes a number",[277,349,351],{"className":279,"code":350,"language":281,"meta":27,"style":27},"caracteres = sorted(list(set(texto)))\ntamanho_vocabulario = len(caracteres)\nprint(\"Vocabulário:\", ''.join(caracteres), \"\\n\")\nprint(\"Tamanho do Vocabulário: \", tamanho_vocabulario)\n",[250,352,353,358,363,368],{"__ignoreMap":27},[285,354,355],{"class":287,"line":34},[285,356,357],{},"caracteres = sorted(list(set(texto)))\n",[285,359,360],{"class":287,"line":28},[285,361,362],{},"tamanho_vocabulario = len(caracteres)\n",[285,364,365],{"class":287,"line":161},[285,366,367],{},"print(\"Vocabulário:\", ''.join(caracteres), \"\\n\")\n",[285,369,370],{"class":287,"line":188},[285,371,372],{},"print(\"Tamanho do Vocabulário: \", tamanho_vocabulario)\n",[12,374,375,378,379,382,383,386,387,390],{},[250,376,377],{},"set(texto)"," grabs every unique character in the file, ",[250,380,381],{},"sorted()"," orders them, and the size of that set is the vocabulary. Output: ",[16,384,385],{},"112 characters",", against the original Shakespeare's ",[16,388,389],{},"65",", nearly double because of accent marks (á, ã, ç, é, í, ó, ô, õ, ú), curly quotes, and a few leftover typographic symbols from the Gutenberg text.",[12,392,393],{},"With the vocabulary set, the character-to-number translator can go both ways:",[277,395,397],{"className":279,"code":396,"language":281,"meta":27,"style":27},"string_to_integer = {ch:i for i, ch in enumerate(caracteres)}\ninteger_to_string = {i:ch for i, ch in enumerate(caracteres)}\n\nencode = lambda s: [string_to_integer[c] for c in s]\ndecode = lambda s: ''.join([integer_to_string[i] for i in s])\n\nprint(encode(\"Teste de encode\"))\nprint(decode(encode(\"Teste de decode\")))\n",[250,398,399,404,409,413,418,423,427,433],{"__ignoreMap":27},[285,400,401],{"class":287,"line":34},[285,402,403],{},"string_to_integer = {ch:i for i, ch in enumerate(caracteres)}\n",[285,405,406],{"class":287,"line":28},[285,407,408],{},"integer_to_string = {i:ch for i, ch in enumerate(caracteres)}\n",[285,410,411],{"class":287,"line":161},[285,412,309],{"emptyLinePlaceholder":33},[285,414,415],{"class":287,"line":188},[285,416,417],{},"encode = lambda s: [string_to_integer[c] for c in s]\n",[285,419,420],{"class":287,"line":229},[285,421,422],{},"decode = lambda s: ''.join([integer_to_string[i] for i in s])\n",[285,424,425],{"class":287,"line":322},[285,426,309],{"emptyLinePlaceholder":33},[285,428,430],{"class":287,"line":429},7,[285,431,432],{},"print(encode(\"Teste de encode\"))\n",[285,434,436],{"class":287,"line":435},8,[285,437,438],{},"print(decode(encode(\"Teste de decode\")))\n",[12,440,441,444,445,448,449,452,453,456,457,460,461,464,465,467],{},[250,442,443],{},"string_to_integer"," maps each character to its position in the sorted vocabulary, ",[250,446,447],{},"integer_to_string"," is the reverse. ",[250,450,451],{},"encode"," swaps each letter of a string for its position, ",[250,454,455],{},"decode"," reverses the trip. Running ",[250,458,459],{},"encode(\"Teste de encode\")"," returns ",[250,462,463],{},"[49, 63, 77, 78, 63, 1, 62, 63, 1, 63, 72, 61, 73, 62, 63]",", pure numbers, and ",[250,466,455],{}," back recovers the exact text.",[12,469,470],{},"Last step: push all of this into the shape PyTorch expects:",[277,472,474],{"className":279,"code":473,"language":281,"meta":27,"style":27},"#Armazenando o texto encodado em um torch.Tensor\nimport torch\ndata = torch.tensor(encode(texto), dtype=torch.long)\n\nprint(data.shape, data.dtype)\nprint(data[:1000])\n\n# Re-definindo a função decode para lidar com tensores PyTorch\ndef decode(s):\n    # Converte cada elemento para Python int se for um tensor\n    return ''.join([integer_to_string[i.item()] if isinstance(i, torch.Tensor) else integer_to_string[i] for i in s])\n",[250,475,476,481,486,491,495,500,505,509,514,520,526],{"__ignoreMap":27},[285,477,478],{"class":287,"line":34},[285,479,480],{},"#Armazenando o texto encodado em um torch.Tensor\n",[285,482,483],{"class":287,"line":28},[285,484,485],{},"import torch\n",[285,487,488],{"class":287,"line":161},[285,489,490],{},"data = torch.tensor(encode(texto), dtype=torch.long)\n",[285,492,493],{"class":287,"line":188},[285,494,309],{"emptyLinePlaceholder":33},[285,496,497],{"class":287,"line":229},[285,498,499],{},"print(data.shape, data.dtype)\n",[285,501,502],{"class":287,"line":322},[285,503,504],{},"print(data[:1000])\n",[285,506,507],{"class":287,"line":429},[285,508,309],{"emptyLinePlaceholder":33},[285,510,511],{"class":287,"line":435},[285,512,513],{},"# Re-definindo a função decode para lidar com tensores PyTorch\n",[285,515,517],{"class":287,"line":516},9,[285,518,519],{},"def decode(s):\n",[285,521,523],{"class":287,"line":522},10,[285,524,525],{},"    # Converte cada elemento para Python int se for um tensor\n",[285,527,529],{"class":287,"line":528},11,[285,530,531],{},"    return ''.join([integer_to_string[i.item()] if isinstance(i, torch.Tensor) else integer_to_string[i] for i in s])\n",[12,533,534,537,538,540,541,544,545,548,549,552],{},[250,535,536],{},"data"," becomes a tensor of 400,944 integers, the entire book encoded into a single strip. Notice ",[250,539,455],{}," gets rewritten here: the first version only knew how to handle plain Python ",[250,542,543],{},"int"," lists, this second version also accepts PyTorch tensors (checking ",[250,546,547],{},"isinstance"," and calling ",[250,550,551],{},".item()"," when needed), because everything coming out of the model from here on arrives as a tensor.",[272,554,556],{"id":555},"trainvalidation-split-and-the-context-window-size","Train\u002Fvalidation split and the context window size",[277,558,560],{"className":279,"code":559,"language":281,"meta":27,"style":27},"#Split dados para treino e para avaliação\nn = int(0.9*len(data))\ntreino = data[:n]\navaliacao = data[n:]\n",[250,561,562,567,572,577],{"__ignoreMap":27},[285,563,564],{"class":287,"line":34},[285,565,566],{},"#Split dados para treino e para avaliação\n",[285,568,569],{"class":287,"line":28},[285,570,571],{},"n = int(0.9*len(data))\n",[285,573,574],{"class":287,"line":161},[285,575,576],{},"treino = data[:n]\n",[285,578,579],{"class":287,"line":188},[285,580,581],{},"avaliacao = data[n:]\n",[12,583,584],{},"90% for training, 10% for validation, a simple positional cut (no shuffling, so the model never sees the end of the book during training).",[277,586,588],{"className":279,"code":587,"language":281,"meta":27,"style":27},"#Definição do bloco de contexto para as previsões do modelo\n# Isso significa que, para prever o próximo caractere,\n#o modelo considerará no máximo os 8 caracteres anteriores.\n#É um hiperparâmetro chave para a arquitetura do Transformer.\nbloco_contexto = 8\n\n# Este é um exemplo de como uma sequência de entrada x\n#e seu alvo y (que é a mesma sequência deslocada em um token)\n#seriam construídos a partir dos dados de treinamento.\ntreino[:bloco_contexto+1]\nprint(decode(treino[:bloco_contexto+1]))\n",[250,589,590,595,600,605,610,615,619,624,629,634,639],{"__ignoreMap":27},[285,591,592],{"class":287,"line":34},[285,593,594],{},"#Definição do bloco de contexto para as previsões do modelo\n",[285,596,597],{"class":287,"line":28},[285,598,599],{},"# Isso significa que, para prever o próximo caractere,\n",[285,601,602],{"class":287,"line":161},[285,603,604],{},"#o modelo considerará no máximo os 8 caracteres anteriores.\n",[285,606,607],{"class":287,"line":188},[285,608,609],{},"#É um hiperparâmetro chave para a arquitetura do Transformer.\n",[285,611,612],{"class":287,"line":229},[285,613,614],{},"bloco_contexto = 8\n",[285,616,617],{"class":287,"line":322},[285,618,309],{"emptyLinePlaceholder":33},[285,620,621],{"class":287,"line":429},[285,622,623],{},"# Este é um exemplo de como uma sequência de entrada x\n",[285,625,626],{"class":287,"line":435},[285,627,628],{},"#e seu alvo y (que é a mesma sequência deslocada em um token)\n",[285,630,631],{"class":287,"line":516},[285,632,633],{},"#seriam construídos a partir dos dados de treinamento.\n",[285,635,636],{"class":287,"line":522},[285,637,638],{},"treino[:bloco_contexto+1]\n",[285,640,641],{"class":287,"line":528},[285,642,643],{},"print(decode(treino[:bloco_contexto+1]))\n",[12,645,646,647,650,651,654,655,660],{},"Output: ",[250,648,649],{},"The Proje",", the book's first 9 characters (after the header gets sliced off by indexing). ",[250,652,653],{},"bloco_contexto = 8"," sets the ",[656,657,659],"glossary-term",{"definition":658},"the maximum number of prior tokens the model can look at to make a prediction, a fixed hyperparameter of the architecture","context window",", the model won't see more than 8 characters back at this early testing stage.",[272,662,664],{"id":663},"bug-1-mixing-training-and-validation","Bug #1: mixing training and validation",[277,666,668],{"className":279,"code":667,"language":281,"meta":27,"style":27},"#cria sequencia de entrada x\nx = treino[:bloco_contexto]\n\n#cria sequência de targets y\n#y é a mesma sequência que x mas deslocada 1 token para frente\n#O modelo prevê y[t] dado x[:t+1]\ny = avaliacao[1:bloco_contexto+1]\n\n#loop para simular como o modelo vê a sequência\nfor tensor in range(bloco_contexto):\n    contexto = x[:tensor+1]\n    target = y[tensor]\n    print(f\"Quando tensor é {contexto} o target é {target}\")\n",[250,669,670,675,680,684,689,694,699,704,708,713,718,723,729],{"__ignoreMap":27},[285,671,672],{"class":287,"line":34},[285,673,674],{},"#cria sequencia de entrada x\n",[285,676,677],{"class":287,"line":28},[285,678,679],{},"x = treino[:bloco_contexto]\n",[285,681,682],{"class":287,"line":161},[285,683,309],{"emptyLinePlaceholder":33},[285,685,686],{"class":287,"line":188},[285,687,688],{},"#cria sequência de targets y\n",[285,690,691],{"class":287,"line":229},[285,692,693],{},"#y é a mesma sequência que x mas deslocada 1 token para frente\n",[285,695,696],{"class":287,"line":322},[285,697,698],{},"#O modelo prevê y[t] dado x[:t+1]\n",[285,700,701],{"class":287,"line":429},[285,702,703],{},"y = avaliacao[1:bloco_contexto+1]\n",[285,705,706],{"class":287,"line":435},[285,707,309],{"emptyLinePlaceholder":33},[285,709,710],{"class":287,"line":516},[285,711,712],{},"#loop para simular como o modelo vê a sequência\n",[285,714,715],{"class":287,"line":522},[285,716,717],{},"for tensor in range(bloco_contexto):\n",[285,719,720],{"class":287,"line":528},[285,721,722],{},"    contexto = x[:tensor+1]\n",[285,724,726],{"class":287,"line":725},12,[285,727,728],{},"    target = y[tensor]\n",[285,730,732],{"class":287,"line":731},13,[285,733,734],{},"    print(f\"Quando tensor é {contexto} o target é {target}\")\n",[12,736,737,738,741,742,745,746,741,749,752,753,756,757,760,761,764],{},"Found the bug right here: ",[250,739,740],{},"x"," comes from ",[250,743,744],{},"treino",", but ",[250,747,748],{},"y",[250,750,751],{},"avaliacao",". Two completely different chunks of the book, so the target the print shows has no real relationship to the context that came before it. Compare with the original notebook, which uses ",[250,754,755],{},"train_data"," on both lines: ",[250,758,759],{},"x = train_data[:block_size]"," and ",[250,762,763],{},"y = train_data[1:block_size+1]",", never switching splits mid-example.",[12,766,767,768,771,772,775],{},"The output matches the mistake: ",[250,769,770],{},"Quando tensor é tensor([49]) o target é 12"," makes no sense at all (12 is the code for some random digit or symbol in the vocabulary, unrelated to \"T\" from \"The\"). The code doesn't crash (both splits share the same tensor shape), it just produces a nonsensical teaching example. Good news: this bug lives only in this manual illustration cell, the ",[250,773,774],{},"get_batch"," function coming up next never mixes splits.",[272,777,779],{"id":778},"packing-real-batches","Packing real batches",[277,781,783],{"className":279,"code":782,"language":281,"meta":27,"style":27},"torch.manual_seed(1337)\nbatch_size = 4 # Quantas sequências independentes serão processadas em paralelo\nblock_size = 8 # Qual tamanho máximo de contexto para predições\n\ndef get_batch(split):\n  #Criando um lote pequeno de dados de entrada x e targets y\n  dados = treino if split == 'treino' else avaliacao\n  ix = torch.randint(len(dados) - block_size, (batch_size,))\n  x = torch.stack([data[i:i+block_size] for i in ix])\n  y = torch.stack([dados[i+1:i+block_size+1] for i in ix])\n  return x, y\n\nxb, yb = get_batch('treino')\nprint(\"Inputs:\")\nprint(xb.shape)\nprint(xb)\nprint('----------')\nprint(\"Targets:\")\nprint(yb.shape)\nprint(yb)\n\nfor b in range(batch_size): #Dimensão batch\n  for t in range(block_size): #Dimensão time\n    contexto_time = xb[b, :t+1]\n    target_time = yb[b,t]\n    print(f\"Quando input é {contexto_time.tolist()} - o target é {target_time}\")\n",[250,784,785,790,795,800,804,809,814,819,824,829,834,839,843,848,854,860,866,872,878,884,890,895,901,907,913,919],{"__ignoreMap":27},[285,786,787],{"class":287,"line":34},[285,788,789],{},"torch.manual_seed(1337)\n",[285,791,792],{"class":287,"line":28},[285,793,794],{},"batch_size = 4 # Quantas sequências independentes serão processadas em paralelo\n",[285,796,797],{"class":287,"line":161},[285,798,799],{},"block_size = 8 # Qual tamanho máximo de contexto para predições\n",[285,801,802],{"class":287,"line":188},[285,803,309],{"emptyLinePlaceholder":33},[285,805,806],{"class":287,"line":229},[285,807,808],{},"def get_batch(split):\n",[285,810,811],{"class":287,"line":322},[285,812,813],{},"  #Criando um lote pequeno de dados de entrada x e targets y\n",[285,815,816],{"class":287,"line":429},[285,817,818],{},"  dados = treino if split == 'treino' else avaliacao\n",[285,820,821],{"class":287,"line":435},[285,822,823],{},"  ix = torch.randint(len(dados) - block_size, (batch_size,))\n",[285,825,826],{"class":287,"line":516},[285,827,828],{},"  x = torch.stack([data[i:i+block_size] for i in ix])\n",[285,830,831],{"class":287,"line":522},[285,832,833],{},"  y = torch.stack([dados[i+1:i+block_size+1] for i in ix])\n",[285,835,836],{"class":287,"line":528},[285,837,838],{},"  return x, y\n",[285,840,841],{"class":287,"line":725},[285,842,309],{"emptyLinePlaceholder":33},[285,844,845],{"class":287,"line":731},[285,846,847],{},"xb, yb = get_batch('treino')\n",[285,849,851],{"class":287,"line":850},14,[285,852,853],{},"print(\"Inputs:\")\n",[285,855,857],{"class":287,"line":856},15,[285,858,859],{},"print(xb.shape)\n",[285,861,863],{"class":287,"line":862},16,[285,864,865],{},"print(xb)\n",[285,867,869],{"class":287,"line":868},17,[285,870,871],{},"print('----------')\n",[285,873,875],{"class":287,"line":874},18,[285,876,877],{},"print(\"Targets:\")\n",[285,879,881],{"class":287,"line":880},19,[285,882,883],{},"print(yb.shape)\n",[285,885,887],{"class":287,"line":886},20,[285,888,889],{},"print(yb)\n",[285,891,893],{"class":287,"line":892},21,[285,894,309],{"emptyLinePlaceholder":33},[285,896,898],{"class":287,"line":897},22,[285,899,900],{},"for b in range(batch_size): #Dimensão batch\n",[285,902,904],{"class":287,"line":903},23,[285,905,906],{},"  for t in range(block_size): #Dimensão time\n",[285,908,910],{"class":287,"line":909},24,[285,911,912],{},"    contexto_time = xb[b, :t+1]\n",[285,914,916],{"class":287,"line":915},25,[285,917,918],{},"    target_time = yb[b,t]\n",[285,920,922],{"class":287,"line":921},26,[285,923,924],{},"    print(f\"Quando input é {contexto_time.tolist()} - o target é {target_time}\")\n",[12,926,927,930,931,934,935,938,939,941,942,945,946,949],{},[250,928,929],{},"ix"," draws ",[250,932,933],{},"batch_size"," random starting positions inside the chosen split, and ",[250,936,937],{},"torch.stack"," piles up ",[250,940,933],{}," windows of ",[250,943,944],{},"block_size"," characters into a single matrix. Real output: ",[250,947,948],{},"Inputs: torch.Size([4, 8])",", a batch of 4 sequences of 8 characters each, processed in parallel. The loop below is just for visualization: for each sequence in the batch, it shows the 8 progressive predictions it contains (predict the 2nd character knowing the 1st, predict the 3rd knowing the first 2, and so on), so a 4×8 batch actually teaches the model 32 prediction examples at once.",[277,951,953],{"className":279,"code":952,"language":281,"meta":27,"style":27},"# Este output nos permite visualizar a forma exata\n#e os valores numéricos das sequências de entrada\n#que o modelo começará a processar.\nprint(xb) # Input em forma de tensor\n",[250,954,955,960,965,970],{"__ignoreMap":27},[285,956,957],{"class":287,"line":34},[285,958,959],{},"# Este output nos permite visualizar a forma exata\n",[285,961,962],{"class":287,"line":28},[285,963,964],{},"#e os valores numéricos das sequências de entrada\n",[285,966,967],{"class":287,"line":161},[285,968,969],{},"#que o modelo começará a processar.\n",[285,971,972],{"class":287,"line":188},[285,973,974],{},"print(xb) # Input em forma de tensor\n",[12,976,977,978,981],{},"Just reprints ",[250,979,980],{},"xb",", nothing new, a useful visual checkpoint before assembling the model.",[272,983,985],{"id":984},"the-first-model-and-the-bug-2-hiding-inside-it","The first model, and the Bug #2 hiding inside it",[277,987,989],{"className":279,"code":988,"language":281,"meta":27,"style":27},"import torch\nimport torch.nn as nn\nfrom torch.nn import functional as F\ntorch.manual_seed(1337)\n\nclass BigramLanguageModel(nn.Module):\n  def __init__(self, tam_vocabulario):\n    super().__init__()\n    #cada token lê diretamente os logits para o\n    #próximo token a partir de uma tabela de consulta\n    self.token_embeeding_table = nn.Embedding(tam_vocabulario, tam_vocabulario)\n\n  def forward(self, idx, targets=None):\n    # idx e targets são ambos tensores de\n    #inteiros de formato (B, T)\n    logits = self.token_embeeding_table(idx) # (B, T, C)\n\n    if targets is None:\n      loss = None\n    else:\n      B, T, C = logits.shape\n      logits = logits.view(B*T, C)\n      targets = targets.view(B*T)\n      loss = F.cross_entropy(logits, targets)\n\n    return logits, loss\n\n  def generate(self, idx, max_new_tokens):\n    # idx é um array de índices (B, T)\n    #no contexto atual\n    for _ in range(max_new_tokens):\n      #\"pega\" as predições\n      logits, loss = self(idx)\n      #foca apenas no último passo de tempo\n      logits = logits[:, -1, : ] #Vira B, C\n      # aplica softmax para obter probabilidades\n      probabilidades = F.softmax(logits, dim=-1) # (B, C)\n      # amostra da distribuição\n      idx_next = torch.multinomial(probabilidades, num_samples=1) # (B, T+1)\n    return idx\n\nmodelo = BigramLanguageModel(tamanho_vocabulario)\nsaida, loss = modelo(xb, yb)\nprint(saida.shape)\nprint(loss)\n\nprint(\"-----\")\n\nprint(decode(modelo.generate(idx = torch.zeros((1,1), dtype=torch.long), max_new_tokens=100)[0].tolist()))\n",[250,990,991,995,1000,1005,1009,1013,1018,1023,1028,1033,1038,1043,1047,1052,1057,1062,1067,1071,1076,1081,1086,1091,1096,1101,1106,1110,1115,1120,1126,1132,1138,1144,1150,1156,1162,1168,1174,1180,1186,1192,1198,1203,1209,1215,1221,1227,1232,1238,1243],{"__ignoreMap":27},[285,992,993],{"class":287,"line":34},[285,994,485],{},[285,996,997],{"class":287,"line":28},[285,998,999],{},"import torch.nn as nn\n",[285,1001,1002],{"class":287,"line":161},[285,1003,1004],{},"from torch.nn import functional as F\n",[285,1006,1007],{"class":287,"line":188},[285,1008,789],{},[285,1010,1011],{"class":287,"line":229},[285,1012,309],{"emptyLinePlaceholder":33},[285,1014,1015],{"class":287,"line":322},[285,1016,1017],{},"class BigramLanguageModel(nn.Module):\n",[285,1019,1020],{"class":287,"line":429},[285,1021,1022],{},"  def __init__(self, tam_vocabulario):\n",[285,1024,1025],{"class":287,"line":435},[285,1026,1027],{},"    super().__init__()\n",[285,1029,1030],{"class":287,"line":516},[285,1031,1032],{},"    #cada token lê diretamente os logits para o\n",[285,1034,1035],{"class":287,"line":522},[285,1036,1037],{},"    #próximo token a partir de uma tabela de consulta\n",[285,1039,1040],{"class":287,"line":528},[285,1041,1042],{},"    self.token_embeeding_table = nn.Embedding(tam_vocabulario, tam_vocabulario)\n",[285,1044,1045],{"class":287,"line":725},[285,1046,309],{"emptyLinePlaceholder":33},[285,1048,1049],{"class":287,"line":731},[285,1050,1051],{},"  def forward(self, idx, targets=None):\n",[285,1053,1054],{"class":287,"line":850},[285,1055,1056],{},"    # idx e targets são ambos tensores de\n",[285,1058,1059],{"class":287,"line":856},[285,1060,1061],{},"    #inteiros de formato (B, T)\n",[285,1063,1064],{"class":287,"line":862},[285,1065,1066],{},"    logits = self.token_embeeding_table(idx) # (B, T, C)\n",[285,1068,1069],{"class":287,"line":868},[285,1070,309],{"emptyLinePlaceholder":33},[285,1072,1073],{"class":287,"line":874},[285,1074,1075],{},"    if targets is None:\n",[285,1077,1078],{"class":287,"line":880},[285,1079,1080],{},"      loss = None\n",[285,1082,1083],{"class":287,"line":886},[285,1084,1085],{},"    else:\n",[285,1087,1088],{"class":287,"line":892},[285,1089,1090],{},"      B, T, C = logits.shape\n",[285,1092,1093],{"class":287,"line":897},[285,1094,1095],{},"      logits = logits.view(B*T, C)\n",[285,1097,1098],{"class":287,"line":903},[285,1099,1100],{},"      targets = targets.view(B*T)\n",[285,1102,1103],{"class":287,"line":909},[285,1104,1105],{},"      loss = F.cross_entropy(logits, targets)\n",[285,1107,1108],{"class":287,"line":915},[285,1109,309],{"emptyLinePlaceholder":33},[285,1111,1112],{"class":287,"line":921},[285,1113,1114],{},"    return logits, loss\n",[285,1116,1118],{"class":287,"line":1117},27,[285,1119,309],{"emptyLinePlaceholder":33},[285,1121,1123],{"class":287,"line":1122},28,[285,1124,1125],{},"  def generate(self, idx, max_new_tokens):\n",[285,1127,1129],{"class":287,"line":1128},29,[285,1130,1131],{},"    # idx é um array de índices (B, T)\n",[285,1133,1135],{"class":287,"line":1134},30,[285,1136,1137],{},"    #no contexto atual\n",[285,1139,1141],{"class":287,"line":1140},31,[285,1142,1143],{},"    for _ in range(max_new_tokens):\n",[285,1145,1147],{"class":287,"line":1146},32,[285,1148,1149],{},"      #\"pega\" as predições\n",[285,1151,1153],{"class":287,"line":1152},33,[285,1154,1155],{},"      logits, loss = self(idx)\n",[285,1157,1159],{"class":287,"line":1158},34,[285,1160,1161],{},"      #foca apenas no último passo de tempo\n",[285,1163,1165],{"class":287,"line":1164},35,[285,1166,1167],{},"      logits = logits[:, -1, : ] #Vira B, C\n",[285,1169,1171],{"class":287,"line":1170},36,[285,1172,1173],{},"      # aplica softmax para obter probabilidades\n",[285,1175,1177],{"class":287,"line":1176},37,[285,1178,1179],{},"      probabilidades = F.softmax(logits, dim=-1) # (B, C)\n",[285,1181,1183],{"class":287,"line":1182},38,[285,1184,1185],{},"      # amostra da distribuição\n",[285,1187,1189],{"class":287,"line":1188},39,[285,1190,1191],{},"      idx_next = torch.multinomial(probabilidades, num_samples=1) # (B, T+1)\n",[285,1193,1195],{"class":287,"line":1194},40,[285,1196,1197],{},"    return idx\n",[285,1199,1201],{"class":287,"line":1200},41,[285,1202,309],{"emptyLinePlaceholder":33},[285,1204,1206],{"class":287,"line":1205},42,[285,1207,1208],{},"modelo = BigramLanguageModel(tamanho_vocabulario)\n",[285,1210,1212],{"class":287,"line":1211},43,[285,1213,1214],{},"saida, loss = modelo(xb, yb)\n",[285,1216,1218],{"class":287,"line":1217},44,[285,1219,1220],{},"print(saida.shape)\n",[285,1222,1224],{"class":287,"line":1223},45,[285,1225,1226],{},"print(loss)\n",[285,1228,1230],{"class":287,"line":1229},46,[285,1231,309],{"emptyLinePlaceholder":33},[285,1233,1235],{"class":287,"line":1234},47,[285,1236,1237],{},"print(\"-----\")\n",[285,1239,1241],{"class":287,"line":1240},48,[285,1242,309],{"emptyLinePlaceholder":33},[285,1244,1246],{"class":287,"line":1245},49,[285,1247,1248],{},"print(decode(modelo.generate(idx = torch.zeros((1,1), dtype=torch.long), max_new_tokens=100)[0].tolist()))\n",[12,1250,1251,1252,1255,1256,1259,1260,1263,1264,1267,1268,1271,1272,1275],{},"The dumbest possible model: ",[250,1253,1254],{},"nn.Embedding(tam_vocabulario, tam_vocabulario)"," is a lookup table where every character points straight to a vector as wide as the whole vocabulary (those vectors become the ",[250,1257,1258],{},"logits",", with no context beyond the current character itself). ",[250,1261,1262],{},"forward"," computes ",[250,1265,1266],{},"loss"," via ",[250,1269,1270],{},"cross_entropy"," only when ",[250,1273,1274],{},"targets"," exists.",[12,1277,1278,1281,1282,1285,1286,1289,1290,1299,1300,1303,1304,1306,1307,1309],{},[250,1279,1280],{},"generate"," is where the bug lives: look at the ",[250,1283,1284],{},"for"," loop that samples and computes ",[250,1287,1288],{},"idx_next",", but it ",[16,1291,1292,1293,1295,1296],{},"never glues that ",[250,1294,1288],{}," back onto ",[250,1297,1298],{},"idx",". The line ",[250,1301,1302],{},"idx = torch.cat((idx, idx_next), dim=1)"," is missing. The original notebook has that line, my version dropped it while typing. Result: ",[250,1305,1280],{}," always returns the original ",[250,1308,1298],{}," unchanged, so \"generating 100 new tokens\" produces zero in practice.",[12,1311,1312,1313,760,1316,1319,1320,1322,1323,1326,1327,1330,1331,1333,1334,1337],{},"The real output confirms the damage: ",[250,1314,1315],{},"torch.Size([32, 112])",[250,1317,1318],{},"tensor(5.0673, ...)"," for ",[250,1321,1266],{}," look normal, but after the \"-----\" the ",[250,1324,1325],{},"print(decode(...))"," comes out ",[16,1328,1329],{},"empty",". Not a runtime error, it's the exact symptom of the bug: ",[250,1332,1298],{}," is still the original ",[250,1335,1336],{},"[[0]]"," tensor, decoding that is just the vocabulary's character at code 0 (a newline), invisible in the print.",[272,1339,1341],{"id":1340},"training-badly-the-bigram-for-only-100-steps","Training (badly) the bigram, for only 100 steps",[277,1343,1345],{"className":279,"code":1344,"language":281,"meta":27,"style":27},"#Criando um otimizador com pytorch\noptimizer = torch.optim.AdamW(modelo.parameters(), lr=1e-3)\n",[250,1346,1347,1352],{"__ignoreMap":27},[285,1348,1349],{"class":287,"line":34},[285,1350,1351],{},"#Criando um otimizador com pytorch\n",[285,1353,1354],{"class":287,"line":28},[285,1355,1356],{},"optimizer = torch.optim.AdamW(modelo.parameters(), lr=1e-3)\n",[277,1358,1360],{"className":279,"code":1359,"language":281,"meta":27,"style":27},"tamanho_batch = 32\nfor passo in range(100):\n  # amostra de um lote de dados\n  xb, yb = get_batch('treino')\n\n  #avaliando o loss\n  logits, loss = modelo(xb, yb)\n  optimizer.zero_grad(set_to_none=True)\n  loss.backward()\n  optimizer.step()\n\nprint(loss.item())\n",[250,1361,1362,1367,1372,1377,1382,1386,1391,1396,1401,1406,1411,1415],{"__ignoreMap":27},[285,1363,1364],{"class":287,"line":34},[285,1365,1366],{},"tamanho_batch = 32\n",[285,1368,1369],{"class":287,"line":28},[285,1370,1371],{},"for passo in range(100):\n",[285,1373,1374],{"class":287,"line":161},[285,1375,1376],{},"  # amostra de um lote de dados\n",[285,1378,1379],{"class":287,"line":188},[285,1380,1381],{},"  xb, yb = get_batch('treino')\n",[285,1383,1384],{"class":287,"line":229},[285,1385,309],{"emptyLinePlaceholder":33},[285,1387,1388],{"class":287,"line":322},[285,1389,1390],{},"  #avaliando o loss\n",[285,1392,1393],{"class":287,"line":429},[285,1394,1395],{},"  logits, loss = modelo(xb, yb)\n",[285,1397,1398],{"class":287,"line":435},[285,1399,1400],{},"  optimizer.zero_grad(set_to_none=True)\n",[285,1402,1403],{"class":287,"line":516},[285,1404,1405],{},"  loss.backward()\n",[285,1407,1408],{"class":287,"line":522},[285,1409,1410],{},"  optimizer.step()\n",[285,1412,1413],{"class":287,"line":528},[285,1414,309],{"emptyLinePlaceholder":33},[285,1416,1417],{"class":287,"line":725},[285,1418,1419],{},"print(loss.item())\n",[12,1421,1422,1423,1426,1427,1430,1431,1434,1435,1438],{},"With 112 possible classes and a random guess, the expected loss for an untrained model is ",[250,1424,1425],{},"-ln(1\u002F112) ≈ 4.72",". The untrained bigram started at ",[16,1428,1429],{},"5.07"," (seen in the previous cell), the same order of magnitude. After only 100 steps of crude training, the loss went to ",[16,1432,1433],{},"5.19"," (real output: ",[250,1436,1437],{},"5.18871545791626","), it actually ticked up slightly, because a bigram has no real memory to work with yet and 100 steps is barely anything.",[277,1440,1442],{"className":279,"code":1441,"language":281,"meta":27,"style":27},"print(decode(modelo.generate(idx = torch.zeros((1,1), dtype=torch.long), max_new_tokens=500)[0].tolist()))\n",[250,1443,1444],{"__ignoreMap":27},[285,1445,1446],{"class":287,"line":34},[285,1447,1441],{},[12,1449,1450,1451,1454],{},"Same bug from before, same consequence: empty output. This bigram model never actually generated real text in this notebook, the full model further down (which has the correct ",[250,1452,1453],{},"torch.cat",") is the one that truly learns to write.",[272,1456,1458],{"id":1457},"the-self-attention-trick-step-by-step","The self-attention trick, step by step",[12,1460,1461],{},"The chapter's core idea: every character needs to \"talk\" to the previous characters. The slow way is a loop, averaging everything that came before:",[277,1463,1465],{"className":279,"code":1464,"language":281,"meta":27,"style":27},"#O truque matemático na autoatenção\n# exemplo simplificado que ilustra como\n# a multiplicação de matrizes pode ser usada\n# para uma \"agregação ponderada\"\ntorch.manual_seed(42)\na = torch.tril(torch.ones(3,3))\na = a \u002F torch.sum(a, 1, keepdim=True)\nb = torch.randint(0,10,(3,2)).float()\nc = a @ b\nprint(\"a=\")\nprint(a)\nprint('--')\nprint('b=')\nprint(b)\nprint('--')\nprint('c=')\nprint(c)\n",[250,1466,1467,1472,1477,1482,1487,1492,1497,1502,1507,1512,1517,1522,1527,1532,1537,1541,1546],{"__ignoreMap":27},[285,1468,1469],{"class":287,"line":34},[285,1470,1471],{},"#O truque matemático na autoatenção\n",[285,1473,1474],{"class":287,"line":28},[285,1475,1476],{},"# exemplo simplificado que ilustra como\n",[285,1478,1479],{"class":287,"line":161},[285,1480,1481],{},"# a multiplicação de matrizes pode ser usada\n",[285,1483,1484],{"class":287,"line":188},[285,1485,1486],{},"# para uma \"agregação ponderada\"\n",[285,1488,1489],{"class":287,"line":229},[285,1490,1491],{},"torch.manual_seed(42)\n",[285,1493,1494],{"class":287,"line":322},[285,1495,1496],{},"a = torch.tril(torch.ones(3,3))\n",[285,1498,1499],{"class":287,"line":429},[285,1500,1501],{},"a = a \u002F torch.sum(a, 1, keepdim=True)\n",[285,1503,1504],{"class":287,"line":435},[285,1505,1506],{},"b = torch.randint(0,10,(3,2)).float()\n",[285,1508,1509],{"class":287,"line":516},[285,1510,1511],{},"c = a @ b\n",[285,1513,1514],{"class":287,"line":522},[285,1515,1516],{},"print(\"a=\")\n",[285,1518,1519],{"class":287,"line":528},[285,1520,1521],{},"print(a)\n",[285,1523,1524],{"class":287,"line":725},[285,1525,1526],{},"print('--')\n",[285,1528,1529],{"class":287,"line":731},[285,1530,1531],{},"print('b=')\n",[285,1533,1534],{"class":287,"line":850},[285,1535,1536],{},"print(b)\n",[285,1538,1539],{"class":287,"line":856},[285,1540,1526],{},[285,1542,1543],{"class":287,"line":862},[285,1544,1545],{},"print('c=')\n",[285,1547,1548],{"class":287,"line":868},[285,1549,1550],{},"print(c)\n",[12,1552,1553,1554,1556,1557,1560,1561,1564,1565,1568,1569,1572,1573,760,1575,1578],{},"This warm-up example (3×3, small numbers) shows the pure trick: ",[250,1555,53],{}," is a lower-triangular matrix normalized by row (each row sums to 1), and multiplying ",[250,1558,1559],{},"a @ b"," already returns the running average of ",[250,1562,1563],{},"b"," row by row, no loop needed. Real output confirms it: the first row of ",[250,1566,1567],{},"c"," is just ",[250,1570,1571],{},"b[0]"," (average of 1 element), the second is the average of ",[250,1574,1571],{},[250,1576,1577],{},"b[1]",", and so on.",[12,1580,1581],{},"Now with real data, in three versions that should be equivalent:",[277,1583,1585],{"className":279,"code":1584,"language":281,"meta":27,"style":27},"# considere o seguinte exemplo simplificado:\ntorch.manual_seed(1337)\nB,T,C = 4,8,2 # batch, time, channels\nx = torch.randn(B,T,C)\nx.shape\n",[250,1586,1587,1592,1596,1601,1606],{"__ignoreMap":27},[285,1588,1589],{"class":287,"line":34},[285,1590,1591],{},"# considere o seguinte exemplo simplificado:\n",[285,1593,1594],{"class":287,"line":28},[285,1595,789],{},[285,1597,1598],{"class":287,"line":161},[285,1599,1600],{},"B,T,C = 4,8,2 # batch, time, channels\n",[285,1602,1603],{"class":287,"line":188},[285,1604,1605],{},"x = torch.randn(B,T,C)\n",[285,1607,1608],{"class":287,"line":229},[285,1609,1610],{},"x.shape\n",[277,1612,1614],{"className":279,"code":1613,"language":281,"meta":27,"style":27},"# Nós queremos x[b,t] = mean_{i\u003C=t} x[b,i]\nxbow = torch.zeros((B,T,C))\nfor b in range(B):\n  for t in range(T):\n    xprev = x[b,:t+1] # (t,C)\n    xbow[b,t] = torch.mean(xprev, 0)\n",[250,1615,1616,1621,1626,1631,1636,1641],{"__ignoreMap":27},[285,1617,1618],{"class":287,"line":34},[285,1619,1620],{},"# Nós queremos x[b,t] = mean_{i\u003C=t} x[b,i]\n",[285,1622,1623],{"class":287,"line":28},[285,1624,1625],{},"xbow = torch.zeros((B,T,C))\n",[285,1627,1628],{"class":287,"line":161},[285,1629,1630],{},"for b in range(B):\n",[285,1632,1633],{"class":287,"line":188},[285,1634,1635],{},"  for t in range(T):\n",[285,1637,1638],{"class":287,"line":229},[285,1639,1640],{},"    xprev = x[b,:t+1] # (t,C)\n",[285,1642,1643],{"class":287,"line":322},[285,1644,1645],{},"    xbow[b,t] = torch.mean(xprev, 0)\n",[12,1647,1648,1651,1652,1654],{},[16,1649,1650],{},"Version 1",", the slow way: two nested ",[250,1653,1284],{}," loops, averaging everything before each position, one batch and time step at a time.",[277,1656,1658],{"className":279,"code":1657,"language":281,"meta":27,"style":27},"# versão 2: usando multiplicação de matrizes para\n# uma agregação ponderada\nwei = torch.tril(torch.ones(T, T))\nwei = wei \u002F wei.sum(1, keepdim=True)\nxbow2 = wei @ x # (B, T, T) @ (B, T, C) ----> (B, T, C)\ntorch.allclose(xbow, xbow2)\n",[250,1659,1660,1665,1670,1675,1680,1685],{"__ignoreMap":27},[285,1661,1662],{"class":287,"line":34},[285,1663,1664],{},"# versão 2: usando multiplicação de matrizes para\n",[285,1666,1667],{"class":287,"line":28},[285,1668,1669],{},"# uma agregação ponderada\n",[285,1671,1672],{"class":287,"line":161},[285,1673,1674],{},"wei = torch.tril(torch.ones(T, T))\n",[285,1676,1677],{"class":287,"line":188},[285,1678,1679],{},"wei = wei \u002F wei.sum(1, keepdim=True)\n",[285,1681,1682],{"class":287,"line":229},[285,1683,1684],{},"xbow2 = wei @ x # (B, T, T) @ (B, T, C) ----> (B, T, C)\n",[285,1686,1687],{"class":287,"line":322},[285,1688,1689],{},"torch.allclose(xbow, xbow2)\n",[12,1691,1692,1695,1696,1699,1700,1703,1704,156],{},[16,1693,1694],{},"Version 2",", the same math via matrix multiplication, the same trick as the warm-up above. ",[250,1697,1698],{},"torch.allclose"," should return ",[250,1701,1702],{},"True"," (both versions compute the same thing). Real output: ",[16,1705,1706],{},[250,1707,1708],{},"False",[277,1710,1712],{"className":279,"code":1711,"language":281,"meta":27,"style":27},"# versão 3: usando softmax\ntril = torch.tril(torch.ones(T, T))\nwei = torch.zeros((T,T))\nwei = wei.masked_fill(tril == 0, float('-inf'))\nwei = F.softmax(wei, dim=-1)\nxbow3 = wei @ x\ntorch.allclose(xbow, xbow3)\n",[250,1713,1714,1719,1724,1729,1734,1739,1744],{"__ignoreMap":27},[285,1715,1716],{"class":287,"line":34},[285,1717,1718],{},"# versão 3: usando softmax\n",[285,1720,1721],{"class":287,"line":28},[285,1722,1723],{},"tril = torch.tril(torch.ones(T, T))\n",[285,1725,1726],{"class":287,"line":161},[285,1727,1728],{},"wei = torch.zeros((T,T))\n",[285,1730,1731],{"class":287,"line":188},[285,1732,1733],{},"wei = wei.masked_fill(tril == 0, float('-inf'))\n",[285,1735,1736],{"class":287,"line":229},[285,1737,1738],{},"wei = F.softmax(wei, dim=-1)\n",[285,1740,1741],{"class":287,"line":322},[285,1742,1743],{},"xbow3 = wei @ x\n",[285,1745,1746],{"class":287,"line":429},[285,1747,1748],{},"torch.allclose(xbow, xbow3)\n",[12,1750,1751,1754,1755,1758,1759,1762,1763,156],{},[16,1752,1753],{},"Version 3",", the same thing again, this time building the weights via ",[250,1756,1757],{},"softmax"," of a matrix masked with ",[250,1760,1761],{},"-inf"," above the diagonal (this matters in a moment, it's how the model will eventually learn weights that aren't just fixed averages). Real output again: ",[16,1764,1765],{},[250,1766,1708],{},[12,1768,1769,1770,1772,1773,1775,1776,1779],{},"That bugged me enough to check the original notebook, and it shows the exact same result, ",[250,1771,1708],{}," on both comparisons, with the exact same code. Not a bug either of us introduced: ",[250,1774,1698],{}," uses a fairly tight default tolerance, and the accumulated sum in a matrix multiply doesn't follow the same floating-point operation order as a loop-computed mean, so the two results land close but not bit-identical. It's a reminder worth more than a lot of theory: ",[16,1777,1778],{},"mathematically equivalent isn't the same as numerically identical",", floating point has its own opinion about operation order.",[272,1781,1783],{"id":1782},"real-self-attention","Real self-attention",[277,1785,1787],{"className":279,"code":1786,"language":281,"meta":27,"style":27},"#versão 4: self-attention\ntorch.manual_seed(1337)\nB,T,C = 4,8,32 #batch, time, channels\nx = torch.randn(B,T,C)\n\n#Vendo uma única \"cabeça\" performando a self-attention\nhead_size = 16\nkey = nn.Linear(C, head_size, bias=False)\nquery = nn.Linear(C, head_size, bias=False)\nvalue = nn.Linear(C, head_size, bias=False)\n\nk = key(x)   # (B, T, 16)\nq = query(x) # (B, T, 16)\nwei =  q @ k.transpose(-2, -1) # (B, T, 16) @ (B, 16, T) ---> (B, T, T)\n\ntril = torch.tril(torch.ones(T, T))\n#wei = torch.zeros((T,T))\nwei = wei.masked_fill(tril == 0, float('-inf'))\nwei = F.softmax(wei, dim=-1)\n\nv = value(x)\nout = wei @ v\n#out = wei @ x\n\nout.shape\n",[250,1788,1789,1794,1798,1803,1807,1811,1816,1821,1826,1831,1836,1840,1845,1850,1855,1859,1863,1868,1872,1876,1880,1885,1890,1895,1899],{"__ignoreMap":27},[285,1790,1791],{"class":287,"line":34},[285,1792,1793],{},"#versão 4: self-attention\n",[285,1795,1796],{"class":287,"line":28},[285,1797,789],{},[285,1799,1800],{"class":287,"line":161},[285,1801,1802],{},"B,T,C = 4,8,32 #batch, time, channels\n",[285,1804,1805],{"class":287,"line":188},[285,1806,1605],{},[285,1808,1809],{"class":287,"line":229},[285,1810,309],{"emptyLinePlaceholder":33},[285,1812,1813],{"class":287,"line":322},[285,1814,1815],{},"#Vendo uma única \"cabeça\" performando a self-attention\n",[285,1817,1818],{"class":287,"line":429},[285,1819,1820],{},"head_size = 16\n",[285,1822,1823],{"class":287,"line":435},[285,1824,1825],{},"key = nn.Linear(C, head_size, bias=False)\n",[285,1827,1828],{"class":287,"line":516},[285,1829,1830],{},"query = nn.Linear(C, head_size, bias=False)\n",[285,1832,1833],{"class":287,"line":522},[285,1834,1835],{},"value = nn.Linear(C, head_size, bias=False)\n",[285,1837,1838],{"class":287,"line":528},[285,1839,309],{"emptyLinePlaceholder":33},[285,1841,1842],{"class":287,"line":725},[285,1843,1844],{},"k = key(x)   # (B, T, 16)\n",[285,1846,1847],{"class":287,"line":731},[285,1848,1849],{},"q = query(x) # (B, T, 16)\n",[285,1851,1852],{"class":287,"line":850},[285,1853,1854],{},"wei =  q @ k.transpose(-2, -1) # (B, T, 16) @ (B, 16, T) ---> (B, T, T)\n",[285,1856,1857],{"class":287,"line":856},[285,1858,309],{"emptyLinePlaceholder":33},[285,1860,1861],{"class":287,"line":862},[285,1862,1723],{},[285,1864,1865],{"class":287,"line":868},[285,1866,1867],{},"#wei = torch.zeros((T,T))\n",[285,1869,1870],{"class":287,"line":874},[285,1871,1733],{},[285,1873,1874],{"class":287,"line":880},[285,1875,1738],{},[285,1877,1878],{"class":287,"line":886},[285,1879,309],{"emptyLinePlaceholder":33},[285,1881,1882],{"class":287,"line":892},[285,1883,1884],{},"v = value(x)\n",[285,1886,1887],{"class":287,"line":897},[285,1888,1889],{},"out = wei @ v\n",[285,1891,1892],{"class":287,"line":903},[285,1893,1894],{},"#out = wei @ x\n",[285,1896,1897],{"class":287,"line":909},[285,1898,309],{"emptyLinePlaceholder":33},[285,1900,1901],{"class":287,"line":915},[285,1902,1903],{},"out.shape\n",[12,1905,1906,1907,1910,1911,1914,1915,760,1918,1921,1922,1925,1926,1929,1930,1933,1934,1937,1938,1940,1941,1944,1945,1948,1949,1952],{},"Here the averaging weights stop being fixed (1\u002F2, 1\u002F3...) and become ",[16,1908,1909],{},"learned and data-dependent",": ",[250,1912,1913],{},"key",", ",[250,1916,1917],{},"query",[250,1919,1920],{},"value"," are three trainable linear layers. ",[250,1923,1924],{},"Query"," asks \"what am I looking for\", ",[250,1927,1928],{},"Key"," answers \"what do I have to offer\", the product ",[250,1931,1932],{},"q @ k.transpose()"," decides how much each earlier position matters, ",[250,1935,1936],{},"masked_fill"," + ",[250,1939,1757],{}," turns that into weights that sum to 1 and never peek at the future, and ",[250,1942,1943],{},"wei @ v"," runs the weighted average using ",[250,1946,1947],{},"Value"," (the actual content each position carries, not the key it uses to get found). Output: ",[250,1950,1951],{},"torch.Size([4, 8, 16])",", a 16-dimensional vector per position, already mixing in context information.",[277,1954,1956],{"className":279,"code":1955,"language":281,"meta":27,"style":27},"wei[0]\n",[250,1957,1958],{"__ignoreMap":27},[285,1959,1960],{"class":287,"line":34},[285,1961,1955],{},[12,1963,1964],{},"Prints the attention weight matrix for the first item in the batch. Real output (trimmed):",[277,1966,1969],{"className":1967,"code":1968,"language":334},[332],"tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.1574, 0.8426, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.2088, 0.1646, 0.6266, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n        ...\n",[250,1970,1968],{"__ignoreMap":27},[12,1972,1973,1974,1977],{},"Notice the first row is ",[250,1975,1976],{},"[1, 0, 0, ...]"," (the first position can only attend to itself, it has no past) and each following row distributes weight quite differently from \"simple average\", the second row gives 84% weight to the second token and only 16% to the first. That's the model deciding, not a fixed formula.",[272,1979,1981],{"id":1980},"why-divide-by-the-square-root-of-head_size","Why divide by the square root of head_size",[277,1983,1985],{"className":279,"code":1984,"language":281,"meta":27,"style":27},"k = torch.randn(B,T, head_size)\nq = torch.randn(B,T, head_size)\nwei = q @ k.transpose(-2, -1) * head_size**-0.5\n",[250,1986,1987,1992,1997],{"__ignoreMap":27},[285,1988,1989],{"class":287,"line":34},[285,1990,1991],{},"k = torch.randn(B,T, head_size)\n",[285,1993,1994],{"class":287,"line":28},[285,1995,1996],{},"q = torch.randn(B,T, head_size)\n",[285,1998,1999],{"class":287,"line":161},[285,2000,2001],{},"wei = q @ k.transpose(-2, -1) * head_size**-0.5\n",[277,2003,2005],{"className":279,"code":2004,"language":281,"meta":27,"style":27},"k.var()\n",[250,2006,2007],{"__ignoreMap":27},[285,2008,2009],{"class":287,"line":34},[285,2010,2004],{},[12,2012,646,2013],{},[250,2014,2015],{},"tensor(1.0449)",[277,2017,2019],{"className":279,"code":2018,"language":281,"meta":27,"style":27},"q.var()\n",[250,2020,2021],{"__ignoreMap":27},[285,2022,2023],{"class":287,"line":34},[285,2024,2018],{},[12,2026,646,2027],{},[250,2028,2029],{},"tensor(1.0700)",[277,2031,2033],{"className":279,"code":2032,"language":281,"meta":27,"style":27},"wei.var()\n",[250,2034,2035],{"__ignoreMap":27},[285,2036,2037],{"class":287,"line":34},[285,2038,2032],{},[12,2040,646,2041],{},[250,2042,2043],{},"tensor(1.0918)",[12,2045,2046,760,2049,2052,2053,2056,2057,2060,2061,2063,2064,2067,2068,2070,2071,2073,2074,2077,2078,2080],{},[250,2047,2048],{},"k",[250,2050,2051],{},"q"," start with variance near 1 (default initialization). Without the ",[250,2054,2055],{},"head_size**-0.5"," factor multiplying ",[250,2058,2059],{},"wei",", the variance of the ",[250,2062,1932],{}," product would explode proportionally to ",[250,2065,2066],{},"head_size",", because each element of ",[250,2069,2059],{}," is the sum of ",[250,2072,2066],{}," multiplications. Scaling by ",[250,2075,2076],{},"1\u002F√head_size"," keeps ",[250,2079,2059],{},"'s variance near 1 too, and that matters because:",[277,2082,2084],{"className":279,"code":2083,"language":281,"meta":27,"style":27},"torch.softmax(torch.tensor([0.1, -0.2, 0.3, -0.2, 0.5]), dim=-1)\n",[250,2085,2086],{"__ignoreMap":27},[285,2087,2088],{"class":287,"line":34},[285,2089,2083],{},[12,2091,646,2092,2095],{},[250,2093,2094],{},"tensor([0.1925, 0.1426, 0.2351, 0.1426, 0.2872])",", a relatively smooth distribution.",[277,2097,2099],{"className":279,"code":2098,"language":281,"meta":27,"style":27},"torch.softmax(torch.tensor([0.1, -0.2, 0.3, -0.2, 0.5])*8, dim=-1) # gets too peaky, converges to one-hot\n",[250,2100,2101],{"__ignoreMap":27},[285,2102,2103],{"class":287,"line":34},[285,2104,2098],{},[12,2106,646,2107,2110],{},[250,2108,2109],{},"tensor([0.0326, 0.0030, 0.1615, 0.0030, 0.8000])",", almost all the weight (80%) stuck on one value.",[12,2112,2113,2114,2116,2117,2121,2122,2124],{},"Same input numbers, just multiplied by 8, and ",[250,2115,1757],{}," already turns into nearly a \"pick one winner\" (",[656,2118,2120],{"definition":2119},"a vector where one position is 1 and every other is 0, the opposite of a smooth probability distribution","one-hot","). If ",[250,2123,2059],{}," weren't scaled, the attention weights would always live in that \"peaky\" regime, the model would attend to a single token and ignore the rest, instead of learning a real mixture.",[272,2126,2128],{"id":2127},"layer-normalization","Layer normalization",[277,2130,2132],{"className":279,"code":2131,"language":281,"meta":27,"style":27},"class LayerNorm1d:\n\n  def __init__(self, dim, eps=1e-5, momentum=0.1):\n    self.eps = eps\n    self.gamma = torch.ones(dim)\n    self.beta = torch.zeros(dim)\n\n  def __call__(self, x):\n    # calcular a passagem direta (forward pass)\n    xmean = x.mean(1, keepdim=True) # média do lote\n    xvar = x.var(1, keepdim=True) # Variância do lote\n    # normalizar para variância unitária\n    xhat = (x - xmean) \u002F torch.sqrt(xvar + self.eps)\n    self.out = self.gamma * xhat + self.beta\n    return self.out\n\n  def parameters(self):\n    return [self.gamma, self.beta]\n\ntorch.manual_seed(1337)\nmodule = LayerNorm1d(100)\n# tamanho de lote de 32 vetores de 100 dimensões\nx = torch.randn(32, 100)\nx = module(x)\nx.shape\n",[250,2133,2134,2139,2143,2148,2153,2158,2163,2167,2172,2177,2182,2187,2192,2197,2202,2207,2211,2216,2221,2225,2229,2234,2239,2244,2249],{"__ignoreMap":27},[285,2135,2136],{"class":287,"line":34},[285,2137,2138],{},"class LayerNorm1d:\n",[285,2140,2141],{"class":287,"line":28},[285,2142,309],{"emptyLinePlaceholder":33},[285,2144,2145],{"class":287,"line":161},[285,2146,2147],{},"  def __init__(self, dim, eps=1e-5, momentum=0.1):\n",[285,2149,2150],{"class":287,"line":188},[285,2151,2152],{},"    self.eps = eps\n",[285,2154,2155],{"class":287,"line":229},[285,2156,2157],{},"    self.gamma = torch.ones(dim)\n",[285,2159,2160],{"class":287,"line":322},[285,2161,2162],{},"    self.beta = torch.zeros(dim)\n",[285,2164,2165],{"class":287,"line":429},[285,2166,309],{"emptyLinePlaceholder":33},[285,2168,2169],{"class":287,"line":435},[285,2170,2171],{},"  def __call__(self, x):\n",[285,2173,2174],{"class":287,"line":516},[285,2175,2176],{},"    # calcular a passagem direta (forward pass)\n",[285,2178,2179],{"class":287,"line":522},[285,2180,2181],{},"    xmean = x.mean(1, keepdim=True) # média do lote\n",[285,2183,2184],{"class":287,"line":528},[285,2185,2186],{},"    xvar = x.var(1, keepdim=True) # Variância do lote\n",[285,2188,2189],{"class":287,"line":725},[285,2190,2191],{},"    # normalizar para variância unitária\n",[285,2193,2194],{"class":287,"line":731},[285,2195,2196],{},"    xhat = (x - xmean) \u002F torch.sqrt(xvar + self.eps)\n",[285,2198,2199],{"class":287,"line":850},[285,2200,2201],{},"    self.out = self.gamma * xhat + self.beta\n",[285,2203,2204],{"class":287,"line":856},[285,2205,2206],{},"    return self.out\n",[285,2208,2209],{"class":287,"line":862},[285,2210,309],{"emptyLinePlaceholder":33},[285,2212,2213],{"class":287,"line":868},[285,2214,2215],{},"  def parameters(self):\n",[285,2217,2218],{"class":287,"line":874},[285,2219,2220],{},"    return [self.gamma, self.beta]\n",[285,2222,2223],{"class":287,"line":880},[285,2224,309],{"emptyLinePlaceholder":33},[285,2226,2227],{"class":287,"line":886},[285,2228,789],{},[285,2230,2231],{"class":287,"line":892},[285,2232,2233],{},"module = LayerNorm1d(100)\n",[285,2235,2236],{"class":287,"line":897},[285,2237,2238],{},"# tamanho de lote de 32 vetores de 100 dimensões\n",[285,2240,2241],{"class":287,"line":903},[285,2242,2243],{},"x = torch.randn(32, 100)\n",[285,2245,2246],{"class":287,"line":909},[285,2247,2248],{},"x = module(x)\n",[285,2250,2251],{"class":287,"line":915},[285,2252,1610],{},[12,2254,2255,2256,2260,2261,2264,2265,760,2268,2271,2272,2275,2276,2279,2280,2283],{},"A from-scratch implementation of ",[656,2257,2259],{"definition":2258},"a technique that recenters and rescales each example's activations individually, keeping mean near 0 and variance near 1 to stabilize training in deep networks","layer normalization",", without using the built-in ",[250,2262,2263],{},"nn.LayerNorm",", just to see the math up close. ",[250,2266,2267],{},"xmean",[250,2269,2270],{},"xvar"," compute mean and variance, ",[250,2273,2274],{},"xhat"," normalizes, ",[250,2277,2278],{},"gamma","\u002F",[250,2281,2282],{},"beta"," let the model relearn a different scale\u002Fshift if it needs to (they start at 1 and 0, meaning no effect at all, until training adjusts them).",[277,2285,2287],{"className":279,"code":2286,"language":281,"meta":27,"style":27},"# média e desvio padrão de uma característica\n#em todas as entradas do lote\nx[:,0].mean(), x[:,0].std()\n",[250,2288,2289,2294,2299],{"__ignoreMap":27},[285,2290,2291],{"class":287,"line":34},[285,2292,2293],{},"# média e desvio padrão de uma característica\n",[285,2295,2296],{"class":287,"line":28},[285,2297,2298],{},"#em todas as entradas do lote\n",[285,2300,2301],{"class":287,"line":161},[285,2302,2303],{},"x[:,0].mean(), x[:,0].std()\n",[12,2305,646,2306,2309,2310,2313],{},[250,2307,2308],{},"(tensor(0.1469), tensor(0.8803))",", the first ",[16,2311,2312],{},"feature",", looking across all 32 entries in the batch, isn't exactly mean 0\u002Fstd 1.",[277,2315,2317],{"className":279,"code":2316,"language":281,"meta":27,"style":27},"# média e desvio padrão de uma única entrada do\n# lote, considerando suas características\nx[0,:].mean(), x[0,:].std()\n",[250,2318,2319,2324,2329],{"__ignoreMap":27},[285,2320,2321],{"class":287,"line":34},[285,2322,2323],{},"# média e desvio padrão de uma única entrada do\n",[285,2325,2326],{"class":287,"line":28},[285,2327,2328],{},"# lote, considerando suas características\n",[285,2330,2331],{"class":287,"line":161},[285,2332,2333],{},"x[0,:].mean(), x[0,:].std()\n",[12,2335,646,2336,2339,2340,2343],{},[250,2337,2338],{},"(tensor(-9.5367e-09), tensor(1.0000))",", practically 0 and exactly 1. That's the key difference between layer norm and batch norm: here normalization happens ",[16,2341,2342],{},"per individual example"," (looking at that one row's 100 dimensions), not per feature across the whole batch, which is why the second check lands perfectly and the first one doesn't.",[272,2345,2347,2348,2351],{"id":2346},"bug-3-the-fold-gotcha-and-the-full-model","Bug #3: the ",[250,2349,2350],{},"Fold"," gotcha, and the full model",[277,2353,2355],{"className":279,"code":2354,"language":281,"meta":27,"style":27},"import torch\nimport torch.nn as nn\nfrom torch.nn import functional as Fold\n\n#hiperparâmetros\nbatch_size = 16 # Número de sequências independentes que serão processadas em paralelo\nblock_size = 32 # qual o contexto máximo de predições\nmax_iters = 5000 #máximo de iterações\neval_interval = 100\nlearning_rate = 1e-3\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\neval_iters = 200\nn_embd = 64\nn_head = 4\nn_layer = 4\ndropout = 0.0\n\ntorch.manual_seed(1337)\n\n#!wget https:\u002F\u002Fwww.gutenberg.org\u002Fcache\u002Fepub\u002F55752\u002Fpg55752.txt\nwith open('pg55752.txt', 'r') as f:\n  texto = f.read()\n\n#Apenas caracteres únicos(limpeza)\nchars = sorted(list(set(texto)))\ntam_vocabulario = len(chars)\n# criando um mapeamento de caracteres para inteiros\nstring_to_int = { ch:i for i,ch in enumerate(chars) }\nint_to_string = { i:ch for i,ch in enumerate(chars) }\n\n# encoder: toma uma string, devolve uma lista de inteiros\nencode = lambda s: [string_to_int[c] for c in s]\n\n# decoder: toma uma lista de inteiros, devolve uma string\ndecode = lambda l: ''.join([int_to_string[i] for i in l])\n\n#splitando o dataset\ndata = torch.tensor(encode(texto), dtype=torch.long)\nn = int(0.9*len(data))\ndados_treino = data[:n]\ndados_avaliacao = data[n:]\nprint(f\"Dispositivo: {device}\")\nprint(f\"Tamanho do texto: {len(texto)} caracteres\")\nprint(f\"Tamanho do vocabulario: {tam_vocabulario}\")\nprint(f\"Treino: {len(dados_treino)} | Avaliacao: {len(dados_avaliacao)}\")\n\n#carregamento dos dados\ndef get_batch(split):\n  #gerando um lote pequeno de dados de input x e targets y\n  dados = dados_treino if split == 'treino' else dados_avaliacao\n  ix = torch.randint(len(dados) - block_size, (batch_size,))\n  x = torch.stack([dados[i:i+block_size] for i in ix])\n  y = torch.stack([dados[i+1:i+block_size+1] for i in ix])\n  x, y = x.to(device), y.to(device)\n  return x, y\n\n@torch.no_grad()\ndef perda_estimada():\n  out = {}\n  modelo.eval()\n  for split in ['treino', 'avaliacao']:\n    losses = torch.zeros(eval_iters)\n    for k in range(eval_iters):\n      X, Y = get_batch(split)\n      logits, loss = modelo(X, Y)\n      losses[k] = loss.item()\n    out[split] = losses.mean()\n\n  modelo.train()\n  return out\n\nclass Head(nn.Module):\n  \"\"\" uma cabeça de self-attention \"\"\"\n\n  def __init__(self, head_size):\n    super().__init__()\n    self.key = nn.Linear(n_embd, head_size, bias=False)\n    self.query = nn.Linear(n_embd, head_size, bias=False)\n    self.value = nn.Linear(n_embd, head_size, bias=False)\n    self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n\n    self.dropout = nn.Dropout(dropout)\n\n  def forward(self, x):\n    B,T,C = x.shape\n    k = self.key(x)   # (B,T,C)\n    q = self.query(x) # (B,T,C)\n    # calcular pontuações de atenção (\"afinidades\")\n    wei = q @ k.transpose(-2,-1) * C**-0.5 # (B, T, C) @ (B, C, T) -> (B, T, T)\n    wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)\n    wei = F.softmax(wei, dim=-1) # (B, T, T)\n    wei = self.dropout(wei)\n    # performar a média ponderada das valores\n    v = self.value(x) # (B,T,C)\n    out = wei @ v # (B, T, T) @ (B, T, C) -> (B, T, C)\n    return out\n\nclass MultiHeadAttention(nn.Module):\n  \"\"\" Múltiplas cabeças de self-attention em paralelo \"\"\"\n\n  def __init__(self, numero_cabecas, tamanho_cabeca):\n    super().__init__()\n    self.heads = nn.ModuleList([Head(tamanho_cabeca) for _ in range(numero_cabecas)])\n    self.proj = nn.Linear(n_embd, n_embd)\n    self.dropout = nn.Dropout(dropout)\n\n  def forward(self, x):\n    out = torch.cat([h(x) for h in self.heads], dim=-1)\n    out = self.dropout(self.proj(out))\n    return out\n\nclass FeedFoward(nn.Module):\n  \"\"\" Uma camada linear simples seguida de uma função de ativação ReLU \"\"\"\n\n  def __init__(self, n_embd):\n    super().__init__()\n    self.net = nn.Sequential(\n        nn.Linear(n_embd, 4 * n_embd),\n        nn.ReLU(),\n        nn.Linear( 4 * n_embd, n_embd),\n        nn.Dropout(dropout)\n    )\n\n  def forward(self, x):\n    return self.net(x)\n\nclass Block(nn.Module):\n  \"\"\"Bloco Transformer: comunicação seguida de computação\"\"\"\n\n  def __init__(self, n_embd, n_head):\n    # n_embd: embedding dimension, n_head:\n    #quantidade de cabeças desejadas\n    super().__init__()\n    head_size = n_embd \u002F\u002F n_head\n    self.sa = MultiHeadAttention(n_head, head_size)\n    self.ffwd = FeedFoward(n_embd)\n    self.ln1 = nn.LayerNorm(n_embd)\n    self.ln2 = nn.LayerNorm(n_embd)\n\n  def forward(self, x):\n    x = x + self.sa(self.ln1(x))\n    x = x + self.ffwd(self.ln2(x))\n    return x\n",[250,2356,2357,2361,2365,2370,2374,2379,2384,2389,2394,2399,2404,2409,2414,2419,2424,2429,2434,2438,2442,2446,2451,2456,2461,2465,2470,2475,2480,2485,2490,2495,2499,2504,2509,2513,2518,2523,2527,2532,2536,2540,2545,2550,2555,2560,2565,2570,2574,2579,2583,2588,2594,2599,2605,2610,2616,2621,2626,2632,2638,2644,2650,2656,2662,2668,2674,2680,2686,2692,2697,2703,2709,2714,2720,2726,2731,2737,2742,2748,2754,2760,2766,2771,2777,2782,2788,2794,2800,2806,2812,2818,2824,2830,2836,2842,2848,2854,2860,2865,2871,2877,2882,2888,2893,2899,2905,2910,2915,2920,2926,2932,2937,2942,2948,2954,2959,2965,2970,2976,2982,2988,2994,3000,3006,3011,3016,3022,3027,3033,3039,3044,3050,3056,3062,3067,3073,3079,3085,3091,3097,3102,3107,3113,3119],{"__ignoreMap":27},[285,2358,2359],{"class":287,"line":34},[285,2360,485],{},[285,2362,2363],{"class":287,"line":28},[285,2364,999],{},[285,2366,2367],{"class":287,"line":161},[285,2368,2369],{},"from torch.nn import functional as Fold\n",[285,2371,2372],{"class":287,"line":188},[285,2373,309],{"emptyLinePlaceholder":33},[285,2375,2376],{"class":287,"line":229},[285,2377,2378],{},"#hiperparâmetros\n",[285,2380,2381],{"class":287,"line":322},[285,2382,2383],{},"batch_size = 16 # Número de sequências independentes que serão processadas em paralelo\n",[285,2385,2386],{"class":287,"line":429},[285,2387,2388],{},"block_size = 32 # qual o contexto máximo de predições\n",[285,2390,2391],{"class":287,"line":435},[285,2392,2393],{},"max_iters = 5000 #máximo de iterações\n",[285,2395,2396],{"class":287,"line":516},[285,2397,2398],{},"eval_interval = 100\n",[285,2400,2401],{"class":287,"line":522},[285,2402,2403],{},"learning_rate = 1e-3\n",[285,2405,2406],{"class":287,"line":528},[285,2407,2408],{},"device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",[285,2410,2411],{"class":287,"line":725},[285,2412,2413],{},"eval_iters = 200\n",[285,2415,2416],{"class":287,"line":731},[285,2417,2418],{},"n_embd = 64\n",[285,2420,2421],{"class":287,"line":850},[285,2422,2423],{},"n_head = 4\n",[285,2425,2426],{"class":287,"line":856},[285,2427,2428],{},"n_layer = 4\n",[285,2430,2431],{"class":287,"line":862},[285,2432,2433],{},"dropout = 0.0\n",[285,2435,2436],{"class":287,"line":868},[285,2437,309],{"emptyLinePlaceholder":33},[285,2439,2440],{"class":287,"line":874},[285,2441,789],{},[285,2443,2444],{"class":287,"line":880},[285,2445,309],{"emptyLinePlaceholder":33},[285,2447,2448],{"class":287,"line":886},[285,2449,2450],{},"#!wget https:\u002F\u002Fwww.gutenberg.org\u002Fcache\u002Fepub\u002F55752\u002Fpg55752.txt\n",[285,2452,2453],{"class":287,"line":892},[285,2454,2455],{},"with open('pg55752.txt', 'r') as f:\n",[285,2457,2458],{"class":287,"line":897},[285,2459,2460],{},"  texto = f.read()\n",[285,2462,2463],{"class":287,"line":903},[285,2464,309],{"emptyLinePlaceholder":33},[285,2466,2467],{"class":287,"line":909},[285,2468,2469],{},"#Apenas caracteres únicos(limpeza)\n",[285,2471,2472],{"class":287,"line":915},[285,2473,2474],{},"chars = sorted(list(set(texto)))\n",[285,2476,2477],{"class":287,"line":921},[285,2478,2479],{},"tam_vocabulario = len(chars)\n",[285,2481,2482],{"class":287,"line":1117},[285,2483,2484],{},"# criando um mapeamento de caracteres para inteiros\n",[285,2486,2487],{"class":287,"line":1122},[285,2488,2489],{},"string_to_int = { ch:i for i,ch in enumerate(chars) }\n",[285,2491,2492],{"class":287,"line":1128},[285,2493,2494],{},"int_to_string = { i:ch for i,ch in enumerate(chars) }\n",[285,2496,2497],{"class":287,"line":1134},[285,2498,309],{"emptyLinePlaceholder":33},[285,2500,2501],{"class":287,"line":1140},[285,2502,2503],{},"# encoder: toma uma string, devolve uma lista de inteiros\n",[285,2505,2506],{"class":287,"line":1146},[285,2507,2508],{},"encode = lambda s: [string_to_int[c] for c in s]\n",[285,2510,2511],{"class":287,"line":1152},[285,2512,309],{"emptyLinePlaceholder":33},[285,2514,2515],{"class":287,"line":1158},[285,2516,2517],{},"# decoder: toma uma lista de inteiros, devolve uma string\n",[285,2519,2520],{"class":287,"line":1164},[285,2521,2522],{},"decode = lambda l: ''.join([int_to_string[i] for i in l])\n",[285,2524,2525],{"class":287,"line":1170},[285,2526,309],{"emptyLinePlaceholder":33},[285,2528,2529],{"class":287,"line":1176},[285,2530,2531],{},"#splitando o dataset\n",[285,2533,2534],{"class":287,"line":1182},[285,2535,490],{},[285,2537,2538],{"class":287,"line":1188},[285,2539,571],{},[285,2541,2542],{"class":287,"line":1194},[285,2543,2544],{},"dados_treino = data[:n]\n",[285,2546,2547],{"class":287,"line":1200},[285,2548,2549],{},"dados_avaliacao = data[n:]\n",[285,2551,2552],{"class":287,"line":1205},[285,2553,2554],{},"print(f\"Dispositivo: {device}\")\n",[285,2556,2557],{"class":287,"line":1211},[285,2558,2559],{},"print(f\"Tamanho do texto: {len(texto)} caracteres\")\n",[285,2561,2562],{"class":287,"line":1217},[285,2563,2564],{},"print(f\"Tamanho do vocabulario: {tam_vocabulario}\")\n",[285,2566,2567],{"class":287,"line":1223},[285,2568,2569],{},"print(f\"Treino: {len(dados_treino)} | Avaliacao: {len(dados_avaliacao)}\")\n",[285,2571,2572],{"class":287,"line":1229},[285,2573,309],{"emptyLinePlaceholder":33},[285,2575,2576],{"class":287,"line":1234},[285,2577,2578],{},"#carregamento dos dados\n",[285,2580,2581],{"class":287,"line":1240},[285,2582,808],{},[285,2584,2585],{"class":287,"line":1245},[285,2586,2587],{},"  #gerando um lote pequeno de dados de input x e targets y\n",[285,2589,2591],{"class":287,"line":2590},50,[285,2592,2593],{},"  dados = dados_treino if split == 'treino' else dados_avaliacao\n",[285,2595,2597],{"class":287,"line":2596},51,[285,2598,823],{},[285,2600,2602],{"class":287,"line":2601},52,[285,2603,2604],{},"  x = torch.stack([dados[i:i+block_size] for i in ix])\n",[285,2606,2608],{"class":287,"line":2607},53,[285,2609,833],{},[285,2611,2613],{"class":287,"line":2612},54,[285,2614,2615],{},"  x, y = x.to(device), y.to(device)\n",[285,2617,2619],{"class":287,"line":2618},55,[285,2620,838],{},[285,2622,2624],{"class":287,"line":2623},56,[285,2625,309],{"emptyLinePlaceholder":33},[285,2627,2629],{"class":287,"line":2628},57,[285,2630,2631],{},"@torch.no_grad()\n",[285,2633,2635],{"class":287,"line":2634},58,[285,2636,2637],{},"def perda_estimada():\n",[285,2639,2641],{"class":287,"line":2640},59,[285,2642,2643],{},"  out = {}\n",[285,2645,2647],{"class":287,"line":2646},60,[285,2648,2649],{},"  modelo.eval()\n",[285,2651,2653],{"class":287,"line":2652},61,[285,2654,2655],{},"  for split in ['treino', 'avaliacao']:\n",[285,2657,2659],{"class":287,"line":2658},62,[285,2660,2661],{},"    losses = torch.zeros(eval_iters)\n",[285,2663,2665],{"class":287,"line":2664},63,[285,2666,2667],{},"    for k in range(eval_iters):\n",[285,2669,2671],{"class":287,"line":2670},64,[285,2672,2673],{},"      X, Y = get_batch(split)\n",[285,2675,2677],{"class":287,"line":2676},65,[285,2678,2679],{},"      logits, loss = modelo(X, Y)\n",[285,2681,2683],{"class":287,"line":2682},66,[285,2684,2685],{},"      losses[k] = loss.item()\n",[285,2687,2689],{"class":287,"line":2688},67,[285,2690,2691],{},"    out[split] = losses.mean()\n",[285,2693,2695],{"class":287,"line":2694},68,[285,2696,309],{"emptyLinePlaceholder":33},[285,2698,2700],{"class":287,"line":2699},69,[285,2701,2702],{},"  modelo.train()\n",[285,2704,2706],{"class":287,"line":2705},70,[285,2707,2708],{},"  return out\n",[285,2710,2712],{"class":287,"line":2711},71,[285,2713,309],{"emptyLinePlaceholder":33},[285,2715,2717],{"class":287,"line":2716},72,[285,2718,2719],{},"class Head(nn.Module):\n",[285,2721,2723],{"class":287,"line":2722},73,[285,2724,2725],{},"  \"\"\" uma cabeça de self-attention \"\"\"\n",[285,2727,2729],{"class":287,"line":2728},74,[285,2730,309],{"emptyLinePlaceholder":33},[285,2732,2734],{"class":287,"line":2733},75,[285,2735,2736],{},"  def __init__(self, head_size):\n",[285,2738,2740],{"class":287,"line":2739},76,[285,2741,1027],{},[285,2743,2745],{"class":287,"line":2744},77,[285,2746,2747],{},"    self.key = nn.Linear(n_embd, head_size, bias=False)\n",[285,2749,2751],{"class":287,"line":2750},78,[285,2752,2753],{},"    self.query = nn.Linear(n_embd, head_size, bias=False)\n",[285,2755,2757],{"class":287,"line":2756},79,[285,2758,2759],{},"    self.value = nn.Linear(n_embd, head_size, bias=False)\n",[285,2761,2763],{"class":287,"line":2762},80,[285,2764,2765],{},"    self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n",[285,2767,2769],{"class":287,"line":2768},81,[285,2770,309],{"emptyLinePlaceholder":33},[285,2772,2774],{"class":287,"line":2773},82,[285,2775,2776],{},"    self.dropout = nn.Dropout(dropout)\n",[285,2778,2780],{"class":287,"line":2779},83,[285,2781,309],{"emptyLinePlaceholder":33},[285,2783,2785],{"class":287,"line":2784},84,[285,2786,2787],{},"  def forward(self, x):\n",[285,2789,2791],{"class":287,"line":2790},85,[285,2792,2793],{},"    B,T,C = x.shape\n",[285,2795,2797],{"class":287,"line":2796},86,[285,2798,2799],{},"    k = self.key(x)   # (B,T,C)\n",[285,2801,2803],{"class":287,"line":2802},87,[285,2804,2805],{},"    q = self.query(x) # (B,T,C)\n",[285,2807,2809],{"class":287,"line":2808},88,[285,2810,2811],{},"    # calcular pontuações de atenção (\"afinidades\")\n",[285,2813,2815],{"class":287,"line":2814},89,[285,2816,2817],{},"    wei = q @ k.transpose(-2,-1) * C**-0.5 # (B, T, C) @ (B, C, T) -> (B, T, T)\n",[285,2819,2821],{"class":287,"line":2820},90,[285,2822,2823],{},"    wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)\n",[285,2825,2827],{"class":287,"line":2826},91,[285,2828,2829],{},"    wei = F.softmax(wei, dim=-1) # (B, T, T)\n",[285,2831,2833],{"class":287,"line":2832},92,[285,2834,2835],{},"    wei = self.dropout(wei)\n",[285,2837,2839],{"class":287,"line":2838},93,[285,2840,2841],{},"    # performar a média ponderada das valores\n",[285,2843,2845],{"class":287,"line":2844},94,[285,2846,2847],{},"    v = self.value(x) # (B,T,C)\n",[285,2849,2851],{"class":287,"line":2850},95,[285,2852,2853],{},"    out = wei @ v # (B, T, T) @ (B, T, C) -> (B, T, C)\n",[285,2855,2857],{"class":287,"line":2856},96,[285,2858,2859],{},"    return out\n",[285,2861,2863],{"class":287,"line":2862},97,[285,2864,309],{"emptyLinePlaceholder":33},[285,2866,2868],{"class":287,"line":2867},98,[285,2869,2870],{},"class MultiHeadAttention(nn.Module):\n",[285,2872,2874],{"class":287,"line":2873},99,[285,2875,2876],{},"  \"\"\" Múltiplas cabeças de self-attention em paralelo \"\"\"\n",[285,2878,2880],{"class":287,"line":2879},100,[285,2881,309],{"emptyLinePlaceholder":33},[285,2883,2885],{"class":287,"line":2884},101,[285,2886,2887],{},"  def __init__(self, numero_cabecas, tamanho_cabeca):\n",[285,2889,2891],{"class":287,"line":2890},102,[285,2892,1027],{},[285,2894,2896],{"class":287,"line":2895},103,[285,2897,2898],{},"    self.heads = nn.ModuleList([Head(tamanho_cabeca) for _ in range(numero_cabecas)])\n",[285,2900,2902],{"class":287,"line":2901},104,[285,2903,2904],{},"    self.proj = nn.Linear(n_embd, n_embd)\n",[285,2906,2908],{"class":287,"line":2907},105,[285,2909,2776],{},[285,2911,2913],{"class":287,"line":2912},106,[285,2914,309],{"emptyLinePlaceholder":33},[285,2916,2918],{"class":287,"line":2917},107,[285,2919,2787],{},[285,2921,2923],{"class":287,"line":2922},108,[285,2924,2925],{},"    out = torch.cat([h(x) for h in self.heads], dim=-1)\n",[285,2927,2929],{"class":287,"line":2928},109,[285,2930,2931],{},"    out = self.dropout(self.proj(out))\n",[285,2933,2935],{"class":287,"line":2934},110,[285,2936,2859],{},[285,2938,2940],{"class":287,"line":2939},111,[285,2941,309],{"emptyLinePlaceholder":33},[285,2943,2945],{"class":287,"line":2944},112,[285,2946,2947],{},"class FeedFoward(nn.Module):\n",[285,2949,2951],{"class":287,"line":2950},113,[285,2952,2953],{},"  \"\"\" Uma camada linear simples seguida de uma função de ativação ReLU \"\"\"\n",[285,2955,2957],{"class":287,"line":2956},114,[285,2958,309],{"emptyLinePlaceholder":33},[285,2960,2962],{"class":287,"line":2961},115,[285,2963,2964],{},"  def __init__(self, n_embd):\n",[285,2966,2968],{"class":287,"line":2967},116,[285,2969,1027],{},[285,2971,2973],{"class":287,"line":2972},117,[285,2974,2975],{},"    self.net = nn.Sequential(\n",[285,2977,2979],{"class":287,"line":2978},118,[285,2980,2981],{},"        nn.Linear(n_embd, 4 * n_embd),\n",[285,2983,2985],{"class":287,"line":2984},119,[285,2986,2987],{},"        nn.ReLU(),\n",[285,2989,2991],{"class":287,"line":2990},120,[285,2992,2993],{},"        nn.Linear( 4 * n_embd, n_embd),\n",[285,2995,2997],{"class":287,"line":2996},121,[285,2998,2999],{},"        nn.Dropout(dropout)\n",[285,3001,3003],{"class":287,"line":3002},122,[285,3004,3005],{},"    )\n",[285,3007,3009],{"class":287,"line":3008},123,[285,3010,309],{"emptyLinePlaceholder":33},[285,3012,3014],{"class":287,"line":3013},124,[285,3015,2787],{},[285,3017,3019],{"class":287,"line":3018},125,[285,3020,3021],{},"    return self.net(x)\n",[285,3023,3025],{"class":287,"line":3024},126,[285,3026,309],{"emptyLinePlaceholder":33},[285,3028,3030],{"class":287,"line":3029},127,[285,3031,3032],{},"class Block(nn.Module):\n",[285,3034,3036],{"class":287,"line":3035},128,[285,3037,3038],{},"  \"\"\"Bloco Transformer: comunicação seguida de computação\"\"\"\n",[285,3040,3042],{"class":287,"line":3041},129,[285,3043,309],{"emptyLinePlaceholder":33},[285,3045,3047],{"class":287,"line":3046},130,[285,3048,3049],{},"  def __init__(self, n_embd, n_head):\n",[285,3051,3053],{"class":287,"line":3052},131,[285,3054,3055],{},"    # n_embd: embedding dimension, n_head:\n",[285,3057,3059],{"class":287,"line":3058},132,[285,3060,3061],{},"    #quantidade de cabeças desejadas\n",[285,3063,3065],{"class":287,"line":3064},133,[285,3066,1027],{},[285,3068,3070],{"class":287,"line":3069},134,[285,3071,3072],{},"    head_size = n_embd \u002F\u002F n_head\n",[285,3074,3076],{"class":287,"line":3075},135,[285,3077,3078],{},"    self.sa = MultiHeadAttention(n_head, head_size)\n",[285,3080,3082],{"class":287,"line":3081},136,[285,3083,3084],{},"    self.ffwd = FeedFoward(n_embd)\n",[285,3086,3088],{"class":287,"line":3087},137,[285,3089,3090],{},"    self.ln1 = nn.LayerNorm(n_embd)\n",[285,3092,3094],{"class":287,"line":3093},138,[285,3095,3096],{},"    self.ln2 = nn.LayerNorm(n_embd)\n",[285,3098,3100],{"class":287,"line":3099},139,[285,3101,309],{"emptyLinePlaceholder":33},[285,3103,3105],{"class":287,"line":3104},140,[285,3106,2787],{},[285,3108,3110],{"class":287,"line":3109},141,[285,3111,3112],{},"    x = x + self.sa(self.ln1(x))\n",[285,3114,3116],{"class":287,"line":3115},142,[285,3117,3118],{},"    x = x + self.ffwd(self.ln2(x))\n",[285,3120,3122],{"class":287,"line":3121},143,[285,3123,3124],{},"    return x\n",[12,3126,3127,3128,3131,3132,3134,3135,3138,3139,3142,3143,3146,3147,3150,3151,3153,3154,3157,3158,3161,3162,3165],{},"Found the third bug right on the second line: ",[250,3129,3130],{},"from torch.nn import functional as Fold",". ",[250,3133,2350],{},", not ",[250,3136,3137],{},"F",". Yet the entire ",[250,3140,3141],{},"Head"," class uses ",[250,3144,3145],{},"F.softmax",", with no error at all. Why? Because a much earlier cell (the first bigram) had already run ",[250,3148,3149],{},"from torch.nn import functional as F",", and that ",[250,3152,3137],{}," stays alive in the Colab kernel's memory even without re-running that earlier cell. If this were a ",[250,3155,3156],{},".py"," script run from scratch, it would've crashed with ",[250,3159,3160],{},"NameError"," immediately. It's the classic notebook trap: the order you ",[66,3163,3164],{},"run"," cells in matters more than the order they appear on screen.",[12,3167,3168,3169,1914,3172,1914,3175,1914,3178,156],{},"The real output from this cell confirms the rest ran fine: ",[250,3170,3171],{},"Dispositivo: cuda",[250,3173,3174],{},"Tamanho do texto: 400944 caracteres",[250,3176,3177],{},"Tamanho do vocabulario: 112",[250,3179,3180],{},"Treino: 360849 | Avaliacao: 40095",[12,3182,3183,3184,3186,3187,3190,3191,3193,3194,3197,3198,3201,3202,3205,3206,3209,3210,3213,3214,3217,3218,3221],{},"The rest of the cell stacks the pieces: ",[250,3185,3141],{}," is one attention head (what we already saw above, now made reusable). ",[250,3188,3189],{},"MultiHeadAttention"," runs several ",[250,3192,3141],{},"s in parallel and concatenates the result. ",[250,3195,3196],{},"FeedFoward"," is a two-layer linear network with ",[250,3199,3200],{},"ReLU"," in between, processing each position on its own (not looking at its neighbors). ",[250,3203,3204],{},"Block"," wraps everything with a ",[16,3207,3208],{},"residual connection"," (the ",[250,3211,3212],{},"x = x + ..."," instead of ",[250,3215,3216],{},"x = ...",", letting gradients flow straight through the layers) and ",[16,3219,3220],{},"layer norm"," before each sub-layer.",[277,3223,3225],{"className":279,"code":3224,"language":281,"meta":27,"style":27},"#um super simples modelo bigram\nclass BigramLanguageModel(nn.Module):\n  def __init__(self):\n    super().__init__()\n    # cada token lê diretamente os logits para o próximo token a partir de uma\n    # tabela de consulta\n    self.token_embeeding_table = nn.Embedding(tam_vocabulario, n_embd)\n    self.position_embeeding_table =nn.Embedding(block_size, n_embd)\n    self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])\n    self.ln_f = nn.LayerNorm(n_embd)\n    self.lm_head = nn.Linear(n_embd, tam_vocabulario)\n\n  def forward(self, idx, targets=None):\n    B, T = idx.shape\n\n    # idx e targets são ambos tensores de inteiros de formato (B, T)\n    tok_emb = self.token_embeeding_table(idx) # (B, T,C)\n    pos_emb = self.position_embeeding_table(torch.arange(T, device=device)) # (T, C)\n    x = tok_emb + pos_emb # (B, T,C)\n    x = self.blocks(x) # (B, T,C)\n    x = self.ln_f(x) # (B, T,C)\n    logits = self.lm_head(x) # (B, T, vocab_size)\n\n    if targets is None:\n      loss = None\n    else:\n      B,T,C = logits.shape\n      logits = logits.view(B*T, C)\n      targets = targets.view(B*T)\n      loss = F.cross_entropy(logits, targets)\n\n    return logits, loss\n\n  def generate(self, idx, max_new_tokens):\n    # idx é um array de índices (B, T)\n    # no contexto atual\n    for _ in range(max_new_tokens):\n      # corta idx para os últimos tokens de tamanho block_size\n      idx_cond = idx[:, -block_size:]\n      #captura predições\n      logits, loss = self(idx_cond)\n      #foca apenas no último passo de tempo\n      logits = logits[:, -1, : ] # (B, C)\n      #aplica softmax para capturar probabilidades\n      probabilidades = F.softmax(logits, dim=-1) # (B, C)\n      # amostra da distribuição\n      idx_next = torch.multinomial(probabilidades, num_samples=1) # (B, 1)\n      # anexa o índice amostrado à sequência em formação\n      idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n    return idx\n",[250,3226,3227,3232,3236,3241,3245,3250,3255,3260,3265,3270,3275,3280,3284,3288,3293,3297,3302,3307,3312,3317,3322,3327,3332,3336,3340,3344,3348,3353,3357,3361,3365,3369,3373,3377,3381,3385,3390,3394,3399,3404,3409,3414,3418,3423,3428,3432,3436,3441,3446,3451],{"__ignoreMap":27},[285,3228,3229],{"class":287,"line":34},[285,3230,3231],{},"#um super simples modelo bigram\n",[285,3233,3234],{"class":287,"line":28},[285,3235,1017],{},[285,3237,3238],{"class":287,"line":161},[285,3239,3240],{},"  def __init__(self):\n",[285,3242,3243],{"class":287,"line":188},[285,3244,1027],{},[285,3246,3247],{"class":287,"line":229},[285,3248,3249],{},"    # cada token lê diretamente os logits para o próximo token a partir de uma\n",[285,3251,3252],{"class":287,"line":322},[285,3253,3254],{},"    # tabela de consulta\n",[285,3256,3257],{"class":287,"line":429},[285,3258,3259],{},"    self.token_embeeding_table = nn.Embedding(tam_vocabulario, n_embd)\n",[285,3261,3262],{"class":287,"line":435},[285,3263,3264],{},"    self.position_embeeding_table =nn.Embedding(block_size, n_embd)\n",[285,3266,3267],{"class":287,"line":516},[285,3268,3269],{},"    self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])\n",[285,3271,3272],{"class":287,"line":522},[285,3273,3274],{},"    self.ln_f = nn.LayerNorm(n_embd)\n",[285,3276,3277],{"class":287,"line":528},[285,3278,3279],{},"    self.lm_head = nn.Linear(n_embd, tam_vocabulario)\n",[285,3281,3282],{"class":287,"line":725},[285,3283,309],{"emptyLinePlaceholder":33},[285,3285,3286],{"class":287,"line":731},[285,3287,1051],{},[285,3289,3290],{"class":287,"line":850},[285,3291,3292],{},"    B, T = idx.shape\n",[285,3294,3295],{"class":287,"line":856},[285,3296,309],{"emptyLinePlaceholder":33},[285,3298,3299],{"class":287,"line":862},[285,3300,3301],{},"    # idx e targets são ambos tensores de inteiros de formato (B, T)\n",[285,3303,3304],{"class":287,"line":868},[285,3305,3306],{},"    tok_emb = self.token_embeeding_table(idx) # (B, T,C)\n",[285,3308,3309],{"class":287,"line":874},[285,3310,3311],{},"    pos_emb = self.position_embeeding_table(torch.arange(T, device=device)) # (T, C)\n",[285,3313,3314],{"class":287,"line":880},[285,3315,3316],{},"    x = tok_emb + pos_emb # (B, T,C)\n",[285,3318,3319],{"class":287,"line":886},[285,3320,3321],{},"    x = self.blocks(x) # (B, T,C)\n",[285,3323,3324],{"class":287,"line":892},[285,3325,3326],{},"    x = self.ln_f(x) # (B, T,C)\n",[285,3328,3329],{"class":287,"line":897},[285,3330,3331],{},"    logits = self.lm_head(x) # (B, T, vocab_size)\n",[285,3333,3334],{"class":287,"line":903},[285,3335,309],{"emptyLinePlaceholder":33},[285,3337,3338],{"class":287,"line":909},[285,3339,1075],{},[285,3341,3342],{"class":287,"line":915},[285,3343,1080],{},[285,3345,3346],{"class":287,"line":921},[285,3347,1085],{},[285,3349,3350],{"class":287,"line":1117},[285,3351,3352],{},"      B,T,C = logits.shape\n",[285,3354,3355],{"class":287,"line":1122},[285,3356,1095],{},[285,3358,3359],{"class":287,"line":1128},[285,3360,1100],{},[285,3362,3363],{"class":287,"line":1134},[285,3364,1105],{},[285,3366,3367],{"class":287,"line":1140},[285,3368,309],{"emptyLinePlaceholder":33},[285,3370,3371],{"class":287,"line":1146},[285,3372,1114],{},[285,3374,3375],{"class":287,"line":1152},[285,3376,309],{"emptyLinePlaceholder":33},[285,3378,3379],{"class":287,"line":1158},[285,3380,1125],{},[285,3382,3383],{"class":287,"line":1164},[285,3384,1131],{},[285,3386,3387],{"class":287,"line":1170},[285,3388,3389],{},"    # no contexto atual\n",[285,3391,3392],{"class":287,"line":1176},[285,3393,1143],{},[285,3395,3396],{"class":287,"line":1182},[285,3397,3398],{},"      # corta idx para os últimos tokens de tamanho block_size\n",[285,3400,3401],{"class":287,"line":1188},[285,3402,3403],{},"      idx_cond = idx[:, -block_size:]\n",[285,3405,3406],{"class":287,"line":1194},[285,3407,3408],{},"      #captura predições\n",[285,3410,3411],{"class":287,"line":1200},[285,3412,3413],{},"      logits, loss = self(idx_cond)\n",[285,3415,3416],{"class":287,"line":1205},[285,3417,1161],{},[285,3419,3420],{"class":287,"line":1211},[285,3421,3422],{},"      logits = logits[:, -1, : ] # (B, C)\n",[285,3424,3425],{"class":287,"line":1217},[285,3426,3427],{},"      #aplica softmax para capturar probabilidades\n",[285,3429,3430],{"class":287,"line":1223},[285,3431,1179],{},[285,3433,3434],{"class":287,"line":1229},[285,3435,1185],{},[285,3437,3438],{"class":287,"line":1234},[285,3439,3440],{},"      idx_next = torch.multinomial(probabilidades, num_samples=1) # (B, 1)\n",[285,3442,3443],{"class":287,"line":1240},[285,3444,3445],{},"      # anexa o índice amostrado à sequência em formação\n",[285,3447,3448],{"class":287,"line":1245},[285,3449,3450],{},"      idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n",[285,3452,3453],{"class":287,"line":2590},[285,3454,1197],{},[12,3456,3457,3458,3461,3462,3465,3466,3469,3470,3473,3474,3477,3478,3481,3482,3484,3485,3488,3489,3491],{},"Despite the name (kept the original \"bigram\" comment even though it's not a bigram anymore), this is the full class: ",[250,3459,3460],{},"token_embeeding_table"," plus ",[250,3463,3464],{},"position_embeeding_table"," (summing the two lets every position know both \"who I am\" and \"where I am\"), ",[250,3467,3468],{},"blocks"," stacks ",[250,3471,3472],{},"n_layer"," Transformer blocks in sequence, ",[250,3475,3476],{},"ln_f"," normalizes at the end, ",[250,3479,3480],{},"lm_head"," projects back to vocabulary size. And this time ",[250,3483,1280],{}," ",[16,3486,3487],{},"does have"," the ",[250,3490,1302],{}," line, fixing the first bigram's bug, which is why only this model can actually generate real text.",[272,3493,3495],{"id":3494},"running-it-for-real-hyperparameters-training-generation","Running it for real: hyperparameters, training, generation",[277,3497,3499],{"className":279,"code":3498,"language":281,"meta":27,"style":27},"modelo = BigramLanguageModel()\nm = modelo.to(device)\n\n# imprimir o número de parâmetros no modelo\nprint(sum(p.numel() for p in m.parameters())\u002F1e6, 'M Número de parâmetros')\n\n#criando um otimizador PyTorch\noptimizer = torch.optim.AdamW(modelo.parameters(), lr=learning_rate)\n\nfor  iter in range(max_iters):\n  # de tempos em tempos, avalie a perda nos conjuntos de treino e validação\n  if iter % eval_interval == 0 or iter == max_iters - 1:\n    losses = perda_estimada()\n    print(f\"step {iter}: train loss {losses['treino']:.4f}, val loss {losses['avaliacao']:.4f}\")\n\n  # amostra de um lote de dados\n  xb, yb = get_batch('treino')\n\n  #avaliando o loss\n  logits, loss = modelo(xb, yb)\n  optimizer.zero_grad(set_to_none=True)\n  loss.backward()\n  optimizer.step()\n\n# gerar a partir do modelo\ncontext = torch.zeros((1,1), dtype=torch.long, device=device)\nprint(decode(modelo.generate(context, max_new_tokens=2000)[0].tolist()))\n",[250,3500,3501,3506,3511,3515,3520,3525,3529,3534,3539,3543,3548,3553,3558,3563,3568,3572,3576,3580,3584,3588,3592,3596,3600,3604,3608,3613,3618],{"__ignoreMap":27},[285,3502,3503],{"class":287,"line":34},[285,3504,3505],{},"modelo = BigramLanguageModel()\n",[285,3507,3508],{"class":287,"line":28},[285,3509,3510],{},"m = modelo.to(device)\n",[285,3512,3513],{"class":287,"line":161},[285,3514,309],{"emptyLinePlaceholder":33},[285,3516,3517],{"class":287,"line":188},[285,3518,3519],{},"# imprimir o número de parâmetros no modelo\n",[285,3521,3522],{"class":287,"line":229},[285,3523,3524],{},"print(sum(p.numel() for p in m.parameters())\u002F1e6, 'M Número de parâmetros')\n",[285,3526,3527],{"class":287,"line":322},[285,3528,309],{"emptyLinePlaceholder":33},[285,3530,3531],{"class":287,"line":429},[285,3532,3533],{},"#criando um otimizador PyTorch\n",[285,3535,3536],{"class":287,"line":435},[285,3537,3538],{},"optimizer = torch.optim.AdamW(modelo.parameters(), lr=learning_rate)\n",[285,3540,3541],{"class":287,"line":516},[285,3542,309],{"emptyLinePlaceholder":33},[285,3544,3545],{"class":287,"line":522},[285,3546,3547],{},"for  iter in range(max_iters):\n",[285,3549,3550],{"class":287,"line":528},[285,3551,3552],{},"  # de tempos em tempos, avalie a perda nos conjuntos de treino e validação\n",[285,3554,3555],{"class":287,"line":725},[285,3556,3557],{},"  if iter % eval_interval == 0 or iter == max_iters - 1:\n",[285,3559,3560],{"class":287,"line":731},[285,3561,3562],{},"    losses = perda_estimada()\n",[285,3564,3565],{"class":287,"line":850},[285,3566,3567],{},"    print(f\"step {iter}: train loss {losses['treino']:.4f}, val loss {losses['avaliacao']:.4f}\")\n",[285,3569,3570],{"class":287,"line":856},[285,3571,309],{"emptyLinePlaceholder":33},[285,3573,3574],{"class":287,"line":862},[285,3575,1376],{},[285,3577,3578],{"class":287,"line":868},[285,3579,1381],{},[285,3581,3582],{"class":287,"line":874},[285,3583,309],{"emptyLinePlaceholder":33},[285,3585,3586],{"class":287,"line":880},[285,3587,1390],{},[285,3589,3590],{"class":287,"line":886},[285,3591,1395],{},[285,3593,3594],{"class":287,"line":892},[285,3595,1400],{},[285,3597,3598],{"class":287,"line":897},[285,3599,1405],{},[285,3601,3602],{"class":287,"line":903},[285,3603,1410],{},[285,3605,3606],{"class":287,"line":909},[285,3607,309],{"emptyLinePlaceholder":33},[285,3609,3610],{"class":287,"line":915},[285,3611,3612],{},"# gerar a partir do modelo\n",[285,3614,3615],{"class":287,"line":921},[285,3616,3617],{},"context = torch.zeros((1,1), dtype=torch.long, device=device)\n",[285,3619,3620],{"class":287,"line":1117},[285,3621,3622],{},"print(decode(modelo.generate(context, max_new_tokens=2000)[0].tolist()))\n",[12,3624,3625,3626,3629,3630,3633,3634,156],{},"This is the cell that actually trains: ",[250,3627,3628],{},"0.215792 M"," parameters (215,792, to be exact), 5,000 iterations, evaluating train and validation loss every 100 steps. First output line: ",[250,3631,3632],{},"step 0: train loss 4.8673, val loss 4.8625",", near the expected random guess. Last: ",[250,3635,3636],{},"step 4999: train loss 1.6304, val loss 2.6720",[272,3638,3640],{"id":3639},"the-two-training-runs-side-by-side","The two training runs, side by side",[12,3642,3643,3644,1914,3647,1914,3650,1914,3653,1914,3656,1914,3659,3662],{},"Same hyperparameters in both notebooks (",[250,3645,3646],{},"batch_size=16",[250,3648,3649],{},"block_size=32",[250,3651,3652],{},"n_embd=64",[250,3654,3655],{},"n_head=4",[250,3657,3658],{},"n_layer=4",[250,3660,3661],{},"max_iters=5000","), the only thing that changes is the text:",[3664,3665,3666,3682],"table",{},[3667,3668,3669],"thead",{},[3670,3671,3672,3676,3679],"tr",{},[3673,3674],"th",{"align":3675},"left",[3673,3677,3678],{"align":3675},"Shakespeare (original)",[3673,3680,3681],{"align":3675},"Dom Casmurro (mine)",[3683,3684,3685,3697,3707,3718,3729],"tbody",{},[3670,3686,3687,3691,3694],{},[3688,3689,3690],"td",{"align":3675},"Characters in dataset",[3688,3692,3693],{"align":3675},"1,115,394",[3688,3695,3696],{"align":3675},"400,944",[3670,3698,3699,3702,3704],{},[3688,3700,3701],{"align":3675},"Vocabulary size",[3688,3703,389],{"align":3675},[3688,3705,3706],{"align":3675},"112",[3670,3708,3709,3712,3715],{},[3688,3710,3711],{"align":3675},"Model parameters",[3688,3713,3714],{"align":3675},"0.2097 M",[3688,3716,3717],{"align":3675},"0.2158 M",[3670,3719,3720,3723,3726],{},[3688,3721,3722],{"align":3675},"Training loss (step 4999)",[3688,3724,3725],{"align":3675},"1.6645",[3688,3727,3728],{"align":3675},"1.6304",[3670,3730,3731,3734,3737],{},[3688,3732,3733],{"align":3675},"Validation loss (step 4999)",[3688,3735,3736],{"align":3675},"1.8286",[3688,3738,3739],{"align":3675},"2.6720",[12,3741,3742],{},"Training loss ended up similar for both, but Dom Casmurro's validation loss came out noticeably worse. Two real reasons, no invention: first, I trained on less than a third of the text volume (400 thousand characters against 1.1 million), and the validation set shrank proportionally too (about 40 thousand characters against about 111 thousand), so the model had fewer examples to generalize from. Second, the bigger vocabulary (112 against 65) means more possible classes to get right at every position, and Portuguese morphology (rich verb conjugation, a lot more suffix variation than Shakespeare's early-modern English) is simply harder for a model this size to absorb. Not a code bug, the identical architecture just hit a harder dataset with less ammunition.",[272,3744,3746],{"id":3745},"what-comes-out-of-generation","What comes out of generation",[12,3748,3749],{},"After 5,000 steps, generating from an empty context, the Dom Casmurro model spits this out (no punctuation or full-sentence sense, but look closely):",[3751,3752,3753],"blockquote",{},[12,3754,3755],{},[66,3756,3757],{},"...abnoha (força-lhe o sappagorla, affendado escontraga-se, não vei nelles, prima Justina, como ella devi enferiu o não dizia uma paertada continueira... de Escobar cegos de enho qué está acconterno gostaria...",[12,3759,3760],{},"\"Justina\" and \"Escobar\" are real character names from the novel. The model doesn't know who Escobar is, doesn't understand the plot, it's 215 thousand parameters of pure character-level statistical pattern. But it learned that those letter sequences show up together often enough to reproduce whole proper names out of the noise. The same thing happens in the English version, which learned to put \"KING RICHARD III:\" and \"QUEEN VINCENTIO:\" in the right all-caps play-script format, without understanding a word of what the character says next.",[272,3762,3764],{"id":3763},"wrapping-up","Wrapping up",[3664,3766,3767,3777],{},[3667,3768,3769],{},[3670,3770,3771,3774],{},[3673,3772,3773],{"align":3675},"What I already knew",[3673,3775,3776],{"align":3675},"What this chapter settled",[3683,3778,3779,3787,3795],{},[3670,3780,3781,3784],{},[3688,3782,3783],{"align":3675},"Self-attention is \"some way of comparing tokens\"",[3688,3785,3786],{"align":3675},"It's literally a learned weighted average, and the \"magic\" is just a triangular matrix plus softmax",[3670,3788,3789,3792],{},[3688,3790,3791],{"align":3675},"Notebooks and scripts are the same thing",[3688,3793,3794],{"align":3675},"Notebooks keep state across out-of-order cells, scripts don't, and that can hide an entire import bug",[3670,3796,3797,3800],{},[3688,3798,3799],{"align":3675},"\"Mathematically equivalent\" means \"numerically identical\"",[3688,3801,3802,3803,3805,3806,3808],{"align":3675},"It doesn't: ",[250,3804,1698],{}," returned ",[250,3807,1708],{}," on two computations that should've matched, because of floating-point operation order",[272,3810,3812],{"id":3811},"practical-application","Practical application",[12,3814,3815],{},"The numbers in the table above come from the two real training runs, step by step, logged during each notebook's 5,000-iteration run (every 100 steps). Zoom the chart and watch the gap between train and validation: on Shakespeare the two lines stay glued together right to the end, on Dom Casmurro they peel apart early and never close back up, the exact visual signature of what the table above already said in numbers.",[3817,3818],"slm-training-loss-compare",{"dom-casmurro-train-label":3819,"dom-casmurro-val-label":3820,"shakespeare-train-label":3821,"shakespeare-val-label":3822,"x-label":3823,"y-label":1266},"Dom Casmurro · train","Dom Casmurro · val","Shakespeare · train","Shakespeare · val","training step",[12,3825,3826],{},"Same architecture, same hyperparameters, two different languages, and the chart alone already shows which one gave the tiny 215-thousand-parameter model more trouble to learn.",[3828,3829,3830],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":27,"searchDepth":28,"depth":28,"links":3832},[3833,3834,3835,3836,3837,3838,3839,3840,3841,3842,3843,3844,3846,3847,3848,3849,3850],{"id":274,"depth":28,"text":275},{"id":346,"depth":28,"text":347},{"id":555,"depth":28,"text":556},{"id":663,"depth":28,"text":664},{"id":778,"depth":28,"text":779},{"id":984,"depth":28,"text":985},{"id":1340,"depth":28,"text":1341},{"id":1457,"depth":28,"text":1458},{"id":1782,"depth":28,"text":1783},{"id":1980,"depth":28,"text":1981},{"id":2127,"depth":28,"text":2128},{"id":2346,"depth":28,"text":3845},"Bug #3: the Fold gotcha, and the full model",{"id":3494,"depth":28,"text":3495},{"id":3639,"depth":28,"text":3640},{"id":3745,"depth":28,"text":3746},{"id":3763,"depth":28,"text":3764},{"id":3811,"depth":28,"text":3812},"2026-08-23","Chapter 2 builds a tiny GPT from scratch, the same classic exercise from Karpathy's video. I clean-transcribe my entire notebook here, cell by cell, full code, comparing against the original notebook and finding real bugs along the way.",{},"how-to-build-and-finetune-slm",{"title":237,"description":3852},"en\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Fbuilding-gpt-from-scratch-dom-casmurro",[3858,3859,3860,3861],"transformers","self-attention","gpt","pytorch","wBmuonS_bJkv2APRlPeWWJmtBPz0O_aZS1OAJmPsZGc",{"id":3864,"title":3865,"body":3866,"cover":3,"date":3851,"description":4009,"extension":31,"meta":4010,"navigation":33,"order":34,"path":214,"playlist":3854,"seo":4011,"status":37,"stem":4012,"tags":4013,"__hash__":4018},"posts\u002Fen\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Fsmall-vs-giant-language-models.md","SLM vs LLM: the Delivery Bike and the Moving Truck",{"type":9,"value":3867,"toc":4003},[3868,3877,3890,3894,3897,3903,3909,3915,3921,3924,3931,3935,3946,3952,3954,3990,3992,3995,4000],[12,3869,3870,3871,3876],{},"Picture this: you need to deliver a pizza a few blocks away. Do you call a moving truck for that? Of course not, you call a delivery bike. The truck can haul a fridge, a couch, an entire household, but it's expensive, slow to turn corners, and you pay the full fare even if your \"move\" is one pizza box. That's basically the argument in Chapter 1 of ",[53,3872,3874],{"href":203,"rel":3873},[77],[66,3875,207],{},", just swap \"move\" for \"language model\".",[12,3878,3879,3880,3884,3885,3889],{},"An ",[656,3881,3883],{"definition":3882},"Large Language Model, a giant language model like GPT-4\u002F5, Claude or Gemini, with hundreds of billions of parameters trained to know a bit of everything","LLM"," is the moving truck. It writes poetry, solves advanced math, explains quantum physics, knows a bit of everything. But it also costs a lot to run at scale, sends your data to someone else's servers, and you can't pop the hood and swap a part when it keeps failing at the same kind of question. A ",[656,3886,3888],{"definition":3887},"Small Language Model, a small model, typically between 125 million and 7 billion parameters, specialized in one task instead of being a generalist","SLM"," is the delivery bike. It can't do everything, but for a fast, specialized, cheap delivery, it beats the truck every time.",[272,3891,3893],{"id":3892},"why-small-beats-giant-sometimes","Why \"small\" beats \"giant\" sometimes",[12,3895,3896],{},"The book lists five practical reasons, and I'd rather retell each one with a real example than stay theoretical:",[12,3898,3899,3902],{},[16,3900,3901],{},"Cost."," Running a giant-LLM API at high volume costs anywhere from hundreds to thousands of dollars a month. Training or fine-tuning the giant model from scratch? Millions. The book gives a concrete example I liked: a support center processing 10,000 tickets a day would spend 500 to 1,000 dollars a month on an API. A local model running on your own machine costs about 50 dollars a month, and that's just the electricity bill.",[12,3904,3905,3908],{},[16,3906,3907],{},"Privacy."," Patient records, legal case files, unpublished research data, none of that should just travel to a third party's server. If a hospital sends patient notes to an outside API, that's a HIPAA violation right there (the US healthcare privacy law). A local model solves that instantly: the data never leaves the building.",[12,3910,3911,3914],{},[16,3912,3913],{},"Control."," If the giant LLM keeps getting your field's terminology wrong, you have no way to fix it, you just complain and wait for the next version. Your own model, you adjust.",[12,3916,3917,3920],{},[16,3918,3919],{},"Availability and speed."," APIs go down, change pricing, or slow to a crawl during peak hours. A local model doesn't depend on the internet or anyone else's business decisions but your own.",[12,3922,3923],{},"The book gives two \"when it's worth it\" examples I liked a lot: a geologist mining mineral data out of thousands of research papers will do better with a model trained on geology vocabulary (\"plagioclase feldspar\", \"phenocryst\") than with a generalist. And a farm using a small vision model to spot plant disease on-site, with no internet dependency, wins on both speed and not paying an API fee per photo.",[12,3925,3926,3927,3930],{},"But the rule isn't \"SLM always wins\". If you need broad general knowledge, complex multi-step reasoning, or real-time up-to-date information, the moving truck still wins. The book sums it up in a line I fully agree with: ",[16,3928,3929],{},"giant LLMs are generalists, SLMs are specialists."," If you can clearly define your task and have relevant data to train on, the small model tends to serve you better. If you can't, the generalist giant covers the gap.",[272,3932,3934],{"id":3933},"what-the-book-will-have-you-build","What the book will have you build",[12,3936,3937,3938,3941,3942,3945],{},"Most of Chapter 1 is really the roadmap for the whole book, and I'll summarize it quickly because it helps place where the two posts I'm writing fit in. The book uses two reference architectures: ",[16,3939,3940],{},"GPT-2"," (2019, from OpenAI, 125 million to 1.5 billion parameters, the classic starting point) and ",[16,3943,3944],{},"MiniMind"," (more recent, 26 to 218 million parameters, optimized to run on smaller hardware with a three-stage training pipeline: pretraining, supervised fine-tuning, and direct preference optimization). It's 12 chapters total, organized into four phases: foundations (chapters 1-3, where you understand the architecture and do your first 30-minute fine-tune), deep-dive training from scratch (chapters 4-7), the complete MiniMind case study (chapters 8-10), and production (chapters 11-12, real deployment).",[12,3947,3948,3949,3951],{},"I'm only covering ",[53,3950,220],{"href":219}," for now, which is \"Phase 1\" in the book's own words: build a tiny GPT from scratch to understand the engine before fine-tuning anything pre-built.",[272,3953,3764],{"id":3763},[3664,3955,3956,3964],{},[3667,3957,3958],{},[3670,3959,3960,3962],{},[3673,3961,3773],{"align":3675},[3673,3963,3776],{"align":3675},[3683,3965,3966,3974,3982],{},[3670,3967,3968,3971],{},[3688,3969,3970],{"align":3675},"Bigger models are more powerful",[3688,3972,3973],{"align":3675},"More powerful isn't the same as \"better for your specific task\"",[3670,3975,3976,3979],{},[3688,3977,3978],{"align":3675},"Training a model is expensive",[3688,3980,3981],{"align":3675},"Only if it's the wrong model for the size of the problem: 50 dollars a month local vs 500 to 1,000 dollars a month API is a math anyone can do",[3670,3983,3984,3987],{},[3688,3985,3986],{"align":3675},"\"Fine-tuning\" is vague jargon",[3688,3988,3989],{"align":3675},"It's literally taking a model that already knows \"language in general\" and adjusting it just enough to know your specific task",[272,3991,3812],{"id":3811},[12,3993,3994],{},"The chapter's most concrete number is that support-center example, 10,000 tickets a day. I didn't invent anything here, these are the two values the book itself cites: a giant-LLM API sits between 500 and 1,000 dollars a month, a local model sits at 50 dollars a month (the electricity bill, since you already own or rent the hardware once). I put them side by side so you can see the scale of the difference at a glance, using the low end of the API range to keep it fair to the smaller number.",[3996,3997],"slm-cost-bar-chart",{":bars":3998,"y-label":3999},"[{\"label\":\"API (giant LLM)\",\"cost\":500,\"color\":\"#cc3300\"},{\"label\":\"Local model (SLM)\",\"cost\":50,\"color\":\"#0033cc\"}]","US$\u002Fmonth",[12,4001,4002],{},"Ten times cheaper, every month, for the same ticket volume. That's the kind of math that separates \"it'd be nice to have my own model\" from \"it makes financial sense to have my own model\". Next post, we build the engine.",{"title":27,"searchDepth":28,"depth":28,"links":4004},[4005,4006,4007,4008],{"id":3892,"depth":28,"text":3893},{"id":3933,"depth":28,"text":3934},{"id":3763,"depth":28,"text":3764},{"id":3811,"depth":28,"text":3812},"Chapter 1 of J. Paul Liu's book argues the giant model isn't always the right call. I tell it through the most Brazilian metaphor I know (a delivery bike vs a moving truck) and check the real cost numbers the book uses.",{},{"title":3865,"description":4009},"en\u002Fplaylists\u002Fhow-to-build-and-finetune-slm\u002Fsmall-vs-giant-language-models",[4014,4015,4016,4017],"slm","llm","cost","hardware","w629qGi_LbJOFTHqaUX4z2WSPvv4Q6GSV-hJ4XvaC6g",{"id":4020,"title":4021,"body":4022,"cover":3,"date":4673,"description":4674,"extension":31,"meta":4675,"navigation":33,"order":188,"path":4676,"playlist":4677,"seo":4678,"status":37,"stem":4679,"tags":4680,"__hash__":4684},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta.md","Adaline: Train on the Line, Classify on the Sign",{"type":9,"value":4023,"toc":4665},[4024,4037,4041,4089,4120,4135,4142,4197,4222,4236,4265,4269,4282,4294,4297,4301,4348,4367,4383,4387,4422,4442,4456,4463,4500,4512,4514,4558,4560,4571,4586,4609,4652,4663],[12,4025,4026,4027,4030,4031,760,4034,156],{},"Lectures 2e and 2f, and the model's name changes to ",[16,4028,4029],{},"Adaline"," (ADAptive LInear NEuron), by Bernard Widrow and Ted Hoff, 1960, just two years after Rosenblatt. Their core idea is subtle, but it splits apart two things I'd been treating as one so far: ",[16,4032,4033],{},"what the model optimizes during training",[16,4035,4036],{},"what it computes at prediction time",[272,4038,4040],{"id":4039},"pre-activation-and-post-activation-the-distinction-adaline-introduces","Pre-activation and post-activation: the distinction Adaline introduces",[277,4042,4044],{"className":279,"code":4043,"language":281,"meta":27,"style":27},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n",[250,4045,4046,4051,4056,4061,4066,4071,4075,4080,4084],{"__ignoreMap":27},[285,4047,4048],{"class":287,"line":34},[285,4049,4050],{},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n",[285,4052,4053],{"class":287,"line":28},[285,4054,4055],{},"  def fit(self, X, y):\n",[285,4057,4058],{"class":287,"line":161},[285,4059,4060],{},"    X = include_bias(X)\n",[285,4062,4063],{"class":287,"line":188},[285,4064,4065],{},"    self.w_ = np.linalg.pinv(X) @ y\n",[285,4067,4068],{"class":287,"line":229},[285,4069,4070],{},"    return self\n",[285,4072,4073],{"class":287,"line":322},[285,4074,309],{"emptyLinePlaceholder":33},[285,4076,4077],{"class":287,"line":429},[285,4078,4079],{},"  def predict(self, X):\n",[285,4081,4082],{"class":287,"line":435},[285,4083,4060],{},[285,4085,4086],{"class":287,"line":516},[285,4087,4088],{},"    return X @ self.w_\n",[12,4090,4091,4092,3484,4095,4099,4100,4102,4103,4106,4107,4110,4111,4113,4114,4116,4117,156],{},"Notice: this is ",[16,4093,4094],{},"exactly",[53,4096,4098],{"href":4097},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal","last post's normal equation",", without changing a line, except now ",[250,4101,748],{}," is ",[250,4104,4105],{},"-1"," or ",[250,4108,4109],{},"+1"," instead of continuous. The professor is treating classification as if it were regression: fitting the line to land as close as possible to ",[250,4112,4105],{}," on one class's points and ",[250,4115,4109],{}," on the other's, never applying ",[250,4118,4119],{},"sign()",[3751,4121,4122],{},[12,4123,4124,4127,4128,4131,4132,156],{},[16,4125,4126],{},"Output:"," RMSE ",[250,4129,4130],{},"0.4148",", weights ",[250,4133,4134],{},"[-1.76, 1.01, 2.74]",[12,4136,4137,4138,4141],{},"RMSE makes sense here because ",[250,4139,4140],{},"predict"," returns a continuous number, not a class. But to actually classify, there's one last step missing:",[277,4143,4145],{"className":279,"code":4144,"language":281,"meta":27,"style":27},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def pre_activation(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n\n  def predict(self, X):\n    return np.sign(self.pre_activation(X))\n",[250,4146,4147,4151,4155,4159,4163,4167,4171,4176,4180,4184,4188,4192],{"__ignoreMap":27},[285,4148,4149],{"class":287,"line":34},[285,4150,4050],{},[285,4152,4153],{"class":287,"line":28},[285,4154,4055],{},[285,4156,4157],{"class":287,"line":161},[285,4158,4060],{},[285,4160,4161],{"class":287,"line":188},[285,4162,4065],{},[285,4164,4165],{"class":287,"line":229},[285,4166,4070],{},[285,4168,4169],{"class":287,"line":322},[285,4170,309],{"emptyLinePlaceholder":33},[285,4172,4173],{"class":287,"line":429},[285,4174,4175],{},"  def pre_activation(self, X):\n",[285,4177,4178],{"class":287,"line":435},[285,4179,4060],{},[285,4181,4182],{"class":287,"line":516},[285,4183,4088],{},[285,4185,4186],{"class":287,"line":522},[285,4187,309],{"emptyLinePlaceholder":33},[285,4189,4190],{"class":287,"line":528},[285,4191,4079],{},[285,4193,4194],{"class":287,"line":725},[285,4195,4196],{},"    return np.sign(self.pre_activation(X))\n",[12,4198,4199,4200,4203,4204,4206,4207,4209,4210,4213,4214,4217,4218,4221],{},"Now there are two methods: ",[250,4201,4202],{},"pre_activation"," (the continuous value, before any threshold) and ",[250,4205,4140],{}," (applies ",[250,4208,4119],{}," on top). Aggarwal calls exactly these two things the ",[16,4211,4212],{},"pre-activation value"," and the ",[16,4215,4216],{},"post-activation value"," (chapter 1): everything a neuron computes happens in two steps, first the weighted sum, then the activation function on top of it. Adaline trains on the pre-activation (it's continuous, so you can measure \"how far off\" each prediction landed from the target) and only applies the activation (",[250,4219,4220],{},"sign",") when deciding the final class.",[3751,4223,4224],{},[12,4225,4226,4228,4229,4232,4233,4235],{},[16,4227,4126],{}," accuracy ",[250,4230,4231],{},"1.0",", the same ",[250,4234,4134],{}," weights as before (it's the same computation, just evaluated by accuracy instead of RMSE this time).",[12,4237,4238,4239,121,4242,4245,4246,4249,4250,4253,4254,4256,4257,4260,4261,4264],{},"This is the ",[16,4240,4241],{},"delta rule",[53,4243,4244],{"href":119},"I've already seen it before, under that exact name",": \"update the weight proportionally to the error times the input\" is Widrow-Hoff's signature. The difference from Rosenblatt's perceptron (which I covered ",[53,4247,4248],{"href":154},"two posts ago",") is exactly this: the perceptron measures error ",[16,4251,4252],{},"after"," applying ",[250,4255,4119],{}," (error in ",[250,4258,4259],{},"{-2,0,+2}","), Adaline measures it ",[16,4262,4263],{},"before",", on the continuous pre-activation. That sounds like a small detail, but it changes everything: a continuous error gives a real, smooth gradient that points toward the better direction even when a prediction is already on the right side but still a bit \"unsure\". The perceptron's binary error, by contrast, only fires on an outright misclassification, with no notion of \"how wrong.\"",[272,4266,4268],{"id":4267},"interactive-nudging-the-pre-activation-and-watching-rmse-and-accuracy-move","Interactive: nudging the pre-activation and watching RMSE (and accuracy) move",[12,4270,4271,4272,1914,4275,121,4278,4281],{},"Instead of training automatically, drag the ",[250,4273,4274],{},"w0",[250,4276,4277],{},"w1",[250,4279,4280],{},"bias"," sliders by hand and watch two readouts at once: RMSE (continuous, changes smoothly with every drag) and accuracy (discrete, only jumps when a point crosses the decision boundary).",[4283,4284],"adaline-explorer",{":classes":4285,":points":4286,":x-max":4287,":x-min":4288,":y-max":4287,":y-min":4288,"negative-label":4289,"positive-label":4290,"readout-label":4291,"x-label":4292,"y-label":4293},"[-1, 1, -1, -1, -1, -1, 1, 1, -1, -1, -1, -1, 1, 1, -1, -1, 1, -1, 1, -1]","[[0.0856, 0.2368], [0.8013, 0.5822], [0.0941, 0.4331], [0.4791, 0.1597], [0.7346, 0.1137], [0.3912, 0.5167], [0.4306, 0.5868], [0.7378, 0.9563], [0.2842, 0.6485], [0.6962, 0.2927], [0.0015, 0.9735], [0.2984, 0.314], [0.8917, 0.5852], [0.4713, 0.7733], [0.0303, 0.707], [0.3742, 0.0909], [0.6605, 0.9315], [0.2072, 0.6301], [0.2982, 0.7418], [0.7222, 0.2187]]","1","0","class -1","class +1","RMSE (continuous pre-activation vs. ±1 label): {rmse} · accuracy (post-activation, sign): {acc}%","x0","x1",[12,4295,4296],{},"Notice how RMSE almost always keeps changing a little even after accuracy already hit 100%: you can push the boundary further into the empty gap between the two classes (RMSE drops more) without gaining or losing a single point (accuracy stays put). That's precisely the difference between \"finding some line that separates\" (what the perceptron does) and \"finding the line that separates with room to spare\" (what you get by optimizing RMSE instead of just counting mistakes).",[272,4298,4300],{"id":4299},"lecture-2f-the-same-computation-just-iterating-and-the-notebook-calls-it-sgd","Lecture 2f: the same computation, just iterating (and the notebook calls it \"SGD\")",[277,4302,4304],{"className":279,"code":4303,"language":281,"meta":27,"style":27},"class Adaline(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.zeros(X.shape[1])\n    for _ in range(self.max_iter):\n      y_pred = X @ self.w_\n      error = y - y_pred\n      self.w_ += self.learning_rate * error @ X\n    return self\n",[250,4305,4306,4311,4315,4319,4324,4329,4334,4339,4344],{"__ignoreMap":27},[285,4307,4308],{"class":287,"line":34},[285,4309,4310],{},"class Adaline(BaseEstimator, ClassifierMixin):\n",[285,4312,4313],{"class":287,"line":28},[285,4314,4055],{},[285,4316,4317],{"class":287,"line":161},[285,4318,4060],{},[285,4320,4321],{"class":287,"line":188},[285,4322,4323],{},"    self.w_ = np.zeros(X.shape[1])\n",[285,4325,4326],{"class":287,"line":229},[285,4327,4328],{},"    for _ in range(self.max_iter):\n",[285,4330,4331],{"class":287,"line":322},[285,4332,4333],{},"      y_pred = X @ self.w_\n",[285,4335,4336],{"class":287,"line":429},[285,4337,4338],{},"      error = y - y_pred\n",[285,4340,4341],{"class":287,"line":435},[285,4342,4343],{},"      self.w_ += self.learning_rate * error @ X\n",[285,4345,4346],{"class":287,"line":516},[285,4347,4070],{},[12,4349,4350,4351,4354,4355,248,4358,4366],{},"One honest note about the notebook's name (",[250,4352,4353],{},"aula02f adaline with SGD","): the code shown here is ",[16,4356,4357],{},"batch gradient descent",[53,4359,4361,4362,4365],{"href":4360},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias","the same ",[250,4363,4364],{},"X.T @ error"," as always","), computing the error over the entire dataset every iteration, not real SGD (which would update on one example at a time, in shuffled order). It's a common informal way of talking (\"it's kind of like gradient descent, so I call it SGD\"), but the technical difference is worth noting, since the names carry precise meaning.",[3751,4368,4369],{},[12,4370,4371,4228,4373,4375,4376,4379,4380,156],{},[16,4372,4126],{},[250,4374,4231],{}," on training, weights ",[250,4377,4378],{},"[-2.92, 3.32, 2.63]",". Tested on 1000 new points: accuracy ",[16,4381,4382],{},"0.953",[272,4384,4386],{"id":4385},"the-bad-dataset-what-it-actually-proves","The \"bad\" dataset: what it actually proves",[277,4388,4390],{"className":279,"code":4389,"language":281,"meta":27,"style":27},"X_bad = np.concatenate((X,np.ones_like(X)))\ny_bad = np.concatenate((y,np.ones_like(y)))\nX_bad = np.concatenate((X_bad,np.ones_like(X)))\ny_bad = np.concatenate((y_bad,np.ones_like(y)))\nclf_bad = Adaline()\nclf_bad.fit(X_bad, y_bad)\n",[250,4391,4392,4397,4402,4407,4412,4417],{"__ignoreMap":27},[285,4393,4394],{"class":287,"line":34},[285,4395,4396],{},"X_bad = np.concatenate((X,np.ones_like(X)))\n",[285,4398,4399],{"class":287,"line":28},[285,4400,4401],{},"y_bad = np.concatenate((y,np.ones_like(y)))\n",[285,4403,4404],{"class":287,"line":161},[285,4405,4406],{},"X_bad = np.concatenate((X_bad,np.ones_like(X)))\n",[285,4408,4409],{"class":287,"line":188},[285,4410,4411],{},"y_bad = np.concatenate((y_bad,np.ones_like(y)))\n",[285,4413,4414],{"class":287,"line":229},[285,4415,4416],{},"clf_bad = Adaline()\n",[285,4418,4419],{"class":287,"line":322},[285,4420,4421],{},"clf_bad.fit(X_bad, y_bad)\n",[12,4423,4424,4425,4428,4429,4432,4433,4435,4436,4438,4439,4441],{},"The professor concatenates the original dataset with ",[16,4426,4427],{},"two extra blocks"," of artificial points: everyone at ",[250,4430,4431],{},"(1,1)",", everyone labeled ",[250,4434,4109],{},". This isn't noise, it's a deliberate bias, nudging training to \"believe\" the region near ",[250,4437,4431],{}," is even more strongly class ",[250,4440,4109],{}," than it really is.",[3751,4443,4444],{},[12,4445,4446,4228,4448,4451,4452,4455],{},[16,4447,4126],{},[250,4449,4450],{},"0.967"," on training (over the biased dataset), but only ",[16,4453,4454],{},"0.811"," on the same 1000 clean test points as before.",[12,4457,4458,4459,4462],{},"Dropped from 0.953 to 0.811. Before writing this post, my working hypothesis was that this cell would show the normal equation (pseudo-inverse) breaking on this problematic dataset, with SGD holding up better. I reproduced the experiment myself, comparing the pseudo-inverse against batch gradient descent ",[16,4460,4461],{},"on the same biased dataset",", and that hypothesis didn't hold up:",[3664,4464,4465,4479],{},[3667,4466,4467],{},[3670,4468,4469,4472,4476],{},[3673,4470,4471],{"align":3675},"Method",[3673,4473,4475],{"align":4474},"right","Train accuracy (biased)",[3673,4477,4478],{"align":4474},"Test accuracy (clean)",[3683,4480,4481,4491],{},[3670,4482,4483,4486,4488],{},[3688,4484,4485],{"align":3675},"Pseudo-inverse",[3688,4487,4450],{"align":4474},[3688,4489,4490],{"align":4474},"0.783",[3670,4492,4493,4496,4498],{},[3688,4494,4495],{"align":3675},"Batch gradient descent",[3688,4497,4450],{"align":4474},[3688,4499,4490],{"align":4474},[12,4501,4502,4503,4506,4507,4511],{},"Both drop by the exact same amount, with essentially identical weights between them. The real lesson of this cell isn't about which training algorithm is more robust, it's about ",[16,4504,4505],{},"training data quality",": biasing the training distribution (even without adding \"noise\" in the random-error sense) shifts the learned boundary somewhere that no longer represents the real distribution, and it hurts both methods equally, because both are solving the exact same optimization problem under the hood. ",[53,4508,4510],{"href":4509},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","This echoes a lesson I already saw in the other playlist",": training on a distribution that doesn't match the real world is a data problem, not an algorithm problem.",[272,4513,3764],{"id":3763},[3664,4515,4516,4525],{},[3667,4517,4518],{},[3670,4519,4520,4522],{},[3673,4521,3773],{"align":3675},[3673,4523,4524],{"align":3675},"What this lecture settled",[3683,4526,4527,4538,4550],{},[3670,4528,4529,4532],{},[3688,4530,4531],{"align":3675},"The perceptron classifies and updates on the binary error",[3688,4533,4534,4535,4537],{"align":3675},"Adaline separates pre-activation (continuous, used in training) from post-activation (",[250,4536,4220],{},", used only to decide the final class)",[3670,4539,4540,4543],{},[3688,4541,4542],{"align":3675},"The delta rule already showed up in the other playlist",[3688,4544,4545,4546,4549],{"align":3675},"Adaline ",[16,4547,4548],{},"is"," the delta rule applied to classification, training as if it were regression on the ±1 labels",[3670,4551,4552,4555],{},[3688,4553,4554],{"align":3675},"Finding a line that separates seems like enough",[3688,4556,4557],{"align":3675},"Optimizing RMSE (not just counting mistakes) keeps finding a better boundary even after accuracy already hit 100%",[272,4559,3812],{"id":3811},[12,4561,4562,4563,4566,4567,4570],{},"I reproduced pseudo-inverse vs. batch gradient descent on Iris (",[66,4564,4565],{},"setosa"," vs. ",[66,4568,4569],{},"versicolor","), by now a familiar face in this playlist, but this time training as Adaline, on the continuous ±1 target, not as a perceptron.",[277,4572,4574],{"className":279,"code":4573,"language":281,"meta":27,"style":27},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\nw_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[250,4575,4576,4581],{"__ignoreMap":27},[285,4577,4578],{"class":287,"line":34},[285,4579,4580],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[285,4582,4583],{"class":287,"line":28},[285,4584,4585],{},"w_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[12,4587,4588,4589,4592,4593,4596,4597,4600,4601,4604,4605,4608],{},"On the first attempt, I used the notebook's original ",[250,4590,4591],{},"learning_rate=0.01"," for batch gradient descent, and it ",[16,4594,4595],{},"diverged"," (weight became ",[250,4598,4599],{},"NaN",") straight away on raw Iris data. No surprise at this point: ",[53,4602,4603],{"href":4097},"it's the same scale lesson from the normal equation post",", just rediscovered again, this time needing a much smaller rate (",[250,4606,4607],{},"0.001",") to not blow up.",[3664,4610,4611,4623],{},[3667,4612,4613],{},[3670,4614,4615,4617,4620],{},[3673,4616,4471],{"align":3675},[3673,4618,4619],{"align":4474},"RMSE (train)",[3673,4621,4622],{"align":4474},"Accuracy (test)",[3683,4624,4625,4637],{},[3670,4626,4627,4630,4633],{},[3688,4628,4629],{"align":3675},"Pseudo-inverse (Adaline)",[3688,4631,4632],{"align":4474},"0.2414",[3688,4634,4635],{"align":4474},[16,4636,4231],{},[3670,4638,4639,4646,4648],{},[3688,4640,4641,4642,4645],{"align":3675},"Batch gradient descent, ",[250,4643,4644],{},"learning_rate=0.001"," (Adaline)",[3688,4647,4632],{"align":4474},[3688,4649,4650],{"align":4474},[16,4651,4231],{},[12,4653,4654,4655,4658,4659,4662],{},"With the rate adjusted, same result from both methods again, RMSE identical to the fourth decimal, and both accuracies hitting 100%, ",[53,4656,4657],{"href":4360},"the same generous margin from Iris that already favored the perceptron with bias",". The real difference between the methods, on this easy dataset, remains just convergence speed (and sensitivity to the choice of ",[250,4660,4661],{},"learning_rate","), not the quality of the final solution.",[3828,4664,3830],{},{"title":27,"searchDepth":28,"depth":28,"links":4666},[4667,4668,4669,4670,4671,4672],{"id":4039,"depth":28,"text":4040},{"id":4267,"depth":28,"text":4268},{"id":4299,"depth":28,"text":4300},{"id":4385,"depth":28,"text":4386},{"id":3763,"depth":28,"text":3764},{"id":3811,"depth":28,"text":3812},"2026-08-20","Lectures 2e and 2f: the professor trains Adaline by fitting a continuous line to the ±1 labels (instead of fitting directly on the sign), and I connect this to the delta rule that already showed up in the other playlist. In the end, a deliberately biased dataset reveals the problem was never the training algorithm.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta","neural-networks",{"title":4021,"description":4674},"en\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta",[4681,4682,4683],"adaline","delta-rule","widrow-hoff","tDtwjGKmRJJOKzWjkAyQ72-9DNH1O8sc9YdTGBFjxlk",[4686,4724],{"id":4687,"title":4688,"body":4689,"date":4709,"description":4710,"extension":31,"meta":4711,"navigation":33,"path":4712,"photos":4713,"seo":4718,"status":37,"stem":4719,"tags":4720,"__hash__":4723},"timeline\u002Fen\u002Ftimeline\u002F2026-08-19-estudando-go.md","Getting shaken up by Go",{"type":9,"value":4690,"toc":4707},[4691,4698,4701,4704],[12,4692,4693,4694,4697],{},"I've been studying a lot of Go these past few weeks, and I mean that literally: I brought ",[66,4695,4696],{},"The Go Programming Language"," (the classic Donovan and Kernighan book, Brazilian edition) to the beach. Between waves, I got through a fair amount of it.",[12,4699,4700],{},"I'm getting a real shake-up trying to adapt to the \"way of Go\". After years used to Python, my brain has to relearn a much more explicit way of thinking: errors handled by hand instead of exceptions, static typing, concurrency that's part of the language itself instead of a framework bolted on top. Nothing absurd, just a different way of solving the same problem.",[12,4702,4703],{},"And the enthusiasm is real. I'm feeling a strong pull to gradually migrate toward Golang, mainly for its robustness: the compiler catches a ton of silly mistakes before the code even runs, and the performance is on another level. Unfortunately the Machine Learning and AI ecosystem in Go is still pretty thin on mature libraries, so I can't switch everything over at once. But I think in the long run the language has real potential to become an important piece in that space, especially for serving models in production.",[12,4705,4706],{},"And yes, the cat decided to show up in one of the photos too. He doesn't study Go with me, he just supervises.",{"title":27,"searchDepth":28,"depth":28,"links":4708},[],"2026-08-19","I've been studying a lot of Go these past few weeks, and I mean that literally: I brought The Go Programming Language (the classic Donovan and Kernighan book, Brazilian edition) to the beach. Between waves, I got through a fair amount of it.",{},"\u002Fen\u002Ftimeline\u002F2026-08-19-estudando-go",[4714,4715,4716,4717],"\u002Fimages\u002Ftimeline\u002F20260819\u002Fgolang_book.jpeg","\u002Fimages\u002Ftimeline\u002F20260819\u002Fgolang_cup.jpeg","\u002Fimages\u002Ftimeline\u002F20260819\u002Fgolang_learn.jpeg","\u002Fimages\u002Ftimeline\u002F20260819\u002Fmy_cat.jpeg",{"title":4688,"description":4710},"en\u002Ftimeline\u002F2026-08-19-estudando-go",[4721,4722],"golang","go","AQlTyWKLXttO1qD9BnNoCctqX-52ZcX0X1DOuF5zWzY",{"id":4725,"title":4726,"body":4727,"date":4746,"description":4747,"extension":31,"meta":4748,"navigation":33,"path":4749,"photos":4750,"seo":4754,"status":37,"stem":4755,"tags":4756,"__hash__":4759},"timeline\u002Fen\u002Ftimeline\u002F2026-08-18-slm-dom-casmurro.md","Chapter 2 of the SLM book, but with Dom Casmurro",{"type":9,"value":4728,"toc":4744},[4729,4735,4741],[12,4730,4731,4732,4734],{},"Started Chapter 2 of ",[66,4733,207],{},", by Prof. Paul Liu, and this time the exercise is building a small GPT from scratch, along the lines of Karpathy's \"build a GPT\", around 10 million parameters.",[12,4736,4737,4738,4740],{},"Instead of using the book's own example dataset though, I swapped it for a text I genuinely love: ",[66,4739,266],{},", by Machado de Assis. Same code, same architecture, but now tokenizing and training on Bentinho and Capitu instead of some generic corpus.",[12,4742,4743],{},"More updates coming soon once I wrap up this study.",{"title":27,"searchDepth":28,"depth":28,"links":4745},[],"2026-08-18","Started Chapter 2 of How to Build and Fine-Tune a Small Language Model, by Prof. Paul Liu, and this time the exercise is building a small GPT from scratch, along the lines of Karpathy's \"build a GPT\", around 10 million parameters.",{},"\u002Fen\u002Ftimeline\u002F2026-08-18-slm-dom-casmurro",[4751,4752,4753],"\u002Fimages\u002Ftimeline\u002F20260818\u002Fslms_book.jpeg","\u002Fimages\u002Ftimeline\u002F20260818\u002Fcapitulo_2_implementacao.jpeg","\u002Fimages\u002Ftimeline\u002F20260818\u002Ftrecho_livro.jpeg",{"title":4726,"description":4747},"en\u002Ftimeline\u002F2026-08-18-slm-dom-casmurro",[4014,3860,4757,4758],"nlp","dom-casmurro","qR1jn7cqvL9YLI9xWeuqbHfPou4oBgnKlDZ1xLA-T4E",1787605212613]