[{"data":1,"prerenderedAt":1126},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmlp-resolve-o-xor":3,"post-en-neural-networks-mlp-resolve-o-xor":4},"\u002Fplaylists\u002Fneural-networks\u002Fmlp-resolve-o-xor",{"id":5,"title":6,"body":7,"cover":1110,"date":1111,"description":1112,"extension":1113,"meta":1114,"navigation":78,"order":99,"path":1115,"playlist":1116,"seo":1117,"status":1118,"stem":1119,"tags":1120,"__hash__":1125},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmlp-resolve-o-xor.md","Two Hidden Lines Solve XOR (and Don't Even Need Training)",{"type":8,"value":9,"toc":1099},"minimark",[10,33,38,145,194,205,209,238,379,425,456,469,473,487,498,508,567,594,602,604,614,618,631,640,670,674,728,763,780,793,797,800,803,872,878,882,929,933,951,1017,1032,1042,1057,1068,1080,1095],[11,12,13,14,19,20,24,25,28,29,32],"p",{},"I'm closing this first big batch of the playlist exactly where it got stuck: ",[15,16,18],"a",{"href":17},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron","the XOR post"," ended with a single neuron banging its head against the wall, and a hook for whenever the professor published the MLP lecture. There's no ",[21,22,23],"code",{},"04b"," in the repository (he jumped straight from ",[21,26,27],{},"04a"," to ",[21,30,31],{},"04c","), but the rest showed up, and it pays off the whole promise: first an MLP with hand-picked hidden weights solves XOR, then the same architecture swaps the hidden activation, and finally a third way of setting the hidden weights shows up, no training at all.",[34,35,37],"h2",{"id":36},"the-architecture-two-hidden-questions-instead-of-one","The architecture: two hidden questions instead of one",[39,40,45],"pre",{"className":41,"code":42,"language":43,"meta":44,"style":44},"language-python shiki shiki-themes github-light github-dark","class XORClassifer(BaseEstimator, ClassifierMixin):\n  def __init__(self):\n    self.ihw = np.array([[0.5, -1, 1], [0.5, 1, -1]]).T\n    self.how = np.array([0.5, -1, -1])\n\n  def fit(self, X, y):\n    pass\n\n  def predict(self, X):\n    self.Xb = include_bias(X)\n    self.pH = self.Xb @ self.ihw\n    self.H = np.sign(self.pH)\n    self.Hb = include_bias(self.H)\n    logits = self.Hb @ self.how\n    y_pred = np.sign(logits)\n    return y_pred\n","python","",[21,46,47,55,61,67,73,80,86,92,97,103,109,115,121,127,133,139],{"__ignoreMap":44},[48,49,52],"span",{"class":50,"line":51},"line",1,[48,53,54],{},"class XORClassifer(BaseEstimator, ClassifierMixin):\n",[48,56,58],{"class":50,"line":57},2,[48,59,60],{},"  def __init__(self):\n",[48,62,64],{"class":50,"line":63},3,[48,65,66],{},"    self.ihw = np.array([[0.5, -1, 1], [0.5, 1, -1]]).T\n",[48,68,70],{"class":50,"line":69},4,[48,71,72],{},"    self.how = np.array([0.5, -1, -1])\n",[48,74,76],{"class":50,"line":75},5,[48,77,79],{"emptyLinePlaceholder":78},true,"\n",[48,81,83],{"class":50,"line":82},6,[48,84,85],{},"  def fit(self, X, y):\n",[48,87,89],{"class":50,"line":88},7,[48,90,91],{},"    pass\n",[48,93,95],{"class":50,"line":94},8,[48,96,79],{"emptyLinePlaceholder":78},[48,98,100],{"class":50,"line":99},9,[48,101,102],{},"  def predict(self, X):\n",[48,104,106],{"class":50,"line":105},10,[48,107,108],{},"    self.Xb = include_bias(X)\n",[48,110,112],{"class":50,"line":111},11,[48,113,114],{},"    self.pH = self.Xb @ self.ihw\n",[48,116,118],{"class":50,"line":117},12,[48,119,120],{},"    self.H = np.sign(self.pH)\n",[48,122,124],{"class":50,"line":123},13,[48,125,126],{},"    self.Hb = include_bias(self.H)\n",[48,128,130],{"class":50,"line":129},14,[48,131,132],{},"    logits = self.Hb @ self.how\n",[48,134,136],{"class":50,"line":135},15,[48,137,138],{},"    y_pred = np.sign(logits)\n",[48,140,142],{"class":50,"line":141},16,[48,143,144],{},"    return y_pred\n",[11,146,147,148,151,152,155,156,160,161,164,165,168,169,173,174,177,178,182,183,186,187,168,190,193],{},"Notice ",[21,149,150],{},"fit",": there's just a ",[21,153,154],{},"pass"," in there. This model ",[157,158,159],"strong",{},"doesn't train",", the weights already come ready-made in ",[21,162,163],{},"__init__",". ",[21,166,167],{},"ihw"," (",[170,171,172],"em",{},"input-to-hidden weights",") is a ",[21,175,176],{},"(3,2)"," matrix: each column is one hidden-layer neuron, with its own bias and its own two weights, ",[15,179,181],{"href":180},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta","the same pre-activation as always",", except now there are ",[157,184,185],{},"two"," neurons computing it in parallel, not just one. ",[21,188,189],{},"how",[170,191,192],{},"hidden-to-output weights",") is the third neuron, which takes the first two's output as if it were two brand-new input variables.",[11,195,196,197,200,201,204],{},"This is the first time in this entire playlist that one output becomes another one's input. That's what separates an MLP (",[170,198,199],{},"multilayer perceptron",") from everything I've seen so far: a ",[157,202,203],{},"hidden layer"," sitting between input and output, each of its neurons seeing the raw data, but nothing outside it seeing what it computes directly, only the next layer does.",[34,206,208],{"id":207},"what-each-hidden-neuron-is-actually-asking","What each hidden neuron is actually asking",[11,210,211,212,168,215,218,219,168,222,225,226,229,230,233,234,237],{},"The two hidden neurons, written as boundaries: ",[21,213,214],{},"0.5 - x0 + x1 = 0",[21,216,217],{},"x1 = x0 - 0.5",") and ",[21,220,221],{},"0.5 + x0 - x1 = 0",[21,223,224],{},"x1 = x0 + 0.5","). Two ",[157,227,228],{},"parallel"," lines (same slope, ",[21,231,232],{},"1","), one shifted half a unit above the other. I computed the pre-activation (",[21,235,236],{},"pH",") of both hidden neurons at exactly the four corners XOR uses:",[239,240,241,272],"table",{},[242,243,244],"thead",{},[245,246,247,252,256,262,267],"tr",{},[248,249,251],"th",{"align":250},"left","Corner",[248,253,255],{"align":254},"center","XOR (label)",[248,257,259,261],{"align":258},"right",[21,260,236],{}," neuron 1",[248,263,264,266],{"align":258},[21,265,236],{}," neuron 2",[248,268,269],{"align":250},[21,270,271],{},"H = sign(pH)",[273,274,275,304,332,356],"tbody",{},[245,276,277,283,290,295,299],{},[278,279,280],"td",{"align":250},[21,281,282],{},"(0.1, 0.1)",[278,284,285,286,289],{"align":254},"false (",[21,287,288],{},"-1",")",[278,291,292],{"align":258},[21,293,294],{},"0.5",[278,296,297],{"align":258},[21,298,294],{},[278,300,301],{"align":250},[21,302,303],{},"[1, 1]",[245,305,306,311,317,322,327],{},[278,307,308],{"align":250},[21,309,310],{},"(0.9, 0.1)",[278,312,313,314,289],{"align":254},"true (",[21,315,316],{},"+1",[278,318,319],{"align":258},[21,320,321],{},"-0.3",[278,323,324],{"align":258},[21,325,326],{},"1.3",[278,328,329],{"align":250},[21,330,331],{},"[-1, 1]",[245,333,334,339,343,347,351],{},[278,335,336],{"align":250},[21,337,338],{},"(0.1, 0.9)",[278,340,313,341,289],{"align":254},[21,342,316],{},[278,344,345],{"align":258},[21,346,326],{},[278,348,349],{"align":258},[21,350,321],{},[278,352,353],{"align":250},[21,354,355],{},"[1, -1]",[245,357,358,363,367,371,375],{},[278,359,360],{"align":250},[21,361,362],{},"(0.9, 0.9)",[278,364,285,365,289],{"align":254},[21,366,288],{},[278,368,369],{"align":258},[21,370,294],{},[278,372,373],{"align":258},[21,374,294],{},[278,376,377],{"align":250},[21,378,303],{},[11,380,381,382,385,386,388,389,392,393,396,397,400,401,404,405,408,409,412,413,416,417,420,421,424],{},"Notice something remarkable: the two ",[157,383,384],{},"false"," corners land on the exact same hidden point, ",[21,387,303],{},". Each hidden neuron is, in practice, a yes\u002Fno question (\"am I above this line?\"), and XOR turns out true exactly when the two answers ",[157,390,391],{},"disagree"," (one yes, one no), and false when they ",[157,394,395],{},"agree"," (both yes, in this case, since no corner falls below both lines). That's not a dataset coincidence, it's the definition of XOR reappearing one level up: the output layer only needs to separate \"the two answers agree\" from \"they disagree\", and that, ",[15,398,399],{"href":17},"look at that",", is XOR again, just now between ",[157,402,403],{},"two binary answers"," instead of two continuous variables. The difference is that summing two disagreeing answers always gives ",[21,406,407],{},"0",", and summing two agreeing ones always gives ",[21,410,411],{},"2"," or ",[21,414,415],{},"-2",": that ",[157,418,419],{},"is"," linearly separable (just split on the ",[21,422,423],{},"sum","), which the original continuous variable wasn't.",[11,426,427,428,431,432,435,436,438,439,442,443,445,446,449,450,445,452,455],{},"With ",[21,429,430],{},"how = [0.5, -1, -1]",", the output is ",[21,433,434],{},"0.5 - H₀ - H₁",", that is, ",[21,437,294],{}," minus the sum of the two hidden answers. False corners (",[21,440,441],{},"[1,1]",", sum ",[21,444,411],{},"): ",[21,447,448],{},"0.5 - 2 = -1.5",", negative. True corners (sum ",[21,451,407],{},[21,453,454],{},"0.5 - 0 = 0.5",", positive. Matches the label exactly, with no training at all, because the weights were already chosen knowing this.",[457,458,459],"blockquote",{},[11,460,461,464,465,468],{},[157,462,463],{},"Output:"," accuracy ",[157,466,467],{},"1.0",".",[34,470,472],{"id":471},"the-final-boundary-two-lines-become-a-band","The final boundary: two lines become a band",[474,475],"mlp-decision-region-chart",{":classes":476,":hidden-weights":477,":output-weights":478,":points":479,":x-max":480,":x-min":481,":y-max":480,":y-min":481,"activation":482,"negative-label":483,"positive-label":484,"x-label":485,"y-label":486},"[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]","[[0.5,-1,1],[0.5,1,-1]]","[0.5,-1,-1]","[[0.097, 0.094], [0.152, 0.037], [0.137, 0.014], [0.09, 0.088], [0.156, 0.099], [0.069, 0.169], [0.181, 0.066], [0.135, 0.078], [0.108, 0.113], [0.036, 0.103], [0.847, 0.051], [0.877, 0.001], [0.826, 0.112], [0.932, 0.143], [0.877, 0.135], [0.978, 0.159], [0.831, 0.013], [0.92, 0.162], [0.899, 0.148], [0.912, 0.111], [0.115, 0.911], [0.079, 0.992], [0.146, 0.872], [0.086, 0.848], [0.124, 0.829], [0.083, 0.937], [0.128, 0.929], [0.041, 0.938], [0.093, 0.966], [0.086, 0.984], [0.921, 0.937], [0.88, 0.88], [0.881, 0.95], [0.878, 0.894], [0.958, 0.926], [0.887, 0.914], [0.83, 0.825], [0.922, 1.038], [1.001, 1.026], [0.929, 0.874]]","1.5","-0.5","sign","false (XOR)","true (XOR)","x0","x1",[11,488,489,490,493,494,497],{},"The blue region is no longer a half-plane cut by one line, it's a ",[157,491,492],{},"diagonal band"," between the two hidden neurons' parallel lines. That's exactly the shape a single neuron ",[15,495,496],{"href":17},"could never draw",": a band has two boundaries, a single line only has one. Two linear boundaries, combined, produce a shape neither of them could alone.",[34,499,501,502,504,505],{"id":500},"lecture-4d-swapping-sign-for-tanh","Lecture 4d: swapping ",[21,503,482],{}," for ",[21,506,507],{},"tanh",[39,509,511],{"className":41,"code":510,"language":43,"meta":44,"style":44},"class XORClassifer(BaseEstimator, ClassifierMixin):\n  def __init__(self):\n    self.ihw = np.array([[0.5, -1, 1], [0.5, 1, -1]]).T\n    self.how = np.array([0.75, -0.9, -0.9])\n\n  def predict(self, X):\n    self.Xb = include_bias(X)\n    self.pH = self.Xb @ self.ihw\n    self.H = np.tanh(self.pH)\n    self.Hb = include_bias(self.H)\n    logits = self.Hb @ self.how\n    y_pred = np.sign(logits)\n    return y_pred\n",[21,512,513,517,521,525,530,534,538,542,546,551,555,559,563],{"__ignoreMap":44},[48,514,515],{"class":50,"line":51},[48,516,54],{},[48,518,519],{"class":50,"line":57},[48,520,60],{},[48,522,523],{"class":50,"line":63},[48,524,66],{},[48,526,527],{"class":50,"line":69},[48,528,529],{},"    self.how = np.array([0.75, -0.9, -0.9])\n",[48,531,532],{"class":50,"line":75},[48,533,79],{"emptyLinePlaceholder":78},[48,535,536],{"class":50,"line":82},[48,537,102],{},[48,539,540],{"class":50,"line":88},[48,541,108],{},[48,543,544],{"class":50,"line":94},[48,545,114],{},[48,547,548],{"class":50,"line":99},[48,549,550],{},"    self.H = np.tanh(self.pH)\n",[48,552,553],{"class":50,"line":105},[48,554,126],{},[48,556,557],{"class":50,"line":111},[48,558,132],{},[48,560,561],{"class":50,"line":117},[48,562,138],{},[48,564,565],{"class":50,"line":123},[48,566,144],{},[11,568,569,570,572,573,576,577,580,581,583,584,28,586,589,590,593],{},"Same ",[21,571,167],{},", same two hidden lines. The only change is ",[21,574,575],{},"H = np.tanh(self.pH)"," instead of ",[21,578,579],{},"H = np.sign(self.pH)"," (and ",[21,582,189],{}," adjusted, from ",[21,585,478],{},[21,587,588],{},"[0.75,-0.9,-0.9]",", to compensate for the different scale). ",[15,591,592],{"href":180},"As I already saw with Adaline",", the difference between pre-activation and post-activation isn't just theoretical: here it decides whether the final boundary has a sharp edge or is smooth.",[457,595,596],{},[11,597,598,464,600,468],{},[157,599,463],{},[157,601,467],{},[474,603],{":classes":476,":hidden-weights":477,":output-weights":588,":points":479,":x-max":480,":x-min":481,":y-max":480,":y-min":481,"activation":507,"negative-label":483,"positive-label":484,"x-label":485,"y-label":486},[11,605,606,607,609,610,613],{},"Compare with the chart above: same overall diagonal band, but now with rounded edges instead of sharp cuts. ",[21,608,507],{}," saturates smoothly near ",[21,611,612],{},"±1"," instead of jumping straight there, so the transition between \"inside the band\" and \"outside the band\" is smooth too.",[34,615,617],{"id":616},"what-the-hidden-layer-is-actually-doing-it-transforms-the-space","What the hidden layer is actually doing: it transforms the space",[11,619,620,621,624,625,627,628,630],{},"The notebook plots the same point cloud after passing through the two hidden neurons (",[21,622,623],{},"H",", not ",[21,626,485],{},"\u002F",[21,629,486],{}," anymore), and that's where the trick becomes obvious:",[632,633],"perceptron-boundary-snapshot",{":bias":634,":classes":476,":points":635,":w":636,":x-max":232,":x-min":637,":y-max":232,":y-min":637,"negative-label":483,"positive-label":484,"x-label":638,"y-label":639},"0.75","[[0.46, 0.464], [0.367, 0.548], [0.36, 0.553], [0.461, 0.463], [0.416, 0.506], [0.537, 0.381], [0.367, 0.548], [0.416, 0.506], [0.466, 0.458], [0.513, 0.408], [-0.288, 0.861], [-0.36, 0.88], [-0.211, 0.838], [-0.282, 0.859], [-0.237, 0.846], [-0.309, 0.867], [-0.308, 0.866], [-0.252, 0.85], [-0.246, 0.849], [-0.292, 0.862], [0.861, -0.288], [0.888, -0.391], [0.841, -0.222], [0.852, -0.256], [0.835, -0.202], [0.875, -0.34], [0.862, -0.292], [0.885, -0.378], [0.879, -0.357], [0.885, -0.378], [0.475, 0.449], [0.462, 0.463], [0.515, 0.406], [0.475, 0.449], [0.436, 0.487], [0.483, 0.441], [0.458, 0.466], [0.548, 0.366], [0.482, 0.442], [0.418, 0.504]]","[-0.9, -0.9]","-0.6","H0 (hidden neuron 1)","H1 (hidden neuron 2)",[11,641,642,643,627,645,647,648,651,652,654,655,659,660,576,662,665,666,669],{},"In this new space, the two \"false\" clusters (which were opposite corners in ",[21,644,485],{},[21,646,486],{},") collapse to practically ",[157,649,650],{},"the same spot",", and the two \"true\" clusters split into two clearly distinct groups, one on each side. And look at the boundary: it's a ",[157,653,50],{},", exactly ",[15,656,658],{"href":657},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","the same old linear boundary",", just drawn over ",[21,661,623],{},[21,663,664],{},"x",". That's the central idea behind every network with a hidden layer: it doesn't \"learn to bend a line\", it ",[157,667,668],{},"transforms the space"," until the problem is linearly separable again, and then solves it with the simplest tool there is.",[34,671,673],{"id":672},"lecture-4e-what-if-the-hidden-weights-were-just-drawn-at-random","Lecture 4e: what if the hidden weights were just drawn at random?",[39,675,677],{"className":41,"code":676,"language":43,"meta":44,"style":44},"class ELM(BaseEstimator, ClassifierMixin):\n  def __init__(self, n_hidden=2):\n    self.ihw = np.random.randn(3,n_hidden)\n\n  def fit(self, X, y):\n    self.Xb = include_bias(X)\n    self.pH = self.Xb @ self.ihw\n    self.H = np.tanh(self.pH)\n    self.Hb = include_bias(self.H)\n    self.how = np.linalg.pinv(self.Hb) @ y\n    return self\n",[21,678,679,684,689,694,698,702,706,710,714,718,723],{"__ignoreMap":44},[48,680,681],{"class":50,"line":51},[48,682,683],{},"class ELM(BaseEstimator, ClassifierMixin):\n",[48,685,686],{"class":50,"line":57},[48,687,688],{},"  def __init__(self, n_hidden=2):\n",[48,690,691],{"class":50,"line":63},[48,692,693],{},"    self.ihw = np.random.randn(3,n_hidden)\n",[48,695,696],{"class":50,"line":69},[48,697,79],{"emptyLinePlaceholder":78},[48,699,700],{"class":50,"line":75},[48,701,85],{},[48,703,704],{"class":50,"line":82},[48,705,108],{},[48,707,708],{"class":50,"line":88},[48,709,114],{},[48,711,712],{"class":50,"line":94},[48,713,550],{},[48,715,716],{"class":50,"line":99},[48,717,126],{},[48,719,720],{"class":50,"line":105},[48,721,722],{},"    self.how = np.linalg.pinv(self.Hb) @ y\n",[48,724,725],{"class":50,"line":111},[48,726,727],{},"    return self\n",[11,729,730,731,168,734,737,738,740,741,744,745,748,749,751,752,754,755,744,758,762],{},"This is an ",[157,732,733],{},"ELM",[170,735,736],{},"Extreme Learning Machine","). The difference from everything I've seen so far: ",[21,739,167],{}," (input-to-hidden weights) is purely ",[157,742,743],{},"random",", ",[21,746,747],{},"np.random.randn",", and it never changes, ",[21,750,150],{}," never adjusts it at all. Only ",[21,753,189],{}," (output layer) actually gets computed, and look how: ",[21,756,757],{},"np.linalg.pinv(self.Hb) @ y",[15,759,761],{"href":760},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal","the very same pseudo-inverse from the normal equation, back in post 3",". ELM doesn't train the whole network, it draws a random space transformation and solves only the last layer, in one shot, no gradient at all.",[457,764,765],{},[11,766,767,464,774,776,777,468],{},[157,768,769,770,773],{},"Notebook output (",[21,771,772],{},"n_hidden=4","):",[157,775,467],{},", output weights ",[21,778,779],{},"[-216.55, -14.43, -13.51, -6.63, 232.97]",[11,781,782,783,785,786,744,789,792],{},"I reproduced this myself, with a seed that also hits ",[21,784,467],{},", so I could plot the final boundary (notice the output weights, also quite large, ",[21,787,788],{},"265.96, -158.78, 3.17, 61.15, -167.19",[15,790,791],{"href":760},"the same pseudo-inverse behavior I already saw before",": it always finds an exact answer, even if that means huge weights):",[474,794],{":classes":476,":hidden-weights":795,":output-weights":796,":points":479,":x-max":480,":x-min":481,":y-max":480,":y-min":481,"activation":507,"negative-label":483,"positive-label":484,"x-label":485,"y-label":486},"[[1.7641,1.8676,-0.1032],[0.4002,-0.9773,0.4106],[0.9787,0.9501,0.144],[2.2409,-0.1514,1.4543]]","[265.9576,-158.7802,3.1749,61.1483,-167.1908]",[11,798,799],{},"Nothing like the tidy band above, it's an irregular shape, four hidden neurons pointing in random directions, each slicing the plane a different way. But it separates all four XOR corners just the same, without me (or the professor) hand-picking a single direction.",[11,801,802],{},"That raises the obvious question: how many random hidden neurons does it take to have a decent shot at solving XOR? I ran 15 different seeds for each hidden-layer size:",[239,804,805,818],{},[242,806,807],{},[245,808,809,812,815],{},[248,810,811],{"align":254},"Hidden neurons",[248,813,814],{"align":258},"Mean accuracy",[248,816,817],{"align":258},"Perfect runs",[273,819,820,830,839,850,861],{},[245,821,822,824,827],{},[278,823,232],{"align":254},[278,825,826],{"align":258},"0.522",[278,828,829],{"align":258},"0 of 15",[245,831,832,834,837],{},[278,833,411],{"align":254},[278,835,836],{"align":258},"0.548",[278,838,829],{"align":258},[245,840,841,844,847],{},[278,842,843],{"align":254},"3",[278,845,846],{"align":258},"0.853",[278,848,849],{"align":258},"3 of 15",[245,851,852,855,858],{},[278,853,854],{"align":254},"4",[278,856,857],{"align":258},"0.977",[278,859,860],{"align":258},"10 of 15",[245,862,863,866,869],{},[278,864,865],{"align":254},"8",[278,867,868],{"align":258},"1.000",[278,870,871],{"align":258},"15 of 15",[11,873,874,875,877],{},"With 1 or 2 neurons, the draw almost never lands two directions that, combined, separate XOR, mean accuracy sits near chance (",[21,876,294],{},"). From 3 or 4 onward, the odds of at least two random directions landing \"well positioned\" grow fast, and with 8 they never failed, across the 15 seeds I tried. Sheer number of random directions makes up for the lack of smart choice, the central idea behind the whole family of random-projection methods.",[34,879,881],{"id":880},"wrapping-up","Wrapping up",[239,883,884,894],{},[242,885,886],{},[245,887,888,891],{},[248,889,890],{"align":250},"What I already knew",[248,892,893],{"align":250},"What this lecture settled",[273,895,896,904,918],{},[245,897,898,901],{},[278,899,900],{"align":250},"A single neuron can't solve XOR",[278,902,903],{"align":250},"Two hidden lines combined can, transforming the space until the problem is linearly separable again",[245,905,906,909],{},[278,907,908],{"align":250},"Pre-activation vs. post-activation already showed up with Adaline",[278,910,911,912,914,915,917],{"align":250},"The hidden activation choice (",[21,913,482],{}," vs. ",[21,916,507],{},") changes the final boundary's shape, from a sharp-edged band to a smooth one",[245,919,920,923],{},[278,921,922],{"align":250},"The normal equation solves regression in one shot",[278,924,925,926,928],{"align":250},"ELM applies the same computation to a ",[157,927,743],{}," hidden layer: only the last layer needs a closed-form solution, the rest can be drawn at random",[34,930,932],{"id":931},"practical-application","Practical application",[11,934,935,936,744,942,945,946,950],{},"I tested ELM on the two-moons dataset (",[15,937,938,941],{"href":17},[21,939,940],{},"make_moons",", the same one from the XOR post",[21,943,944],{},"noise=0.25",", the same one already used ",[15,947,949],{"href":948},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fsigmoide-consertada-hinge-perdeu-margem","in the last post","), varying the number of hidden neurons, 10 seeds per size, always with a 70\u002F30 train\u002Ftest split:",[239,952,953,962],{},[242,954,955],{},[245,956,957,959],{},[248,958,811],{"align":250},[248,960,961],{"align":258},"Test accuracy (mean of 10 seeds)",[273,963,964,975,983,991,999,1009],{},[245,965,966,972],{},[278,967,968,969,289],{"align":250},"Perceptron (no hidden layer, ",[15,970,971],{"href":17},"XOR post",[278,973,974],{"align":258},"0.867",[245,976,977,980],{},[278,978,979],{"align":250},"ELM, 1 neuron",[278,981,982],{"align":258},"0.714",[245,984,985,988],{},[278,986,987],{"align":250},"ELM, 5 neurons",[278,989,990],{"align":258},"0.881",[245,992,993,996],{},[278,994,995],{"align":250},"ELM, 20 neurons",[278,997,998],{"align":258},"0.927",[245,1000,1001,1004],{},[278,1002,1003],{"align":250},"ELM, 50 neurons",[278,1005,1006],{"align":258},[157,1007,1008],{},"0.930",[245,1010,1011,1014],{},[278,1012,1013],{"align":250},"ELM, 100 neurons",[278,1015,1016],{"align":258},"0.843",[11,1018,1019,1020,1023,1024,1027,1028,468],{},"With just 1 random neuron, ELM ends up ",[157,1021,1022],{},"worse"," than the properly trained perceptron: a single drawn direction, with no adjustment at all, is rarely as good as a direction the algorithm itself optimized. From 5 neurons on, ELM already catches up to the perceptron, and at 20 to 50 it pulls well ahead, several random directions combined manage to bend the boundary around the moon shape, something no single line can do. But at 100 neurons accuracy ",[157,1025,1026],{},"drops",": a hidden layer too big for only 210 training points becomes an easy space to fit exactly, the pseudo-inverse nails the training set \"perfectly\" at the cost of worse generalization, the same complexity-versus-generalization trade-off ",[15,1029,1031],{"href":1030},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab04-feature-engineering","that already showed up in the other playlist",[11,1033,1034,1035,1037,1038,1041],{},"The code that produced this table is the same ",[21,1036,733],{}," from above, just now inside a loop varying ",[21,1039,1040],{},"n_hidden"," and the seed:",[39,1043,1045],{"className":41,"code":1044,"language":43,"meta":44,"style":44},"elm = ELM(n_hidden=50)\nelm.fit(X_train, y_train)\n",[21,1046,1047,1052],{"__ignoreMap":44},[48,1048,1049],{"class":50,"line":51},[48,1050,1051],{},"elm = ELM(n_hidden=50)\n",[48,1053,1054],{"class":50,"line":57},[48,1055,1056],{},"elm.fit(X_train, y_train)\n",[11,1058,1059,1060,1063,1064,1067],{},"Taking the best of the 10 seeds for 50 neurons (test accuracy ",[21,1061,1062],{},"0.956","), plotting the boundary over the ",[157,1065,1066],{},"90 test points",", never seen during training:",[474,1069],{":classes":1070,":hidden-weights":1071,":output-weights":1072,":points":1073,":x-max":1074,":x-min":1075,":y-max":1076,":y-min":1077,"activation":507,"negative-label":1078,"positive-label":1079,"x-label":485,"y-label":486},"[1, 1, 1, 1, 1, -1, 1, -1, 1, 1, -1, -1, 1, -1, 1, 1, -1, 1, -1, -1, -1, 1, -1, -1, 1, 1, -1, 1, -1, 1, 1, -1, -1, -1, -1, 1, -1, 1, 1, 1, 1, -1, -1, 1, -1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, 1, -1, 1, -1, 1, 1, -1, -1, 1, -1, -1, 1, 1, -1, -1, 1, 1, 1, -1, -1, -1, 1, -1, 1, 1, -1, 1, 1, 1, 1, 1, 1]","[[1.764, -0.895, 1.883], [0.4, 0.387, -1.348], [0.979, -0.511, -1.27], [2.241, -1.181, 0.969], [1.868, -0.028, -1.173], [-0.977, 0.428, 1.944], [0.95, 0.067, -0.414], [-0.151, 0.302, -0.747], [-0.103, -0.634, 1.923], [0.411, -0.363, 1.481], [0.144, -0.672, 1.868], [1.454, -0.36, 0.906], [0.761, -0.813, -0.861], [0.122, -1.726, 1.91], [0.444, 0.177, -0.268], [0.334, -0.402, 0.802], [1.494, -1.63, 0.947], [-0.205, 0.463, -0.155], [0.313, -0.907, 0.614], [-0.854, 0.052, 0.922], [-2.553, 0.729, 0.376], [0.654, 0.129, -1.099], [0.864, 1.139, 0.298], [-0.742, -1.235, 1.326], [2.27, 0.402, -0.695], [-1.454, -0.685, -0.15], [0.046, -0.871, -0.435], [-0.187, -0.579, 1.849], [1.533, -0.312, 0.672], [1.469, 0.056, 0.407], [0.155, -1.165, -0.77], [0.378, 0.901, 0.539], [-0.888, 0.466, -0.674], [-1.981, -1.536, 0.032], [-0.348, 1.488, -0.636], [0.156, 1.896, 0.676], [1.23, 1.179, 0.577], [1.202, -0.18, -0.208], [-0.387, -1.071, 0.396], [-0.302, 1.054, -1.093], [-1.049, -0.403, -1.491], [-1.42, 1.222, 0.439], [-1.706, 0.208, 0.167], [1.951, 0.977, 0.635], [-0.51, 0.356, 2.383], [-0.438, 0.707, 0.944], [-1.253, 0.011, -0.913], [0.777, 1.786, 1.117], [-1.614, 0.127, -1.316], [-0.213, 0.402, -0.462]]","[-2152.198, -2.811, -80.853, 3.077, -17.459, -24.08, -0.653, 2029.747, 717.624, 68.737, -22.837, -11.59, -235.063, 11.619, 0.06, -2401.914, 604.646, -17.405, -575.25, 7.077, 91.445, -117.75, 105.179, 3.536, 22.877, 288.569, -179.658, -67.765, -78.265, 132.702, 1055.225, 10.531, 44.293, -236.05, 22.344, 23.606, -1.939, 16.423, -3762.274, -42.702, -47.89, 3.27, 28.381, -3796.323, 32.026, -0.036, 1.681, 490.96, -6.227, -142.406, 947.283]","[[0.459, -0.307], [2.533, 0.563], [0.775, -0.587], [0.313, 0.02], [1.919, -0.722], [-0.467, 0.153], [1.938, 0.643], [-0.059, 0.977], [1.001, -0.314], [1.285, -0.493], [-0.569, 0.954], [-0.516, 0.844], [0.062, 0.484], [0.626, 1.302], [1.613, 0.5], [1.921, 0.04], [-0.883, 0.769], [1.574, 0.382], [1.2, -0.04], [-1.196, 0.011], [0.07, 1.035], [1.268, -0.456], [0.976, 0.071], [-0.776, 0.618], [1.839, -0.179], [1.958, -0.004], [1.424, 0.352], [0.508, -0.064], [0.089, 1.376], [1.53, -0.02], [0.725, -0.242], [0.735, 0.453], [0.764, 0.067], [-1.03, 0.902], [0.265, 1.636], [0.992, -0.205], [-0.404, 0.661], [0.519, -0.287], [1.167, -0.596], [0.576, -0.121], [1.29, -0.046], [-0.235, 1.188], [-1.088, 0.061], [1.0, -0.268], [-0.046, 1.319], [0.093, 0.118], [0.234, -0.23], [0.364, -0.531], [1.369, -0.069], [1.319, -0.777], [0.356, -0.179], [0.542, -1.002], [1.024, -0.46], [1.414, 0.592], [-0.73, 0.946], [0.149, 0.648], [-0.741, -0.41], [0.029, 1.456], [1.178, -0.145], [0.529, 0.794], [2.187, 0.293], [0.955, 0.648], [0.321, -0.131], [1.787, 0.148], [0.508, 0.51], [1.277, 0.558], [1.535, -0.584], [0.924, 0.607], [-0.236, 0.773], [1.57, -0.272], [0.271, -0.032], [0.692, 0.917], [0.399, 0.695], [0.374, -0.947], [2.224, -0.191], [0.046, 0.335], [0.412, 0.782], [-0.265, 0.983], [0.897, 0.68], [0.494, 0.467], [0.625, 0.516], [2.185, 0.273], [0.382, 0.436], [-1.1, 0.543], [2.113, 0.488], [1.479, 0.104], [0.197, -0.404], [1.417, -0.792], [1.153, -0.648], [1.418, -0.025]]","2.6","-1.3","1.7","-1.1","moon 0","moon 1",[11,1081,1082,1083,1085,1086,1088,1089,1091,1092,1094],{},"50 random directions combined really do manage to curve around the two moons, nothing like the perceptron's single line or the XOR's tidy two-line band: it's an irregular boundary, full of folds, but one that follows the data's real shape in a way no single line ever could. Comparing with the two-hidden-line ",[21,1084,474],{}," from above, line by line: it's the same component, the same math (",[21,1087,507],{}," on the hidden pre-activation, ",[21,1090,482],{}," on the final logit), just with 50 hidden weight columns instead of 2, and those 50 columns came from ",[21,1093,747],{},", not hand-picked numbers.",[1096,1097,1098],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":44,"searchDepth":57,"depth":57,"links":1100},[1101,1102,1103,1104,1106,1107,1108,1109],{"id":36,"depth":57,"text":37},{"id":207,"depth":57,"text":208},{"id":471,"depth":57,"text":472},{"id":500,"depth":57,"text":1105},"Lecture 4d: swapping sign for tanh",{"id":616,"depth":57,"text":617},{"id":672,"depth":57,"text":673},{"id":880,"depth":57,"text":881},{"id":931,"depth":57,"text":932},null,"2026-08-27","Lectures 4c, 4d, and 4e: the hook I left hanging in the XOR post finally pays off. A real MLP solves the problem, first with hand-picked hidden weights, then with randomly drawn hidden weights (ELM), the output layer always solved by the same normal equation from post 3.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmlp-resolve-o-xor","neural-networks",{"title":6,"description":1112},"published","en\u002Fplaylists\u002Fneural-networks\u002Fmlp-resolve-o-xor",[1121,1122,1123,1124],"mlp","hidden-layer","elm","xor","7u2r_ZQbjw0AVvTCQzmniFw9CRG3e2v_QT7NDFEg2RI",1788965100311]