{"id":988,"date":"2026-06-15T12:50:15","date_gmt":"2026-06-15T12:50:15","guid":{"rendered":"https:\/\/tensorzen.online\/?p=988"},"modified":"2026-06-15T12:50:15","modified_gmt":"2026-06-15T12:50:15","slug":"expected-prediction-error-and-the-regression-function","status":"publish","type":"post","link":"https:\/\/tensorzen.online\/?p=988","title":{"rendered":"Expected Prediction Error and the Regression Function"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><strong>Source: Elements of Statistical Learning, Section 2.4<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This notebook walks through the theoretical framework for supervised learning: why squared error loss is natural, what EPE is, and why the optimal predictor is always the conditional expection $f(x) = \\mathbb{E}[Y \\mid X = x]$.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">1. Setup: the prediction problem<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We have:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>$X \\in \\mathbb{R}^p$ \u2014 a random input vector<\/li>\n\n\n\n<li>$Y \\in \\mathbb{R}$ \u2014 a random real-valued output<\/li>\n\n\n\n<li>Joint distribution $\\Pr(X, Y)$ \u2014 we never know this fully, only observe samples from it<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">We want a function $f(X)$ that predicts $Y$ well.<br>To measure &#8220;well&#8221; we need a <strong>loss function<\/strong> $L(Y, f(X))$.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The most common choice is <strong>squared error loss<\/strong>:<br>$$L(Y, f(X)) = (Y &#8211; f(X))^2$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It penalizes large errors more than small ones, is symmetric, and\u2014crucially\u2014is<br>mathematically tractable.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"521\" src=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1024x521.png\" alt=\"\" class=\"wp-image-992\" srcset=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1024x521.png 1024w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-300x153.png 300w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-768x391.png 768w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1536x782.png 1536w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image.png 1740w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<pre class=\"wp-block-preformatted\">Squared loss grows quadratically \u2014 large errors are penalized much more heavily.<\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">2. Expected Prediction Error (EPE) \u2014 Equations 2.9 &amp; 2.10<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We don&#8217;t want $f$ to do well on one specific $(x, y)$ pair.<br>We want it to do well <strong>on average over the whole joint distribution<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s the <strong>Expected Prediction Error<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$<br>\\text{EPE}(f) = \\mathbb{E}\\bigl[(Y &#8211; f(X))^2\\bigr]<br>= \\int \\bigl[y &#8211; f(x)\\bigr]^2 \\Pr(dx,\\, dy)<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The integral is over the full joint distribution of $(X, Y)$.<br>We want to find the $f$ that minimizes this.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Deep dive: what does EPE actually mean?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Your intuition is exactly right, but let&#8217;s be precise about <em>what<\/em> we&#8217;re averaging over.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">The setup<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">We imagine a process that generates data:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Draw $(X, Y)$ from the joint distribution $\\Pr(X, Y)$.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">This is the same process that generates your training set and any future test point.<br>We never see $\\Pr(X, Y)$ directly \u2014 we only observe samples from it.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">EPE unpacked<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">$$<br>\\text{EPE}(f) = \\mathbb{E}\\bigl[(Y &#8211; f(X))^2\\bigr]<br>= \\int \\bigl[y &#8211; f(x)\\bigr]^2 \\,\\Pr(dx,\\, dy)<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The <strong>expectation<\/strong> is over the <strong>joint distribution of $(X, Y)$<\/strong> \u2014 both inputs and outputs vary.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Think of it concretely:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Draw a random input $X = x$ from wherever inputs come from.<\/li>\n\n\n\n<li>Draw the corresponding output $Y = y$ from the conditional distribution $\\Pr(Y \\mid X=x)$ \u2014 which may be noisy.<\/li>\n\n\n\n<li>Compute the squared error $(y &#8211; f(x))^2$.<\/li>\n\n\n\n<li><strong>Average this over infinitely many such draws.<\/strong><\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">EPE is that long-run average squared error.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Is it &#8220;average error over a dataset&#8221;?<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Almost \u2014 but with an important distinction:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>What you described<\/th><th>What EPE is<\/th><\/tr><\/thead><tbody><tr><td>Average loss over a <strong>finite training set<\/strong><\/td><td>Average loss over the <strong>true population<\/strong> (infinite draws)<\/td><\/tr><tr><td>$\\frac{1}{n}\\sum_{i=1}^n (y_i &#8211; f(x_i))^2$ \u2014 the <strong>training MSE<\/strong><\/td><td>$\\mathbb{E}_{(X,Y)}[(Y-f(X))^2]$ \u2014 the <strong>population MSE<\/strong><\/td><\/tr><tr><td>Depends on which $n$ samples you happened to draw<\/td><td>Fixed property of $f$ and the true distribution<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The training MSE is an <strong>estimate<\/strong> of EPE using a finite sample.<br>As $n \\to \\infty$, training MSE $\\to$ EPE (by the law of large numbers).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Why minimize EPE, not training MSE?<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Because we care about <strong>future predictions<\/strong>, not memorizing the training set.<br>A function that memorizes training data perfectly (training MSE = 0) can have terrible EPE on new data \u2014 that&#8217;s <strong>overfitting<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The best $f$ is the one with minimum <strong>EPE<\/strong> \u2014 minimum expected error on a fresh draw from the same distribution.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"272\" src=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1-1024x272.png\" alt=\"\" class=\"wp-image-993\" srcset=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1-1024x272.png 1024w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1-300x80.png 300w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1-768x204.png 768w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1-1536x408.png 1536w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-1-2048x543.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">3. Conditioning on X \u2014 Equation 2.11<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The joint expectation can always be decomposed using the <strong>law of total expectation<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$\\mathbb{E}<em>{X,Y}[\\cdot] = \\mathbb{E}_X\\bigl[\\mathbb{E}<\/em>{Y|X}[\\cdot \\mid X]\\bigr]$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Applying this to EPE:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$<br>\\text{EPE}(f) = \\mathbb{E}<em>X\\Bigl[\\mathbb{E}<\/em>{Y|X}\\bigl[(Y &#8211; f(X))^2 \\mid X\\bigr]\\Bigr]<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now we have an outer expectation over $X$, and an inner expectation over $Y$ given $X$.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Key insight:<\/strong> the outer $\\mathbb{E}_X[\\cdot]$ is an average of non-negative terms.<br>To minimize the whole thing, it suffices to minimize the <strong>inner term pointwise<\/strong> \u2014<br>for each fixed value of $X = x$.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">4. Pointwise minimization \u2014 Equation 2.12<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Fix $X = x$. We ask: what constant $c$ minimizes<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$\\mathbb{E}_{Y|X}\\bigl[(Y &#8211; c)^2 \\mid X = x\\bigr]\\,?$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is a function of $c$ alone. Take the derivative and set to zero:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$<br>\\frac{d}{dc}\\,\\mathbb{E}[(Y-c)^2 \\mid X=x]<br>= \\mathbb{E}\\bigl[2(Y-c)(-1) \\mid X=x\\bigr]<br>= -2\\,\\mathbb{E}[Y &#8211; c \\mid X=x] = 0<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$\\Rightarrow\\quad \\mathbb{E}[Y \\mid X=x] = c$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So the minimizing constant at each point $x$ is exactly the <strong>conditional mean<\/strong> of $Y$ given $X=x$.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore:<br>$$<br>f(x) = \\mathbb{E}[Y \\mid X = x] \\qquad \\text{(Eq. 2.13)}<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is called the <strong>regression function<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"545\" src=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-2-1024x545.png\" alt=\"\" class=\"wp-image-994\" srcset=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-2-1024x545.png 1024w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-2-300x160.png 300w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-2-768x408.png 768w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-2-1536x817.png 1536w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-2.png 1756w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">5. Bias-Variance decomposition: why the minimum is the conditional mean<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">There&#8217;s a cleaner algebraic way to see the same result.<br>Add and subtract $\\mathbb{E}[Y \\mid X=x]$, letting $\\mu(x) = \\mathbb{E}[Y \\mid X=x]$:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$<br>(Y &#8211; c)^2 = \\bigl[(Y &#8211; \\mu(x)) + (\\mu(x) &#8211; c)\\bigr]^2<br>= (Y &#8211; \\mu(x))^2 + 2(Y &#8211; \\mu(x))(\\mu(x) &#8211; c) + (\\mu(x) &#8211; c)^2<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Taking $\\mathbb{E}[\\cdot \\mid X=x]$, the cross term vanishes because $\\mathbb{E}[Y &#8211; \\mu(x) \\mid X=x] = 0$:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$<br>\\mathbb{E}[(Y-c)^2 \\mid X=x] = \\underbrace{\\text{Var}(Y \\mid X=x)}<em>{\\text{irreducible noise}} + \\underbrace{(\\mu(x) &#8211; c)^2}<\/em>{\\text{bias}^2}<br>$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The first term is fixed \u2014 no choice of $c$ can remove the inherent randomness of $Y$.<br>The second term is zero exactly when $c = \\mu(x) = \\mathbb{E}[Y \\mid X=x]$.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is the <strong>bias-variance decomposition<\/strong> in its simplest form.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"468\" src=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-3-1024x468.png\" alt=\"\" class=\"wp-image-995\" srcset=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-3-1024x468.png 1024w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-3-300x137.png 300w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-3-768x351.png 768w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-3-1536x702.png 1536w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-3-2048x936.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">6. The full picture: putting it together<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Step<\/th><th>What we do<\/th><th>Result<\/th><\/tr><\/thead><tbody><tr><td>Define problem<\/td><td>Want $f$ to predict $Y$ from $X$<\/td><td>Need a loss function<\/td><\/tr><tr><td>Choose loss<\/td><td>Squared error $(Y &#8211; f(X))^2$<\/td><td>Tractable, penalizes large errors<\/td><\/tr><tr><td>Define criterion<\/td><td>$\\text{EPE}(f) = \\mathbb{E}[(Y-f(X))^2]$<\/td><td>Global average loss<\/td><\/tr><tr><td>Condition on $X$<\/td><td>Law of total expectation<\/td><td>$\\text{EPE} = \\mathbb{E}<em>X[\\mathbb{E}<\/em>{Y<\/td><\/tr><tr><td>Minimize pointwise<\/td><td>At each $x$, minimize over constant $c$<\/td><td>$c^* = \\mathbb{E}[Y \\mid X=x]$<\/td><\/tr><tr><td>Conclusion<\/td><td>Optimal predictor<\/td><td>$f(x) = \\mathbb{E}[Y \\mid X=x]$<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The regression function $f(x) = \\mathbb{E}[Y \\mid X=x]$ is the theoretical gold standard.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Every regression method (linear regression, kNN, neural networks, gradient boosting)<br>is an attempt to <em>approximate<\/em> this function from finite training data.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">7. Why nearest-neighbor methods follow from this<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The text ends by saying nearest-neighbor methods try to <strong>directly implement this recipe<\/strong>.<br>Here&#8217;s the idea:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$f(x) = \\mathbb{E}[Y \\mid X = x] \\approx \\frac{1}{k} \\sum_{x_i \\in N_k(x)} y_i$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">where $N_k(x)$ is the set of $k$ training points closest to $x$.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This approximation relies on two hopes:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Averaging<\/strong> approximates expectation (law of large numbers)<\/li>\n\n\n\n<li><strong>Nearby points<\/strong> $x_i \\approx x$, so conditioning on $X = x_i$ \u2248 conditioning on $X = x$<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Both hopes fail in high dimensions (the curse of dimensionality) \u2014 but that&#8217;s ESL Section 2.5.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"291\" src=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-4-1024x291.png\" alt=\"\" class=\"wp-image-996\" srcset=\"https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-4-1024x291.png 1024w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-4-300x85.png 300w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-4-768x218.png 768w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-4-1536x437.png 1536w, https:\/\/tensorzen.online\/wp-content\/uploads\/2026\/06\/image-4-2048x582.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<pre class=\"wp-block-preformatted\">k=1: low bias, high variance (wiggly)\nk=30: high bias, low variance (smooth but misses local structure)\nThe true E[Y|X=x] = sin(x) is what all k choices try to approximate.<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Source: Elements of Statistical Learning, Section 2.4 This notebook walks through the theoretical framework for supervised learning: why squared error loss is natural, what EPE is, and why [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,25,1],"tags":[],"class_list":["post-988","post","type-post","status-publish","format-standard","hentry","category-base","category-in-english","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/tensorzen.online\/index.php?rest_route=\/wp\/v2\/posts\/988","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tensorzen.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tensorzen.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tensorzen.online\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tensorzen.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=988"}],"version-history":[{"count":4,"href":"https:\/\/tensorzen.online\/index.php?rest_route=\/wp\/v2\/posts\/988\/revisions"}],"predecessor-version":[{"id":997,"href":"https:\/\/tensorzen.online\/index.php?rest_route=\/wp\/v2\/posts\/988\/revisions\/997"}],"wp:attachment":[{"href":"https:\/\/tensorzen.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=988"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tensorzen.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=988"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tensorzen.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=988"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}