https://yahel1216.github.io/tags/gradient-descent/Recent content in Gradient-Descent on Yahel UffenheimerHugoen-USSun, 11 Jan 2026 00:00:00 +0000https://yahel1216.github.io/posts/conj-gradient/Sun, 11 Jan 2026 00:00:00 +0000https://yahel1216.github.io/posts/conj-gradient/<div class="alert alert-info my-6 overflow-hidden rounded-lg transition-all duration-200 ease-out hover:-translate-y-0.5 hover:shadow-md" style="background-color: color-mix(in srgb, var(--color-) 10%, transparent); border-left-color: var(--color-); --hover-bg: color-mix(in srgb, var(--color-) 15%, transparent);" onmouseover="this.style.backgroundColor = this.style.getPropertyValue('--hover-bg')" onmouseout="this.style.backgroundColor = 'color-mix(in srgb, var(--color-) 10%, transparent)'" role="alert" aria-labelledby="alert-0-title"> <div class=" flex items-center justify-between px-6 py-6" > <div class="flex items-center gap-3"> <h4 id="alert-0-title" class="m-0 font-semibold text-foreground/90"> Prerequisites </h4> </div> </div> <div id="alert-0-content" class="alert-content px-6 pb-6"> <div class="prose prose-sm text-foreground/90 max-w-none"> <ul> <li><strong>Linear Algebra:</strong> Eigenvalues, eigenvectors, positive (semi-)definite (PSD) matrices, and the notion of orthogonality.</li> <li><strong>Calculus:</strong> Gradients and basic convexity.</li> <li><strong>Optimization:</strong> Gradient Descent (GD) basics.</li> </ul> </div> </div> </div><script> function toggleAlert(alertId) { const content = document.getElementById(alertId + '-content'); const chevron = document.getElementById(alertId + '-chevron'); const header = content.previousElementSibling; if (content.classList.contains('hidden')) { content.classList.remove('hidden'); chevron.style.transform = 'rotate(0deg)'; header.setAttribute('aria-expanded', 'true'); } else { content.classList.add('hidden'); chevron.style.transform = 'rotate(-90deg)'; header.setAttribute('aria-expanded', 'false'); } } document.addEventListener('DOMContentLoaded', function() { const collapsedAlerts = document.querySelectorAll('.alert-content.hidden'); collapsedAlerts.forEach(function(content) { const alertId = content.id.replace('-content', ''); const chevron = document.getElementById(alertId + '-chevron'); if (chevron) { chevron.style.transform = 'rotate(-90deg)'; } }); }); </script> <p>One of the most common tasks in numerical algorithms is to solve a linear equation—that is, find $x$ for which $$Ax=b$$ for a given matrix $A$ and vector $b$. This can be solved via Gaussian elimination, which generally has a high runtime ($O(n^3)$). We will show how to improve upon this using optimization ideas. This is one instance of a problem for which we can find an <strong>approximate</strong> solution much faster using <strong>calculus</strong> tools, compared with using a close-form exact algebraic solution.</p>