Optimization on Yahel Uffenheimer

Clarkson's Algorithm for Linear Programming

Tue, 31 Mar 2026 00:00:00 +0000

Clarkson’s Algorithm for Linear Programming

Prerequisites

Linear Programming: Feasibility, optimality, and the standard form of an LP.
Linear Algebra: Rank, linear independence, matrix inverses, and spanning sets.
Probability: Expectation and Markov’s inequality.

Linear programs arise throughout combinatorial optimization, machine learning, and operations research. In many practical settings the number of constraints $n$ is enormous compared to the ambient dimension $d$ — think of $n = 10^6$ constraints in $d = 50$ dimensions. Standard algorithms like the simplex method take time proportional to $n$ per pivot, and interior-point methods scale as $O(n^{3.5})$ in the worst case. When $n \gg d$, most constraints are redundant: the optimal solution is determined by only $d$ of them.

Approximating the Cut-Norm - Part 1

Mon, 12 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, PSD matrices, Tensor products.
Convex Optimization: Basic familiarity with Semidefinite Programming (SDP).
Probability: Expectations, Markov’s inequality.
Graph Theory: Basic definitions, Cuts, Regularity.

Consider the following problem: Given an undirected graph $G=(V,E)$, let $A,B\subset V$ denote non-empty disjoint sets. Let $E(A,B)$ denote the set of edges in $E$ that cross from $A$ to $B$. Denote $D(A,B)=\frac{\left|E(A,B)\right|}{\left|A\right|\left|B\right|}$ to be the density of edges crossing from $A$ to $B$.

Approximating the Cut-Norm - Part 2

Mon, 12 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, PSD matrices, Tensor products.
Convex Optimization: Basic familiarity with Semidefinite Programming (SDP).
Probability: Expectations, Markov’s inequality.
Graph Theory: Basic definitions, Cuts, Regularity.

We are interested in computing the cut norm of a matrix, defined by $$\|A\|_C=\max_{I\subset R,J\subset S}\left|\sum_{i\in I,j\in J}a_{i,j}\right|$$ where $A=(a_{i,j})_{i\in R,j\in S}$. We’ve seen this is a hard problem, and it is often equivalent to computing the $\infty\mapsto 1$ norm, defined by $$\|A\|_{\infty\mapsto 1}=\max_{x\in \set{\pm 1}^R, y\in \set{\pm1}^S} \sum_{i\in R,j\in S} a_{i,j}\cdot x_i\cdot y_j$$ We’ve already seen that computing the latter norm can be done by solving an integer quadratic program, which has a relaxation to a quadratically constrained quadratic program given by $$\max \sum_{i,j}a_{i,j}\cdot \langle u_i, v_j\rangle \quad\text{subject to}\quad \|u_i \|^2 = \|v_j\|^2=1$$ where the optimization is over vectors $u_1,\ldots,u_n$ and $v_1,\ldots ,v_m$. We’ve seen that this problem can be solved using semi-definite programming, and we’ve seen one method to round the solution, giving an approximation factor of $\approx 0.03$. In this post, we’ll see another method, which is much cleaner, and uses randomized rounding of this SDP.

The Conjugate Gradient Method for Linear Equations

Sun, 11 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, eigenvectors, positive (semi-)definite (PSD) matrices, and the notion of orthogonality.
Calculus: Gradients and basic convexity.
Optimization: Gradient Descent (GD) basics.

One of the most common tasks in numerical algorithms is to solve a linear equation—that is, find $x$ for which $$Ax=b$$ for a given matrix $A$ and vector $b$. This can be solved via Gaussian elimination, which generally has a high runtime ($O(n^3)$). We will show how to improve upon this using optimization ideas. This is one instance of a problem for which we can find an approximate solution much faster using calculus tools, compared with using a close-form exact algebraic solution.