Optimization on Yahel Uffenheimer

Clarkson's Algorithm for Linear Programming

Tue, 31 Mar 2026 00:00:00 +0000

Clarkson’s Algorithm for Linear Programming

Prerequisites

Linear Programming: Feasibility, optimality, and the standard form of an LP.
Linear Algebra: Rank, linear independence, matrix inverses, and spanning sets.
Probability: Expectation and Markov’s inequality.

Linear programs arise throughout combinatorial optimization, machine learning, and operations research. In many practical settings the number of constraints $n$ is enormous compared to the ambient dimension $d$ — think of $n = 10^6$ constraints in $d = 50$ dimensions. Standard algorithms like the simplex method take time proportional to $n$ per pivot, and interior-point methods scale as $O(n^{3.5})$ in the worst case. When $n \gg d$, most constraints are redundant: the optimal solution is determined by only $d$ of them.

The Nystrom Method: Spectral Action

Tue, 20 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigen-decompositions, positive definite matrices, rank.
Functional Analysis: Hilbert spaces, $L^2$ spaces, operators.
Kernel Methods: Previous posts in the series.

Continuing our series on kernel methods, recall that these methods allow us to operate in high-dimensional spaces using the “kernel trick”. However, they suffer from a major computational bottleneck: constructing and manipulating the Gram matrix requires $O(N^2)$ memory and $O(N^3)$ time for operations like inversion or eigen decomposition, where $N$ is the dataset size. When $N$ reaches hundreds of thousands, exact computation becomes infeasible. In this post, I will explore the Nystrom method, a powerful technique for constructing low-rank approximations of these matrices. Rather than treating it merely as a linear algebra heuristic, I want to derive it from first principles: starting with the spectral properties of integral operators on Hilbert spaces and showing how the discretization of these operators naturally leads to the matrix approximation formulas we use in practice.

Approximating the Cut-Norm - Part 1

Mon, 12 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, PSD matrices, Tensor products.
Convex Optimization: Basic familiarity with Semidefinite Programming (SDP).
Probability: Expectations, Markov’s inequality.
Graph Theory: Basic definitions, Cuts, Regularity.

Consider the following problem: Given an undirected graph $G=(V,E)$, let $A,B\subset V$ denote non-empty disjoint sets. Let $E(A,B)$ denote the set of edges in $E$ that cross from $A$ to $B$. Denote $D(A,B)=\frac{\left|E(A,B)\right|}{\left|A\right|\left|B\right|}$ to be the density of edges crossing from $A$ to $B$.

Approximating the Cut-Norm - Part 2

Mon, 12 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, PSD matrices, Tensor products.
Convex Optimization: Basic familiarity with Semidefinite Programming (SDP).
Probability: Expectations, Markov’s inequality.
Graph Theory: Basic definitions, Cuts, Regularity.

We are interested in computing the cut norm of a matrix, defined by $$\|A\|_C=\max_{I\subset R,J\subset S}\left|\sum_{i\in I,j\in J}a_{i,j}\right|$$ where $A=(a_{i,j})_{i\in R,j\in S}$. We’ve seen this is a hard problem, and it is often equivalent to computing the $\infty\mapsto 1$ norm, defined by $$\|A\|_{\infty\mapsto 1}=\max_{x\in \set{\pm 1}^R, y\in \set{\pm1}^S} \sum_{i\in R,j\in S} a_{i,j}\cdot x_i\cdot y_j$$ We’ve already seen that computing the latter norm can be done by solving an integer quadratic program, which has a relaxation to a quadratically constrained quadratic program given by $$\max \sum_{i,j}a_{i,j}\cdot \langle u_i, v_j\rangle \quad\text{subject to}\quad \|u_i \|^2 = \|v_j\|^2=1$$ where the optimization is over vectors $u_1,\ldots,u_n$ and $v_1,\ldots ,v_m$. We’ve seen that this problem can be solved using semi-definite programming, and we’ve seen one method to round the solution, giving an approximation factor of $\approx 0.03$. In this post, we’ll see another method, which is much cleaner, and uses randomized rounding of this SDP.

The Conjugate Gradient Method for Linear Equations

Sun, 11 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, eigenvectors, positive (semi-)definite (PSD) matrices, and the notion of orthogonality.
Calculus: Gradients and basic convexity.
Optimization: Gradient Descent (GD) basics.

One of the most common tasks in numerical algorithms is to solve a linear equation—that is, find $x$ for which $$Ax=b$$ for a given matrix $A$ and vector $b$. This can be solved via Gaussian elimination, which generally has a high runtime ($O(n^3)$). We will show how to improve upon this using optimization ideas. This is one instance of a problem for which we can find an approximate solution much faster using calculus tools, compared with using a close-form exact algebraic solution.

Random Fourier Features

Fri, 09 Jan 2026 00:00:00 +0000

Prerequisites

Analysis: Basic Fourier Analysis (transforms, exponentials).
Probability: Concentration inequalities (Hoeffding), expectation, and Gaussian distributions.
Kernel Methods: Familiarity with the basic kernel trick (see previous post).

In the previous post, we introduced the idea of kernels as a way to lift a separation problem to a much larger space (potentially infinite-dimensional) while keeping the computation tractable via the “Kernel Trick.” We also mentioned that when the number of points in the dataset is very large—which is the case in most modern applications—the kernel method is less useful, as it requires computing and storing a huge $n \times n$ matrix.

Tensor Sketch: Polynomial Kernels

Fri, 09 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Inner products, tensor products, and vectorization.
Probability: Hash functions, independence, and variance.
Algorithms: Fast Fourier Transform (FFT) and basic convolution.
Kernel Methods: Familiarity with the Polynomial Kernel.

In previous posts, we discussed the Radial Basis Function (RBF) kernel and how to approximate it using Random Fourier Features. Today, we turn our attention to another fundamental kernel—the Polynomial Kernel—and a powerful algebraic technique to approximate it called Tensor Sketching.

The Kernel Method

Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Inner product spaces, positive definite matrices, spectral decomposition, projections.

In this post, we will explore the idea of kernels in machine learning. In future posts, we will explore different ways to approximate specific kernel computations. Approximation is useful for big data applications due to the prohibitively high cost of exact kernel computations.