Posts on Yahel Uffenheimer

Clarkson's Algorithm for Linear Programming

Tue, 31 Mar 2026 00:00:00 +0000

Clarkson’s Algorithm for Linear Programming

Prerequisites

Linear Programming: Feasibility, optimality, and the standard form of an LP.
Linear Algebra: Rank, linear independence, matrix inverses, and spanning sets.
Probability: Expectation and Markov’s inequality.

Linear programs arise throughout combinatorial optimization, machine learning, and operations research. In many practical settings the number of constraints $n$ is enormous compared to the ambient dimension $d$ — think of $n = 10^6$ constraints in $d = 50$ dimensions. Standard algorithms like the simplex method take time proportional to $n$ per pivot, and interior-point methods scale as $O(n^{3.5})$ in the worst case. When $n \gg d$, most constraints are redundant: the optimal solution is determined by only $d$ of them.

The Nystrom Method: Spectral Action

Tue, 20 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigen-decompositions, positive definite matrices, rank.
Functional Analysis: Hilbert spaces, $L^2$ spaces, operators.
Kernel Methods: Previous posts in the series.

Continuing our series on kernel methods, recall that these methods allow us to operate in high-dimensional spaces using the “kernel trick”. However, they suffer from a major computational bottleneck: constructing and manipulating the Gram matrix requires $O(N^2)$ memory and $O(N^3)$ time for operations like inversion or eigen decomposition, where $N$ is the dataset size. When $N$ reaches hundreds of thousands, exact computation becomes infeasible. In this post, I will explore the Nystrom method, a powerful technique for constructing low-rank approximations of these matrices. Rather than treating it merely as a linear algebra heuristic, I want to derive it from first principles: starting with the spectral properties of integral operators on Hilbert spaces and showing how the discretization of these operators naturally leads to the matrix approximation formulas we use in practice.

Algebraic Techniques for Fast Integer Multiplication

Sun, 18 Jan 2026 00:00:00 +0000

Prerequisites

Ring Theory: Homomorphisms, Ideals, Quotient Rings, Chinese Remainder Theorem (see Appendix for definitions).
Polynomials: Polynomial multiplication, division, convolutions.
Algorithm Analysis: Basic recursive complexity and $O$-notation.

Given two integers $a,b\in \mathbb{N}$, how fast can we multiply them?

Fast Fourier Transform over Finite Fields

Sun, 18 Jan 2026 00:00:00 +0000

Prerequisites

Ring Theory: Basic definitions (Rings, Fields, Polynomials).
Group Theory: Cyclic groups and generators.
Basic FFT: Familiarity with the standard divide-and-conquer strategy on $\mathbb{C}$ is helpful but not required.

The Fast Fourier Transform (FFT) is one of the most important algorithms in history. Usually, it is introduced over the field of complex numbers $\mathbb{C}$, relying on geometric intuition about roots of unity lying on the unit circle.

Fast Matrix Multiplication - Part 4

Fri, 16 Jan 2026 00:00:00 +0000

Prerequisites

Part 3: Familiarity with Tensor Rank, Direct Sums, and Tensor Products.
Combinatorics: Basic understanding of the Multinomial Theorem.
Asymptotic Analysis: Limits and roots of polynomials.

In this post, we will formalize the tools needed to compare different tensor algorithms and prove the powerful $\tau$-Theorem (also known as the Asymptotic Sum Inequality). This theorem is the engine behind many modern improvements in the exponent of matrix multiplication, allowing us to derive bounds on $\omega$ from sums of disparate tensors.

Fast Matrix Multiplication - Part 3

Thu, 15 Jan 2026 00:00:00 +0000

Prerequisites

Part 2 of this series: Familiarity with the tensor product of vector spaces and bilinear algorithms.
Linear Algebra: Vector spaces, Bases, Dual spaces, and Tensor Products (basic definition).

In this post, we continue building the foundation for fast matrix multiplication algorithms. We will discuss essential tensor operations—product, sum, and restriction—and establish the Triple Product Condition, setting the stage for the group-theoretic approach.

Fast Matrix Multiplication - Part 2

Wed, 14 Jan 2026 00:00:00 +0000

Prerequisites

Part 1 of this series: Familiarity with standard Matrix Multiplication (MM) and $\omega$.
Linear Algebra: Vector spaces, Bases, Dual spaces, and Tensor Products (basic definition).
Abstract Algebra: Fields and Polynomial rings (helpful for the tensor intuition).

In the previous post, we discussed Strassen’s algorithm and the definition of the exponent $\omega$. We demonstrated that the complexity of matrix multiplication is dominated by the number of multiplications in the base algorithm, assuming a recursive approach.

Fast Matrix Multiplication - Part 1

Tue, 13 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Basic definitions (matrix multiplication, inner products, block matrices).
Asymptotic Notation: Big-O notation and the Master Theorem.
No prior knowledge of Group Theory or Representation Theory is required for this post.

In this series of posts, we will discuss a fascinating line of work that aims to use group theory and representation theory to design fast algorithms for matrix multiplication. We will start with the basics of matrix multiplication algorithms and build up to the group theoretic approach. This post is intended for readers without any prior knowledge of groups or representation theory, or of matrix multiplication algorithms. My goal is to build an intuitive understanding of the problem space before we dive deeper in future posts.

Approximating the Cut-Norm - Part 1

Mon, 12 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, PSD matrices, Tensor products.
Convex Optimization: Basic familiarity with Semidefinite Programming (SDP).
Probability: Expectations, Markov’s inequality.
Graph Theory: Basic definitions, Cuts, Regularity.

Consider the following problem: Given an undirected graph $G=(V,E)$, let $A,B\subset V$ denote non-empty disjoint sets. Let $E(A,B)$ denote the set of edges in $E$ that cross from $A$ to $B$. Denote $D(A,B)=\frac{\left|E(A,B)\right|}{\left|A\right|\left|B\right|}$ to be the density of edges crossing from $A$ to $B$.

Approximating the Cut-Norm - Part 2

Mon, 12 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, PSD matrices, Tensor products.
Convex Optimization: Basic familiarity with Semidefinite Programming (SDP).
Probability: Expectations, Markov’s inequality.
Graph Theory: Basic definitions, Cuts, Regularity.

We are interested in computing the cut norm of a matrix, defined by $$\|A\|_C=\max_{I\subset R,J\subset S}\left|\sum_{i\in I,j\in J}a_{i,j}\right|$$ where $A=(a_{i,j})_{i\in R,j\in S}$. We’ve seen this is a hard problem, and it is often equivalent to computing the $\infty\mapsto 1$ norm, defined by $$\|A\|_{\infty\mapsto 1}=\max_{x\in \set{\pm 1}^R, y\in \set{\pm1}^S} \sum_{i\in R,j\in S} a_{i,j}\cdot x_i\cdot y_j$$ We’ve already seen that computing the latter norm can be done by solving an integer quadratic program, which has a relaxation to a quadratically constrained quadratic program given by $$\max \sum_{i,j}a_{i,j}\cdot \langle u_i, v_j\rangle \quad\text{subject to}\quad \|u_i \|^2 = \|v_j\|^2=1$$ where the optimization is over vectors $u_1,\ldots,u_n$ and $v_1,\ldots ,v_m$. We’ve seen that this problem can be solved using semi-definite programming, and we’ve seen one method to round the solution, giving an approximation factor of $\approx 0.03$. In this post, we’ll see another method, which is much cleaner, and uses randomized rounding of this SDP.

The Conjugate Gradient Method for Linear Equations

Sun, 11 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigenvalues, eigenvectors, positive (semi-)definite (PSD) matrices, and the notion of orthogonality.
Calculus: Gradients and basic convexity.
Optimization: Gradient Descent (GD) basics.

One of the most common tasks in numerical algorithms is to solve a linear equation—that is, find $x$ for which $$Ax=b$$ for a given matrix $A$ and vector $b$. This can be solved via Gaussian elimination, which generally has a high runtime ($O(n^3)$). We will show how to improve upon this using optimization ideas. This is one instance of a problem for which we can find an approximate solution much faster using calculus tools, compared with using a close-form exact algebraic solution.

Basic Matroid Theory

Sat, 10 Jan 2026 00:00:00 +0000

Basic Matroid Theory and Infinite Extensions

Prerequisites:

Linear Algebra: Vector spaces, basis, dimension, and linear independence.
Graph Theory: Graphs, connected components, cycles, and spanning trees.
Set Theory: Basic cardinality and Zorn’s Lemma (for the infinite section).

In this post, we will discuss the basics of Matroid Theory. While often taught merely as a theoretical framework for Greedy Algorithms, matroids are a rich combinatorial structure in their own right. We will start with the finite foundations and then explore how these definitions behave (and break) when extended to the infinite case.

Random Fourier Features

Fri, 09 Jan 2026 00:00:00 +0000

Prerequisites

Analysis: Basic Fourier Analysis (transforms, exponentials).
Probability: Concentration inequalities (Hoeffding), expectation, and Gaussian distributions.
Kernel Methods: Familiarity with the basic kernel trick (see previous post).

In the previous post, we introduced the idea of kernels as a way to lift a separation problem to a much larger space (potentially infinite-dimensional) while keeping the computation tractable via the “Kernel Trick.” We also mentioned that when the number of points in the dataset is very large—which is the case in most modern applications—the kernel method is less useful, as it requires computing and storing a huge $n \times n$ matrix.

Tensor Sketch: Polynomial Kernels

Fri, 09 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Inner products, tensor products, and vectorization.
Probability: Hash functions, independence, and variance.
Algorithms: Fast Fourier Transform (FFT) and basic convolution.
Kernel Methods: Familiarity with the Polynomial Kernel.

In previous posts, we discussed the Radial Basis Function (RBF) kernel and how to approximate it using Random Fourier Features. Today, we turn our attention to another fundamental kernel—the Polynomial Kernel—and a powerful algebraic technique to approximate it called Tensor Sketching.

The Kernel Method

Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Inner product spaces, positive definite matrices, spectral decomposition, projections.

In this post, we will explore the idea of kernels in machine learning. In future posts, we will explore different ways to approximate specific kernel computations. Approximation is useful for big data applications due to the prohibitively high cost of exact kernel computations.