Machine Learning on Yahel Uffenheimer

The Nystrom Method: Spectral Action

Tue, 20 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Eigen-decompositions, positive definite matrices, rank.
Functional Analysis: Hilbert spaces, $L^2$ spaces, operators.
Kernel Methods: Previous posts in the series.

Continuing our series on kernel methods, recall that these methods allow us to operate in high-dimensional spaces using the “kernel trick”. However, they suffer from a major computational bottleneck: constructing and manipulating the Gram matrix requires $O(N^2)$ memory and $O(N^3)$ time for operations like inversion or eigen decomposition, where $N$ is the dataset size. When $N$ reaches hundreds of thousands, exact computation becomes infeasible. In this post, I will explore the Nystrom method, a powerful technique for constructing low-rank approximations of these matrices. Rather than treating it merely as a linear algebra heuristic, I want to derive it from first principles: starting with the spectral properties of integral operators on Hilbert spaces and showing how the discretization of these operators naturally leads to the matrix approximation formulas we use in practice.

Random Fourier Features

Fri, 09 Jan 2026 00:00:00 +0000

Prerequisites

Analysis: Basic Fourier Analysis (transforms, exponentials).
Probability: Concentration inequalities (Hoeffding), expectation, and Gaussian distributions.
Kernel Methods: Familiarity with the basic kernel trick (see previous post).

In the previous post, we introduced the idea of kernels as a way to lift a separation problem to a much larger space (potentially infinite-dimensional) while keeping the computation tractable via the “Kernel Trick.” We also mentioned that when the number of points in the dataset is very large—which is the case in most modern applications—the kernel method is less useful, as it requires computing and storing a huge $n \times n$ matrix.

Tensor Sketch: Polynomial Kernels

Fri, 09 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Inner products, tensor products, and vectorization.
Probability: Hash functions, independence, and variance.
Algorithms: Fast Fourier Transform (FFT) and basic convolution.
Kernel Methods: Familiarity with the Polynomial Kernel.

In previous posts, we discussed the Radial Basis Function (RBF) kernel and how to approximate it using Random Fourier Features. Today, we turn our attention to another fundamental kernel—the Polynomial Kernel—and a powerful algebraic technique to approximate it called Tensor Sketching.

The Kernel Method

Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites

Linear Algebra: Inner product spaces, positive definite matrices, spectral decomposition, projections.

In this post, we will explore the idea of kernels in machine learning. In future posts, we will explore different ways to approximate specific kernel computations. Approximation is useful for big data applications due to the prohibitively high cost of exact kernel computations.