Projects & Thoughts

State Space Models & Mamba

A deep dive deconstructing State Space Models (SSMs). Rooted in Control Theory, SSMs offer linear scaling and infinite context potential compared to traditional Transformers. This series covers the fundamental math, writing a 1D State Space layer in PyTorch, and scaling it against baseline Transformers.

Kolmogorov-Arnold Networks (KANs)

An exploration of Kolmogorov-Arnold Networks (KANs), replacing fixed node activations with learnable B-splines on the edges. This series covers the mathematics of the Kolmogorov-Arnold representation theorem, a pure PyTorch implementation, and benchmarks scaling KANs versus MLPs for symbolic regression.