Stochastic Gradient Descent

Iterative stochastic-approximation method for finding the root/optimum of a function from noisy observations (Robbins & Monro, "A Stochastic Approximation Method," Annals of Mathematical Statistics, 1951). Pure statistics with no ML framing at origin -- introduced decades before any machine-learning application existed, and became the algorithmic ancestor of the optimizer used to train virtually all modern neural networks.