Markov Categorical Framework

for Language Modeling
A unifying framework using Markov Categories that reveals NLL training as an implicit form of spectral contrastive learning.

Yifan Zhang
ICML 2025 AI4Math Workshop  ·  arXiv:2507.19247
Markov Categories Spectral Contrastive Information Geometry NLL Objective

Abstract

Auto-regressive language models are incredibly powerful, yet a deep theoretical understanding of why the simple negative log-likelihood (NLL) objective works so well remains elusive. This work introduces a unifying framework using Markov Categories to deconstruct the generation process and the NLL objective.

We model the single-step generation map as a composition of Markov kernels, which lets us precisely analyze information flow and the geometry of the learned representation space. Our core finding is that NLL training is an implicit form of spectral contrastive learning: it forces the model's representation space to align with the eigenspectrum of a predictive similarity operator, learning a geometrically structured space without explicit contrastive pairs. This perspective reveals deep structural principles underlying the effectiveness of modern LMs.

Read the Paper Code

Conceptual Overview

Conceptual overview of the Markov Categorical framework.
A conceptual overview of the framework. Center: the AR generation step is modeled as a composition of Markov kernels $k_{\text{gen}} = k_{\text{head}} \circ k_{\text{bb}} \circ k_{\text{emb}}$. Top: the NLL objective implicitly forces the model to learn the data's intrinsic stochasticity and geometric structure — equivalent to spectral contrastive learning. Bottom: the framework endows the representation space $\mathcal{H}$ with an information geometry that explains modern speculative decoding.

Citation

If you find this work useful, please cite:

@article{zhang2025markov,
  title   = {A Markov Categorical Framework for Language Modeling},
  author  = {Zhang, Yifan},
  journal = {arXiv preprint arXiv:2507.19247},
  year    = {2025}
}