Statistical and Mathematical Foundations for Human Preference Alignment in LLMs

Jiancong Xiao is a tenure-track Assistant Professor in the Department of Computer Science, School of Computing, at the National University of Singapore (NUS). Prior to joining NUS, he was a postdoctoral researcher at the University of Pennsylvania, working with Prof. Qi Long and Prof. Weijie Su. He received his Ph.D. from The Chinese University of Hong Kong, Shenzhen, advised by Prof. Zhi-Quan (Tom) Luo. His research lies at the intersection of learning theory, statistics, and optimization, with a focus on building theoretical foundations for responsible and trustworthy machine learning models. His recent work studies mathematical and statistical foundations of large language models, including preference alignment, fine-tuning and post-training, calibration, algorithmic bias. Previously, he worked on adversarial robustness, and robust generalization. His research has been published in leading journals, including JMLR, the Annals of Statistics, and JASA, as well as top machine learning conferences, including COLT, NeurIPS, ICML, and ICLR.

Aligning LLMs with diverse human preferences is essential for value alignment and for ensuring that AI systems interact responsibly with users from different backgrounds. However, we argue that the predominant reward-model-based approach, reinforcement learning from human feedback (RLHF), suffers from an inherent algorithmic bias. In extreme cases, this bias can cause preference collapse, where minority preferences are effectively ignored. To address this, we introduce preference matching, a framework that provably aligns LLMs with diverse preferences under the Bradley–Terry (BT) model. Beyond the BT setting, we study fundamental statistical limits of preference alignment. We show that human preferences can be represented by a reward model if and only if they contain no Condorcet cycle. Yet under a probabilistic preference model, Condorcet cycles occur with probability approaching one exponentially fast, demonstrating an inherent impossibility in fully aligning preferences through reward-based methods such as RLHF. We then examine when non-reward-based approaches such as Nash learning from human feedback (NLHF) produce mixed strategies rather than collapsing to a single response. We identify a necessary and sufficient condition: the absence of a response that is preferred by a majority over all others. This condition holds with high probability under the probabilistic model, highlighting the statistical possibility of preserving minority preferences without explicit regularization. Our work establishes the fundamental limits of whether large language models can fully align with human preferences.