Pre-trained Large Language Models Learn <em>Hidden</em> Markov Models In-context

Abstract

Hidden Markov Models (HMMs) are foundational tools for modeling sequential data with latent Markovian structure, yet fitting them to real-world data remains computationally challenging.

In this work, we show that pre-trained large language models (LLMs) can effectively model data generated by HMMs via in-context learning (ICL)—their ability to infer patterns from examples within a prompt.

On a diverse set of synthetic HMMs, LLMs achieve predictive accuracy approaching the theoretical optimum. We uncover novel scaling trends influenced by HMM properties, and offer theoretical conjectures for these empirical observations.

We also provide practical guidelines for scientists on using ICL as a diagnostic tool for complex data. On real-world animal decision-making tasks, ICL achieves competitive performance with models designed by human experts.

To our knowledge, this is the first demonstration that ICL can learn and predict HMM-generated sequences—an advance that deepens our understanding of in-context learning in LLMs and establishes its potential as a powerful tool for uncovering hidden structure in complex scientific data.

Convergence to theoretical optimum

We conduct systematic, controlled experiments on synthetic HMMs and empirically show that pre-trained LLMs outperform traditional statistical methods such as Baum–Welch. Moreover, their prediction accuracy consistently converges to the theoretical optimum—as given by the Viterbi algorithm with ground-truth model parameters—across a wide range of HMM configurations.

Scaling trends of in-context learning on HMMs

We varied properties of HMMs, such as mixing rate and entropy.

We identify and characterize empirical scaling trends showing that LLM performance improves with longer context windows, and that these trends are shaped by fundamental HMM properties such as mixing rate and entropy. We further provide theoretical conjectures to explain these phenomena, drawing connections to—and highlighting distinctions from—classical HMM learning paradigms, including spectral methods. These findings offer important insights into the learnability of stochastic systems through in-context learning.

Connection to real world

What's the connection between in-context learning and real-world HMMs? We translate our findings into practical guidelines for scientists, demonstrating how LLM in-context learning can serve as a diagnostic tool for assessing data complexity and uncovering underlying structure.

We tried two real-world tasks: animal decision-making and animal reward-learning.

We found that LLM ICL can perform competitively (under conditions found in the scaling trends) with domain-specific models developed by human experts.

Citation

@misc{dai2025pretrainedlargelanguagemodels,
      title={Pre-trained Large Language Models Learn Hidden Markov Models In-context}, 
      author={Yijia Dai and Zhaolin Gao and Yahya Satter and Sarah Dean and Jennifer J. Sun},
      year={2025},
      eprint={2506.07298},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2506.07298}, 
}