November 08, 2024, Henley Hall

Machine Learning has progressed significantly as an empirical discipline, accelerated in recent years by the advent of Large Language Models (LLMs), including so-called World Models. The tools to interpret and guide such progress have been rooted in the theory of inductive learning, based on concepts of generalization and regularization. These tools are ill-fit to describe LLMs which are instead (inductively-trained) transductive inference engines. Transductive inference requires memorization and test-time computation, and has as its limit Solomonoff Induction, which is optimal with respect to any physically realizable (computable) process. Rudimentary forms of transductive inference, known as in-context learning, emerge when training sequence predictors at scale. As we keep scaling these models up, are they inching towards Solomonoff Induction and becoming unbeatable? Are they developing “superintelligence”? Will we lose our ability to control them? Have we already? What does that even mean? Can LLMs even represent and manipulate abstract concepts and “meanings”? To begin addressing some of these questions, I will review some basic ideas from Stochastic Realization Theory from the 1970s, and describe how current Foundation Models can be understood within that framework. I will then show how the ability to represent abstract concepts and meanings emerges in these models, and discuss whether and how they can be observed and controlled.

Stefano Soatto is a Professor of Computer Science at UCLA and Vice President of Applied Science and Distinguished Scientist at AWS, where he has led the global science teams that built AI Application Services in the areas of vision (e.g., Rekognition, Textract), speech (e.g., Transcribe), language (e.g., Lex, Transcribe, Comprehend), verticals (e.g., Forecast, Personalize), and most recently foundation models (Bedrock, Q).