Abstract: How should we act under incomplete information? We must integrate over missing values to assess the risk of a downstream decision, or the expected risk reduction from gathering further information. Applications include NLP annotation, LLM workflows, clinical medicine, experimental design, and many others. The classical approach to predicting missing values involves fitting some generative model to the incomplete data. Alas, classical methods are slow and often inaccurate due to local optima, poor mixing, or model misspecification. We propose a low-fuss, efficient alternative with fewer assumptions: given incomplete data, train a neural network to directly predict masked values. Our generic network architecture resembles BERT: it constructs a representation for each random variable's posterior marginal distribution and iteratively refines it through attention on other random variables and entities. We find that this posterior marginal Transformer ("Marformer") works much better than EM and almost as well as MCMC even when those methods know the true data generating model, and it outperforms them on naturally occurring data. We outline future methods for adaptive masking and a future design for managing data collection efforts.
Bio: Jason Eisner is Professor of Computer Science at Johns Hopkins University and a Fellow of the Association for Computational Linguistics. At Johns Hopkins, he is also affiliated with the Center for Language and Speech Processing, the Mathematical Institute for Data Science, the Cognitive Science Department, and the Data Science and AI Institute. His goal is to develop the probabilistic modeling, inference, and learning techniques needed for a unified model of all kinds of linguistic structure, and to connect existing models (such as LLMs) to commonsense reasoning, formal reasoning, and downstream applications. His 180+ papers have presented various algorithms for parsing, machine translation, and weighted finite-state machines; formalizations, algorithms, theorems, and empirical results in computational phonology; unsupervised or semi-supervised learning methods for syntax, morphology, and word-sense disambiguation; and principled methods for conversational AI, including neural language modeling and semantic parsing. From 2019-2024 he was Director of Research at Microsoft Semantic Machines, which developed new approaches to conversational AI. He is also the lead designer of Dyna, a declarative programming language that provides an infrastructure for AI algorithms. He has received 3 school-wide awards for excellence in teaching, most recently in 2025, as well as recent
For more information, or for CODA guest access, please contact shatcher8@gatech.edu at least 2 business days prior to the event.
]]>