In this blog post, I’ll give a bite-sized summary of our paper Learning to Plan for Language Modeling from Unlabeled Data, which was accepted at COLM 2024 🥳. Problem Setting Language Models (LMs) have revolutionized various NLP tasks, largely due to their self-supervised pretraining objectives that scale effectively with data. However, traditional LMs predict the next word in a sequence purely left-to-right, which contrasts with how humans approach writing complex texts.

This blog post will give a less technical explanation of the main ideas from my (first!) paper Critical Analysis of Deconfounded Pretraining to Improve Visio-Linguistic Models. Let’s dive right into it. The goal: out-of-distribution performance Machine learning has improved a lot in its ability to learn an accurate predictive model of various kinds of data. Specifically for data that is visual (in the form of pixels) or linguistic (in the form of natural language) great strides have been made.