Language models (LMs) have become ubiquitous in both AI research and commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the significance of these details in scientifically studying these models, including their biases and potential risks, I argue that it is essential for the research community to have access to powerful, truly open LMs.  In this talk, I present our OLMo project aimed at building strong language models and making them fully accessible to researchers along with open-source code for data, training, and inference. I describe our efforts in building language modeling from scratch, expanding their scope to make them applicable and useful for real-world applications, and investigating a new generation of LMs that address fundamental challenges inherent in current models.