Introducing AI2 OLMo 2: A Significant Upgrade in Open Language Models
About the Author
By Ryan Daws
Categories:
Ryan Daws is a senior editor at TechForge Media with over a decade of experience in crafting compelling narratives and making complex topics accessible. His articles and interviews with industry leaders have earned him recognition as a key influencer by organisations like Onalytica. Under his leadership, publications have been praised by analyst firms such as Forrester for their excellence and performance. Connect with him on X (@gadget_ry), Bluesky (@gadgetry.bsky.social), and/or Mastodon (@gadgetry@techhub.social)
Ai2 Releases OLMo 2: A Family of Open-Source Language Models
Ai2 is releasing OLMo 2, a family of open-source language models that advances the democratization of AI and narrows the gap between open and proprietary solutions. The new models, available in 7B and 13B parameter versions, are trained on up to 5 trillion tokens and demonstrate performance levels that match or exceed comparable fully open models while remaining competitive with open-weight models such as Llama 3.1 on English academic benchmarks.
"Since the release of the first OLMo in February 2024, we’ve seen rapid growth in the open language model ecosystem, and a narrowing of the performance gap between open and proprietary models," explained Ai2.
The development team achieved these improvements through several innovations, including enhanced training stability measures, staged training approaches, and state-of-the-art post-training methodologies derived from their Tülu 3 framework. Notable technical improvements include the switch from nonparametric layer norm to RMSNorm and the implementation of rotary positional embedding.
OLMo 2 Model Training Breakthrough
The training process employed a sophisticated two-stage approach. The initial stage utilized the OLMo-Mix-1124 dataset of approximately 3.9 trillion tokens, sourced from DCLM, Dolma, Starcoder, and Proof Pile II. The second stage incorporated a carefully curated mixture of high-quality web data and domain-specific content through the Dolmino-Mix-1124 dataset.
Particularly noteworthy is the OLMo 2-Instruct-13B variant, which is the most capable model in the series. The model demonstrates superior performance compared to Qwen 2.5 14B instruct, Tülu 3 8B, and Llama 3.1 8B instruct models across various benchmarks.
(Credit: Ai2)
Committing to Open Science
Reinforcing its commitment to open science, Ai2 has released comprehensive documentation including weights, data, code, recipes, intermediate checkpoints, and instruction-tuned models. This transparency allows for full inspection and reproduction of results by the wider AI community.
The release also introduces an evaluation framework called OLMES (Open Language Modeling Evaluation System), comprising 20 benchmarks designed to assess core capabilities such as knowledge recall, commonsense reasoning, and mathematical reasoning.
OLMo 2 Raises the Bar in Open-Source AI Development
OLMo 2 raises the bar in open-source AI development, potentially accelerating the pace of innovation in the field while maintaining transparency and accessibility.
See Also:
- OpenAI Enhances AI Safety with New Red Teaming Methods
- Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is co-located with other leading events including:
Explore other upcoming enterprise technology events and webinars powered by TechForge here.
Tags:
View Comments:
Leave a comment and let us know what you think about OLMo 2.