interconnects.ai

Articles

GPT-4.5: "Not a frontier model"?

1 month ago | interconnects.ai | Nathan Lambert |Ethan Mollick

As GPT-4.5 was being released, the first material the public got access to was OpenAI’s system card for the model that details some capability evaluations and mostly safety estimates. Before the live stream and official blog post, we knew things were going to be weird because of this line:GPT-4.5 is not a frontier model. The updated system card in the launch blog post does not have this.
Building on evaluation quicksand

Oct 16, 2024 | interconnects.ai | Nathan Lambert

Other than the pace of progress on individual evaluations being extremely high, the landscape of evaluating leading language models has not changed substantially in the last year. The biggest change is the required level of detail in reporting results, where more information must be communicated to contextualize the capabilities of your models. The new types of models that OpenAI's o1 heralds welcome a new axis to this, evaluation-time compute.
A post-training approach to AI regulation with Model Specs

Sep 9, 2024 | interconnects.ai | Nathan Lambert

The central point of AI regulation and policy over the last few years, everything from the Biden Executive Order to California’s SB 1047 bill, has been model size. The most common tool for AI enforcement proportional to model size has been thresholds that kick in once an AI system uses more than a certain amount of compute (or money) to be trained. The use of thresholds for regulation is the subject of substantial pushback and is likely fading in relevance due to it.
On Nous Hermes 3 and classifying a "frontier model"

Aug 16, 2024 | interconnects.ai | Nathan Lambert

Edit 1, 16 August: I made a few errors in this original post, which makes the Nous Research team, and by extension the EluetherAI evaluation harness, look bad. This isn’t my intention. As a summary:The Nous report is clear on their evaluation recipe. The open question of how best to evaluate Llama 3.1 models still holds. I’ve changed the title of the post. I’ve toned down the text in a few places to better reflect this. Sorry for the error.
AI for the rest of us

Jun 14, 2024 | interconnects.ai | Nathan Lambert

By being quiet in the race for the biggest and most rad foundation models, many people assumed that Apple didn’t have anything to contribute to the AI race. After their new announcements, many in the heart of the AI labs still will argue this, forecasting that . Apple is betting that AI follows paths paved by previous technological revolutions: incremental progress to transformational results.
We aren’t running out of training data, we are running out of open training data

May 29, 2024 | interconnects.ai | Nathan Lambert

For months we’ve been getting stories about how the leading teams training language models (LMs) are running out of data for their next generation of models — vaguely insinuating a struggle for big tech’s darling industry with no strategic claims beyond the fact that the second derivative on training dataset size is negative.
ChatBotArena: The peoples’ LLM evaluation, the future of evaluation, the incentives of evaluation, and gpt2chatbot

May 8, 2024 | interconnects.ai | Nathan Lambert

Evaluation is no longer equally accessible to academics and everyone who is involved in language model development. There are obvious reasons we shouldn’t let this happen, from transparency to science to regulation. Broadly, language model evaluation these days largely is reduced to three things:The best academic benchmarks: The massive multitask language understanding (MMLU) benchmark, which is used to test general language model knowledge along with a few static and open benchmarks.
Phi 3 and Arctic: Outlier LMs are hints

Apr 29, 2024 | interconnects.ai | Nathan Lambert

Last week, two very interesting open models were released into the world, Phi 3 from Microsoft and Arctic from Snowflake, and they both stand on their own in training techniques. The Phi series of models is known for taking an opinionated stance on synthetic, high-quality data and Arctic is a new entry that has a very sparse Mixture of Experts (MoE) architecture that makes it useful exclusively for the coding-focused, VRAM-rich inference population.
Stop "reinventing" everything to solve alignment

Apr 17, 2024 | interconnects.ai | Nathan Lambert

Editors note: This week marks 1 year since the rebrand to Interconnects. Things are going great. I have so much to write about, engaged readers, and a few additions to the offerings. The paid member discord is a wonderful small community that helps me spend less time on Twitter!The audio of posts, created via 11Labs to sound just like me, for podcasts or in video on YouTube is becoming increasingly stable and has a solid, smaller audience.
Why we disagree on what open-source AI should be

Apr 3, 2024 | interconnects.ai | Nathan Lambert

We’re so early in the process of defining community norms and standards around open LLMs that the goals of the people involved are not clear at all. Open source means a lot of things to a lot of different people, and this is by design. The guiding principles and rules of open source make it succeed when multiple groups dictate its path.