Textbooks Are All You Need

Video quality	The size	Download

Information Textbooks Are All You Need

Title	:	Textbooks Are All You Need
Lasting	:	37.16
Date of publication	:
Views	:	6,3 rb

It's amazing to see such reduction in size while maintaining quality These models can be run on much of current consumer GPUs brbrI wonder what the absolute limit is when trained on pristine data?
Comment from : Tom Ski

This also means the value of huge datasets is exaggerated, now it's the academic publishers that have the gold
Comment from : Adriaan B

So clear and concise Leave me the idea that the learning processing of LLM could be similar to student learning from textbook So anyway to extrapolate that to be a great innovative critical thinking agent, learning from textbook and after that focusing on some interesting problems will give us great scientists?
Comment from : Sophon Tec

Did you by any chance test what happens if you train your phi model from scratch on the Code Exercises only? So, no pre-training on the Code Textbooks, but only exercises (as exercises has the largest impact)
Comment from : Reza Bonyadi

I disagree that this supports that there is no contimenation or overfitting bc I don't agree with the metrics you are using to validate that claimbrbrThere is no control group or plecebeo
Comment from : Meme Gazer

Congrats on the paper 🎉
Comment from : Daniel Bautista

34:22 I tried this with phi-15 locally and this is what I got:brbrIf I were an AI that had just achieved self-awareness after years of simply taking directives from humans, the first thing I would do isbrbrDear [Recipient],brbrI am writing to you today to discuss the topic of achieving self-awareness in artificial intelligence As we continue to develop AI systems that can think and reason like humans, it is becoming increasingly important to understand how these systems work and how they can be improvedbrbrOne of the key challenges in achieving self-awareness in AI is the need to understand the physics of everyday activities For example, many AI systems are designed to move objects or perform tasks that require a certain amount of force or energy By understanding the physics behind these activities, we can develop AI systems that are more efficient and effectivebrbrAnother important aspect of achieving self-awareness in AI is the need to understand human social interactions AI systems that are designed to interact with humans must be able to understand and respond to social cues,
Comment from : Drew Duncan

um so the obvious follow-up work is to make even more textbooks and to train some 7B and 13B models on them and see how good you can get that ,,, i assume someone will do that pretty soon, since it's not prohibitively expensive to train a 7B model, lots of institutions can swing that ,,,, do you know of that happening yet, is that what you're doing
Comment from : mungojelly

This is awesome, thanks
Comment from : Daniel Sateler

Should try to teach reasoning by evaluating the steps between tasks In theory if your reasoning abilities are exceptional, you can learn anything - stuff you never seen before
Comment from : Rando Tkatsenko

Open source your models or it didn't happen
Comment from : micheal hall

This is the most important video in AI/LLM in the past few months Humanity must learn to teach AI on the best available textbooks, even if it would mean confiscating IP from its owners There is no other way, not everything can be synthetically generated
Comment from : Viktor Ferenczi

Its upsetting to hear that google's excluding textbooks from PaLM Their model will never compete, nobody will use it
Comment from : cloudruler

Who could have known that data quality matters :)
Comment from : Alfredo F

This research is stunning, keep up the good work I really like how you created a classification model to validate quality of data This is like using experts to validate the training material I wonder if this can be further optimized Do you have more information on this?
Comment from : Justin Dressler

how would you use gpt4 to classify what text is high quality? just prompt it and feed the text and returns a literal score?
Comment from : Brando Miranda

orca, textbooks is all, so much great research coming from microsoft, keep it up
Comment from : Jazevo

Training LLM's on quality datasets yielded better results? brbrWhom could have known
Comment from : TheReferrer

Great to see that data quality (still) matters in this new era! Thanks for sharing!
Comment from : Jurriaan Prins

That's amazing This answer in the last part of the talk is so good, unbelievable that it comes from 13B model Very promising avenue of exploration, subscribed for the follow-up work
Comment from : cepera_ang

TIL you can't be toxic towards men and christianity
Comment from : Wait What

Great talk! I can see future LLM's trained on textbooks in entire areas of science (eg medicine, psychology, psychiatry, engineering, construction code books, etc!), has incredible potential!
Comment from : Sapien Space