Andrej Karpathy Let's reproduce GPT-2 (124M)

🎯 Загружено автоматически через бота: 🚫 Оригинал видео: 📺 Данное видео принадлежит каналу «Andrej Karpathy» (@AndrejKarpathy). Оно представлено в нашем сообществе исключительно в информационных, научных, образовательных или культурных целях. Наше сообщество не утверждает никаких прав на данное видео. Пожалуйста, поддержите автора, посетив его оригинальный канал. ✉️ Если у вас есть претензии к авторским правам на данное видео, пожалуйста, свяжитесь с нами по почте support@, и мы немедленно удалим его. 📃 Оригинальное описание: We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links: build-nanogpt GitHub repo, with all the changes in this video as individual commits: nanoGPT repo: llm.c repo: my website: my twitter: our Discord channel: Supplementary links: Attention is All You Need paper: OpenAI GPT-3 paper: - OpenAI GPT-2 paper: The GPU I’m training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: Chapters: intro: Let’s reproduce GPT-2 (124M) exploring the GPT-2 (124M) OpenAI checkpoint SECTION 1: implementing the GPT-2 loading the huggingface/GPT-2 parameters implementing the forward pass to get logits sampling init, prefix tokens, tokenization sampling loop sample, auto-detect the device let’s train: data batches (B,T) → logits (B,T,C) cross entropy loss optimization loop: overfit a single batch data loader lite parameter sharing wte and lm_head model initialization: std , residual init SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms Tensor Cores, timing the code, TF32 precision, 333ms float16, gradient scalers, bfloat16, 300ms , Python overhead, kernel fusion, 130ms flash attention, 96ms nice/ugly numbers. vocab size 50257 → 50304, 93ms SECTION 3: hyperpamaters, AdamW, gradient clipping learning rate scheduler: warmup cosine decay batch size schedule, weight decay, FusedAdamW, 90ms gradient accumulation distributed data parallel (DDP) datasets used in GPT-2, GPT-3, FineWeb (EDU) validation data split, validation loss, sampling revive evaluation: HellaSwag, starting the run SECTION 4: results in the morning! GPT-2, GPT-3 repro shoutout to llm.c, equivalent but faster code in raw C/CUDA summary, phew, build-nanogpt github repo Corrections: I will post all errata and followups to the build-nanogpt GitH