Discussions

• 148 You must be logged in to vote

  💬

  Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20

  notable

  karpathy started May 28, 2024 in General

  

  

  

  

  127
• 88 You must be logged in to vote

  💬

  Let's reproduce GPT-2 (1.6B): one 8XH100 node, 24 hours, $672, in llm.c

  karpathy started Jul 11, 2024 in General

  

  

  

  

  33
• 2 You must be logged in to vote

  💬

  Is there a open continuation of this for gpt 3.5/4/ llama3.1/o1 ?

  manuvector started Dec 21, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  💡

  using training data more efficiently

  Majdoddin started Dec 7, 2024 in Ideas

  

  

  0
• 3 You must be logged in to vote

  🙏

  Fineweb vs Openwebtext: why use Fineweb?

  shehper asked Nov 21, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Is it possible to load gpt2-xl?

  scarburato asked Oct 15, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 2 You must be logged in to vote

  💬

  How to swap tinysheakspere for custom data?

  GalanthusLufen started Oct 11, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Training fails on small vocabulary (V<8192)

  austinleedavis asked Oct 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Why accumulation?

  rickojn asked Sep 26, 2024 in Q&A · Unanswered

  

  

  0
• 4 You must be logged in to vote

  💡

  Reducing the cuDNN Download and Installation sizes - cuDNN JIT

  jhalabi-nv started Aug 28, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  💬

  Precision (deterministic) floating point operations

  bitmarkcc started Aug 1, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  💬

  Is there an issue with Conda environments?

  Denizdius started Jul 30, 2024 in General

  

  

  0
• 2 You must be logged in to vote

  🗳️

  LLMs on any GPU: True Open Source to run LLMs

  wbmlr started Jul 5, 2024 in Polls

  

  

  

  

  5
• 14 You must be logged in to vote

  💡

  Intra-document masking - cross contamination and distractions during pre-training

  biuq started Jul 16, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  🙏

  How to train a little dataset ? What format to use and how to call train_gpt2 executable to generate the LLM ? Then, how to test it ?

  FiveTechSoft asked Jul 13, 2024 in Q&A · Unanswered

  

  

  3
• 2 You must be logged in to vote

  🙏

  No dropout in this version

  Phil-U-U asked Jul 9, 2024 in Q&A · Unanswered

  

  

  0
• 12 You must be logged in to vote

  🙌

  Surpassing(?) FineWeb-edu with the DCLM-Baseline Dataset

  robbiegwald started Jul 1, 2024 in Show and tell

  

  

  1
• 5 You must be logged in to vote

  💬

  Understand this entire codebase with the help of a custom GPT [LLM-C by Andrej Karpathy]

  ehzawad started Apr 17, 2024 in General

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  running CUDA on mac or any non-nvidia gpu

  vinayakarsh asked Apr 22, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💡

  Intel SYCL standard runtime support with oneAPI

  abhilash1910 started Jun 7, 2024 in Ideas

  

  

  6
• 11 You must be logged in to vote

  💬

  GPT-2 (774M) reproduced

  karpathy started Jun 12, 2024 in General

  

  

  

  

  9
• 2 You must be logged in to vote

  💬

  Translator layer to cut on vocab size

  IlyaGazman started Jun 13, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to solve the dx of the /doc/layernorm.py?

  amnesiiac asked Jun 18, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Self-attention in O(N) while still letting all the tokens to talk to each other

  IlyaGazman started Jun 13, 2024 in General

  

  

  0
• 4 You must be logged in to vote

  💬

  CPU: llm.c vs Pytorch Benchmarks

  dagelf started Apr 25, 2024 in General

  

  

  3