Scaling Data-Constrained Language Models

Reproducible scaling laws for contrastive language-image learning (https://arxiv.org/abs/2212.07143)

🔥🔥🔥 Latest Advances on Large Recommendation Models

[NeurIPS'24 Spotlight] Observational Scaling Laws

A toolkit for scaling law research ⚖

Dimensionless learning codes for our paper called "Data-driven discovery of dimensionless numbers and governing laws from scarce measurements".

Official code for the ICLR 2025 paper, "Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining"

Code for reproducing the experiments on large-scale pre-training and transfer learning for the paper "Effect of large-scale pre-training on full and few-shot transfer learning for natural and medical images" (https://arxiv.org/abs/2106.00116)

[NeurIPS 2023] Multi-fidelity hyperparameter optimization with deep power laws that achieves state-of-the-art results across diverse benchmarks.

[ICML 2023] "Data Efficient Neural Scaling Law via Model Reusing" by Peihao Wang, Rameswar Panda, Zhangyang Wang

[ICLR 2025] Official implementation of "Towards Neural Scaling Laws for Time Series Foundation Models"

code for Scaling Laws for Language Transfer Learning

A method for calculating scaling laws for LLMs from publicly available models

[ACL2025 Oral] Cuckoo: A Series of IE Free Riders Using LLM's Resources to Scale up Themselves.

Code for CoNLL BabyLM workshop Mini Minds: Exploring Bebeshka and Zlata Baby Models

Model Hemorrhage and the Robustness Limits of Large Language Models: A Perspective

🌹[ICML 2024] Selecting Large Language Model to Fine-tune via Rectified Scaling Law

RSRC Calculator is a practical tool designed to evaluate the efficiency of AI models in the post-scaling era: Recursive Self-Referential Compression (RSRC), this tool computes training efficiency metrics by analyzing factors such as training FLOPs, energy consumption, and model architecture details.

Presentation on Scaling Laws for Neural Language Models​