Sign-SGD via Parameter-Free Optimization

This repository contains the code for experiments applying a range of optimization methods

Requirements

To install requirements:

pip install -r requirements.txt

To train and evaluate the model in the paper, run this command:

./run_script.sh

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
checkpoints		checkpoints
configs		configs
frugal		frugal
logs		logs
scripts		scripts
utils		utils
wandb		wandb
README.md		README.md
__init__.py		__init__.py
adam_like_run.sh		adam_like_run.sh
adamw_delayed.py		adamw_delayed.py
adamw_run.sh		adamw_run.sh
alias_run.sh		alias_run.sh
download_c4_val.py		download_c4_val.py
environment.yml		environment.yml
exp_requirements.txt		exp_requirements.txt
load_dataset_test.py		load_dataset_test.py
nirvana_utils.py		nirvana_utils.py
normalized_sgd_run.sh		normalized_sgd_run.sh
parameter_free_signsgd.py		parameter_free_signsgd.py
prodigy_run.sh		prodigy_run.sh
requirements.txt		requirements.txt
run_glue.py		run_glue.py
sgd_with_adam.py		sgd_with_adam.py
signsgd_run.sh		signsgd_run.sh
steepest_descent_run.sh		steepest_descent_run.sh
torchrun_main.py		torchrun_main.py