A recreation of the Geneformer model described by Theodoris et al. using AttentionSmithy.
This file is the pre-training script for the geneformer foundation model.
This file is a training script for fine-tuning a pretrained geneformer model for a cell classification task.
The code for the (pre-trained) geneformer foundation model. It was written using pytorch lightning for readability, and thus outlines the construction of the model, the forward pass process, and how that looks for training and validation steps.
The code for preparing the data module used in training and validating the geneformer foundation model. It is made to be used with the pytorch lightning Trainer class, as called in model training scripts.
The code for the fine-tuned geneformer model. It was also written using pytorch lightning.
The code for preparing the data module used in training and validating the fine-tuned geneformer model.
Theodoris, C.V., Xiao, L., Chopra, A. et al. Transfer learning enables predictions in network biology. Nature 618, 616–624 (2023). https://doi.org/10.1038/s41586-023-06139-9