Train Gemma from Scratch

[samanbiook]

Hi,

Aside from fine-tuning capability, is it possible to train Gemma from scratch, using whatever dataset we wish to use?

[Conchylicultor]

In theory yes, just use one of the fine tuning example but remove the init_transform to keep the random initialization rather than the pertained weights.

However in practice training a model like Gemma require huge amount of compute/data and the sharing hasn't been optimized for such large use case