Fine tuning batch parameter increases training time?

[Tachyon5]

I am fine tuning a mistral model using finetune and I am perplexed why increasing batch size for training greatly increases the time to train. This seems at odds with what I have understood for a long time. Can someone help explain?

[Tachyon5]

@ggerganov do you happen to know why increasing batch during ./finetune would cause a very large increase in training time? I'm a bit concerned that something strange is happening that I do not understand. In 'normal' machine learning increasing the batch size during training would decrease training time perhaps to the detriment of generalization if batch is too large. currently I can't map what the trade off is, just seems training with a batch size of 1 is fastest. I will comb through the code to try and make sense but was hoping someone involved with the project could offer a high-level overview.

[ggerganov]

Hm not sure, probably @xaedes could provide some insights

[Tachyon5]

Thank you for responding. I'm looking at source code and trying to make sense. I'll report back if I can figure it out.

[Tachyon5]

I feel dumb. Took me a bit to realize the adam-iter = epochs * samples / batch_size. I was just setting batch to a larger number and keeping adam-iter the same. :(