Random discussions

[cutoken]

Hi,
Assert fails in tensor/mod.rs when number of heads = number of layers. Not sure if it's not a good number but in nanogpt, equal values are supported - see the cpu section (apples to oranges so please ignore if it's not a good comparison):

https://github.com/karpathy/nanoGPT

[keyvank]

@cutoken Have you tried removing the train_data/ folder before starting the training process on a different architecture?

[cutoken]

Oops. That seems to be the reason.
Btw, the default learning rate given for Adam seems to be really low. With around 0.001 it seems to be converging faster. That's the beauty of this project though: to play around and see how the training and quality is affected.

[keyvank]

@cutoken Be careful! With a higher learning-rate, you may get trapped into a sub-optimal solution :)

[cutoken]

Will keep that in mind!
Is there any possibility to quantize with this model ? I'm curious on where we should make the necessary changes to make it use 8 bit for example.

Btw, here is what it produces at step 1000 while training on a micro sample of nursery rhymes :D

Lis tt ucame lintle dor bamy
“Quack, quack.”
Butwont one  s litttle o o ducaour two dame q
A fhe the

[cutoken]

Okay one more question:
Let's say I'm training it on one dataset.txt. After some time I stop the training and switch it to another dataset.txt. Is it possible to port only the weights from previous training and start training on new data set? Currently it asserts in the same place as above. I'm guessing it's because more things than weights are getting stored to disk.

[keyvank]

@cutoken You mean using 8-bit floating-point instead of f32?

Wow I love that "Quack, quack." LOL 😆

Yes you can, but it's a bit tricky. You have to make sure the vocab and vocab-sizes of previous dataset and new dataset are equal, otherwise, you have to somehow make sure the char->int mappings are the same in both datasets. (And also change the shape of token_embedding tensor, which is tricky)

[cutoken]

Yes 8 bit Float instead of f32.

Can you explain a little on embedding.dat, optimizer.dat, pos_embedding.dat and tensor_num.dat files. Where do the weights for the layers get stored and is vocab same as character set or different. Sorry too many naive questions. If there is some paper or article that makes sense to get the basic understanding, you drop that instead :)

[keyvank]

Right now, the implementation is not generic, but you can just find/replace f32 with other types 🙂

Watch this: https://www.youtube.com/watch?v=kCc8FmEb1nY

embedding.dat contains the token-embeddings
pos_embedding.dat contrains the positional embeddings
optimizer.dat contains the state of the Adam optimizer
tensor_{tensor_id}.dat contains weights, biases, ... of different operations (E.g Linear, LayerNorm, ...) in the network

[cutoken]

Got it. If I could get the weights alone loadable so that we can switch the model around different data sets, I'll raise a PR.

[cutoken]

@keyvank
Solved the lack of interchangeability of weights between datasets issue by the simpler solution of limiting the vocab to a range of ascii characters. Unfortunately as you can guess, it would not only impose restrictions on datasets having other kinds of characters but also impose a penalty on simpler data sets. However it would let same model be used across similar data sets. So no PR is coming but I can raise one with a optional command line switch like --vocab=ascii if needed.

[keyvank]

@cutoken I just added a ASCII tokenizer! :)

[cutoken]

Awesome @keyvank !

[cutoken]

I'm trying to use the ascii tokenizer but may be a bit of implementation is missing. Shouldn't the new Tokenizer trait be used instead of SimpleTokenizer everywhere ?

[keyvank]

@cutoken It's not used by default. In src/main.rs you should replace let tokenizer = ... to let tokenizer = AsciiTokenizer {}

Btw, have you been able to get good results with a larger model?