how to convert .pt models into .nemo format[Question]

[Ashbajawed]

I have following blog Jump-start Training for Speech Recognition Models in Different Languages with NVIDIA NeMo and successfully finetune the model on my custom data. Now I have to test the models but in checkpoint directory I have 3 .pt files as output (JasperDecoderForCTC-STEP-80000.pt JasperEncoder-STEP-80000.pt trainer-STEP-80000.pt)

I am confuse how would I convert them into .nemo format or is there any other way to use those models for inference.

code : quartznet.py
nemo version = 0.11

Also it will be helpful if anyone can share tutorial for finetuning on latest release

[ericharper]

We have completely changed NeMo since .11. We highly recommend to start using our latest stable version. Please see our User Guide: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/ and our Tutorials: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/starthere/tutorials.html for how to get started with NeMo.

[chrisby]

Would you mind pointing out explicitly which tutorial helps you with converting .pt to .nemo? It seems like I can't find it.

[XuesongYang]

@Ashbajawed Please let us know if it is still a blocker for you. Thanks.

[manjuke]

Hi All, I still Have this issue. I have done transfer learning on confermer-ctc-model and I have ckpt files generated. But, I am unable to load them as they are not in .nemo format.
I am getting error, if I try to load them using model = nemo_asr.models.EncDecCTCModelBPE.restore_from(restore_path="/checkpoints/epoch_5-step_82700.ckpt")

So, How do I convert .ckpt to .nemo format.
I could not find any relevant answers in internet as well. Thanks.

[titu1994]

You cannot use restore_from to work with ckpt files. You can use load_from_checkpoint() which is the PTL method to load PTL checkpoints. Then you can call save_to() to generate a Nemo file c

[manjuke]

Thanks, But, I am getting below error:
TypeError: expected str, bytes or os.PathLike object, not NoneType

When I tried to load as below:
ckp='/home/manjunath/asr/lightning_logs/version_0/checkpoints/ep10.ckpt'
hps='/home/manjunath/asr/lightning_logs/version_0/hparams.yaml'
model2 = nemo_asr.models.EncDecCTCModelBPE.load_from_checkpoint(checkpoint_path=ckp, hparams_file=hps)

I have tried several ways such as os.join, Path(ckp) and many more to convert to os.PathLike object, but no luck. Please suggest. Thanks

[titu1994]

Hparams file can you skip ? Just pass the ckpt fild

[manjuke]

Thanks.. I tried that also but it is giving the same error..

[titu1994]

Paste the full stack trace

[manjuke]

Hi, Thanks for the response.

code:

import nemo
import nemo.collections.asr as nemo_asr
import os
chk_path = '/home/manjunath/asr/lightning_logs/version_0/checkpoints/epoch=17-step=258891.ckpt'
model2 = nemo_asr.models.EncDecCTCModelBPE.load_from_checkpoint(chk_path)

Full Stack-Trace:

TypeError Traceback (most recent call last)
/home/manjunath/asr/scripts/test.ipynb Cell 2 in <cell line: 5>()
3 import os
4 chk_path = '/home/manjunath/asr/lightning_logs/version_0/checkpoints/epoch=17-step=258891.ckpt'
----> 5 model2 = nemo_asr.models.EncDecCTCModelBPE.load_from_checkpoint(chk_path)

File ~/anaconda3/lib/python3.9/site-packages/nemo/core/classes/modelPT.py:336, in ModelPT.load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, strict, *args, **kwargs)
333 try:
334 cls._set_model_restore_state(is_being_restored=True)
--> 336 checkpoint = super().load_from_checkpoint(
337 checkpoint_path=checkpoint_path,
338 *args,
339 map_location=map_location,
340 hparams_file=hparams_file,
341 strict=strict,
342 **kwargs,
343 )
345 finally:
346 cls._set_model_restore_state(is_being_restored=False)

File ~/anaconda3/lib/python3.9/site-packages/pytorch_lightning/core/saving.py:137, in ModelIO.load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, strict, **kwargs)
57 @classmethod
58 def load_from_checkpoint(
59 cls,
(...)
64 **kwargs,
65 ):
66 r"""
67 Primary way of loading a model from a checkpoint. When Lightning saves a checkpoint
68 it stores the arguments passed to __init__ in the checkpoint under "hyper_parameters".
(...)
135 y_hat = pretrained_model(x)
136 """
--> 137 return _load_from_checkpoint(
138 cls,
139 checkpoint_path,
140 map_location,
141 hparams_file,
142 strict,
143 **kwargs,
144 )

File ~/anaconda3/lib/python3.9/site-packages/pytorch_lightning/core/saving.py:205, in _load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, strict, **kwargs)
203 if issubclass(cls, pl.LightningDataModule):
204 return _load_state(cls, checkpoint, **kwargs)
--> 205 return _load_state(cls, checkpoint, strict=strict, **kwargs)

File ~/anaconda3/lib/python3.9/site-packages/pytorch_lightning/core/saving.py:250, in _load_state(cls, checkpoint, strict, cls_kwargs_new)
246 if not cls_spec.varkw:
247 # filter kwargs according to class init unless it allows any argument via kwargs
248 _cls_kwargs = {k: v for k, v in _cls_kwargs.items() if k in cls_init_args_name}
--> 250 obj = cls(_cls_kwargs)
252 # give model a chance to load something
253 obj.on_load_checkpoint(checkpoint)

File ~/anaconda3/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_bpe_models.py:46, in EncDecCTCModelBPE.init(self, cfg, trainer)
43 raise ValueError("cfg must have tokenizer config to create a tokenizer !")
45 # Setup the tokenizer
---> 46 self._setup_tokenizer(cfg.tokenizer)
48 # Initialize a dummy vocabulary
49 vocabulary = self.tokenizer.tokenizer.get_vocab()

File ~/anaconda3/lib/python3.9/site-packages/nemo/collections/asr/parts/mixins/mixins.py:61, in ASRBPEMixin._setup_tokenizer(self, tokenizer_cfg)
59 self._setup_aggregate_tokenizer(tokenizer_cfg)
60 else:
---> 61 self._setup_monolingual_tokenizer(tokenizer_cfg)

File ~/anaconda3/lib/python3.9/site-packages/nemo/collections/asr/parts/mixins/mixins.py:98, in ASRBPEMixin._setup_monolingual_tokenizer(self, tokenizer_cfg)
96 else:
97 model_path = os.path.join(self.tokenizer_dir, 'tokenizer.model')
---> 98 model_path = self.register_artifact('tokenizer.model_path', model_path)
99 self.model_path = model_path
101 if 'special_tokens' in self.tokenizer_cfg:

File ~/anaconda3/lib/python3.9/site-packages/nemo/core/classes/modelPT.py:222, in ModelPT.register_artifact(self, config_path, src, verify_src_exists)
216 if config_path in self.artifacts.keys():
217 logging.warning(
218 f"You tried to register an artifact under config key={config_path} but an artifact for "
219 f"it has already been registered."
220 )
--> 222 return self._save_restore_connector.register_artifact(self, config_path, src, verify_src_exists)

File ~/anaconda3/lib/python3.9/site-packages/nemo/core/connectors/save_restore_connector.py:369, in SaveRestoreConnector.register_artifact(self, model, config_path, src, verify_src_exists)
366 # this is the case when artifact must be retried from the nemo file
367 # we are assuming that the location of the right nemo file is available from _MODEL_RESTORE_PATH
368 elif src.startswith("nemo:"):
--> 369 return_path = os.path.abspath(os.path.join(app_state.nemo_file_folder, src[5:]))
370 artifact_item.path_type = model_utils.ArtifactPathType.TAR_PATH
372 # backward compatibility implementation

File ~/anaconda3/lib/python3.9/posixpath.py:76, in join(a, *p)
71 def join(a, *p):
72 """Join two or more pathname components, inserting '/' as needed.
73 If any component is an absolute path, all previous path components
74 will be discarded. An empty last part will result in a path that
75 ends with a separator."""
---> 76 a = os.fspath(a)
77 sep = _get_sep(a)
78 path = a

TypeError: expected str, bytes or os.PathLike object, not NoneType

[manjuke]

@titu1994 Any suggestions on resolving this error? Thanks

[titu1994]

The tokenizer cannot be found because the path you used during training is no longer valid (in the environment where you are trying to restore), so it's complaining that the file does not exist.

This is why we suggest not to use ckpt files but instead use Nemo files - because they are portable.

The only solution is to have the tokenizer in the exact same path as the ckpt model that was saved. Or you use torch.load() on the ckpt file, modify the config to update the path of the config and then save a new ckpt file, then use that to restore the model (so that it finds the tokenizer correctly).

[manjuke]

Thanks a lot. Is it possible to save in nemo file format instead of ckpt during training. If yes, could you pls suggest me on how to do it? Like passing some parameter etc. Thanks

[titu1994]

ExpManager does this if you set it's flag for always_save_nemo to True. You can still recover your ckpt file using above tricks but I suggest just using Nemo file after you finish training by using model.save_to("xyz.nemo")

[manjuke]

Sure.. thank you

[Akshaya-Devadiga]

code:
import nemo
import nemo.collections.nlp as nemo_nlp
model = nemo_nlp.models.distilbert-base-uncased.from_pretrained(model_name="pre_trained_bert")

Error:

AttributeError Traceback (most recent call last)
in
----> 1 model = nemo_nlp.models.distilbert-base-uncased.from_pretrained(model_name="pre_trained_bert")

AttributeError: module 'nemo.collections.nlp' has no attribute 'models'

@titu1994 Could you please help me solve this error

[VahidooX]

#2730 (reply in thread)

[Akshaya-Devadiga]

@VahidooX

Thank You for your response.

I am still having error. Attaching a file for the same

model = nemo.collections.nlp.models.TokenClassificationModel.load_from_checkpoint('final_model/work/checkpoints/SequenceClassifier-STEP-3610.pt')

Awaiting your response.