Baseline Incompatible with other pretrained models

[AtharvanDogra]

I would request the organizers to look into the matter that the baseline predictions are not compatible with other pretrained models.
For instance, the condition shown in the snippet below
elif token.startswith(start_token_pattern) and token != '▁́':
is not applicable to other models as only xlm-roberta is producing tokens in such a way that they have an '_' (underscore) at the beginning while other models are not.
It is present here in the code:

multiconer-baseline/utils/reader_utils.py

Line 92 in a44705c

elif token.startswith(start_token_pattern) and token != '▁́':

Also, running the prediction module on other models requires us to add a separate parameter of encoder_model so that it can run on those models, otherwise gives this error.

[bfetahu]

The baseline and the rest of the functionalities have been to help the participants set their own models and approaches. The participants are by no means requested or restricted in using this baseline only.

We highly encourage you to come up with a more advanced approach, and adopt the existing baseline functionalities for other models.