Promote TorchSharp as One-Stop for .NET tokenization library need

[GeorgeS2019]

The .NET tokenization library to go has been discussed many times.

• Proposal for common .NET tokenization library
• Missing TorchText.NN feature

Many users have suggested the promising solution: BlingFire provided by Microsoft.

It is vital to learn from the vast python community how best to create such solution.

Recently the existing TorchSharp has been re-organized to TorchShrap.Core.

I see this a timely point to explore how to package the tokenization solution under TorchSharp.Text (like PyTorch.Text or Tensorlfow.NET.Text)

A quick preliminary evaluation of PyTorch.Text functionalities suggest BlingFire do cover many of PyTorch.Text features WELL! (performance superiority with respect to Hugging Face Tokenizers)

This is the current state of TorchSharp.Text

By integrating BlingFire to TorchSharp.Text, more essential functionalities commonly needed by .NET community will be available as ONE-STOP

Most important, beyond PyTorch.Text, by integrating BlingFire, TorchSharp.Text will provide many needed HuggingFace Tokenizers as requested here.

By integrating and implementing TorchSharp.Text as suggested above, the community can move towards the End-To-End real world Deep NLP examples as requested in the April 2021 survey.

Instead of the existing example based on the Transformer architecture, the community can contribute to that into the End-To-End example including a better performance Hugging Face tokenizer (BlingFire) provided through TorchSharp.Text.

[GeorgeS2019]

Finally Microsoft.ML.Tokenizers