splitting lib into train and inference(onnx)

[felixdittrich92]

Hi @charlesmindee,
what do you think about following case ?

Split the lib into train and inference parts:

• train part: include datasets and training scripts + models for TF and Pytorch also the option to export trained model in onnx format (do not integrate some model code into inference part)
• inference part: supports the doc/predictor/detector with loading pretrained models via onnxruntime and onnxruntime-gpu so remove the TF/Pytorch specific "duplicated" code (which than is setup in train part) and framework checks with pure onnx model loading

Advantages:

• cleaner and slimmer code base
• faster inference on cpu and gpu
• good model interface (not framework specific)
• much easier to contribute new models / datasets without touching the core code
• can be split with pip only for inference no TF/Pytorch dependency only onnxruntime

Would be nice to work on this in my nearly vacation if you want before i work on adding any model ^^

[fg-mindee]

Hi @felixdittrich92

Thanks for the suggestion! ONNX is indeed on our radar since #5 😅
However, while ONNX should be one possible way to do inference, that should not close the door to others (OpenVINO, etc.)

That being said, the general idea is also something we were planning to do. But that needs in-depth discussion first on a few points:

• how constraining will "being able to export in ONNX" be? (considering we need to make modifications about existing models)
• how can we leave the door open for other IR / inference engines?
• Fully agree on having default setup for inference, and making training an extra 👍

Looking forward to this discussion!

[fg-mindee]

Actually, this is a long-term discussion which might be more suitable for Github "discussions", let's continue over there!

[felixdittrich92]

@fg-mindee
I think a good first point would be which formats are possible and make sense in this case !?
ONNX - CPU and GPU
TensorRT - only GPU (would restrict the inference part of the library quite a bit)
OpenVino - CPU and GPU (I've never used it personally / Opinions about it ? )
... add more
and whether they support the layers, ... ?
I have tested the db_-models for onnx there is no problem only the db_postprocessor only the forward pass have to be refactored also removing the db_postprocessor from this

[fg-mindee]

Another consideration we need to take: as much as possible, we don't want feature to be exclusive to one of the frameworks. So it's best if it's possible in PyTorch & TensorFlow 😅

I agree that a benchmark of Deep Learning inference engine would be valuabe! I'll try to explore a bit, if you find other reasonable options, make sure to let us know :)

[felixdittrich92]

@fg-mindee
only short and ugly to describe it better
What is your opinion on this subject ?
Of course you would lose the framework splitting in the inference part, but do you really need it there? In return, you would get a much better inference part that is clearer, faster and nicely encapsulates the training process

The option to only train models framework-specifically would remain

[fg-mindee]

So yes, that's what I had in mind. But the question remains: while PyTorch pushes for ONNX compatibility, it is not the case for TensorFlow unfortunately. So my point is that I'm not sure we're able to convert all our TF trained models into ONNX.

I need to carry some investigations but here is what I suggest to do:

• benchmark inference engine
• check compatibility with supported frameworks in DocTR
• implement the easiest ONNX support first (PyTorch most likely) then move on to the other before the next release

[felixdittrich92]

@fg-mindee
this sounds very good and yes now i got it what your problem is ^^ sry for this i haven´t used TF since version 2.1 (
I wasn't aware that there is a major incompatibility there) :)

[fg-mindee]

@felixdittrich92 would you mind marking the relevant message as answer for this discussion? It will help potential future visitors to quickly identify the ins & outs of the topic :)

[felixdittrich92]

@fg-mindee I would but I don't have a button to do that

[fg-mindee]

Oh my bad, I think it's because it"s in the "Ideas" category and not Q&A