How to store TensorRT engine file for re-use?

[jonathan-fileread]

Currently my GPU cold starts for PaddleOCR with TensorRT take up to 10 minutes (as I'm assuming it is building the engine)

TRT is able to successfully initialize with this below

self._ocr_model = PaddleOCR( use_tensorrt=True, enable_hpi=True, use_angle_cls=True, lang='en', ocr_version="PP-OCRv5", text_detection_model_name='PP-OCRv5_mobile_det', text_det_limit_type="min", text_det_limit_side_len=960, use_doc_orientation_classify=False, use_doc_unwarping=False, )

However everytime the GPU boots up it takes 10 minutes to load. I'm wondering if I can cache the TRT engine to reuse everytime the GPU gets spun up.

It looks like this block of code (config.enable_tensorrt_engine)

PaddleOCR/tools/infer/utility.py

Line 324 in eaede68

config.enable_tensorrt_engine(

allows a parameter called use_static=True, which documentation
use_static，type：bool, and the default value is False. If it is designated as True, then the optimized TRT information will be serialized to the disk during the first run of the program, and will be directly loaded next time without regeneration.

How do I pass this param through python's paddleocr.PaddleOCR()?

[Bobholamovic]

What paddlepaddle version is used?

[jonathan-fileread]

Hey @Bobholamovic ! Here are the versions across the board.

Python version: 3.10
base_image:
image: paddlepaddle/paddle:3.0.0-gpu-cuda11.8-cudnn8.9-trt8.6

• wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-11.8.tar.gz
• pip install TensorRT-8.6.1.6/python/tensorrt-8.6.1-cp310-none-linux_x86_64.whl
• pip install paddleocr==3.2.0

[jonathan-fileread]

Preferentially my ideal usage would be something like this

    self._ocr_model = PaddleOCR(
        use_tensorrt=True,
        tensorrt_config={use_static:True, ...}
        enable_hpi=True,
        ....
    )

[Bobholamovic]

That’s a bit strange, because if you’re using PaddlePaddle 3.0.0, it should be using PIR-TRT, but the issue you’re encountering occurs in a different conditional branch. I’m not entirely sure what’s causing the problem, but I recommend using PaddleOCR’s HPI feature — it can intelligently select and configure the inference engine, and it will also automatically use TensorRT caching.

[jonathan-fileread]

I believe I've gotten a little farther, I think using the paddlex_config to load into PaddleOCR(paddlex_config). it looks like in hpi_config, there is a backend_config that allows me to put in trt_cfg_setting. However I'm still unsure which is the tensorRT engine file

I searched through my FS and found these trt files - are these the engine files I can cache and reference at PaddleOCR runtime?

/root/.paddlex/official_models/PP-OCRv5_mobile_det/.cache/tensorrt/trt_serialized.trt
/root/.paddlex/official_models/PP-OCRv5_server_rec/.cache/tensorrt/trt_serialized.trt 
/root/.paddlex/official_models/PP-LCNet_x1_0_textline_ori/.cache/tensorrt/trt_serialized.trt 

    paddlex_config = {
        "pipeline_name": "OCR",
        "text_type": "general",
        "use_doc_preprocessor": False,
        "use_textline_orientation": True,
        
        # Global HPI settings
        "use_hpip": True,
        "hpi_config": {
            "auto_config": False,
            "backend": "tensorrt",
            "backend_config": {
                    "trt_cfg_setting": {
                    "use_static": True,  # ← KEY: Serialize engines to disk!
                    },
                "run_mode": "trt_fp16",
                "device_type": "gpu",
                "trt_collect_shape_range_info": False,  # DON'T REBUILD!
                "trt_discard_cached_shape_range_info": False,
                "trt_allow_rebuild_at_runtime": False,
                "trt_shape_range_info_path": "/cache/model/.trt_shape_cache",
                "mkldnn_cache_capacity": 10
            }
        },
        
        # Module configurations
        "SubModules": {
            "TextDetection": {
                "model_name": "PP-OCRv5_mobile_det",
                "model_dir": None,
                "module_name": "text_detection",
                "limit_side_len": 960,
                "limit_type": "min",
                "max_side_limit": 4000,
                "box_thresh": 0.6,
                "thresh": 0.3,
                "unclip_ratio": 1.5,
                "use_hpip": True,
                "hpi_config": {
                    "auto_config": False,
                    "backend": "tensorrt",
                    "backend_config": {
                        "trt_cfg_setting": {
                        "use_static": True,  # ← KEY: Serialize engines to disk!
                        },
                        "run_mode": "trt_fp16",
                        "device_type": "gpu",
                        "trt_collect_shape_range_info": False,
                        "trt_allow_rebuild_at_runtime": False,
                    }
                }
            },
            
            "TextLineOrientation": {
                "model_name": "PP-LCNet_x1_0_textline_ori",
                "model_dir": None,
                "module_name": "textline_orientation",
                "batch_size": 6,
                "use_hpip": True,
                "hpi_config": {
                    "auto_config": False,
                    "backend": "tensorrt",
                    "backend_config": {
                        "trt_cfg_setting": {
                        "use_static": True,  # ← KEY: Serialize engines to disk!
                        },
                        "run_mode": "trt_fp16",
                        "device_type": "gpu",
                        "trt_collect_shape_range_info": False,
                        "trt_allow_rebuild_at_runtime": False,  
                    }
                }
            },
            
            "TextRecognition": {
                "model_name": "PP-OCRv5_server_rec",
                "model_dir": None,
                "module_name": "text_recognition",
                "batch_size": 6,
                "score_thresh": 0.0,
                "use_hpip": True,
                "hpi_config": {
                    "auto_config": False,
                    "backend": "tensorrt",
                    "backend_config": {
                        "trt_cfg_setting": {
                        "use_static": True,  # ← KEY: Serialize engines to disk!
                        },
                        "run_mode": "trt_fp16",
                        "device_type": "gpu",
                        "trt_collect_shape_range_info": False,
                        "trt_allow_rebuild_at_runtime": False,
                    }
                }
            }
        },
        
        "SubPipelines": {
            "DocPreprocessor": {
                "pipeline_name": "doc_preprocessor",
                "use_doc_orientation_classify": False,
                "use_doc_unwarping": False,
                "SubModules": {
                    "DocOrientationClassify": {
                        "model_dir": None,
                        "model_name": "PP-LCNet_x1_0_doc_ori",
                        "module_name": "doc_text_orientation"
                    },
                    "DocUnwarping": {
                        "model_dir": None,
                        "model_name": "UVDoc",
                        "module_name": "image_unwarping"
                    }
                }
            }
        }
    }
    def list_all(path):
        for root, dirs, files in os.walk(path):
            for name in dirs:
                print(os.path.join(root, name))
            for name in files:
                print(os.path.join(root, name))
    print("LIST BEFORE PADDLEOCR CREATED")
    list_all("/root/.paddlex")

    # Initialize PaddleOCR with GPU support
    self._ocr_model = PaddleOCR(
        paddlex_config=paddlex_config,
        use_tensorrt=True,
        enable_hpi=True,
        use_angle_cls=True,
        lang='en',
        ocr_version="PP-OCRv5",
        text_detection_model_name='PP-OCRv5_mobile_det',
        text_det_limit_type="min",
        text_det_limit_side_len=960,
        use_doc_orientation_classify=False,
        use_doc_unwarping=False,
    )

[Bobholamovic]

Yes, the cache is stored in model_dir, and the two steps you listed is the standard way to do this.

[jonathan-fileread]

Thanks @Bobholamovic - this is helpful!

One last question - ( if I use settings with the paddlex_config above)

PaddleOCR(
paddlex_config=paddlex_config,
use_tensorrt=True,
enable_hpi=True,
)

I should expect better performance (high performance mode with tensorrt) versus regular GPU correct?

https://www.paddleocr.ai/main/en/version3.x/module_usage/text_recognition.html#2-list-of-supported-models

[Bobholamovic]

Yes, in our tests, HPI with TensorRT shows better performance.

[jonathan-fileread]

Inference backend: tensorrt 

Inference backend config: precision='fp16' use_dynamic_shapes=True dynamic_shapes={'x': [[1, 3, 80, 160], [1, 3, 80, 160], [8, 3, 80, 160]]} 

[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(567)::BuildTrtEngine	[TrtBackend] Use FP16 to inference. 

[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine	Start to building TensorRT Engine... 

[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(659)::BuildTrtEngine	TensorRT Engine is built successfully. 

[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(661)::BuildTrtEngine	Serialize TensorRTEngine to local file data/.paddlex/official_models/PP-LCNet_x1_0_textline_ori/.cache/tensorrt/trt_serialized.trt. 

[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(672)::BuildTrtEngine	TensorRTEngine is serialized to local file data/.paddlex/official_models/PP-LCNet_x1_0_textline_ori/.cache/tensorrt/trt_serialized.trt, we can load this model from the serialized engine directly next time. 

[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log	3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior. 

 ) 

 [INFO] ultra_infer/runtime/runtime.cc(320)::CreateTrtBackend	Runtime initialized with Backend::TRT in Device::GPU. 

I've stored my model_dir in data/.paddlex/official_models/PP-LCNet_x1_0_textline_ori - however it seems that PaddleOCR + TRT is not able to read from the cache. Previously if the files weren't there it would error, but now they are there and not being read - still building from scratch. please note i built the model on a seperate GPU, then transferred the cached files into another GPU to run.

TRT - 8.6
PaddleOCR==3.2.0
baseimage: paddlepaddle/paddle:3.0.0-gpu-cuda11.8-cudnn8.9-trt8.6

This is my container build

• apt update
• apt install -y wget
• wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-11.8.tar.gz
• tar xvf TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-11.8.tar.gz
• pip install TensorRT-8.6.1.6/python/tensorrt-8.6.1-cp310-none-linux_x86_64.whl
• pip install paddleocr==3.2.0
• export LD_LIBRARY_PATH="/usr/local/cuda-11.8/compat:$LD_LIBRARY_PATH:TensorRT-8.6.1.6/lib" && paddleocr install_hpi_deps gpu

[Bobholamovic]

Are the GPU used for building the model and the GPU used for running the same type?