PaddleX/latest/pipeline_deploy/serving

[giscus[bot]]

PaddleX/latest/pipeline_deploy/serving

https://paddlepaddle.github.io/PaddleX/latest/pipeline_deploy/serving.html

[Danee-wawawa]

您好，paddlex --serve --pipeline image_classification这样启动服务之后，关闭服务的命令是什么呀？

[gaoxiangpost]

ctrl+c

[crazy-1677]

paddlex --serve --pipeline {产线名称或产线配置文件路径} [{其他命令行选项}]

如果在部署多个产线， 是否要启动多个服务吗？

[cuicheng01]

您好，需要启动多个服务

[heliang230]

服务化部署支持实时数据的高性能推理？

[Bobholamovic]

暂时没有支持流式返回的服务。不过，对于实时数据，可以考虑多实例+高性能推理增加并发、降低延迟～

[yaoguang]

我在测试‘高稳定性服务化部署’，使用通用OCR SDK，本地docker启动成功，client.py测试GRPCInferenceService通过，Metrics Service访问成功，但是HTTPService请求失败，一直报错400 Bad Request。想问一下HTTPService的接口调用文档有没有，请求参数是什么，怎么才能正确访问HTTPService

[Bobholamovic]

你好，我们暂时没有提供HTTPService的调用文档，目前只支持gRPC～

[wxz5459]

SDK的下载链接都失效了，麻烦修复下吧

[Bobholamovic]

你好，目前这块正在升级中，文档先更新了，最新的3.0.1 SDK预计今天之内上传～

[Danee-wawawa]

你好，目前这块正在升级中，文档先更新了，最新的3.0.1 SDK预计今天之内上传～

您好，想问下此次更新主要是更新了什么内容呢？

[Bobholamovic]

这次主要是bug修复，去除了一些无效的参数，另外PP-ChatOCR产线也补充了一个use_textline_orientation参数

[LonerangerLR]

您好
ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.1-gpu
和
ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/paddlex:paddlex3.0.0-paddlepaddle3.0.0-gpu-cuda11.8-cudnn8.9-trt8.6
两个镜像的区别是什么

[Bobholamovic]

前者是部署镜像，用于高稳定性服务化部署，内置了相关环境；后者是开发镜像，主要用来开发调试paddlex代码，如果对镜像体积不关心，也可以用在部署（高性能推理、基础服务化部署等）。

[wu2754522801]

docker部署的，为什么λ localhost ~/PaddleX paddlex --serve --pipeline OCR
Creating model: ('PP-LCNet_x1_0_doc_ori', None)
Using official model (PP-LCNet_x1_0_doc_ori), the model files will be automatically downloaded and saved in /root/.paddlex/official_models.
Illegal instruction (core dumped)执行启动服务化命令报错啊，没有看到文档中的输出信息，这是什么原因啊？这个是麒麟的linux系统
[root@localhost ~]# docker --version
Docker version 27.0.3, build 7d4bcd8

[Bobholamovic]

可能需要先确认一下本地是否能跑通

[wu2754522801]

我使用阿里云或者腾讯云的linux系统就可以，同样的命令，但是使用麒麟系统的这个就不太行
[root@localhost ~]# lscpu
架构： x86_64
CPU 运行模式： 32-bit, 64-bit
字节序： Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU: 16
在线 CPU 列表： 0-15
每个核的线程数： 1
每个座的核数： 8
座： 2
NUMA 节点： 1
厂商 ID： AuthenticAMD
CPU 系列： 15
型号： 6
型号名称： Hygon C86 7375 32-core Processor
步进： 3
CPU MHz： 1999.999
BogoMIPS： 3999.99
超管理器厂商： KVM
虚拟化类型： 完全
L1d 缓存： 1 MiB
L1i 缓存： 1 MiB
L2 缓存： 8 MiB
L3 缓存： 32 MiB
NUMA 节点0 CPU： 0-15

是不是不太支持这个版本呀？

[Bobholamovic]

建议可以提一个issue，我们会跟进这个问题

[lilong1988]

高稳定性服务化部署如何通过http方式调用

[Bobholamovic]

暂不支持，后续我们会考虑添加HTTP调用示例～

[yaohongfenglove]

文档中说到 手动构造http请求 的方式，好像不能用呀，端口都没通。
如果别人来调用ocr服务，请问我是需要在grpc的基础上使用flask之类的构造http服务吗？谢谢大佬

[Bobholamovic]

请问具体报什么错呢？

[yaohongfenglove]

已解决，是我机器端口号冲突了。把--network host改为-p 7090:8000 -p 7091:8001 -p 7092:8002就可以了。
端口号冲突时，目前运行启动脚本不会报错，不是很友好呢

[Bobholamovic]

感谢你的反馈，后续我们将考虑增加端口占用检查，并优化错误提示

[ChangDong001]

您好，这个SDK又不能下载了

[Bobholamovic]

不好意思，这几天在更新中，预计今天之内会上传。

[wqcai]

您好，想请教下CUDA版本为12.4的话 怎么获取ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.1-gpu的版本 这个只支持CUDA版本为11.8的吧？

[Bobholamovic]

CUDA驱动支持到12.4应该也可以运行CUDA 11.8的镜像，您可以直接尝试～我们目前暂时没有提供更高版本CUDA Runtime的镜像

[wqcai]

你好，我用paddlex3.0.0rc1-gpu版本能跑成功了 但是OCR识别的结果还是和cpu一样 需要10秒左右；显卡是Tesla T4 请问还有其它调优方式吗？config_gpu.pbtxt配置是这样：
backend: "python"
max_batch_size: 5
input [
{
name: "input"
data_type: TYPE_STRING
dims: [ 1 ]
}
]
output [
{
name: "output"
data_type: TYPE_STRING
dims: [ 1 ]
}
]
instance_group [
{
count: 2
kind: KIND_GPU
gpus: [ 0 ]
}
]

[Bobholamovic]

请使用paddlex和paddle框架最新版本吧 3.0.0rc1应该是很早以前的版本～

[wqcai]

之前只是paddlex3.1-gpu会报如下错误：Traceback (most recent call last):
File "/paddlex/py310/lib/python3.10/site-packages/paddlex_hps_server/base_model.py", line 88, in execute
result_or_output = self.run(input_, log_id)
File "/paddlex/var/paddlex_model_repo/ocr/1/model.py", line 80, in run
images, data_info = utils.file_to_images(
File "/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/serving/infra/utils.py", line 251, in file_to_images
images, data_info = read_pdf(file_bytes, max_num_imgs=max_num_imgs)
File "/paddlex/py310/lib/python3.10/site-packages/paddlex/utils/deps.py", line 135, in wrapper
return func(*args, **kwargs)
File "/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/serving/infra/utils.py", line 190, in read_pdf
doc = pdfium.PdfDocument(bytes)
File "/paddlex/py310/lib/python3.10/site-packages/pypdfium2/_helpers/document.py", line 78, in init
self.raw, to_hold, to_close = _open_pdf(self._input, self._password, self._autoclose)
File "/paddlex/py310/lib/python3.10/site-packages/pypdfium2/_helpers/document.py", line 678, in _open_pdf
raise PdfiumError(f"Failed to load document (PDFium: {pdfium_i.ErrorToStr.get(err_code)}).")
pypdfium2._helpers.misc.PdfiumError: Failed to load document (PDFium: Data format error).

[Bobholamovic]

这个问题看起来是因为我们底层使用的pypdfium2库无法正确解析这个PDF，建议可以到pypdfium2库提一个issue～

[azyhCoding]

服务化部署如何开启高性能推理？

[Bobholamovic]

基础服务化部署在启动服务时指定参数--use_hpip，高稳定性服务化部署设置环境变量PADDLEX_HPS_USE_HPIP为1。

[azyhCoding]

高性能推理插件已经安装，但是加上这个参数启动时报错，不加这个参数是可以正常启动的。

paddlex --serve --use_hpip --pipeline OCR

Creating model: ('PP-LCNet_x1_0_doc_ori', None)
Using official model (PP-LCNet_x1_0_doc_ori), the model files will be automatically downloaded and saved in /root/.paddlex/official_models.
/usr/local/lib/python3.10/dist-packages/setuptools-68.2.2-py3.10.egg/_distutils_hack/init.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the distutils module in sys.modules. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/setuptools-68.2.2-py3.10.egg/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
The Paddle Inference backend is selected with the default configuration. This may not provide optimal performance.
Using Paddle Inference backend
Paddle predictor option: device_type: cpu, device_id: None, trt_dynamic_shapes: {'x': [[1, 3, 224, 224], [1, 3, 224, 224], [8, 3, 224, 224]]}, run_mode: paddle, cpu_threads: 8, delete_pass: [], enable_new_ir: True, enable_cinn: False, trt_cfg_setting: {}, trt_use_dynamic_shapes: True, trt_collect_shape_range_info: True, trt_discard_cached_shape_range_info: False, trt_dynamic_shape_input_data: None, trt_shape_range_info_path: None, trt_allow_rebuild_at_runtime: True
Creating model: ('UVDoc', None)
Using official model (UVDoc), the model files will be automatically downloaded and saved in /root/.paddlex/official_models.
The Paddle Inference backend is selected with the default configuration. This may not provide optimal performance.
Using Paddle Inference backend
Paddle predictor option: device_type: cpu, device_id: None, trt_dynamic_shapes: {'img': [[1, 3, 128, 64], [1, 3, 256, 128], [8, 3, 512, 256]]}, run_mode: paddle, cpu_threads: 8, delete_pass: [], enable_new_ir: True, enable_cinn: False, trt_cfg_setting: {}, trt_use_dynamic_shapes: True, trt_collect_shape_range_info: True, trt_discard_cached_shape_range_info: False, trt_dynamic_shape_input_data: None, trt_shape_range_info_path: None, trt_allow_rebuild_at_runtime: True
Creating model: ('PP-LCNet_x1_0_textline_ori', None)
Using official model (PP-LCNet_x1_0_textline_ori), the model files will be automatically downloaded and saved in /root/.paddlex/official_models.
Traceback (most recent call last):
File "/usr/local/bin/paddlex", line 8, in
sys.exit(console_entry())
File "/root/PaddleX/paddlex/main.py", line 26, in console_entry
main()
File "/root/PaddleX/paddlex/paddlex_cli.py", line 444, in main
serve(
File "/root/PaddleX/paddlex/paddlex_cli.py", line 343, in serve
pipeline = create_pipeline(
File "/root/PaddleX/paddlex/inference/pipelines/init.py", line 165, in create_pipeline
pipeline = BasePipeline.get(pipeline_name)(
File "/root/PaddleX/paddlex/utils/deps.py", line 195, in _wrapper
return old_init_func(self, *args, **kwargs)
File "/root/PaddleX/paddlex/inference/pipelines/_parallel.py", line 103, in init
self._pipeline = self._create_internal_pipeline(config, self.device)
File "/root/PaddleX/paddlex/inference/pipelines/_parallel.py", line 158, in _create_internal_pipeline
return self._pipeline_cls(
File "/root/PaddleX/paddlex/inference/pipelines/ocr/pipeline.py", line 83, in init
self.textline_orientation_model = self.create_model(
File "/root/PaddleX/paddlex/inference/pipelines/base.py", line 107, in create_model
model = create_predictor(
File "/root/PaddleX/paddlex/inference/models/init.py", line 77, in create_predictor
return BasePredictor.get(model_name)(
File "/root/PaddleX/paddlex/inference/models/image_classification/predictor.py", line 49, in init
self.preprocessors, self.infer, self.postprocessors = self._build()
File "/root/PaddleX/paddlex/inference/models/image_classification/predictor.py", line 82, in _build
infer = self.create_static_infer()
File "/root/PaddleX/paddlex/inference/models/base/predictor/base_predictor.py", line 242, in create_static_infer
return HPInfer(
File "/root/PaddleX/paddlex/utils/deps.py", line 148, in _wrapper
return old_init_func(self, *args, **kwargs)
File "/root/PaddleX/paddlex/inference/models/common/static_infer.py", line 575, in init
backend, backend_config = self._determine_backend_and_config()
File "/root/PaddleX/paddlex/inference/models/common/static_infer.py", line 630, in _determine_backend_and_config
raise RuntimeError(
RuntimeError: No inference backend and configuration could be suggested. Reason: 'PP-LCNet_x1_0_textline_ori' is not a known model.

[Bobholamovic]

请确认你使用的是最新版本的paddlex～

[bmmpp]

执行 1.1 paddlex --install serving 一直报错啊，os 版本 AlmaLinux9.6， 在linux 上你们强烈建议使用 docker安装paddlex ，这里怎么没有使用docekr 安装的paddlex 执行服务部署的教程呀， 请大佬指教， 谢谢！

Using cached future-1.0.0-py3-none-any.whl (491 kB)
Downloading pycryptodome-3.23.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ 1.0/2.3 MB 16.1 kB/s eta 0:01:16

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip3.12 install --upgrade pip
ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
yield
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/urllib3/response.py", line 561, in read
data = self._fp_read(amt) if not fp_closed else b""
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 98, in read
data: bytes = self.__fp.read(amt)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/http/client.py", line 479, in read
s = self.fp.read(amt)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/socket.py", line 720, in readinto
return self._sock.recv_into(b)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into
return self.read(nbytes, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/ssl.py", line 1103, in read
return self._sslobj.read(len, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 106, in _run_wrapper
status = _inner_run()
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 97, in _inner_run
return self.run(options, args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 386, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 179, in resolve
self.factory.preparer.prepare_linked_requirements_more(reqs)
File "/usr/local/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 554, in prepare_linked_requirements_more
self._complete_partial_requirements(
File "/usr/local/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 469, in _complete_partial_requirements
for link, (filepath, _) in batch_download:
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/network/download.py", line 184, in call
for chunk in chunks:
^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/progress_bars.py", line 55, in _rich_progress_bar
for chunk in iterable:
^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/network/utils.py", line 65, in response_chunks
for chunk in response.raw.stream(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/urllib3/response.py", line 622, in stream
data = self.read(amt=amt, decode_content=decode_content)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/urllib3/response.py", line 560, in read
with self._error_catcher():
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
Traceback (most recent call last):
File "/home/vision/.local/bin/paddlex", line 8, in
sys.exit(console_entry())
^^^^^^^^^^^^^^^
File "/home/vision/.local/lib/python3.12/site-packages/paddlex/main.py", line 26, in console_entry
main()
File "/home/vision/.local/lib/python3.12/site-packages/paddlex/paddlex_cli.py", line 441, in main
install(args)
File "/home/vision/.local/lib/python3.12/site-packages/paddlex/paddlex_cli.py", line 275, in install
_install_serving_deps()
File "/home/vision/.local/lib/python3.12/site-packages/paddlex/paddlex_cli.py", line 228, in _install_serving_deps
install_packages(reqs)
File "/home/vision/.local/lib/python3.12/site-packages/paddlex/utils/install.py", line 69, in install_packages
return install_packages_from_requirements_file(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vision/.local/lib/python3.12/site-packages/paddlex/utils/install.py", line 58, in install_packages_from_requirements_file
return subprocess.check_call(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python3.12', '-m', 'pip', 'install', '-c', '/tmp/tmpdmh51963.txt', '-r', '/tmp/tmpow6ocinu.txt']' returned non-zero exit status 2.

[bmmpp]

好的，谢谢大佬， 对于 2.4.2 可以使用grpc的方式吗， 监听的有端口， 是不是按http 的方式传入json就可以呢， 如果前台要出图片流， 或者 base64 字符， grpc 如何调用呢，谢谢大佬指导

[Bobholamovic]

不好意思，我刚贴的文档是之前的版本的……可以参考这里：
https://paddlepaddle.github.io/PaddleX/3.1/pipeline_deploy/serving.html#24

grpc的话，建议用Python客户端来调用；HTTP的话，可以手动构造请求。

[bmmpp]

谢谢大佬指导，我们的业务环境比较特殊， 我们前台app 是安卓自动循环调用相机每帧去识别的，所以有大量的请求发送到server端，而且每次识别要尽量快，既然容器监听的有grpc的端口， 那么这种场景grpc 应该比http 性能要好很多吧，可以指导一下grpc的方式吗？

[ethlucky]

这个版本 ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-gpu \ 我运行正常，但是运行
ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.1-cpu \后报以下错误了

docker run
-it
-e PADDLEX_HPS_DEVICE_TYPE=gpu
-e PADDLEX_HPS_USE_HPIP=1
-v "$(pwd)":/app
-w /app
--gpus all
--init
--network host
--shm-size 8g
ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.1-cpu
/bin/bash server.sh

E0716 02:02:04.653384 7 model_repository_manager.cc:1890] Poll failed for model directory 'ocr': instance group ocr_0 of model ocr has kind KIND_GPU but server does not support GPUs
I0716 02:02:04.653437 7 server.cc:522]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0716 02:02:04.653444 7 server.cc:549]
+---------+------+--------+
| Backend | Path | Config |
+---------+------+--------+
+---------+------+--------+

I0716 02:02:04.653449 7 server.cc:592]
+-------+---------+--------+
| Model | Version | Status |
+-------+---------+--------+
+-------+---------+--------+

I0716 02:02:04.653482 7 tritonserver.cc:1920]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.15.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /paddlex/var/paddlex_model_repo |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 0.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0716 02:02:04.653493 7 server.cc:252] Waiting for in-flight requests to complete.
I0716 02:02:04.653494 7 server.cc:267] Timeout 30: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models

[Bobholamovic]

因为你使用的是CPU镜像～请使用GPU镜像吧（以-gpu为后缀）

[bmmpp]

https://paddlepaddle.github.io/PaddleX/latest/pipeline_deploy/serving.html#23
这种方式 docker run 启动的如何调整cpu 线程数， 要在每个 SubModules 中都指定吗
hpi_config：
backend_config：
cpu_num_threads：20 这样吗？

调整后的 pipeline_config.yaml

pipeline_name: OCR

text_type: general

use_doc_preprocessor: True
use_textline_orientation: True

SubPipelines:
DocPreprocessor:
pipeline_name: doc_preprocessor
use_doc_orientation_classify: True
use_doc_unwarping: True
SubModules:
DocOrientationClassify:
module_name: doc_text_orientation
model_name: PP-LCNet_x1_0_doc_ori
model_dir: null
hpi_config：
backend_config：
cpu_num_threads：20
DocUnwarping:
module_name: image_unwarping
model_name: UVDoc
model_dir: null
hpi_config：
backend_config：
cpu_num_threads：20

SubModules:
TextDetection:
module_name: text_detection
model_name: PP-OCRv4_mobile_det
model_dir: null
limit_side_len: 960
limit_type: max
max_side_limit: 4000
thresh: 0.3
box_thresh: 0.6
unclip_ratio: 1.5
hpi_config：
backend_config：
cpu_num_threads：20
TextLineOrientation:
module_name: textline_orientation
model_name: PP-LCNet_x0_25_textline_ori
model_dir: null
batch_size: 6
hpi_config：
backend_config：
cpu_num_threads：20
TextRecognition:
module_name: text_recognition
model_name: en_PP-OCRv4_mobile_rec
model_dir: null
batch_size: 6
score_thresh: 0.0
hpi_config：
backend_config：
cpu_num_threads：20

[Bobholamovic]

明白了，看起来是因为部分模型使用的后端（比如paddle）不支持这个配置，所以出现了上面的报错，在不手动固定后端的情况下，可能还是需要逐模型配置～

[ethlucky]

{
"inputs": [
{
"name": "input",
"shape": [1, 1],
"datatype": "BYTES",
"data": [
{
"file": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/general_ocr_001.png",
"visualize": false
}
]
}
],
"outputs": [
{
"name": "output"
}
]
}

大佬，ocr识别的话，这个file必须必须是个url地址吗？

[Bobholamovic]

不是，请参考产线使用文档吧，其中有详细的API参数说明。

[Kommisaar]

启用hpi时，启动服务发生以下错误

Creating model: ('PP-DocLayout_plus-L', None)
Using official model (PP-DocLayout_plus-L), the model files will be automatically downloaded and saved in /root/.paddlex/official_models.
INFO:root:Create a symbolic link pointing to /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvcaffe_parser.so.8 named /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvcaffe_parser.so.
INFO:root:Create a symbolic link pointing to /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvinfer_plugin.so.8 named /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvinfer_plugin.so.
INFO:root:Create a symbolic link pointing to /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvinfer.so.8 named /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvinfer.so.
INFO:root:Create a symbolic link pointing to /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvonnxparser.so.8 named /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvonnxparser.so.
INFO:root:Create a symbolic link pointing to /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvparsers.so.8 named /paddlex/py310/lib/python3.10/site-packages/ultra_infer/libs/third_libs/tensorrt/lib/libnvparsers.so.
Automatically converting PaddlePaddle model to ONNX format
Inference backend: tensorrt
Inference backend config: precision='fp16' use_dynamic_shapes=True dynamic_shapes={'im_shape': [[1, 2], [1, 2], [8, 2]], 'image': [[1, 3, 800, 800], [1, 3, 800, 800], [8, 3, 800, 800]], 'scale_factor': [[1, 2], [1, 2], [8, 2]]}
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(567)::BuildTrtEngine [TrtBackend] Use FP16 to inference.
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine Start to building TensorRT Engine...
symbolic_global_padding.cpp:929: DCHECK(use->is_use_only()) failed.
symbolic_global_padding.cpp:929: DCHECK(use->is_use_only()) failed.
symbolic_global_padding.cpp:929: DCHECK(use->is_use_only()) failed.
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 10: Could not find any implementation for node {ForeignNode[eager_tmp_2_deepcopy_547...Reshape.248]}.
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[eager_tmp_2_deepcopy_547...Reshape.248]}.)
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(636)::BuildTrtEngine Failed to call buildSerializedNetwork().
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(752)::CreateTrtEngineFromOnnx Failed to build tensorrt engine.
[INFO] ultra_infer/runtime/runtime.cc(320)::CreateTrtBackend Runtime initialized with Backend::TRT in Device::GPU.

[Bobholamovic]

如果这个错误不影响正常使用（只是程序卡住打印这些错误），可以忽略～

[Kommisaar]

对于这种启动服务后还需下载的模型，因为网络原因无法下载怎么办，可以手动挂载吗

I0718 07:56:17.887946 7 grpc_server.cc:4117] Started GRPCInferenceService at 0.0.0.0:8001
I0718 07:56:17.888270 7 http_server.cc:2815] Started HTTPService at 0.0.0.0:8000
I0718 07:56:17.930587 7 http_server.cc:167] Started Metrics Service at 0.0.0.0:8002
Creating model: ('PP-LCNet_x1_0_doc_ori', None)
Using official model (PP-LCNet_x1_0_doc_ori), the model files will be automatically downloaded and saved in /root/.paddlex/official_models.
Connecting to https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-LCNet_x1_0_doc_ori_infer.tar ...
[ ERROR] [2025-07-18 07:56:46,523] [be1e7915a6f040f6a15a0381acaa85bc] [508e2ed5-7a07-4079-ae38-30d8abe620c8] - Unhandled exception
Traceback (most recent call last):
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
conn.connect()
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connection.py", line 704, in connect
self.sock = sock = self._new_conn()
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connection.py", line 205, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7154cfde3550>: Failed to resolve 'paddle-model-ecology.bj.bcebos.com' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/paddlex/py310/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/paddlex/py310/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='paddle-model-ecology.bj.bcebos.com', port=443): Max retries exceeded with url: /paddlex/official_inference_model/paddle3.0.0/PP-LCNet_x1_0_doc_ori_infer.tar (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7154cfde3550>: Failed to resolve 'paddle-model-ecology.bj.bcebos.com' ([Errno -3] Temporary failure in name resolution)"))

[Kommisaar]

再问一下，字体文件又该挂载到哪个位置？
Max retries exceeded with url: /paddlex/PaddleX3.0/fonts/PingFang-SC-Regular.ttf (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fc311ed0940>: Failed to resolve 'paddle-model-ecology.bj.bcebos.com' ([Errno -3] Temporary failure in name resolution)"))

[Kommisaar]

解决了

[lilong1988]

2.4.2 手动构造 HTTP 请求，构造的请求体格式不对，"data"中的json应该是String类型。
{
"inputs": [
{
"name": "input",
"shape": [1, 1],
"datatype": "BYTES",
"data": [
"{ "file": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/general_ocr_001.png","visualize": false }"
]
}
],
"outputs": [
{
"name": "output"
}
]
}

[ethlucky]

pipeline_name: OCR

text_type: general

use_doc_preprocessor: True
use_textline_orientation: True

SubPipelines:
DocPreprocessor:
pipeline_name: doc_preprocessor
use_doc_orientation_classify: True
use_doc_unwarping: True
SubModules:
DocOrientationClassify:
module_name: doc_text_orientation
model_name: PP-LCNet_x1_0_doc_ori
model_dir: null
DocUnwarping:
module_name: image_unwarping
model_name: UVDoc
model_dir: null

SubModules:
TextDetection:
module_name: text_detection
model_name: PP-OCRv5_server_det
model_dir: null
limit_side_len: 960
limit_type: max
max_side_limit: 4000
thresh: 0.3
box_thresh: 0.6
unclip_ratio: 1.5
TextLineOrientation:
module_name: textline_orientation
model_name: PP-LCNet_x0_25_textline_ori
model_dir: null
batch_size: 6
TextRecognition:
module_name: text_recognition
model_name: PP-OCRv5_server_rec
model_dir: null
batch_size: 6
score_thresh: 0.0
~

Inference backend config: precision='fp16' use_dynamic_shapes=True dynamic_shapes={'x': [[1, 3, 224, 224], [1, 3, 224, 224], [8, 3, 224, 224]]}
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(567)::BuildTrtEngine [TrtBackend] Use FP16 to inference.
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine Start to building TensorRT Engine...
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(659)::BuildTrtEngine TensorRT Engine is built successfully.
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(661)::BuildTrtEngine Serialize TensorRTEngine to local file /root/.paddlex/official_models/PP-LCNet_x1_0_doc_ori/.cache/tensorrt/trt_serialized.trt.
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(672)::BuildTrtEngine TensorRTEngine is serialized to local file /root/.paddlex/official_models/PP-LCNet_x1_0_doc_ori/.cache/tensorrt/trt_serialized.trt, we can load this model from the serialized engine directly next time.
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
)

pipeline_config.yaml配置修改下模型为PP-OCRv5_server_det, PP-OCRv5_server_rec 启动报以上错误

原来使用的mobile模型， 我就修改了这个地方，起订报错了，这两个模型不可用吗？

[ethlucky]

NFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine Start to building TensorRT Engine...
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(659)::BuildTrtEngine TensorRT Engine is built successfully.
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(661)::BuildTrtEngine Serialize TensorRTEngine to local file /root/.paddlex/official_models/PP-LCNet_x0_25_textline_ori/.cache/tensorrt/trt_serialized.trt.
[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(672)::BuildTrtEngine TensorRTEngine is serialized to local file /root/.paddlex/official_models/PP-LCNet_x0_25_textline_ori/.cache/tensorrt/trt_serialized.trt, we can load this model from the serialized engine directly next time.
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
)

[INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine Start to building TensorRT Engine...
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 4: Could not find any implementation for node Conv.140 + BatchNormalization.86 + Relu.59 due to insufficient workspace. See verbose log for requested sizes.
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 4: [optimizer.cpp::computeCosts::3867] Error Code 4: Internal Error (Could not find any implementation for node Conv.140 + BatchNormalization.86 + Relu.59 due to insufficient workspace. See verbose log for requested sizes.)
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(636)::BuildTrtEngine Failed to call buildSerializedNetwork().
[ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(752)::CreateTrtEngineFromOnnx Failed to build tensorrt engine.
[INFO] ultra_infer/runtime/runtime.cc(320)::CreateTrtBackend Runtime initialized with Backend::TRT in Device::GPU.

字后还有两个报错，ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 4: Could not find any implementation for node Conv.140 + BatchNormalization.86 + Relu.59 due to insufficient workspace.

这个是空间不足吗？

[Bobholamovic]

这些错误如果不影响正常直行的话可以忽略～

[ethlucky]

我这边调用报错了，没法调用

如果ocr mobile5模型有些内容的话识别不出来，换成ocr server5的话，识别精度会不会好点呢

[Bobholamovic]

请问是在PaddleX官方镜像中安装和使用高性能推理插件的吗？

server模型的识别精度更高，不过可能需要消耗更多的计算资源。

[yaohongfenglove]

请大佬有空了帮忙看看这个问题：

1. 服务启动命令：

docker run -it -e PADDLEX_HPS_DEVICE_TYPE=gpu -v "/opt/paddlex_hps_OCR_sdk/server":/app -w /app --rm --gpus all --init -p 7090:8000 -p 7091:8001 -p 7092:8002 --shm-size 8g ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.1-gpu /bin/bash server.sh

2. ~/paddlex_hps_OCR_sdk/server/pipeline_config.yaml配置文件的内容

pipeline_name: OCR

text_type: general

use_doc_preprocessor: True
use_textline_orientation: True

SubPipelines:
  DocPreprocessor:
    pipeline_name: doc_preprocessor
    use_doc_orientation_classify: True
    use_doc_unwarping: True
    SubModules:
      DocOrientationClassify:
        module_name: doc_text_orientation
        model_name: PP-LCNet_x1_0_doc_ori
        model_dir: null
      DocUnwarping:
        module_name: image_unwarping
        model_name: UVDoc
        model_dir: null

SubModules:
  TextDetection:
    module_name: text_detection
    model_name: PP-OCRv5_server_det
    model_dir: null
    limit_side_len: 1920
    limit_type: max
    max_side_limit: 4000
    thresh: 0.3
    box_thresh: 0.6
    unclip_ratio: 1.5
  TextLineOrientation:
    module_name: textline_orientation
    model_name: PP-LCNet_x1_0_textline_ori 
    model_dir: null
    batch_size: 6    
  TextRecognition:
    module_name: text_recognition
    model_name: PP-OCRv5_server_rec 
    model_dir: null
    batch_size: 6

3. 调用命令

curl --location '127.0.0.1:7090/v2/models/ocr/infer' \
--header 'Content-Type: application/json' \
--data '{
    "inputs": [
        {
            "name": "input",
            "shape": [
                1,
                1
            ],
            "datatype": "BYTES",
            "data": [
                "{ \"file\": \"https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/general_ocr_rec_001.png\",\"visualize\": true,\"use_doc_unwarping\": false }"
            ]
        }
    ],
    "outputs": [
        {
            "name": "output"
        }
    ]
}'

以上运行后，有两个问题。

1. 调用时"visualize": true，返回的base64无法还原为图片，添加的头为data:image/png;base64,
2. 调用时"use_doc_unwarping": false，但根据结果值可以推测实际上运行了use_doc_unwarping

[Bobholamovic]

1. 返回的base64应该是不带有头data:image/png;base64的，可以直接解析内容；
2. 由于历史原因，服务参数和本地API参数的命名风格不一致，就use_doc_unwarping参数而言，对应的服务参数应该是useDocUnwarping，详情可以参考服务API reference。

[yaohongfenglove]

明白了，感谢，useDocUnwarping确实OK了。

图片无法还原的问题我也在找到了，返回了三张图片，而不是一张

[github2136]

使用paddlex --serve --pipeline small_object_detection --device cpu 启动如何指定是使用PP-YOLOE_plus_SOD-S 还是使用 PP-YOLOE_plus_SOD-L 模型？

[yaohongfenglove]

1. paddlex --get_pipeline_config small_object_detection --save_path ./my_path
2. vim ./my_path/small_object_detection.yaml
3. paddlex --serve --pipeline ./my_path/small_object_detection.yaml --device cpu

[github2136]

https://paddlepaddle.github.io/PaddleX/latest/en/support_list/models_list.html?h=small+object+detection+yaml#object-detection-module
是下载这里的PP-YOLOE_plus_SOD-S.yaml 文件嘛？

[yaohongfenglove]

vim ./my_path/small_object_detection.yaml 改small_object_detection.yaml中的model_name配置就好了，会自动下载的