The input shape broadcasting error of the ADD operator and replace.json cannot resolve the issue #784
Description
Issue Type
Others
OS
Linux
onnx2tf version number
1.28.2
onnx version number
1.18.0
onnxruntime version number
1.22.1
onnxsim (onnx_simplifier) version number
0.4.36
tensorflow version number
2.19.0
Download URL for ONNX
https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct/blob/main/onnx/vision_encoder_uint8.onnx
Parameter Replacement JSON
{
"format_version": 1,
"operations": [
{
"op_name": "wa/vision_model/encoder/layers.0/self_attn/q_proj/Add",
"param_target": "inputs",
"input_index": 0,
"pre_process_reshape": [1, 1, 768]
}
]
}Description
My convert command:
onnx2tf
-i vision_encoder_uint8_new.onnx
-o tf_out1
-b 1
-ois "pixel_values:1,1,3,512,512"
-kat pixel_values
-oiqt
-qt per-channel
-cind pixel_values calib/pixel_values_n64_b1_d1_c3_h512_w512_m1p1.npy
-prf replace.json
An error occurred:
Your model contains "Tile" ops or/and "ConstantOfShape" ops. Folding these ops can make the simplified model much
larger. If it is not expected, please specify "--no-large-tensor" (which will lose some optimization chances)
Simplifying...
Finish! Here is the difference:
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ ┃ Original Model ┃ Simplified Model ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ Add │ 172 │ 172 │
│ Cast │ 75 │ 75 │
│ Concat │ 1 │ 1 │
│ Constant │ 329 │ 329 │
│ ConstantOfShape │ 1 │ 1 │
│ ConvInteger │ 1 │ 1 │
│ DequantizeLinear │ 1 │ 1 │
│ Div │ 25 │ 25 │
│ DynamicQuantizeLinear │ 50 │ 50 │
│ Equal │ 2 │ 2 │
│ Gather │ 2 │ 2 │
│ GatherND │ 1 │ 1 │
│ MatMul │ 24 │ 24 │
│ MatMulInteger │ 73 │ 73 │
│ Mul │ 257 │ 257 │
│ NonZero │ 1 │ 1 │
│ Not │ 1 │ 1 │
│ Pow │ 25 │ 25 │
│ ReduceMean │ 50 │ 50 │
│ ReduceSum │ 1 │ 1 │
│ Reshape │ 53 │ 53 │
│ ScatterND │ 1 │ 1 │
│ Shape │ 1 │ 1 │
│ Softmax │ 12 │ 12 │
│ Sqrt │ 25 │ 25 │
│ Sub │ 25 │ 25 │
│ Tanh │ 12 │ 12 │
│ Transpose │ 52 │ 52 │
│ Unsqueeze │ 1 │ 1 │
│ Model Size │ 89.9MiB │ 89.9MiB │
└───────────────────────┴────────────────┴──────────────────┘
Model optimizing complete!
INFO: 50 / 946
INFO: onnx_op_type: Add onnx_op_name: /vision_model/encoder/layers.0/self_attn/q_proj/Add
INFO: input_name.1: vision_model.encoder.layers.0.self_attn.q_proj.bias shape: [768] dtype: float32
INFO: input_name.2: /vision_model/encoder/layers.0/self_attn/q_proj/MatMul_output_0 shape: ['NonZero_7_o0__d1', 1024, 768] dtype: float32
INFO: output_name.1: /vision_model/encoder/layers.0/self_attn/q_proj/Add_output_0 shape: ['NonZero_7_o0__d1', 1024, 768] dtype: float32
ERROR: The trace log is below.
Traceback (most recent call last):
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 314, in print_wrapper_func
result = func(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 388, in inverted_operation_enable_disable_wrapper_func
result = func(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 57, in get_replacement_parameter_wrapper_func
func(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/ops/Add.py", line 281, in make_node
merge_two_consecutive_identical_ops_into_one(
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 5547, in merge_two_consecutive_identical_ops_into_one
tf.math.add(
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tensorflow/python/ops/weak_tensor_ops.py", line 142, in wrapper
return op(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tf_keras/src/layers/core/tf_op_layer.py", line 119, in handle
return TFOpLambda(op)(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tf_keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
ValueError: Exception encountered when calling layer "tf.math.add_6" (type TFOpLambda).
Dimensions must be equal, but are 1024 and 768 for '{{node tf.math.add_6/Add}} = AddV2[T=DT_FLOAT](Placeholder, tf.math.add_6/Add/y)' with input shapes: [?,1024,768], [1,768,1].
Call arguments received by layer "tf.math.add_6" (type TFOpLambda):
• x=tf.Tensor(shape=(None, 1024, 768), dtype=float32)
• y=tf.Tensor(shape=(1, 768, 1), dtype=float32)
• name='wa/vision_model/encoder/layers.0/self_attn/q_proj/Add'
ERROR: input_onnx_file_path: vision_encoder_uint8_new.onnx
ERROR: onnx_op_name: wa/vision_model/encoder/layers.0/self_attn/q_proj/Add
ERROR: Read this and deal with it. https://github.com/PINTO0309/onnx2tf#parameter-replacement
ERROR: Alternatively, if the input OP has a dynamic dimension, use the -b or -ois option to rewrite it to a static shape and try again.
ERROR: If the input OP of ONNX before conversion is NHWC or an irregular channel arrangement other than NCHW, use the -kt or -kat option.
ERROR: Also, for models that include NonMaxSuppression in the post-processing, try the -onwdt option.
Traceback (most recent call last):
File "/opt/conda/envs/onnx2tf/bin/onnx2tf", line 8, in <module>
sys.exit(main())
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/onnx2tf.py", line 3034, in main
model = convert(
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/onnx2tf.py", line 1272, in convert
raise ex
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/onnx2tf.py", line 1186, in convert
op.make_node(
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 314, in print_wrapper_func
result = func(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 388, in inverted_operation_enable_disable_wrapper_func
result = func(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 57, in get_replacement_parameter_wrapper_func
func(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/ops/Add.py", line 281, in make_node
merge_two_consecutive_identical_ops_into_one(
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/onnx2tf/utils/common_functions.py", line 5547, in merge_two_consecutive_identical_ops_into_one
tf.math.add(
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tensorflow/python/ops/weak_tensor_ops.py", line 142, in wrapper
return op(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tf_keras/src/layers/core/tf_op_layer.py", line 119, in handle
return TFOpLambda(op)(*args, **kwargs)
File "/opt/conda/envs/onnx2tf/lib/python3.10/site-packages/tf_keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
ValueError: Exception encountered when calling layer "tf.math.add_6" (type TFOpLambda).
Dimensions must be equal, but are 1024 and 768 for '{{node tf.math.add_6/Add}} = AddV2[T=DT_FLOAT](Placeholder, tf.math.add_6/Add/y)' with input shapes: [?,1024,768], [1,768,1].
Call arguments received by layer "tf.math.add_6" (type TFOpLambda):
• x=tf.Tensor(shape=(None, 1024, 768), dtype=float32)
• y=tf.Tensor(shape=(1, 768, 1), dtype=float32)
• name='wa/vision_model/encoder/layers.0/self_attn/q_proj/Add'
My replace.json attempts to specify the shape (but it doesn't work):
{
"format_version": 1,
"operations": [
{
"op_name": "wa/vision_model/encoder/layers.0/self_attn/q_proj/Add",
"param_target": "inputs",
"input_index": 0,
"pre_process_reshape": [1, 1, 768]
}
]
}
How can I solve this error? Thanks