Issue with DepthwiseConv2D layer

[wilfredkisku]

I have a model with has depthwise convolutions which fails when I try to create a configuration file. As far as I know hls4ml has DepthwiseConv2D layer defined.

Interpreting Model
Topology:
Layer name: input_1, layer type: Input
Layer name: conv1, layer type: QConv2DBatchnorm
Layer name: conv_gb_1_1, layer type: QConv2DBatchnorm
Layer name: relu_conv_gb_1_1, layer type: QActivation
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
[<ipython-input-5-d5e5431b9693>](https://localhost:8080/#) in <module>
      1 import hls4ml
----> 2 config = hls4ml.utils.config_from_keras_model(model, granularity='name')
      3 plotting.print_dict(config)

[/usr/local/lib/python3.9/dist-packages/hls4ml/utils/config.py](https://localhost:8080/#) in config_from_keras_model(model, granularity, default_precision, default_reuse_factor)
    134     for keras_layer in keras_layer_config:
    135         if keras_layer['class_name'] not in supported_layers:
--> 136             raise Exception('ERROR: Unsupported layer type: {}'.format(keras_layer['class_name']))
    137         if keras_layer['class_name'] in skip_layers:
    138             continue

Exception: ERROR: Unsupported layer type: DepthwiseConv2D

[jmduarte]

Hi @wilfredkisku, it looks like you may be using hls4ml 0.6.0 (from pip or conda-forge?). Unfortunately, it's quite out of date (and we're planning to release 0.7.0 very soon.

In the meantime, you can install the main branch by running

pip install git+https://github.com/fastmachinelearning/hls4ml@main

[wilfredkisku]

@jmduarte Thanks, I had been using the pip version. Installing from the main branch off git worked perfectly. Hope v0.7.0 will be released soon.

I still have an architectural issue, I have used a multiplication operations that results in a ERROR: Unsupported layer type: TFOpLambda while profiling.

The code that results in this error is show below:

def QSEBlock(x, filters, id, ratio=2):
    #y = QGlobalAveragePooling2D(average_quantizer="quantized_bits("+str(IFM)+",1,alpha=1)", keepdims=True, data_format='channels_last')(x)
    y = GlobalAveragePooling2D(
                              #average_quantizer="quantized_bits("+str(IFM)+",1,alpha=1)", 
                               keepdims=True, data_format='channels_last')(x)
    y = QDense(filters // ratio,
            kernel_quantizer="quantized_bits("+str(WGT)+",1,alpha=1)",
            kernel_initializer='he_uniform', kernel_regularizer=l1(0.0001),name='dense_1_'+id, use_bias=False)(y)
    #y = Dense(filters // ratio, kernel_initializer='he_normal', use_bias=False)(y)
    y = BatchNormalization()(y)
    y = QActivation('quantized_relu('+str(IFM)+',1)', name='relu_dense1_se_'+id)(y)
    y = QDense(filters,
            kernel_quantizer="quantized_bits("+str(WGT)+",1,alpha=1)",
            kernel_initializer='he_uniform', kernel_regularizer=l1(0.0001),name='dense_2_'+id, use_bias=False)(y)
    #y = Dense(filters, kernel_initializer='he_normal', use_bias=False)(y)
    y = BatchNormalization()(y)
    y = QActivation('quantized_relu('+str(IFM)+',1, use_sigmoid=True)', name='relu_dense2_se_'+id)(y)
    return x * y

The return statement operates on x ([batch,1,1,ch]) multiplies it with y([batch, width, height, ch]). Is there a way how to bypass the TFOpLambda error?

Thanks for the help.

[wilfredkisku]

Another issue that I am facing is during compilation the depthwise layers present in the models throw errors:

firmware/myproject.cpp: In function ‘void myproject(hls::stream<nnet::array<ap_fixed<16, 6>, 1u> >&, hls::stream<nnet::array<ap_fixed<16, 6>, 10u> >&)’:
firmware/myproject.cpp:203:5: error: ‘depthwise_conv_2d_cl’ is not a member of ‘nnet’
     nnet::depthwise_conv_2d_cl<layer200_t, layer7_t, config7>(layer200_out, layer7_out, w7, b7); // depthconv_gb_1_1
     ^
firmware/myproject.cpp:203:42: error: expected primary-expression before ‘,’ token
     nnet::depthwise_conv_2d_cl<layer200_t, layer7_t, config7>(layer200_out, layer7_out, w7, b7); // depthconv_gb_1_1
                                          ^
firmware/myproject.cpp:203:52: error: expected primary-expression before ‘,’ token
     nnet::depthwise_conv_2d_cl<layer200_t, layer7_t, config7>(layer200_out, layer7_out, w7, b7); // depthconv_gb_1_1

...

g++: error: myproject.o: No such file or directory
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[12], line 69
     66 #cfg_q['XilinxPart'] = 'xcu250-figd2104-2L-e'
     68 hls_model_q = hls4ml.converters.keras_to_hls(cfg_q)
---> 69 hls_model_q.compile()

File ~/anaconda3/envs/my_tensorflow_new/lib/python3.9/site-packages/hls4ml/model/graph.py:628, in ModelGraph.compile(self)
    626     dlclose_func.restype = ctypes.c_int
    627     dlclose_func(self._top_function_lib._handle)
--> 628 self._top_function_lib = ctypes.cdll.LoadLibrary(lib_name)

File ~/anaconda3/envs/my_tensorflow_new/lib/python3.9/ctypes/__init__.py:460, in LibraryLoader.LoadLibrary(self, name)
    459 def LoadLibrary(self, name):
--> 460     return self._dlltype(name)

File ~/anaconda3/envs/my_tensorflow_new/lib/python3.9/ctypes/__init__.py:382, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    379 self._FuncPtr = _FuncPtr
    381 if handle is None:
--> 382     self._handle = _dlopen(self._name, mode)
    383 else:
    384     self._handle = handle

OSError: /home/icd02/Documents/tcsvt/MOBILENET_GHOSTNET/ghostnet/cnn_model_A_16_8/firmware/myproject-ccdd390F.so: cannot open shared object file: No such file or directory

Also, here is the detail of the g++ compiler that I have installed in the linux system (RHEL 7).

g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.