Training and Inference of AutoML EfficientDet on OpenVino with EML Tools

In this folder, there is a template project for inference of trained, exported models of AutoML EfficientDet in OpenVino. In the following procedure, instructions are provided to setup and run one or more networks and to extract the evaluations of the executions. All evaluations are compatible with the EML tools.

Setup

Prerequisites

1. Setup the task spooler on the target device. Instructions can be found here: https://github.com/embedded-machine-learning/scripts-and-guides/blob/main/guides/task_spooler_manual.md
2. Install OpenVino. In this implementation, we used openvino_2021.4.582
3. Setup sendmail according to this guide https://github.com/embedded-machine-learning/scripts-and-guides/blob/main/guides/SENDMAIL.md

Dataset

For validating the tool chain, download the small validation set from kaggle: https://www.kaggle.com/alexanderwendt/oxford-pets-cleaned-for-eml-tools

It contains of two small sets that are used for training and inference validation in the structure that is compatible to the EML Tools. Put it in the following folder structure, e.g. /srv/cdl-eml/datasets/dataset-oxford-pets-cleaned/

Generate EML Tools directory structure, Setup the AutoML EfficientDet and OpenVino Environment

The following steps are only necessary if you setup the EML tools for the first time on a device.

1. Create a folder for your datasets. Usually, multiple users use one folder for all datasets to be able to share them. Later on, in the training and inference scripts, you will need the path to the dataset.

2. Create the EML tools folder structure, e.g. eml-tools. The structure can be found here: https://github.com/embedded-machine-learning/eml-tools#interface-folder-structure. Most of the following steps are performed with this script as well: generate_workspace_efficientdet_openvino.sh.

For OpenVino, we need three environments: EfficentDet, TF2ODA and OpenVino, all with different and conflicting requirements.

#!/bin/bash


#1. Create a folder for your datasets. Usually, multiple users use one folder for all datasets to be able to share them. Later on, in the 
#training and inference scripts, you will need the path to the dataset.
#2. Create the EML tools folder structure, e.g. ```eml-tools```. The structure can be found here: https://github.com/embedded-machine-learning/eml-tools#interface-folder-structure
ROOTFOLDER=`pwd`

#In your root directory, create the structure. Sample code
mkdir -p eml_projects
mkdir -p venv

#3. Clone the EML tools repository into your workspace
EMLTOOLSFOLDER=./eml-tools
if [ ! -d "$EMLTOOLSFOLDER" ] ; then
  git clone https://github.com/embedded-machine-learning/eml-tools.git "$EMLTOOLSFOLDER"
else 
  echo $EMLTOOLSFOLDER already exists
fi

# Project setup
#4. Clone the AutoML repository
AUTOMLFOLDER=./automl
if [ ! -d "$AUTOMLFOLDER" ] ; then
  git clone https://github.com/google/automl.git "$AUTOMLFOLDER"
else 
  echo $AUTOMLFOLDER already exists
fi

#5. Create a virtual environment for TF2ODA in your venv folder. The venv folder is put outside of the project folder to 
#avoid copying lots of small files when you copy the project folder. Conda would also be a good alternative.
# From root
cd $ROOTFOLDER

cd ./venv

EFFDETENV=effdet_py36
if [ ! -d "$EFFDETENV" ] ; then
  virtualenv -p python3.8 $EFFDETENV
  source ./$EFFDETENV/bin/activate

  # Install necessary libraries
  python -m pip install --upgrade pip
  pip install --upgrade setuptools cython wheel
  
  # Install EML libraries
  pip install lxml xmltodict tdqm beautifulsoup4 pycocotools numpy tdqm pandas matplotlib pillow
  
  # Install TF2ODA specifics
  #pip install tensorflow==2.4.1
  
  cd $ROOTFOLDER
  
  pip install -r ./$AUTOMLFOLDER/efficientdet/requirements.txt
  
  cd $ROOTFOLDER/automl/efficientdet
  export PYTHONPATH=$PYTHONPATH:`pwd`
  echo New python path $PYTHONPATH
  
  cd $ROOTFOLDER

  echo # Test installation
  # If all tests are OK or skipped, then the installation was successful
  #python object_detection/builders/model_builder_tf2_test.py
  
  echo Test if Tensorflow works with CUDA on the machine. For TF2.4.1, you have to use CUDA 11.0
  python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
  
  echo Installation complete
  
else 
  echo $EFFDETENV already exists
fi

# Create TF2ODA environment that is used for the evaluation and inference measurements
TF2ODAENV=tf24_py36
if [ ! -d "$TF2ODAENV" ] ; then
  virtualenv -p python3.8 $TF2ODAENV
  source ./$TF2ODAENV/bin/activate

  # Install necessary libraries
  python -m pip install --upgrade pip
  pip install --upgrade setuptools cython wheel
  
  # Install EML libraries
  pip install lxml xmltodict tdqm beautifulsoup4 pycocotools numpy tdqm pandas matplotlib pillow
  
  cd $ROOTFOLDER
  
  echo # Install protobuf
  PROTOC_ZIP=protoc-3.14.0-linux-x86_64.zip
  curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/$PROTOC_ZIP
  unzip -o $PROTOC_ZIP -d protobuf
  rm -f $PROTOC_ZIP
  
  echo # Clone tensorflow repository
  git clone https://github.com/tensorflow/models.git
  cd models/research/
  cp object_detection/packages/tf2/setup.py .
  python -m pip install .
  
  echo # Add object detection and slim to python path
  export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
  
  echo # Prepare TF2 Proto Files
  ../../protobuf/bin/protoc object_detection/protos/*.proto --python_out=.

  echo # Test installation
  # If all tests are OK or skipped, then the installation was successful
  python object_detection/builders/model_builder_tf2_test.py
  
  # Install TF2ODA specifics
  #pip install tensorflow==2.4.1
  echo #Test if Tensorflow works with CUDA on the machine. For TF2.4.1, you have to use CUDA 11.0
  python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
  
  echo "Important information: If there are any library errors, you have to install the correct versions manually. TFODAPI does install the latest version of "
  echo "tensorflow. However, in this script Tensorflow 2.4.1 is desired. Then, you have to uninstall the newer versions and replace with current versions."

  echo # Installation complete
  
else 
  echo $TF2ODAENV already exists
fi


#6. Create a virtual environment for OpenVino 2021.4 in Python 3.6 as it does not cope with Python 3.8
# From root
cd $ROOTFOLDER
cd ./venv
OPENVINOVENVFOLDER=openvino_py36
if [ ! -d "$OPENVINOVENVFOLDER" ] ; then
  virtualenv -p python3.6 $OPENVINOVENVFOLDER
  source $OPENVINOVENVFOLDER/bin/activate

  # Install necessary libraries
  python -m pip install --upgrade pip
  pip install --upgrade setuptools cython wheel

  # Install EML libraries
  pip install lxml xmltodict tdqm beautifulsoup4 pycocotools pandas absl-py

  # Install OpenVino libraries
  pip install onnx-simplifier networkx defusedxml progress requests
  
  # Install Tensorflow
  #pip install tensorflow
else 
  echo $OPENVINOVENVFOLDER already exists
fi

cd $ROOTFOLDER
source ./venv/$EFFDETENV/bin/activate

echo Created environment for AutoML EfficientDet and OpenVino inference

Project setup

1. Go to your project folder e.g. ./eml_projects and create a project folder, e.g. ./effdet-oxford-pets

2. Copy the scripts from this repository to that folder and execute chmod 777 *.sh to be able to run the scripts. One of the script is the task spooler script, which could be used by multiple EML projects, ./init_ts.sh.

3. run ./setup_dirs_effdet_openvino.sh to generate all necessary folders.

4. Copy your exported models from your training, e.g from https://github.com/embedded-machine-learning/hwmodule-efficientdet-server to your ./exported-models.

Modification of script files

The next step is to adapt the script files to the current environment.

Adapt Task Spooler Script

In init.ts.sh, either adapt

export TS_SOCKET="/srv/ts_socket/GPU.socket"
chmod 777 /srv/ts_socket/GPU.socket
export TS_TMPDIR=~/logs

to your task spooler path or call another task spooler script in your EML Tools root.

. ../../init_ts.sh

Adapt Environment Scripts for Tensorflow Inference

In init_env.sh, adapt the following part to your venv folder or conda implementation.

source $ENVROOT/venv/effdet_py36/bin/activate
cd $ENVROOT/automl/efficientdet
export PYTHONPATH=`pwd`
echo New python path $PYTHONPATH

In init_env_tf2oda, adapt the following part to your venv folder or conda implementation.

source $ENVROOT/tf24_py36/bin/activate
cd $ENVROOT/models/research/
export PYTHONPATH=`pwd`:`pwd`/slim
echo New python path $PYTHONPATH

In init_env_openvino.sh, adapt source ../../venv/openvino_tf2_py36/bin/activate to match your openvino virtual environment.

OpenVino Conversion of AutoML EfficientDet Exported Models to OpenVino IR

The first script to adapt is convert_tf2effdet_to_ir_TEMPLATE.sh. It converts the exported models into OpenVino IR.

1. Edit the script and adapt the following constants for your conversion:

# Constant Definition
USEREMAIL=alexander.wendt@tuwien.ac.at    #Change to your email address
SCRIPTPREFIX=../../eml-tools   #No need to change this
HARDWARENAME=IntelNUC    #Use your HW identifier

#Openvino installation directory for the model optimizer
OPENVINOINSTALLDIR=/opt/intel/openvino_2021.4.582   #Use your OpenVino installation
PRECISIONLIST="FP16 FP32"   # Set the precisions to convert to. Default: FP16 and FP32.

In the exported-models folder, for each model, you will find the following files:

• saved_model.tflite
• saved_model_simplified.onnx
• saved_model_unsimplified.onnx
• saved_model/saved_model_frozen.pb
• saved_model/saved_model.pb
• saved_model/variables

saved_model/saved_model_frozen.pb is the file that is used.

2. If you want to use the script manually do the following. Else, you can use the add_folder_conv_ir.sh instead. Copy and rename convert_yolo_onnx_to_ir_TEMPLATE.sh to convert_yolo_onnx_to_ir_[MODELNAME].sh, e.g. convert_tf2effdet_to_ir_tf2_efficientdetd0_640x640_oxfordpets.sh, where the MODELNAME is a exact match of the folder name of the model folder in exported-models. Now, this script will only be used to convert this model. Note that the MODELNAME will be extracted from the file name and information about the implementation will be extracted for the evaluation.

Inference Script OpenVino

The script openvino_inf_eval_saved_model_TEMPLATE.sh executes the TEMPLATE network on OpenVino.

Adapt the following constants for your environment:

SCRIPTPREFIX=../../eml-tools   #No need to change this
HARDWARENAME=IntelNUC     #Set your hardware id
DATASET=/home/intel-nuc/eml-tools/datasets/dataset-oxford-pets-val-debug   #Validation dataset

#Openvino installation directory for the inferrer (not necessary the same as the model optimizer)
OPENVINOINSTALLDIR=/opt/intel/openvino_2021.4.689   #Set your OpenVino version
APIMODE=sync   # Use sync mode to get the detailed layer reports too
HARDWARETYPELIST="CPU GPU MYRIAD"  #Set the hardware, which you want to do inference on

Inference Script TensorFlow for EfficientDet

The script tf2_effdet_train_export_inf_TEMPLATE.sh trains and executes the model in the TEMPLATE on TF2 for AutoML EfficientDet.

For inference, TEMPLATE has to be replaced by the network name that shall be trained. In case you don not use the add_folder... scripts, you can manually prepare the scripts. First copy tf2_effdet_train_export_inf_TEMPLATE.sh and rename it to fit your network, e.g. tf2_effdet_train_export_inf_tf2_efficientdet-d0_640x640_oxfordpets.sh.

For each network to be trained, the following constants have to be adapted:

USERNAME=wendt    # Set your user name
USEREMAIL=alexander.wendt@tuwien.ac.at   # Set your email
SCRIPTPREFIX=../../eml-tools    # No need to change this
DATASET_INFERENCE=/home/intel-nuc/eml-tools/datasets/dataset-oxford-pets-val-debug    #Set your debug dataset
HARDWARENAME=IntelNUC   # Set your hardware device id
# Model type: The name is a value used to find the model to used for training. Default is efficientdet-d1. 
MODEL_TYPE=efficientdet-d0    # Not used for inference, only for training
#NUMBEREPOCHS=300    # Not used for inference, only for training
NUMBEREPOCHS=1    # Not used for inference, only for training
NUMBEREXAMPLES=2000    # Not used for inference, only for training
# Set this variable true if the network shall be trained, else only inference shall be performed
TRAINNETWORK=false    # Set false for inference only on the target devices

Send Mail

To get notifications when the jobs start and stop, the program can send mails to the user. Each Sendmail script consists of a prefix sendmail_[TEMPLATE] and a action-hardware part. The script reads the latter part and uses it as subject and content in a mail. Here is an example script for sending a mail when the TF2 inference starts on the IntelNUC: sendmail_Start_TF2_IntelNUC.sh

For the proper scripts, the following constants need to be changed:

USEREMAIL=alexander.wendt@tuwien.ac.at

Add Folder Jobs

add_folder_conv_ir.sh adds conversion jobs to the task spooler. Nothing needs to be changed here.

add_folder_inftf2effdet_jobs.sh loads all exported models ./exported-models and puts their execution scripts in the task spooler. The execution script templates work like this: The add jobs script makes a copy of tf2_effdet_train_export_inf_TEMPLATE.sh and replaces TEMPLATE with the model name. Then, it adds these two scripts to the task spooler. No script adaptions are necessary.

add_folder_infopenvino_jobs.sh loads all converted IR models from ./exported-models-openvino and puts their execution scripts in the task spooler. The execution script templates work like this: The add jobs script makes a copy of openvino_inf_eval_saved_model_TEMPLATE.sh and replaces TEMPLATE with the model name. Then, it adds these two scripts to the task spooler. No script adaptions are necessary.

Running the system

1. Convert the models you want to run with ./add_folder_conv_ir.sh and wait for the job to finish
2. Run add folder job scripts: add_folder_inftf2_jobs.sh to add normal execution and add_folder_infopenvino_jobs.sh to run OpenVino execution

The results can be found in ./results.

Common Problems

Task Spooler Blocked

If the task spooler freezes or is blocked, the following error message is shown:

=== Init task spooler ===
Setup task spooler socket for GPU.
chmod: changing permissions of '/srv/ts_socket/GPU.socket': Operation not permitted
task spooler output directory: /home/wendt/logs
Task spooler initialized /srv/ts_socket/GPU.socket
(tf24) [wendt@eda02 graz-pedestrian]$ ts -l
c: cannot connect to the server
(tf24) [wendt@eda02 graz-pedestrian]$

The cause is the a user blocks the task spooler and nobody else has access rights. It has to be released by the user or a sudo-user. The solution is to put the following command line into the task spooler script: chmod 777 /srv/ts_socket/GPU.socket

Windows Instead of Linux EOL in Files

Note: If you get this error: /bin/bash^M: bad interpreter or other strange execution problems, then you might use Windows EOL. To correct it, change EOL to Unix.

Embedded Machine Learning Laboratory

This repository is part of the Embedded Machine Learning Laboratory at the TU Wien. For more useful guides and various scripts for many different platforms visit our EML-Tools: https://github.com/embedded-machine-learning/eml-tools.

Our newest projects can be viewed on our webpage: https://eml.ict.tuwien.ac.at/