Vitis-ai very important quick start:
https://xilinx.github.io/Vitis-AI/3.0/html/docs/quickstart/mpsoc.html
https://github.com/Xilinx/Vitis-AI/blob/v3.0/board_setup/mpsoc/board_setup_mpsoc.rst
https://github.com/Xilinx/Vitis-AI/tree/v3.0/examples/vai_runtime
Vitis-ai github:
https://github.com/Xilinx/Vitis-AI
Usb3 install:
https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841729/DWC3+Xilinx+Linux+USB+driver
- Clone SD card for your board, get images from
https://github.com/Xilinx/Vitis-AI/blob/v3.0/board_setup/mpsoc/board_setup_mpsoc.rst - Extract video image file into SD (from quick start)
Download the vitis_ai_runtime_r3.0.0_image_video.tar.gz from host to the target using scp - (Optional) Download the model and extract to SD (didn’t need to do it - it is already there)
- Download DPUCZDX8G.tar.gz and extract it and inspect the content
Read prj/vitis/Readme.rd & prj.vivado/readme.rd
This is the TRD for the creating of dpu.xclbin file in design (vitis v++) ! - (Optional) Run
zynqmp_dpu_optimize.shto optimize board settings.
The script runs automatically after the board boots up with the official image. But you can also find thedpu_sw_optimize.tar.gzin DPUCZDX8G.tar.gz (download it)
cd ~/dpu_sw_optimize/zynqmp/
./zynqmp_dpu_optimize.sh - Install petalinux 2022.2
There are some vitis-ai docker in hub
https://hub.docker.com/search?q=xilinx+vitis
Useful:
https://hub.docker.com/r/xilinx/vitis-ai-pytorch-cpu
https://hub.docker.com/r/xilinx/vitis-ai-tensorflow2-cpu
Note: there is also xilinx/vitis-ai-cpu but it seemed older so I did not use it.
But I wanted gpu tf2 and did not find it so built it myself here:
cd~/projects/vitis-ai/Vitis-AI/docker
./docker_build.sh -f tf2 -t gpu
Enter docker using:
sudo ./docker_run.sh xilinx/vitis-ai-tensorflow2-gpu:latest
And then inside docker:
conda activate vitis-ai-tensorflow2
- Zcu102 board
- Camera usb3 - fix switches as described
https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1586626609/USB+Boot+example+using+ZCU102+Host+and+ZCU102+Device
3 switches in total - we had to change the defaults- J7 – 1-2 Open
- J113 - 1-2 Close
- J110 - 1-2 Close
- Once we connect a simple camera we can see in dmesg that it is detected
- SD card with image for zcu102 see above link
- Serial to usb
- Ethernet connection
- Set boot switch: booting from sd
Note: we have 2 zcu102 with rev 1.1, yet on running examples on one of them it fails on failed fingerprints, i.e. 2 cores with correct finger print but the 3rd one has a fault finger print
TODO: should try to run _optimize script see vitis-ai q&a
various examples:
1. stability of image
2. detect of target (person, car, etc)
3. Tracking of objects
Check resolution:
ffprobe video/adas.webm
convert video to tiff in 1920x1080 resolution:
ffmpeg -i *.webm -vf scale=1920:1080 output_%04d.tiff
modify readfile in main.cc of adas_detection
2 options:
- Build on host using cross compiler (https://xilinx.github.io/Vitis-AI/3.0/html/docs/quickstart/mpsoc.html)
[Host] $ cd Vitis-AI/board_setup/mpsoc
Only for the first time:
[Host] $ sudo chmod u+r+x host_cross_compiler_setup.sh
[Host] $ ./host_cross_compiler_setup.sh
When the installation is complete, follow the prompts and execute the following command:
[Host] $ source ~/petalinux_sdk_2022.2/environment-setup-cortexa72-cortexa53-xilinx-linux
cd ~/projects/vitis-ai/Vitis-AI/examples/vai_runtime/adas_detection
bash +x build.sh
- Run docker source cross-compiler and build
cd ~/projects/vitis-ai/Vitis-AI/examples/vai_runtime/adas_detection
bash +x build.sh
SD IMAGE backup
https://drive.google.com/file/d/174lMQa_VEUNpPDdsyLyodPiHVbvvMiBP/view?usp=sharing
https://drive.google.com/file/d/1v2iE4ZZOMogsV0mNDJXu6V1RWs8Xniro/view?usp=sharing
Vitis-AI code change backup
https://github.com/ranshalit/ai_demos
Run Application example
[HOST]
Sudo ifconfig <> 192.168.1.101
scp adas_detection root@192.168.1.100
scp tiff folder to /tmp in target
ssh -X root@192.168.1.100
Run any example according to table in:
https://github.com/Xilinx/Vitis-AI/blob/master/examples/vai_runtime/README.md
E.g.
./adas_detection video/adas.webm /usr/share/vitis_ai_library/models/yolov3_adas_pruned_0_9/yolov3_adas_pruned_0_9.xmodel
Running our own built “adas_demo”:
in target:
"Usage of ADAS detection: "
" <video> <model_file> <input type> <model type>"
"input type: (0-video file (.webm), 1-folder of tiffs, 2-camera (dev/video0)"
"model type: 0-adas, 1-yolo3 2-custom model"
E.g.
Running on 80 class darknet yolo3 model:
~/adas_detection /dev/video0 ~/yolov3_tf2.xmodel 0 1
Running on my own yolo3 trained model (20 classes)
~/adas_detection /dev/video0 ~/test.xmodel 0 2
Download model
cd ~/projects/vitis-ai/Vitis-AI/model_zoo/
python 3 download.py
choose tf2 yolov3_coco_416_416_65.9G_3.0
extract tar and follow the README.md !
If you use docker then no need to do these install from readme:
(Environment requirement
- anaconda3
- tensorflow 2.3
- cython, opencv, tqdm etc. (refer to requirements for more details)
Installation
conda create -n yolov3-tf2 python=3.6
source activate yolov3-tf2
# conda activate yolov3-tf2
pip install python
pip install --user -r requirements.txt
bash code/test/download_data.sh
bash code/test/convert_data.sh
Images will be extracted to
/home/ranshal/projects/vitis-ai/Vitis-AI/model_zoo/tf2_yolov3_coco_416_416_65.9G_3.0/data/val2017/000000500565.jpg
….
And annotation into:
/home/ranshal/projects/vitis-ai/Vitis-AI/model_zoo/tf2_yolov3_coco_416_416_65.9G_3.0/data/annotations/instances_val2017.json
…..
Dataset directory structure:
# val2017 and annotations are unpacked from the downloaded data
+ data
+ val2017
+ 000000000139.jpg
+ 000000000285.jpg
+ ...
+ annotations
+ instances_train2017.json
+ instances_val2017.json
+ ...
+ val2017.txt <- I think this is the result of prepare_data.sh
Download the official darknet weights and convert to tensorflow weights of .h5 format
cd code/test
wget -O ../../float/yolov3.weights https://github.com/AlexeyAB/darknet/releases/download/darknet\_yolo\_v3\_optimal/yolov3.weight
python tools/model_converter/convert.py cfg/yolov3.cfg ../../float/yolov3.weights ../../float/yolov3.h5
mkdir custom_weight
wget https://github.com/AlexeyAB/darknet/releases/download/darknet\_yolo\_v3\_optimal/yolov3.weights
Evaluation Configure the model path and data path in code/test/run_eval.sh
# run the script under 'code/test'
bash run_eval.sh
- Data preprocess
data channel order: RGB(0~255)
resize: keep aspect ratio of the raw image and resize it to make the length of the longer side equal to 416
padding: pad along the short side with pixel value 128 to generate the input image with size = 416 x 416
input = input / 255
- Node information
input node: 'image_input:0'
output nodes: 'conv2d_58:0', 'conv2d_66:0', 'conv2d_74:0'
Evaluate accuracy of quant model (tf2)
To evaluate the quantized model after you have generated it (saved as ./quantized/quantized.h5), you can use the same eval.py script but with the quantized model path and the --quant flag to indicate it's a quantized model.
Here's how to do it:
1st option is to run
/code/test/run_eval_qual.sh
But you might need to edit that file (see the links to files at the start)
The 2nd option is to use these 2 calls:
- python eval.py --model_path=./quantized/quantized.h5 --anchors_path=configs/yolo3_anchors.txt --classes_path=configs/coco_classes.txt --annotation_file=../../data/val2017.txt --model_input_shape=416x416 --quant --eval_only
- python tools/evaluation/pycoco_eval.py --result_txt result/detection_result.txt --coco_annotation_json ../../data/annotations/instances_val2017.json --coco_result_json result/detection_result_iou0.45.json
see https://www.hackster.io/LogicTronix/zcu102-vitis-dpu-trd-vitis-ai-3-0-c51609
vai_c_tensorflow2 \
--model quantized/quantized.h5 \
--arch /opt/vitis_ai/compiler/arch/DPUCZDX8G/ZCU102/arch.json \
--output_dir compiled_model \
--net_name yolov3_tf2 \
--options "{'input_shape':'1,416,416,3'}"
Note: if during running example it complaint about wrong fingerprint, then try to modify arch.json in this command
As a preparation might need to create val2017.txt, a file which has links to dataset and provide annotation to each of the image, i.e. this format
ranshal@D-P-RANSHAL-L-RF-L-RF:~/projects/vitis-ai/Vitis-AI/model_zoo/tf2_yolov3_coco_416_416_65.9G_3.0$ head data/val2017.txt
/workspace/model_zoo/tf2_yolov3_coco_416_416_65.9G_3.0/data/val2017/000000289343.jpg 473,395,511,423,16 204,235,264,412,0 0,499,339,605,13 204,304,256,456,1
/workspace/model_zoo/tf2_yolov3_coco_416_416_65.9G_3.0/data/val2017/000000061471.jpg 272,200,423,479,16 181,86,208,159,39 174,0,435,220,61
But no need to create it yourself! Just run the required preparation script
It depends on annotation format for example for coco use:
python ./tf2_yolov3_coco_416_416_65.9G_3.0/code/test/tools/dataset_converter/coco_annotation_val.py
this will generate a file like
./../data/val2017.txt \
Note:
coco_annotation.py |
Converts the training set (e.g., train2017) COCO annotations into YOLO/other formats |
|---|
coco_annotation_val.py |
Converts the validation set (e.g., val2017) COCO annotations into YOLO/other formats |
|---|
python ./tf2_yolov3_coco_416_416_65.9G_3.0/code/test/tools/dataset_converter/coco_annotation.py
./../data/train2017.txt \
python ./tf2_yolov3_coco_416_416_65.9G_3.0/code/test/tools/dataset_converter/coco_annotation_val.py
this will generate a file like
./../data/val2017.txt \
python eval.py --model_path=../../float/yolov3.h5 --anchors_path=configs/yolo3_anchors.txt --classes_path=configs/coco_classes.txt --annotation_file=../../data/val2017.txt --model_input_shape=416x416 --quant
The quantized model will be saved here:
./quantized/quantized.h5
Note: darknet is alternative to tf2, and originally yolo3 was created in darknet
Do it all from host (no docker):
Doing it with AlexeyAB’s repo and VOC sample:
- Clone AlexeyAB darknet repo:
git clone https://github.com/AlexeyAB/darknet.git
cd darknet
- Compile darknet:
sed -i 's/GPU=0/GPU=1/' Makefile
sed -i 's/CUDNN=0/CUDNN=1/' Makefile
sed -i 's/OPENCV=0/OPENCV=1/' Makefile
make -j$(nproc)
Note: Actually I worked with GPU=1 and CUDNN=0, OPENCV=1, maybe that’s why it was quite slow (18 hours for 50000 iterations)
- Download VOC dataset and config files:
mkdir data/voc
cd data/voc
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval\_11-May-2012.tar
tar xf VOCtrainval_11-May-2012.tar
cd ../..
python3 voc_label.py
cd tools/dataset_converter/
python voc_annotation.py --dataset_path /workspace/darknet/data/voc/VOCdevkit --year 2012 --set train --output_path . --classes_path /workspace/darknet/data/voc.names
this will generate a file like:
2012_train.txt, we then copy to darknet main folder
- Download pretrained weights for darknet53 backbone:
https://sourceforge.net/projects/yolov3.mirror/files/v8/darknet53.conv.74/download
cp darknet53.conv.74 backup/
- Start training on VOC (20 classes):
vi cfg/voc.data
classes= 20
train = /home/ranshal/projects/vitis-ai/Vitis-AI/darknet/data/voc/2012_train.txt
valid = /home/ranshal/projects/vitis-ai/Vitis-AI/darknet/data/voc/2012_val.txt
names = data/voc.names
backup = /home/ranshal/projects/vitis-ai/Vitis-AI/darknet/backup
vi cfg/yolov3-voc.cfg
[net]
# Testing
batch=8 ←-
subdivisions=2 ←-
# Training
# batch=64
# subdivisions=16
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
Also set random=0 in all relevant places, otherwise it may create random size of images
./darknet detector train cfg/voc.data cfg/yolov3-voc.cfg backup/darknet53.conv.74 -dont_show
-
Convert weight(darknet) to h5(tensorflow)
cd workspace/model_zoo/tf2_yolov3_coco_416_416_65.9G_3.0/code/test/tools/model_converter
python3 convert.py /workspace/darknet/cfg/yolov3-voc.cfg /workspace/darknet/yolov3-voc_last.weights /workspace/darknet/yolov3_converted.h5 -
Quantize h5
python eval.py --model_path=/workspace/darknet/yolov3_converted.h5 --anchors_path=/workspace/darknet/yolov3_converted_anchors.txt --classes_path=/workspace/darknet/data/voc.names --annotation_file=/workspace/darknet/2012_train.txt --model_input_shape=416x416 --quant
- Compile the quantized model h5 into xmodel
vai_c_tensorflow2 --model /workspace/darknet/quantized.h5 --arch /opt/vitis_ai/compiler/arch/DPUCZDX8G/ZCU102/arch.json --output_dir compiled_model --net_name yolov3_tf2 --options "{'input_shape':'1,416,416,3'}"
Hooray! We got a model ./compiled_model/yolov3_tf2.xmodel
xdputil xmodel -l compiled_model/yolov3_tf2.xmodel
Now we can use this name in our application !
Build using petalinux https://github.com/Xilinx/Vitis-AI/blob/v3.0/src/vai_petalinux_recipes/recipes-vai-kernel/linux-xlnx/bsp.cfg
Added manually
Also added CONFIG_DMA_PROXY
I used adas_detection example and modified it:
-
Added camera input
-
Added support for different model: yolo3 (tf2 - 80 class), yolo3(darknet 20 class)
-
Added input options: folder of tiffs
"Usage of ADAS detection: "
" <video> <model_file> <input type> <model type>"
"input type: (0-video file (.webm), 1-folder of tiffs, 2-camera (dev/video0)"
"model type: 0-adas, 1-yolo3 2-custome model"
## ✅ Step-by-Step Instructions
### 📁 1. Move the file to your project directory
Assuming your working directory is:
```
/workspace/model_zoo/tf_resnetv1_50_imagenet_224_224_6.97G_3.0/
```
Move the tar file there (if not already there):
mv ~/Downloads/ILSVRC2012_img_val.tar ./data/
After that, it should exist as:
./data/ILSVRC2012_img_val.tar
### 📦 2. Extract the images to the `validation/` folder
```bash
mkdir -p ./data/validation
tar -xf ./data/ILSVRC2012_img_val.tar -C ./data/validation/
```
This creates:
./data/validation/ILSVRC2012_val_00000001.JPEG
./data/validation/ILSVRC2012_val_00000002.JPEG
You should see around **50,000 JPEG files**.
### 🧪 3. Verify the extraction
Run:
```bash
ls ./data/validation | wc -l
```
Should output:
```
50000
```
---
### 🧾 4. Run `get_dataset.sh` to preprocess the dataset
Now go to the script folder and run:
```bash
cd code/gen_data
./get_dataset.sh
```
This script will:
* Take the raw images from `../../data/validation/`
* Generate:
* `../../data/Imagenet/val_dataset/` → resized & cropped images
* `../../data/Imagenet/val.txt` → labels
---
### ✅ 5. Validate Output
Check a few files:
```bash
ls ../../data/Imagenet/val_dataset | head
head ../../data/Imagenet/val.txt
```
Also count:
```bash
ls ../../data/Imagenet/val_dataset | wc -l
wc -l ../../data/Imagenet/val.txt
```
Expect both to be \~50,000.
---
You're now ready to run:
```bash
cd ../test
bash run_eval_float_pb.sh
```
vai_c_tensorflow --arch ./arch_zcu102_2xb512_march2023.json -f quantized/quantized_baseline_6.96B_919.pb --output_dir compile_result_zcu102 -n tf_resnet50