Skip to content

Commit e0c55d6

Browse files
KHao123KHao123
authored
[Feature] Add 4d association method (#16)
* add 4dag method * rename fourd to bottom up, add metric and alignment, change name manner * rename some files and variables * fix a bug * delete Camera class in association * change configs * modify triangulation pipeline * convert data format form pkl to json * align triangulate method, align kps convention * delete fourdag triangulate, add fourdag19 convention pipeline * change some names * resolve conflict * resolve some comments * run pre-commit * align result * refactor code * refactor triangulator and optimization * resolve comments * pre-commit * change limb info * fix bug * add stringdoc * rename joint to kps * add yapf * pre-commit * add docstring * refactor * add limb info json file * fix bug * delete debug call * add readme * rephase term class * rephase associate * add cloud file and path * process fourdag seq5 * resolve comments
1 parent 1d02098 commit e0c55d6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+4101
-12
lines changed

‎configs/fourdag/README.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# 4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras
2+
Note: As a python variable name cannot start with a number, we refer to this method as `FourDAG` in the following text and code.
3+
4+
- [Introduction](#introduction)
5+
- [Prepare limb information and datasets](#prepare-limb-information-and-datasets)
6+
- [Results](#results)
7+
- [Campus](#campus)
8+
- [Shelf](#shelf)
9+
- [FourDAG](#fourdag-1)
10+
11+
## Introduction
12+
13+
We provide the config files for FourDAG: [4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras](https://arxiv.org/abs/2002.12625).
14+
15+
16+
[Official Implementation](https://github.com/zhangyux15/4d_association)
17+
18+
```BibTeX
19+
@inproceedings{Zhang20204DAG,
20+
title={4D Association Graph for Realtime Multi-Person Motion Capture Using Multiple Video Cameras},
21+
author={Yuxiang Zhang and Liang An and Tao Yu and Xiu Li and Kun Li and Yebin Liu},
22+
journal={IEEE/CVF Conference on Computer Vision and Pattern Recognition},
23+
year={2020},
24+
pages={1321-1330}
25+
}
26+
```
27+
## Prepare limb information and datasets
28+
29+
- **Prepare limb information**:
30+
31+
```
32+
sh scripts/download_weight.sh
33+
```
34+
You could find perception models in `weight` file.
35+
36+
- **Prepare the datasets**:
37+
38+
You could download Shelf, Campus or FourDAG datasets, and convert original dataset to our unified meta-data. Considering that it takes long to run a converter, we have done it for you. Please download compressed zip file for converted meta-data from [here](../../docs/en/dataset_preparation.md), and place meta-data under `ROOT/xrmocap_data/DATASET`.
39+
40+
The final file structure would be like:
41+
42+
```text
43+
xrmocap
44+
├── xrmocap
45+
├── docs
46+
├── tools
47+
├── configs
48+
├── weight
49+
| └── limb_info.json
50+
└── xrmocap_data
51+
├── CampusSeq1
52+
├── Shelf
53+
| ├── Camera0
54+
| ├── ...
55+
| ├── Camera4
56+
| └── xrmocap_meta_testset
57+
└── FourDAG
58+
├── seq2
59+
├── seq4
60+
├── seq5
61+
├── xrmocap_meta_seq2
62+
├── xrmocap_meta_seq4
63+
└── xrmocap_meta_seq5
64+
```
65+
You can download just one dataset of Shelf, Campus and FourDAG.
66+
67+
## Results
68+
69+
We evaluate FourDAG on 3 benchmarks, report the Percentage of Correct Parts (PCP) on Shelf/Campus/FourDAG datasets.
70+
71+
You can find the recommended configs in `configs/foudage/*/eval_keypoints3d.py`.
72+
73+
74+
### Campus
75+
76+
The 2D keypoints and pafs data we use is generated by openpose, and you can download it from [here](/docs/en/dataset_preparation.md#download-converted-meta-data).
77+
78+
| Config | Actor 0 | Actor 1 | Actor 2 | Average | Download |
79+
|:------:|:-------:|:--------:|:--------:|:--------:|:--------:|
80+
| [eval_keypoints3d.py](./campus_config/eval_keypoints3d.py) | 64.26 | 90.64 | 86.27 | 80.39 | [log](https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrmocap/logs/FourDAG/campus.zip) |
81+
82+
83+
### Shelf
84+
85+
The 2D keypoints and pafs data we use is generated by fasterrcnn, and you can download it from [here](/docs/en/dataset_preparation.md#download-converted-meta-data).
86+
87+
| Config | Actor 0 | Actor 1 | Actor 2 | Average | Download |
88+
|:------:|:-------:|:--------:|:--------:|:--------:|:--------:|
89+
| [eval_keypoints3d.py](./shelf_config/eval_keypoints3d.py) | 99.61 | 96.76 | 98.20 | 98.19 | [log](https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrmocap/logs/FourDAG/shelf.zip) |
90+
91+
92+
### FourDAG
93+
94+
The 2D keypoints and pafs data we use is generated by mmpose, and you can download it from [here](/docs/en/dataset_preparation.md#download-converted-meta-data).
95+
96+
- **seq2**
97+
98+
| Config | Actor 0 | Actor 1 | Average | PCK@200mm | Download |
99+
|:-------:|:--------:|:--------:|:--------:|:--------:|:--------:|
100+
| [eval_keypoints3d.py](./fourdag_config/eval_keypoints3d_seq2.py) | 92.18 | 87.35 | 89.77 | 83.10 | [log](https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrmocap/logs/FourDAG/fourdag.zip) |
101+
102+
- **seq4**
103+
104+
| Config | Actor 0 | Actor 1 | Actor 1 | Average | PCK@200mm | Download |
105+
|:-------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
106+
| [eval_keypoints3d.py](./fourdag_config/eval_keypoints3d_seq4.py) | 91.85 | 86.48 | 92.92 | 90.42 | 81.29 |[log](https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrmocap/logs/FourDAG/fourdag.zip) |

‎configs/fourdag/campus_config/eval_keypoints3d.py

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
type = 'BottomUpAssociationEvaluation'
2+
3+
__data_root__ = './xrmocap_data/CampusSeq1'
4+
__meta_path__ = __data_root__ + '/xrmocap_meta_testset'
5+
6+
logger = None
7+
output_dir = './output/fourdag/CampusSeq1_fourdag_19_FourDAGOptimization/'
8+
pred_kps3d_convention = 'fourdag_19'
9+
eval_kps3d_convention = 'campus'
10+
selected_limbs_name = [
11+
'left_lower_leg', 'right_lower_leg', 'left_upperarm', 'right_upperarm',
12+
'left_forearm', 'right_forearm', 'left_thigh', 'right_thigh'
13+
]
14+
additional_limbs_names = [['jaw', 'headtop']]
15+
16+
associator = dict(
17+
type='FourDAGAssociator',
18+
kps_convention=pred_kps3d_convention,
19+
min_asgn_cnt=10,
20+
use_tracking_edges=True,
21+
keypoints3d_optimizer=dict(
22+
type='FourDAGOptimizer',
23+
triangulator=dict(type='JacobiTriangulator', ),
24+
active_rate=0.5,
25+
min_track_cnt=20,
26+
bone_capacity=30,
27+
w_bone3d=1.0,
28+
w_square_shape=1e-3,
29+
shape_max_iter=5,
30+
w_kps3d=1.0,
31+
w_regular_pose=1e-4,
32+
pose_max_iter=20,
33+
w_kps2d=1e-5,
34+
w_temporal_trans=1e-1 / pow(512 / 2048, 2),
35+
w_temporal_pose=1e-1 / pow(512 / 2048, 2),
36+
min_triangulate_cnt=15,
37+
init_active=0.9,
38+
triangulate_thresh=0.05,
39+
logger=logger,
40+
),
41+
graph_construct=dict(
42+
type='GraphConstruct',
43+
kps_convention=pred_kps3d_convention,
44+
max_epi_dist=0.15,
45+
max_temp_dist=0.2,
46+
normalize_edges=True,
47+
logger=logger,
48+
),
49+
graph_associate=dict(
50+
type='GraphAssociate',
51+
kps_convention=pred_kps3d_convention,
52+
w_epi=2,
53+
w_temp=2,
54+
w_view=2,
55+
w_paf=4,
56+
w_hier=0.5,
57+
c_view_cnt=1.5,
58+
min_check_cnt=1,
59+
logger=logger,
60+
),
61+
logger=logger,
62+
)
63+
64+
dataset = dict(
65+
type='BottomUpMviewMpersonDataset',
66+
data_root=__data_root__,
67+
img_pipeline=[
68+
dict(type='LoadImagePIL'),
69+
dict(type='ToTensor'),
70+
],
71+
meta_path=__meta_path__,
72+
test_mode=True,
73+
shuffled=False,
74+
kps2d_convention=pred_kps3d_convention,
75+
gt_kps3d_convention='campus',
76+
cam_world2cam=True,
77+
)
78+
79+
dataset_visualization = dict(
80+
type='MviewMpersonDataVisualization',
81+
data_root=__data_root__,
82+
output_dir=output_dir,
83+
meta_path=__meta_path__,
84+
pred_kps3d_paths=None,
85+
vis_percep2d=False,
86+
kps2d_convention=pred_kps3d_convention,
87+
vis_gt_kps3d=False,
88+
vis_bottom_up=True,
89+
gt_kps3d_convention=None,
90+
)

‎configs/fourdag/fourdag_config/eval_keypoints3d_seq2.py

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
type = 'BottomUpAssociationEvaluation'
2+
3+
__data_root__ = './xrmocap_data/FourDAG/'
4+
__meta_path__ = __data_root__ + '/xrmocap_meta_seq2'
5+
6+
logger = None
7+
output_dir = './output/fourdag/fourdag_fourdag_19_FourDAGOptimization/'
8+
pred_kps3d_convention = 'fourdag_19'
9+
eval_kps3d_convention = 'campus'
10+
selected_limbs_name = [
11+
'left_lower_leg', 'right_lower_leg', 'left_upperarm', 'right_upperarm',
12+
'left_forearm', 'right_forearm', 'left_thigh', 'right_thigh'
13+
]
14+
# additional_limbs_names = [['jaw', 'headtop']]
15+
16+
associator = dict(
17+
type='FourDAGAssociator',
18+
kps_convention=pred_kps3d_convention,
19+
min_asgn_cnt=5,
20+
use_tracking_edges=True,
21+
keypoints3d_optimizer=dict(
22+
type='FourDAGOptimizer',
23+
triangulator=dict(type='JacobiTriangulator', ),
24+
active_rate=0.5,
25+
min_track_cnt=20,
26+
bone_capacity=30,
27+
w_bone3d=1.0,
28+
w_square_shape=1e-3,
29+
shape_max_iter=5,
30+
w_kps3d=1.0,
31+
w_regular_pose=1e-4,
32+
pose_max_iter=20,
33+
w_kps2d=1e-5,
34+
w_temporal_trans=1e-1 / pow(512 / 2048, 2),
35+
w_temporal_pose=1e-1 / pow(512 / 2048, 2),
36+
min_triangulate_cnt=15,
37+
init_active=0.9,
38+
triangulate_thresh=0.05,
39+
logger=logger,
40+
),
41+
graph_construct=dict(
42+
type='GraphConstruct',
43+
kps_convention=pred_kps3d_convention,
44+
max_epi_dist=0.15,
45+
max_temp_dist=0.3,
46+
normalize_edges=True,
47+
logger=logger,
48+
),
49+
graph_associate=dict(
50+
type='GraphAssociate',
51+
kps_convention=pred_kps3d_convention,
52+
w_epi=1,
53+
w_temp=2,
54+
w_view=1,
55+
w_paf=2,
56+
w_hier=1,
57+
c_view_cnt=1,
58+
min_check_cnt=10,
59+
logger=logger,
60+
),
61+
logger=logger,
62+
)
63+
64+
dataset = dict(
65+
type='BottomUpMviewMpersonDataset',
66+
data_root=__data_root__,
67+
img_pipeline=[
68+
dict(type='LoadImagePIL'),
69+
dict(type='ToTensor'),
70+
],
71+
meta_path=__meta_path__,
72+
test_mode=True,
73+
shuffled=False,
74+
kps2d_convention=pred_kps3d_convention,
75+
gt_kps3d_convention='campus',
76+
cam_world2cam=True,
77+
)
78+
79+
dataset_visualization = dict(
80+
type='MviewMpersonDataVisualization',
81+
data_root=__data_root__,
82+
output_dir=output_dir,
83+
meta_path=__meta_path__,
84+
pred_kps3d_paths=None,
85+
vis_percep2d=False,
86+
kps2d_convention=pred_kps3d_convention,
87+
vis_gt_kps3d=False,
88+
vis_bottom_up=True,
89+
gt_kps3d_convention=None,
90+
resolution=(368, 368),
91+
)

‎configs/fourdag/fourdag_config/eval_keypoints3d_seq4.py

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
type = 'BottomUpAssociationEvaluation'
2+
3+
__data_root__ = './xrmocap_data/FourDAG'
4+
__meta_path__ = __data_root__ + '/xrmocap_meta_seq4'
5+
6+
logger = None
7+
output_dir = './output/fourdag/fourdag_fourdag_19_FourDAGOptimization/'
8+
pred_kps3d_convention = 'fourdag_19'
9+
eval_kps3d_convention = 'campus'
10+
selected_limbs_name = [
11+
'left_lower_leg', 'right_lower_leg', 'left_upperarm', 'right_upperarm',
12+
'left_forearm', 'right_forearm', 'left_thigh', 'right_thigh'
13+
]
14+
# additional_limbs_names = [['jaw', 'headtop']]
15+
16+
associator = dict(
17+
type='FourDAGAssociator',
18+
kps_convention=pred_kps3d_convention,
19+
min_asgn_cnt=5,
20+
use_tracking_edges=True,
21+
keypoints3d_optimizer=dict(
22+
type='FourDAGOptimizer',
23+
triangulator=dict(type='JacobiTriangulator', ),
24+
active_rate=0.5,
25+
min_track_cnt=20,
26+
bone_capacity=30,
27+
w_bone3d=1.0,
28+
w_square_shape=1e-3,
29+
shape_max_iter=5,
30+
w_kps3d=1.0,
31+
w_regular_pose=1e-4,
32+
pose_max_iter=20,
33+
w_kps2d=1e-5,
34+
w_temporal_trans=1e-1 / pow(512 / 2048, 2),
35+
w_temporal_pose=1e-1 / pow(512 / 2048, 2),
36+
min_triangulate_cnt=15,
37+
init_active=0.9,
38+
triangulate_thresh=0.05,
39+
logger=logger,
40+
),
41+
graph_construct=dict(
42+
type='GraphConstruct',
43+
kps_convention=pred_kps3d_convention,
44+
max_epi_dist=0.15,
45+
max_temp_dist=0.3,
46+
normalize_edges=True,
47+
logger=logger,
48+
),
49+
graph_associate=dict(
50+
type='GraphAssociate',
51+
kps_convention=pred_kps3d_convention,
52+
w_epi=1,
53+
w_temp=2,
54+
w_view=1,
55+
w_paf=2,
56+
w_hier=1,
57+
c_view_cnt=1,
58+
min_check_cnt=10,
59+
logger=logger,
60+
),
61+
logger=logger,
62+
)
63+
64+
dataset = dict(
65+
type='BottomUpMviewMpersonDataset',
66+
data_root=__data_root__,
67+
img_pipeline=[
68+
dict(type='LoadImagePIL'),
69+
dict(type='ToTensor'),
70+
],
71+
meta_path=__meta_path__,
72+
test_mode=True,
73+
shuffled=False,
74+
kps2d_convention=pred_kps3d_convention,
75+
gt_kps3d_convention='campus',
76+
cam_world2cam=True,
77+
)
78+
79+
dataset_visualization = dict(
80+
type='MviewMpersonDataVisualization',
81+
data_root=__data_root__,
82+
output_dir=output_dir,
83+
meta_path=__meta_path__,
84+
pred_kps3d_paths=None,
85+
vis_percep2d=False,
86+
kps2d_convention=pred_kps3d_convention,
87+
vis_gt_kps3d=False,
88+
vis_bottom_up=True,
89+
gt_kps3d_convention=None,
90+
resolution=(368, 368),
91+
)

0 commit comments

Comments
 (0)