How to use my image data to get the cape model

[yangbiaobiao500]

If I know the pose parameters, how can I use the code to obtain the cape model? I am not sure where I should modify the code. I look forward to your reply. I appreciate it.

[qianlim]

Hi! In the simplest scenario, to get a clothed body from SMPL pose params, you can take a look at the function sample_vary_clotype in demos.py as an example (only gives a random clothed body for the specified pose though). Basically: first convert the 72-dim pose param to the 216-dim flattend rotational matrix by using Rodrigues formula; then take only the rot params from clothing-related joints, encode this into pose embeddings (line 184). Then feed the pose and clothing type embeddings, together with the shape latent code z, to the CAPE decoder (line 205). Then just add the predicted displacements to the SMPL body and repose.

To get the CAPE body matching the observed shape in images, the process is a bit more envolved, as one needs to optimize the latent code:

1. Customize the SMPL model by adding the clothing displacements produced by CAPE to the SMPL canonical body template after applying pose-correctives, before doing skinning (i.e. posing the model).
2. Then, to relate this clothed body to the image, assume that the camera parameters are available (it should be there if you have acquired the pose parameters by fitting SMPL to the image via e.g. SMPLify or HMR), one can project the clothed body to the image, and compute the difference in the predicted and ground truth silhouette. Optimize the body shape, pose, clothing shape z, and the clothing type jointly to minimize the silhouette difference. You can see our Supplementary materials for more details.

This is best done when the SMPL model loading and the optimization are done in Tensorflow. Unfortunately our customized tensorflow implementation of SMPL is not available at the moment. I will probably reimplement CAPE in Pytorch, which will then be easily compatible with the smplx package.

Feel free to ping me if you have further questions.

[yangbiaobiao500]

Thank you very much for your reply, but I tried to modify it, but I still get the undesired result. After I get the pose parameters from HMR, where should I modify your code so that I can get the CAPE model, very Looking forward to your reply again, I am very grateful!

[yangbiaobiao500]

Use Rodrigues formula to convert 72-dimensional attitude parameters into 216-dimensional rotation matrix. Is it a direct calculation or is there a part of the code in the smplify and HMR algorithms?

[yangbiaobiao500]

A pose parameter can get a 3×3 matrix through Rodrigues formula, then 72 pose parameters get 72 3×3 matrices, then in the data set you gave, the 72-dimensional pose vector in rot.npy corresponds to 216 dimensions Rot vector, I want to ask, how to get this result

[yangbiaobiao500]

Hello, I can now get the CAPE model of my own data. The first picture shows my own CAPE model, and the second picture shows your model. It can be seen that my model is far inferior to yours, even a bit I can't see the effect of the clothes. I would like to ask you, if you want to get an effect like yours, do you need to do anything with the CAPE model? I look forward to your reply, and I am very grateful!

[qianlim]

Hi, is this result done by fitting the pre-trained model to your image data by optimizing over the z variable to minimize the silhouette loss? If that's the case I'm afraid there's not much that can be done. It's likely due to that after optimization the z variable can minimize the silh loss but goes far from the training latent space, hence showing a mean-like clothing shape.

However the visual effect can be affected by how you render it too. You can try turning wireframe on, and try different shading options in Meshlab and get less smooth visualization.

Re your previous pose-rotation matrix question, see this line.

[yangbiaobiao500]

Thank you very much for your reply, yes, I am trying to use the contour information to further optimize the z hidden space, but the result of the CAPE model is not significant. As you said, It's likely due to that after optimization the z variable can minimize the silh loss but goes far from the training latent space, hence showing a mean-like clothing shape. But the effect of the CAPE model I got with the data you provided is also the same, I also tried to choose other renderings in meshlab, but got The results are still poor, so I would like to ask you, how did you get the results of the CAPE model in your paper? What kind of rendering process has gone through? Finally, the CAPE model of the light blue grid can be obtained. I look forward to your answers very much, I am very happy to communicate with you, I am very grateful!

[yangbiaobiao500]

In addition, I have a small question, how to project CAPE onto the contour of the human body in the picture, thank you!

[yangbiaobiao500]

My CAPE model looks like this

[qianlim]

If you render it with Blender with proper lighting / material setup, your mesh provided above can be rendered as good as the CAPE teaser figure (sorry for not giving further tutorials on Blender here).
Regarding silhouette projection, in SMPLify you get the camera parameters in the first optimization step, then simply project the body mesh with this camera. More details can be found in our supplementary or many relevant papers e.g. SMPLX, HMR (where they project keypoints) and Unite the People (silhouettes too), etc.

[enriccorona]

Hi! I'm trying to fit CAPE on a 3D scan given the cloth segmentations. Do you have a sample code to do something like that? I need to fit the SMPL parameters too, but the smpl-x layer is on pytorch and so it won't be easily integrated with tensorflow to propagate the error to the parameters. What do you think?

I'm very used to pytorch and have used smpl-x before on pytorch. I've read your comment on porting the model to pytorch, did you finally do that? Or have the model definition in pytorch? If that's the case, I would try to port the trained model weights into pytorch and share the final code I used and weights here.

[qianlim]

Hi! I'm trying to fit CAPE on a 3D scan given the cloth segmentations. Do you have a sample code to do something like that? I need to fit the SMPL parameters too, but the smpl-x layer is on pytorch and so it won't be easily integrated with tensorflow to propagate the error to the parameters. What do you think?

I'm very used to pytorch and have used smpl-x before on pytorch. I've read your comment on porting the model to pytorch, did you finally do that? Or have the model definition in pytorch? If that's the case, I would try to port the trained model weights into pytorch and share the final code I used and weights here.

Hi! For fitting to 3D data I do have tf code but they are too messy to release :) But in principle it's the same as fitting to images: load the decoder and optimize over the z, pose and body betas w.r.t. chamfer dist or point to surface distance etc. However since the model preds a single mesh I'm not sure if cloth segmentations would be useful.
I'll try to reimplement the code in pytorch after cvpr deadline together with sample codes for fitting to images / 3D data (sorry for the delay).

[yangbiaobiao500]

Hello, as you said, posture can control clothing. In your code, you only need to provide 72 dimensional posture parameters, and convert them into 216 dimensional rotation matrix, then you can get the Cape model. So I would like to ask, body shape parameters, don't you need to input body shape parameters to control the Cape model? I'm in a lot of trouble. I'm looking forward to your reply. Thank you very much!

[qianlim]

Hi, in this work we do not consider the influence of body shape on the clothing deformation. Of course you can change the SMPL shape parameters to get different underlying bodies, however that won't affect the clothing displacements. Please refer to our CVPR paper for details.