RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle)`

[zhang-hao-yuan]

python demo.py --image-folder samples --exp-cfg data/conf.yaml --output-folder OUTPUT_FOLDER --show False --save-params True --save-vis True --save-mesh True
Processing with R-CNN: 0%| | 0/1 [00:00<?, ?it/s]/home/na/anaconda3/envs/expose/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Processing with R-CNN: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1.05it/s]
2024-09-19 12:28:48.391 | INFO | main:main:241 - Saving results to: OUTPUT_FOLDER
2024-09-19 12:28:48.393 | WARNING | expose.models.attention.predictor:init:91 - Apply hand network on body: True
2024-09-19 12:28:48.393 | WARNING | expose.models.attention.predictor:init:93 - Apply hand network on hands: True
2024-09-19 12:28:48.393 | WARNING | expose.models.attention.predictor:init:95 - Predict hands: True
2024-09-19 12:28:48.393 | WARNING | expose.models.attention.predictor:init:102 - Predict head: True
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:109 - Condition hand on body: True
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:112 - Condition hand wrist pose on body: True
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:117 - Condition hand finger pose on body: True
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:121 - Condition hand shape on body shape: False
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:169 - Condition head on body: True
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:170 - Condition expression on body: True
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:172 - Condition shape on body: False
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:173 - Condition neck pose on body: False
2024-09-19 12:28:48.393 | INFO | expose.models.attention.predictor:init:175 - Condition jaw pose on body: True
2024-09-19 12:28:48.849 | INFO | expose.models.attention.predictor:init:252 - Body model: SMPLXLayer(
Gender: NEUTRAL
Number of joints: 55
Betas: 10
Number of PCA components: 6
Flat hand mean: False
Number of Expression Coefficients: 10
(vertex_joint_selector): VertexJointSelector()
)
2024-09-19 12:28:49.034 | INFO | expose.models.backbone.hrnet:init_weights:487 - => init weights from normal distribution
2024-09-19 12:28:49.293 | WARNING | expose.models.backbone.hrnet:load_weights:519 - => please download pre-trained models first!
2024-09-19 12:28:49.293 | WARNING | expose.models.backbone.hrnet:load_weights:520 - data/network_weights/hrnet/imagenet/hrnet_w48-8ef0771d.pth does not exist!
2024-09-19 12:28:49.305 | INFO | expose.models.common.networks:init:267 - Building iterative regressor with 3 stages
2024-09-19 12:28:49.305 | INFO | expose.models.common.networks:init:282 - Detach mean: False
2024-09-19 12:28:49.363 | INFO | expose.models.backbone.resnet:resnet18:113 - Loading pretrained ResNet-18
2024-09-19 12:28:49.386 | INFO | expose.models.common.networks:init:267 - Building iterative regressor with 3 stages
2024-09-19 12:28:49.387 | INFO | expose.models.common.networks:init:282 - Detach mean: False
2024-09-19 12:28:49.387 | INFO | expose.models.attention.head_predictor:init:81 - Building head predictor with 3 stages
2024-09-19 12:28:49.438 | INFO | expose.models.backbone.resnet:resnet18:113 - Loading pretrained ResNet-18
2024-09-19 12:28:49.462 | INFO | expose.models.common.networks:init:267 - Building iterative regressor with 3 stages
2024-09-19 12:28:49.462 | INFO | expose.models.common.networks:init:282 - Detach mean: False
2024-09-19 12:28:49.462 | INFO | expose.models.attention.predictor:init:490 - 2D Head crop keyps loss: KeypointLoss(Norm type: L1)
2024-09-19 12:28:49.462 | INFO | expose.models.attention.predictor:init:502 - 2D Left hand crop keyps loss: KeypointLoss(Norm type: L1)
2024-09-19 12:28:49.462 | INFO | expose.models.attention.predictor:init:515 - 2D Left hand crop keyps loss: KeypointLoss(Norm type: L1)
2024-09-19 12:28:49.462 | INFO | expose.models.common.smplx_loss_modules:init:48 - Stages to penalize: [-1]
2024-09-19 12:28:49.463 | INFO | expose.models.common.smplx_loss_modules:init:400 - Stages to regularize: [-1]
2024-09-19 12:28:49.558 | INFO | expose.utils.checkpointer:init:44 - Creating directory data/checkpoints
2024-09-19 12:28:49.559 | INFO | expose.utils.checkpointer:load_checkpoint:90 - Load pretrained: False
2024-09-19 12:28:49.559 | WARNING | expose.utils.checkpointer:load_checkpoint:93 - Loading checkpoint from data/checkpoints/model.ckpt!
2024-09-19 12:28:49.841 | WARNING | expose.utils.checkpointer:load_checkpoint:121 - The following keys were not found: ['smplx.head_idxs', 'smplx.body_model.left_hand_components', 'smplx.body_model.right_hand_components', 'smplx.body_model.left_hand_mean', 'smplx.body_model.right_hand_mean', 'smplx.body_model.pose_mean', 'smplx.body_model.dynamic_lmk_bary_coords']
2024-09-19 12:28:49.841 | WARNING | expose.utils.checkpointer:load_checkpoint:124 - The following keys were not expected: ['smplx.body_model.source_idxs', 'smplx.body_model.target_idxs', 'smplx.body_model.extra_joint_regressor', 'smplx.body_model.dynamic_lmk_b_coords', 'smplx.hand_predictor.hand_offset', 'smplx.hand_predictor.hand_model.extra_joints_idxs', 'smplx.hand_predictor.hand_model.faces_tensor', 'smplx.hand_predictor.hand_model.v_template', 'smplx.hand_predictor.hand_model.shapedirs', 'smplx.hand_predictor.hand_model.J_regressor', 'smplx.hand_predictor.hand_model.posedirs', 'smplx.hand_predictor.hand_model.parents', 'smplx.hand_predictor.hand_model.lbs_weights', 'smplx.hand_predictor.pca_decoder.pca_basis', 'smplx.hand_predictor.pca_decoder.inv_pca_basis', 'smplx.hand_predictor.pca_decoder.mean', 'smplx.head_predictor.head_offset', 'smplx.head_predictor.head_vertices_ids', 'smplx.head_predictor.head_model.faces_tensor', 'smplx.head_predictor.head_model.v_template', 'smplx.head_predictor.head_model.shapedirs', 'smplx.head_predictor.head_model.expr_dirs', 'smplx.head_predictor.head_model.J_regressor', 'smplx.head_predictor.head_model.posedirs', 'smplx.head_predictor.head_model.parents', 'smplx.head_predictor.head_model.lbs_weights', 'smplx.head_predictor.head_model.lmk_faces_idx', 'smplx.head_predictor.head_model.lmk_bary_coords', 'smplx.head_predictor.head_model.dynamic_lmk_faces_idx', 'smplx.head_predictor.head_model.dynamic_lmk_b_coords', 'smplx.head_predictor.head_model.neck_kin_chain', 'smplx.body_loss.edge_loss.gt_connections', 'smplx.body_loss.edge_loss.est_connections', 'smplx.hand_loss.edge_loss.gt_connections', 'smplx.hand_loss.edge_loss.est_connections', 'smplx.head_loss.edge_loss.gt_connections', 'smplx.head_loss.edge_loss.est_connections']
0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "demo.py", line 554, in
main(
File "/home/na/anaconda3/envs/expose/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "demo.py", line 288, in main
model_output = model(body_imgs, body_targets, full_imgs=full_imgs,
File "/home/na/anaconda3/envs/expose/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/na/Desktop/smplx-test/expose-master/expose/models/smplx_net.py", line 71, in forward
output = self.smplx(images, targets=targets,
File "/home/na/anaconda3/envs/expose/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/na/Desktop/smplx-test/expose-master/expose/models/attention/predictor.py", line 1094, in forward
left_hand_points_to_crop = self.points_to_crops(
File "/home/na/anaconda3/envs/expose/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/na/Desktop/smplx-test/expose-master/expose/models/common/bbox_sampler.py", line 58, in forward
inv_crop_transforms = torch.inverse(crop_transforms)
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnCreate(handle)