CUDA illegal memory access error while using deepspeed.initialize

[seungyounglim]

I was trying to use init_inference API for my BERT sentiment classification model.
I trained this model with Huggingface Trainer API, without DeepSpeed.

The model was loaded successfully,
but when I forwarded inputs to the model, I got this error message below :

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

The expement was run on Titan-XP and V100 GPU, and the result was same.
I would appreciate it if I can get any help with this.

This is the code that I used.

model = AutoModelForSequenceClassification.from_pretrained(model_path , num_labels=2)

ds_engine = deepspeed.init_inference(model,
                                 mp_size=1,
                                 triangular_masking=False,
                                 dtype=torch.float,
                                 checkpoint=None ,
                                 replace_method='auto',
                                 replace_with_kernel_inject=True)

ds_engine(input_0, 
          token_type_ids = input_1 ,
          attention_mask=input_2)

[xmanners]

hmmm, seems like didn't get helpful solutions ... orz

[xmanners]

and from my oppinion, if you didn't set replace_with_kernel_inject=True it will work,
but, who knows?