problem when using local mistral models

[Asma-droid]

I'am trying to use local mistral models as follows :

llm:
    init_parameters:
      huggingface_pipeline_kwargs:
        model: mistralai/Mixtral-8x7B-v0.1 
        model_kwargs:
          bnb_4bit_compute_dtype: torch.bfloat16
          bnb_4bit_quant_type: nf4
          bnb_4bit_use_double_quant: true
          device_map: auto
          load_in_4bit: true
        task: text-generation
      stop_words: null
      token:
        env_vars:
        - HF_API_TOKEN
        strict: false
        type: env_var
    type: haystack.components.generators.hugging_face_local.HuggingFaceLocalGenerator
    

i have got this error

rest-api_1 | File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
rest-api_1 | return await get_asynclib().run_sync_in_worker_thread(
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
rest-api_1 | return await future
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
rest-api_1 | result = context.run(func, *args)
rest-api_1 | File "/code/src/main.py", line 219, in ask_rag_pipeline
rest-api_1 | result = rag_pipeline.run(
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py", line 671, in run
rest-api_1 | self.warm_up()
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py", line 573, in warm_up
rest-api_1 | self.graph.nodes[node]["instance"].warm_up()
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/haystack/components/generators/hugging_face_local.py", line 145, in warm_up
rest-api_1 | self.pipeline = pipeline(**self.huggingface_pipeline_kwargs)
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/transformers/pipelines/init.py", line 905, in pipeline
rest-api_1 | framework, model = infer_framework_load_model(
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/transformers/pipelines/base.py", line 279, in infer_framework_load_model
rest-api_1 | model = model_class.from_pretrained(model, **kwargs)
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 561, in from_pretrained
rest-api_1 | return model_class.from_pretrained(
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2960, in from_pretrained
rest-api_1 | quantization_config, kwargs = BitsAndBytesConfig.from_dict(
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/transformers/utils/quantization_config.py", line 90, in from_dict
rest-api_1 | config = cls(**config_dict)
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/transformers/utils/quantization_config.py", line 259, in init
rest-api_1 | self.bnb_4bit_compute_dtype = getattr(torch, bnb_4bit_compute_dtype)
rest-api_1 | File "/usr/local/lib/python3.10/site-packages/torch/init.py", line 1938, in getattr
rest-api_1 | raise AttributeError(f"module '{name}' has no attribute '{name}'")

in parallel i have tried

print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.is_bf16_supported())
t = torch.rand(10, device='cuda', dtype=torch.bfloat16)
print(t)

and it works fine

any help please

[julian-risch]

Related to this issue: #7277