Replies: 2 comments 5 replies
-
| 
         I know that 1080ti was Pascal, but are there any chat models that will run on Pascal - or is Volta (and later) a requirement?  | 
  
Beta Was this translation helpful? Give feedback.
                  
                    5 replies
                  
                
            -
| 
         Can you please provide me with a little more info regarding OobaBooga? How to set it up, etc?  | 
  
Beta Was this translation helpful? Give feedback.
                  
                    0 replies
                  
                
            
  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment
  
        
    
Uh oh!
There was an error while loading. Please reload this page.
-
When I update custom_setting.py file (copied from settings.py) with:
'''# Path to chatbot model - download from HuggingFace at runtime by default (gets cached)
chatbot_model_path: str = 'TheBloke/vicuna-7b-v1.5-GPTQ'
And support_chatbot: bool = True and restarted WIS, I noticed these messages:
'''...
willow-inference-server-wis-1 | [2023-11-02 13:49:52 +0000] [99] [INFO] CUDA: Detected 1 device(s)
willow-inference-server-wis-1 | [2023-11-02 13:49:52 +0000] [99] [INFO] CUDA: Device 0 name: NVIDIA GeForce GTX 1080 Ti
willow-inference-server-wis-1 | [2023-11-02 13:49:52 +0000] [99] [INFO] CUDA: Device 0 capability: 61
willow-inference-server-wis-1 | [2023-11-02 13:49:53 +0000] [99] [INFO] CUDA: Device 0 total memory: 11711873024 bytes
willow-inference-server-wis-1 | [2023-11-02 13:49:53 +0000] [99] [INFO] CUDA: Device 0 free memory: 11214716928 bytes
willow-inference-server-wis-1 | [2023-11-02 13:49:53 +0000] [99] [WARNING] CUDA: Device 0 is pre-Volta, forcing int8
willow-inference-server-wis-1 | [2023-11-02 13:49:53 +0000] [99] [WARNING] CUDA: Device 0 is pre-Volta, disabling chatbot
Does this mean the GTX 1080ti is not compatible to do chatbot functions?
Beta Was this translation helpful? Give feedback.
All reactions