a few questions #45

Closed Answered by giladgd

hiqsociety asked this question in Q&A

hiqsociety
Sep 19, 2023

just realised i'm not using gpu vram though i compiled with cuda. how to use the vram?
how to specify how many gpu layers to offload in cuda mode? i've compiled for cuda but i'm not sure where to adjust gpu layer offload settings.
how to make it accepts an input and output for api use like chatgpt etc? i dont mind input with output on command line

Answered by giladgd

llama.cpp supports some parameters you can configure to customize the behavior of CUDA.
You can read more about it here and then set the parameters you need as environment variables when you compile using node-llama-cpp download --cuda and when you run your code
Docs: gpuLayers
Docs: use a model as a chatbot

View full answer

Replies: 1 comment

giladgd
Sep 23, 2023
Maintainer

llama.cpp supports some parameters you can configure to customize the behavior of CUDA.
You can read more about it here and then set the parameters you need as environment variables when you compile using node-llama-cpp download --cuda and when you run your code
Docs: gpuLayers
Docs: use a model as a chatbot

0 replies

Answer selected by giladgd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment