Skip to content

ggml/kompute: Add multi-GPU support #9328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

ggml/kompute: Add multi-GPU support #9328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3666c86
ggml/kompute: Rename ggml_kompute_context to ggml_backend_kompute_con…
Aug 21, 2024
e914ac7
ggml/kompute: Introducing struct ggml_backend_kompute_buffer_context
Aug 21, 2024
74ba851
ggml/kompute: Move butf into struct ggml_backend_kompute_context
wishlee1124 Aug 21, 2024
d94ad56
ggml/kompute: Use the kp::Manager in ggml_backend_kompute_context ins…
wishlee1124 Aug 21, 2024
3676778
ggml/kompute: Implement ggml_backend_i.offload_op interface
Aug 21, 2024
f57f8cb
ggml/kompute: Reimplement kompute_manager
Aug 21, 2024
cc9514f
ggml/kompute: Remove unused ggml_backend_kompute_device_{ref, unref}()
flynnjiang Aug 21, 2024
97efd50
ggml/kompute: Introduce ggml_backend_kompute_get_device_count()
flynnjiang Aug 21, 2024
56c5f98
ggml/kompute: Introduce ggml_backend_kompute_get_device_memory()
flynnjiang Aug 21, 2024
424e3a5
llama/kompute: Add multi-GPU support
flynnjiang Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ggml/include/ggml-kompute.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
extern "C" {
#endif

#define GGML_KOMPUTE_MAX_DEVICES 16

struct ggml_vk_device {
int index;
int type; // same as VkPhysicalDeviceType
Expand All @@ -23,10 +25,10 @@ struct ggml_vk_device {
};

struct ggml_vk_device * ggml_vk_available_devices(size_t memoryRequired, size_t * count);
int ggml_backend_kompute_get_device_count(void);
void ggml_backend_kompute_get_device_memory(int device, size_t * free, size_t * total);
bool ggml_vk_get_device(struct ggml_vk_device * device, size_t memoryRequired, const char * name);
bool ggml_vk_has_vulkan(void);
bool ggml_vk_has_device(void);
struct ggml_vk_device ggml_vk_current_device(void);

//
// backend API
Expand Down
Loading
Loading