Does this dllama work in kubernetes environment?

[phucly01]

Hi everyone,
I was googling for ways to use multiple low end mini PCs for larger llama model and came across this one. My question is will it and if so, how does it work under kubernetes cluster? Are the root node and worker nodes treated as one entity or kubernetes just know about the root node?

Thanks
phucly01

[notfol]

I'm running it on K3s. I'm doing a root deployment and worker StatefulSet with a service.

 ---

  apiVersion: v1
  kind: Service
  metadata:
    name: dllama-worker-external
  spec:
    type: NodePort
    selector:
      app: dllama-worker
    ports:
      - port: 6767
        targetPort: 6767
        nodePort: 30676  # Fixed port across all nodes

root containers spec

          - name: DLLAMA_WORKERS
            value: "{{.Values.rootWorkers}}"

My entrypoint

      "root")
          echo "[INFO] Starting Distributed LLaMA root"
          ./dllama-api \
              --model $DLLAMA_MODEL_PATH \
              --tokenizer $DLLAMA_TOKENIZER_PATH \
              --buffer-float-type $DLLAMA_MODEL_BUFFER_FLOAT_TYPE \
              --steps 16 \
              --nthreads $DLLAMA_THREADS \
              --workers $DLLAMA_WORKERS \ <----
              --port $DLLAMA_API_PORT

[phucly01]

So basically the root node is part of the kubernetes cluster. The other worker nodes are then managed by the root node.
I wonder if the worker nodes can be part of kubernetes too and let the deployment handle high availability.
Thank you I will try to mimic your config