Skip to content

0.13.6
Compare
Choose a tag to compare
@b4rtaz b4rtaz released this 26 Apr 12:24
· 2 commits to main since this release
v0.13.6
a16d2f0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This version adds two new arguments:

  • --net-turbo 0 - allows to disable non blocking sockets,
  • --gpu-segments <from>:<to> - allows specifying which segments of the neural network will be loaded onto the GPU. Currently, this option is dedicated only to skipping the first layer (embedding). Other settings may not work.

These options allowed to run Llama 3.3 70B Q40 on 4 x NVIDIA RTX 3060 12 GB. Check this test.

Assets 2
Loading