Omniforge – Local AI Text, Images & TTS, Plus Grid Integration

Omniforge merges local AI generation (text, images, TTS) with the Grid for decentralized incentives. You enjoy privacy and direct ownership of your hardware, while contributing to a growing open-source AI renaissance.

Launch Omniforge, pick your favorite models, and contribute to a new era of AI accessibility and creativity.

Highlights

• Local & Cross-Platform:
  • One-file executables for Windows and Linux, plus Mac ARM64 builds.
  • Optional GPU offloading (NVIDIA CUDA, AMD/NVIDIA Vulkan, etc.)
  • Legacy CPU fallback for old or minimal systems.
• Text Generation & Chat:
  • Runs modern GGUF/GGML models (Llama, Deepseek, Mistral, Qwen, etc.).
  • Full UI with story mode, chat mode, instruct mode, or adventure mode.
  • Persistent “world info,” memory, scenario editing, advanced samplers, context shifting, etc.
• Stable Diffusion Image Generation:
  • Load any .safetensors stable-diffusion model (Flux, SDXL, etc.) in .gguf format.
  • Optional VAE tiling, advanced model merges, flux/SD3 support, multiple GPU acceleration.
• Whisper Speech Recognition:
  • Integrate local speech-to-text with VAD (Hands Free) or push-to-talk.
  • Choose from many Whisper .gguf models for multilingual voice transcription.
• NEW: OuteTTS Text-To-Speech:
  • Load an OuteTTS GGUF model + WavTokenizer GGUF for local TTS.
  • Optional --ttsgpu to offload TTS to GPU, plus --ttsthreads for thread count.
  • Dynamic voice synthesis with built-in voices or random name “new speaker” generation.
  • OpenAI Speech API & XTTS endpoints for hooking into external TTS interfaces.
• Advanced Performance Settings:
  • GPU layer offloading (--gpulayers)
  • Full or partial GPU usage via CUDA/CLBlast/Vulkan/Metal.
  • High context sizes (--contextsize) with optional RoPE config for beyond-2k token context.
  • New mechs: flash attention, quantized KV caches, NTK scaling, partial layer shifting.
• Rich UI & API Endpoints:
  • Web-based GUI on http://localhost:5001 – no extra dependencies.
  • SSE-based streaming, chat completions adapters, multiple device usage.
  • Command line or config files (.kcpps or .kcppt) for advanced deployments.
  • Interactive or legacy “no-script” UI for minimal browsers/SSH.

Connecting to the Grid

Omniforge was specifically designed to integrate with the Grid, our decentralized AI compute platform that rewards contributors with $AIPG. Here’s how:

1. Enable the Grid Worker: Either launch Omniforge with something like --gridkey YOURKEY --gridmodel MYMODEL --gridworker "My Omniforge Node" or open the GUI and configure the Grid Worker settings.
2. Accept Remote Requests: Once connected, your local hardware can be used by others to handle text generations. You’ll automatically earn $AIPG for your contributions.
3. Monitor Activity: The console or web UI logs will display tasks queued from the Grid, as well as your acceptance rate, generation speeds, and successful completions.

Typical Usage Flow

1. Download Omniforge: Choose your .exe (Windows), Linux binary, or Mac ARM64 build.
2. Obtain a Model: For text, look on Huggingface for .gguf models (Llama, Deepseek, Mistral, etc.).
3. Run:

   ./omniforge.exe --model textmodel.gguf --usecublas --gpulayers 20 \
       --ttsmodel outetts_v0.3.gguf --ttswavtokenizer outetts_tokenizer.gguf \
       --sdmodel stable-diffusion-1.5.gguf --gridkey yourGridKey
   
   
    Open the UI: In a browser, go to http://localhost:5001. Load your scenarios, memory, or TTS voices from the “Extras” tab.
    Start Generating: Talk with your model in Chat Mode, try TTS, or create images under the “Image Generation” tab.
   

Notable New Extras

GGUF File Analyzer: --analyze or the “Extras” tab to see a model’s internal metadata, shapes, and layers.
TAESD: For stable diffusion auto-fixing your VAE, now compressed to ~3MB. Toggle with --sdvaeauto.
Context Shifting: For advanced GPU-based context extension (in GGUF models).
Speculative Decoding & Draft Models: Faster generation if you load a separate “draft” model.
WebSearch & Document Lookup: Optionally local search for references, or text-based “minisearch” to do rudimentary RAG with your pasted docs.

Tips & Tricks

Performance: Launch with --threads (X) matching physical cores, plus --gpulayers if you have enough VRAM.
GPU: NVIDIA users add --usecublas; AMD or older GPUs try --usevulkan or --useclblast.
TTS: Keep TTS model size small if your GPU is limited (the 500MB or 1B voice sets).
Long Context: Use --contextsize 4096 or more if you want deeper memory, with caution on available RAM.
RoPE: Adjust with --ropeconfig [freq_scale] [freq_base] for extended or shifted context.

Omniforge merges local AI generation (text, images, TTS) with the Grid for decentralized incentives. You enjoy privacy and direct ownership of your hardware, while contributing to a growing open-source AI renaissance. Launch Omniforge, pick your favorite models, and join the movement—helping power a new era of AI accessibility and creativity.

Compiling Omniforge From Source Code

Compiling on Linux (Using koboldcpp.sh automated compiler script)

when you can't use the precompiled binary directly, we provide an automated build script which uses conda to obtain all dependencies, and generates (from source) a ready-to-use a pyinstaller binary for linux users.

• Clone the repo with git clone https://github.com/LostRuins/koboldcpp.git
• Simply execute the build script with ./koboldcpp.sh dist and run the generated binary. (Not recommended for systems that already have an existing installation of conda. Dependencies: curl, bzip2)

./koboldcpp.sh # This launches the GUI for easy configuration and launching (X11 required).
./koboldcpp.sh --help # List all available terminal commands for using Koboldcpp, you can use koboldcpp.sh the same way as our python script and binaries.
./koboldcpp.sh rebuild # Automatically generates a new conda runtime and compiles a fresh copy of the libraries. Do this after updating Koboldcpp to keep everything functional.
./koboldcpp.sh dist # Generate your own precompiled binary (Due to the nature of Linux compiling these will only work on distributions equal or newer than your own.)

Compiling on Linux (Manual Method)

• To compile your binaries from source, clone the repo with git clone https://github.com/LostRuins/koboldcpp.git
• A makefile is provided, simply run make.
• Optional Vulkan: Link your own install of Vulkan SDK manually with make LLAMA_VULKAN=1
• Optional CLBlast: Link your own install of CLBlast manually with make LLAMA_CLBLAST=1
• Note: for these you will need to obtain and link OpenCL and CLBlast libraries.
  • For Arch Linux: Install cblas and clblast.
  • For Debian: Install libclblast-dev.
• You can attempt a CuBLAS build with LLAMA_CUBLAS=1, (or LLAMA_HIPBLAS=1 for AMD). You will need CUDA Toolkit installed. Some have also reported success with the CMake file, though that is more for windows.
• For a full featured build (all backends), do make LLAMA_CLBLAST=1 LLAMA_CUBLAS=1 LLAMA_VULKAN=1. (Note that LLAMA_CUBLAS=1 will not work on windows, you need visual studio)
• To make your build sharable and capable of working on other devices, you must use LLAMA_PORTABLE=1
• After all binaries are built, you can run the python script with the command koboldcpp.py [ggml_model.gguf] [port]

Compiling on Windows

• You're encouraged to use the .exe released, but if you want to compile your binaries from source at Windows, the easiest way is:
  • Get the latest release of w64devkit (https://github.com/skeeto/w64devkit). Be sure to use the "vanilla one", not i686 or other different stuff. If you try they will conflit with the precompiled libs!
  • Clone the repo with git clone https://github.com/LostRuins/koboldcpp.git
  • Make sure you are using the w64devkit integrated terminal, then run make at the KoboldCpp source folder. This will create the .dll files for a pure CPU native build.
  • For a full featured build (all backends), do make LLAMA_CLBLAST=1 LLAMA_VULKAN=1. (Note that LLAMA_CUBLAS=1 will not work on windows, you need visual studio)
  • To make your build sharable and capable of working on other devices, you must use LLAMA_PORTABLE=1
  • If you want to generate the .exe file, make sure you have the python module PyInstaller installed with pip (pip install PyInstaller). Then run the script make_pyinstaller.bat
  • The koboldcpp.exe file will be at your dist folder.
• Building with CUDA: Visual Studio, CMake and CUDA Toolkit is required. Clone the repo, then open the CMake file and compile it in Visual Studio. Copy the koboldcpp_cublas.dll generated into the same directory as the koboldcpp.py file. If you are bundling executables, you may need to include CUDA dynamic libraries (such as cublasLt64_11.dll and cublas64_11.dll) in order for the executable to work correctly on a different PC.
• Replacing Libraries (Not Recommended): If you wish to use your own version of the additional Windows libraries (OpenCL, CLBlast, Vulkan), you can do it with:
  • OpenCL - tested with https://github.com/KhronosGroup/OpenCL-SDK . If you wish to compile it, follow the repository instructions. You will need vcpkg.
  • CLBlast - tested with https://github.com/CNugteren/CLBlast . If you wish to compile it you will need to reference the OpenCL files. It will only generate the ".lib" file if you compile using MSVC.
  • Move the respectives .lib files to the /lib folder of your project, overwriting the older files.
  • Also, replace the existing versions of the corresponding .dll files located in the project directory root (e.g. clblast.dll).
  • Make the KoboldCpp project using the instructions above.

Compiling on MacOS

• You can compile your binaries from source. You can clone the repo with git clone https://github.com/LostRuins/koboldcpp.git
• A makefile is provided, simply run make.
• If you want Metal GPU support, instead run make LLAMA_METAL=1, note that MacOS metal libraries need to be installed.
• To make your build sharable and capable of working on other devices, you must use LLAMA_PORTABLE=1
• After all binaries are built, you can run the python script with the command koboldcpp.py --model [ggml_model.gguf] (and add --gpulayers (number of layer) if you wish to offload layers to GPU).

Compiling on Android (Termux Installation)

• Install and run Termux from F-Droid
• Enter the command termux-change-repo and choose Mirror by BFSU
• Install dependencies with pkg install wget git python (plus any other missing packages)
• Install dependencies apt install openssl (if needed)
• Clone the repo git clone https://github.com/LostRuins/koboldcpp.git
• Navigate to the koboldcpp folder cd koboldcpp
• Build the project make
• To make your build sharable and capable of working on other devices, you must use LLAMA_PORTABLE=1, this disables usage of ARM instrinsics.
• Grab a small GGUF model, such as wget https://huggingface.co/concedo/KobbleTinyV2-1.1B-GGUF/resolve/main/KobbleTiny-Q4_K.gguf
• Start the python server python koboldcpp.py --model KobbleTiny-Q4_K.gguf
• Connect to http://localhost:5001 on your mobile browser
• If you encounter any errors, make sure your packages are up-to-date with pkg up
• GPU acceleration for Termux may be possible but I have not explored it. If you find a good cross-device solution, do share or PR it.

./koboldcpp.sh # This launches the GUI for easy configuration and launching (X11 required).
./koboldcpp.sh --help # List all available terminal commands for using Koboldcpp, you can use koboldcpp.sh the same way as our python script and binaries.
./koboldcpp.sh rebuild # Automatically generates a new conda runtime and compiles a fresh copy of the libraries. Do this after updating Koboldcpp to keep everything functional.
./koboldcpp.sh dist # Generate your own precompiled binary (Due to the nature of Linux compiling these will only work on distributions equal or newer than your own.)

AMD Users

• For most users, you can get very decent speeds by selecting the Vulkan option instead, which supports both Nvidia and AMD GPUs.
• Alternatively, you can try the ROCM fork at https://github.com/YellowRoseCx/koboldcpp-rocm

Third Party Resources

• These unofficial resources have been contributed by the community, and may be outdated or unmaintained. No official support will be provided for them!
  • Arch Linux Packages: CUBLAS, and HIPBLAS.
  • Unofficial Dockers: korewaChino and noneabove1182
  • Nix & NixOS: KoboldCpp is available on Nixpkgs and can be installed by adding just koboldcpp to your environment.systemPackages (or it can also be placed in home.packages).
    • Example Nix Setup and further information
    • If you face any issues with running KoboldCpp on Nix, please open an issue here.
• GPTLocalhost - KoboldCpp is supported by GPTLocalhost, a local Word Add-in for you to use KoboldCpp in Microsoft Word. A local alternative to "Copilot in Word."

Questions and Help Wiki

• First, please check out The KoboldCpp FAQ and Knowledgebase which may already have answers to your questions! Also please search through past issues and discussions.
• If you cannot find an answer, open an issue on this github, or find us on the KoboldAI Discord.

KoboldCpp and KoboldAI API Documentation

• Documentation for KoboldAI and KoboldCpp endpoints can be found here

KoboldCpp Public Demo

• A public KoboldCpp demo can be found at our Huggingface Space. Please do not abuse it.

Considerations

• For Windows: No installation, single file executable, (It Just Works)
• Since v1.15, requires CLBlast if enabled, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without CLBlast.
• Since v1.33, you can set the context size to be above what the model supports officially. It does increases perplexity but should still work well below 4096 even on untuned models. (For GPT-NeoX, GPT-J, and Llama models) Customize this with --ropeconfig.
• Since v1.42, supports GGUF models for LLAMA and Falcon
• Since v1.55, lcuda paths on Linux are hardcoded and may require manual changes to the makefile if you do not use koboldcpp.sh for the compilation.
• Since v1.60, provides native image generation with StableDiffusion.cpp, you can load any SD1.5 or SDXL .safetensors model and it will provide an A1111 compatible API to use.
• I try to keep backwards compatibility with ALL past llama.cpp models. But you are also encouraged to reconvert/update your models if possible for best results.
• Since v1.75, openblas has been deprecated and removed in favor of the native CPU implementation.

License

• The original GGML library and llama.cpp by ggerganov are licensed under the MIT License
• However, KoboldAI Lite is licensed under the AGPL v3.0 License
• KoboldCpp code and other files are also under the AGPL v3.0 License unless otherwise stated

Notes

• If you wish, after building the koboldcpp libraries with make, you can rebuild the exe yourself with pyinstaller by using make_pyinstaller.bat
• API documentation available at /api (e.g. http://localhost:5001/api) and https://lite.koboldai.net/koboldcpp_api. An OpenAI compatible API is also provided at /v1 route (e.g. http://localhost:5001/v1).
• All up-to-date GGUF models are supported, and KoboldCpp also includes backward compatibility for older versions/legacy GGML .bin models, though some newer features might be unavailable.
• An incomplete list of architectures is listed, but there are many hundreds of other GGUF models. In general, if it's GGUF, it should work.
• Llama / Llama2 / Llama3 / Alpaca / GPT4All / Vicuna / Koala / Pygmalion / Metharme / WizardLM / Mistral / Mixtral / Miqu / Qwen / Qwen2 / Yi / Gemma / Gemma2 / GPT-2 / Cerebras / Phi-2 / Phi-3 / GPT-NeoX / Pythia / StableLM / Dolly / RedPajama / GPT-J / RWKV4 / MPT / Falcon / Starcoder / Deepseek and many, many more.

Where can I download AI model files?

• The best place to get GGUF text models is huggingface. For image models, CivitAI has a good selection. Here are some to get started.
  • Text Generation: Airoboros Mistral 7B (smaller and weaker) or Tiefighter 13B (larger model) or Beepo 22B (largest and most powerful)
  • Image Generation: Anything v3 or Deliberate V2 or Dreamshaper SDXL
  • Image Recognition MMproj: Pick the correct one for your model architecture here
  • Speech Recognition: Whisper models for Speech-To-Text
  • Text-To-Speech: TTS models for Narration