🚀 T2ITrainer

⚠️ Development Notice: Currently in active development - stability not guaranteed. Frequent updates - check changelogs regularly.

T2ITrainer is a diffusers based training script. It aims to provide simple yet implementation for lora training.

📅 Recent Updates

• 2025-06-27: Support kontext nf4 training

Rehosted an nf4 version flux kontext on https://huggingface.co/lrzjason/flux-kontext-nf4

Rehosted an nf4 version flux fill on https://huggingface.co/lrzjason/flux-fill-nf4

download nf4 version flux fill and use it for training could significantly decrease lora training VRAM requirement.

---

🛡️ Prerequisites

• PyTorch: torch>=2.3.0+cu121 (CUDA 12.1 supported)

---

💻 Supported Training Configurations

Model Type	VRAM Requirements	Status
Kolors	11GB GPU	✅ Supported
SD3.5 (FP16 BS1)	24GB GPU	✅ Supported
Flux Fill,Kontext	24GB GPU	✅ Supported

---

⚙️ Installation Guide

0. System Requirements

❗ Mandatory: Install Microsoft Visual C++ Redistributable if encountering DLL errors

1. Automated Setup

Recommended Method

  git clone https://github.com/lrzjason/T2ITrainer.git
  cd T2ITrainer
  setup.bat

• Handles: Virtual Environment • Dependency Installation • Model Downloads

2. Manual Installation

Clone Repository 🌐

    git clone https://github.com/lrzjason/T2ITrainer.git
    cd T2ITrainer

Virtual Environment 🛠️

    python -m venv venv
    call venv\Scripts\activate
    pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Model Downloads 📥

    # Kolors
    huggingface-cli download Kwai-Kolors/Kolors --local-dir kolors_models/

    # NF4 Flux Fill for low gpu
    huggingface-cli download "lrzjason/flux-fill-nf4" --local-dir flux_models/fill/

    # skip if downloaded nf 4 Flux Fill
    huggingface-cli download "black-forest-labs/FLUX.1-fill-dev" --local-dir flux_models/fill/
    
    # SD3.5 Models
    huggingface-cli download "stabilityai/stable-diffusion-3.5-large" --local-dir "sd3.5L/"

    # NF4 Flux kontext
    huggingface-cli download "lrzjason/flux-kontext-nf4" --local-dir flux_models/kontext/

Folder Structure

flux_models:

kontext:

🚀 Launch Options

Script	Command	Special Notes
Flux kontext	python ui_flux_fill.py	Requires diffusers>=0.32.0, 24GB VRAM Recommended
Flux Fill	python ui_flux_fill.py	Requires diffusers>=0.32.0, 24GB VRAM Recommended
Kolors	python ui.py	Needs Fixed VAE
SD3.5 Large	python ui_sd35.py	24GB VRAM Recommended

🔧 Parameter Configuration Guide

---

🌌 Flux Model Management

Kontext Model Installation

Inpainting Model Setup

  huggingface-cli download "lrzjason/flux-kontext-nf4" --local-dir flux_models/kontext/

For more details (example dataset):

• https://github.com/lrzjason/T2ITrainer/blob/main/doc/flux_kontext.md
• https://huggingface.co/datasets/lrzjason/object_removal_alpha_kontext

Fill Model Installation (Skip if train kontext)

Inpainting Model Setup

  huggingface-cli download "lrzjason/flux-fill-nf4" --local-dir flux_models/fill/ 

For more details (example dataset):

• https://github.com/lrzjason/T2ITrainer/blob/main/doc/flux_fill.md
• https://huggingface.co/datasets/lrzjason/ObjectRemovalAlpha

Dev Model Download (Skip if train fill and kontext)

Dev Model Installation

  huggingface-cli download "black-forest-labs/FLUX.1-dev" --local-dir flux_models/dev/

⚙️ Flux Training Recommended Parameters

Category	Settings
Base Configuration	Rank 16, AdamW, Lr 1e-4
24GB GPU	512 resolution, Batch Size 1
VRAM Optimization	Use nf4 based training
Precision	bf16

💻 VRAM Usage nf4

VRAM Peak

💻 VRAM Usage (bf16, blocks_to_swap=10)

VRAM Peak

VRAM Low

🔧 Kolors Testing & Integration

• Kolors Workflow:

      # ComfyUI Plugins
      git clone https://github.com/kijai/ComfyUI-KwaiKolorsWrapper
      git clone https://github.com/MinusZoneAI/ComfyUI-Kolors-MZ

• Configuration Guide: 📖 CivitAI Article

🆘 Troubleshooting

• Kolors Black Image Issue: Ensure you're using FP16 Fixed VAE
• VRAM Limitations: Adjust blocks_to_swap parameter (higher values reduce memory usage)
• Windows DLL Errors: Verify VC++ Redistributable installation

---

Star History

Old Change logs:

• https://github.com/lrzjason/T2ITrainer/blob/main/doc/change_logs.md

Recent Change Logs:

• 2025-06-27: Support kontxt nf4 training

Sponsor:

• Thanks to @sourceful support me making flux fill lora training script.

📬 Contact

• 𝕏 Twitter: @Lrzjason
• 📧 Email: lrzjason@gmail.com
• 💬 QQ Group: 866612947
• 💬 WeChat ID: fkdeai
• 🎨 CivitAI: xiaozhijason

Sponsors me for more open source projects:

Buy me a coffee:

WeChat:

Acknowledgements:

• Thanks for diffusers and Terminus Research Group
• Thanks to minienglish1 and Freon in EveryDream Discord for the assistance.
• Special thanks to kohya ss for references from the training codebase.
• Thanks to Kblueleaf for coding reference on hunyuandit gradient checkpoint implementation.
• Thanks to Kolors for the open-source checkpoint.
• Thanks to comfyui for the wonderful codebase.
• Thanks to emojiiii for the setup.bat script and other updates.
• Thanks to Rohit Gandikota and related authors of Concept Sliders https://github.com/rohitgandikota/sliders