Kohya brought massive improvements to FLUX LoRA (as low as 4 GB GPUs) and DreamBooth / Fine-Tuning (as low as 6 GB GPUs) training
Added 2024-11-17 13:45:38 +0000 UTCYou can download all configs and full instructions
> https://www.patreon.com/posts/112099700 - Fine Tuning post
> https://www.patreon.com/posts/110879657 - LoRA post
Kohya brought massive improvements to FLUX LoRA and DreamBooth / Fine-Tuning (min 6GB GPU) training.
Now as low as 4GB GPUs can train FLUX LoRA with decent quality and 24GB and below GPUs got a huge speed boost when doing Full DreamBooth / Fine-Tuning training
You need minimum 4GB GPU to do a FLUX LoRA training and minimum 6 GB GPU to do FLUX DreamBooth / Full Fine-Tuning training. It is just mind blowing.
You can download all configs and full instructions > https://www.patreon.com/posts/112099700
The above post also has 1-click installers and downloaders for Windows, RunPod and Massed Compute
The model downloader scripts also updated and downloading 30+GB models takes total 1 minute on Massed Compute
You can read the recent updates here : https://github.com/kohya-ss/sd-scripts/tree/sd3?tab=readme-ov-file#recent-updates
This is the Kohya GUI branch : https://github.com/bmaltais/kohya_ss/tree/sd3-flux.1
Key thing to reduce VRAM usage is using block swap
Kohya implemented the logic of OneTrainer to improve block swapping speed significantly and now it is supported for LoRAs as well
Now you can do FP16 training with LoRAs on 24 GB and below GPUs
Now you can train a FLUX LoRA on a 4 GB GPU - key is FP8, block swap and using certain layers training (remember single layer LoRA training)
It took me more than 1 day to test all newer configs, their VRAM demands, their relative step speeds and prepare the configs :)
Comments
wow i am gonna test this unexpected
Furkan Gözükara
2024-11-19 22:13:52 +0000 UTCawesome
Furkan Gözükara
2024-11-19 22:13:14 +0000 UTCThanks. Really, torch 2.5.1dev is slower than torch 2.5
Vlad
2024-11-19 19:58:14 +0000 UTCOh my god!! It works, I did a fresh installation again, this time without step 2 and the speed improved so much! Now I'n getting 8.60s/it (only waited 100 steps to publish the result). Thank you so much!!
Vicent Guardiola
2024-11-19 19:55:08 +0000 UTCI did a fresh install with 'Windows_Install_Step_1', followed by 'Windows_Update_Kohya_and_Fix_FLUX_Step2'. I completely skipped 'Windows_Install_Torch_2_5_Dev_Huge_Speed_Up' because this step keeps slowing it down for me for whatever reason - i think the 2.5.1 dev torch version in it confuses my setup or something, and the default 2.5 version that gets installed in Step_1 is better for me. Perhaps try doing the same and see if it improves things for you as well
GH
2024-11-19 19:06:32 +0000 UTCI'm using the DreamBooth_Tab_Fine_Tuning_Best_FLUX_Configs/24GB_GPU_22900MB_7.3_second_it_Tier_1.json
Vicent Guardiola
2024-11-19 19:03:29 +0000 UTCI'm only training, I don't use the PC for anything more than training while the process is running.
Vicent Guardiola
2024-11-19 19:00:00 +0000 UTCye it is very slow. which config name you are using exactly?
Furkan Gözükara
2024-11-19 18:58:10 +0000 UTCHi! With previou configs and install I usually get 10.5s/it aprox, now I'm getting 11.03s/it. I waited 600 steps to record the speed. I made a fresh install with Kohya_FLUX_DreamBooth_v14.zip and now with Kohya_FLUX_DreamBooth_v15.zip and the speed is the same.
Vicent Guardiola
2024-11-19 18:57:04 +0000 UTChi Vincent you must have seen some speed improvement with fresh install. can you tell me your step speed? i also have 3090
Furkan Gözükara
2024-11-19 17:50:45 +0000 UTCawesome ty so much for the feedback
Furkan Gözükara
2024-11-19 17:50:19 +0000 UTCHi, have you done a fresh installation? Did you do the installation with Windows_Install_Step_1 and Windows_Install_Torch_2_5_Dev_Huge_Speed_Up and nothing else? I did a fresh installation and it hasn't changed the speed. (I have an 3090) Thanks!
Vicent Guardiola
2024-11-19 12:26:20 +0000 UTCThe speed boost is insane!! Gone from 15s/it to 8.5s/it on my 3090, and I can actually use my PC while training as it doesn't hang up anymore! Thanks for updating your scripts, they make my life so much easier!
GH
2024-11-19 12:03:03 +0000 UTCye he decided to reverse after we replied :D
Furkan Gözükara
2024-11-19 10:22:51 +0000 UTCSeems to be fixed or will be fixed in next pull. Thank you!
Chris
2024-11-19 06:57:58 +0000 UTCok he says that now you have to enter yourself max train steps :( this sucks so bad please leave a comment : https://github.com/bmaltais/kohya_ss/issues/2976#issuecomment-2484164299
Furkan Gözükara
2024-11-18 21:43:59 +0000 UTCThank you 😊
Chris
2024-11-18 21:26:59 +0000 UTCcurrently broken here i opened issue : https://github.com/bmaltais/kohya_ss/issues/2976
Furkan Gözükara
2024-11-18 21:18:39 +0000 UTCyes as you use it more, it will use more RAM
Furkan Gözükara
2024-11-18 21:04:33 +0000 UTCyes currently calculations are broken i think. it shows you are doing 534 epoch. not related to config but related to kohya or kohya gui. run installer again and try again please i can test later today
Furkan Gözükara
2024-11-18 20:28:02 +0000 UTCHas something changed with batch size 7 training, config file or koyha update? batch size 7 training now takes 7 hours with 15 pictures instead of roughly 2 hours on massed compute. i set 15 source pictures, 160 epochs. output gives to much steps and to much epochs for batch size 7? yesterday it worked fine. enable full bf16 training. running training / 学習開始 num examples / サンプル数: 15 num batches per epoch / 1epochのバッチ数: 3 num epochs / epoch数: 534 batch size per device / バッチサイズ: 7 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 1600 steps: 0%| | 0/1600 [00:00
Chris
2024-11-18 18:36:09 +0000 UTC[Edit] figured it out. It was the extra param with --blocks_to_swap 32. Reducing or removing it solved it. [Original] I have 16gb vram and 32gb ram, and for some reason it starts loading data into ram after the "enable full bf16 training" message. Ending with a CUDA out of memory error. I tried both the 16gb lora config and the 12gb lora one for flux. I also allow system managed swap file so it should be able to increase it? Using the v44 installer. enable full bf16 training. Traceback (most recent call last): File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\flux_train_network.py", line 574, in trainer.train(args) File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\train_network.py", line 639, in train unet = self.prepare_unet_with_accelerator(args, accelerator, unet) # accelerator does some magic here File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\flux_train_network.py", line 545, in prepare_unet_with_accelerator accelerator.unwrap_model(flux).prepare_block_swap_before_forward() File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\library\flux_models.py", line 1006, in prepare_block_swap_before_forward self.offloader_single.prepare_block_devices_before_forward(self.single_blocks) File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 210, in prepare_block_devices_before_forward weighs_to_device(b, "cpu") # make sure weights are on cpu File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 91, in weighs_to_device module.weight.data = module.weight.data.to(device, non_blocking=True) RuntimeError: CUDA error: out of memory
Mikael Svenson
2024-11-18 16:50:36 +0000 UTCThank you!!
Luno Lux
2024-11-18 10:59:48 +0000 UTCthanks a lot for the comment
Furkan Gözükara
2024-11-18 10:47:00 +0000 UTCuse attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker
Furkan Gözükara
2024-11-18 10:46:43 +0000 UTCuse attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker
Furkan Gözükara
2024-11-18 10:46:39 +0000 UTCuse attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker
Furkan Gözükara
2024-11-18 10:46:36 +0000 UTCsame here. I also tried with v8 but it creates the same issue. I guess we will have to wait for kohya to fix it (already in issuses on the github page)
Luno Lux
2024-11-18 10:34:29 +0000 UTCI agree with you @renderartist. I'm a computer engineer full-time and can make everything work on my own just fine. I don't need any of these scripts. BUT... do I feel like spending 30 minutes to an hour to see if Rope is as cool as I think? Not really. Furkan makes the installers so easy I can just spend time playing in the playground I would have spend minutes to hours debugging and setting up.
Patrick Major
2024-11-18 07:23:46 +0000 UTCsame issue. please advise v14
UC Arcade
2024-11-18 07:04:53 +0000 UTCI just did a fresh install of Kohya to try this and this is the error I keep getting using the Dreambooth tab, I made sure I selected a Dreambooth config. It happens as soon as I hit start training. Traceback (most recent call last): File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\queueing.py", line 624, in process_events response = await route_utils.call_process_api( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\route_utils.py", line 323, in call_process_api output = await app.get_blocks().process_api( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 2018, in process_api result = await self.call_function( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1567, in call_function prediction = await anyio.to_thread.run_sync( # type: ignore File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2441, in run_sync_in_worker_thread return await future File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 943, in run result = context.run(func, *args) File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\utils.py", line 846, in wrapper response = f(*args, **kwargs) File "D:\AI\kohya\kohya_ss\kohya_gui\dreambooth_gui.py", line 817, in train_model log.info(max_train_steps_info) UnboundLocalError: local variable 'max_train_steps_info' referenced before assignment
Jeff Penny
2024-11-18 02:37:29 +0000 UTCThank you so much. Yes mods just deleted my post which was most up voted one :/
Furkan Gözükara
2024-11-17 21:33:26 +0000 UTCThank you for all that you do Furkan, people can be so negative on Reddit...I really don't understand the hate you get, this is all super insightful information. I find so much value in what you share, please keep going! 👍🏼🔥
renderartist
2024-11-17 20:58:10 +0000 UTCthanks a lot
Furkan Gözükara
2024-11-17 14:36:29 +0000 UTCGreat value as always Doctor. Your raising the bar as always
AiInfluence
2024-11-17 13:48:26 +0000 UTC
