All The LoRA FLUX Training Experiments I Have Done So Far
Added 2024-09-09 21:00:00 +0000 UTCI have been using a machine on RunPod that has 8x RTX A6000 over a week now. I have done all the following experiments with each one being 3000 steps.
Latest configs shared on : https://www.patreon.com/posts/kohya-flux-lora-110879657
All findings, grids, and best configs and 1-click installers for Windows, RunPod and Massed Compute shared here : https://www.patreon.com/posts/kohya-flux-lora-110293257
Best configs got updated and total number of trainings reached to 104
You can see saved safetensors files in below repos. Click load more button at the end to see entire repo files
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part2/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part3/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part4/tree/main
https://huggingface.co/datasets/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part5/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part6_Final/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part6_T5_XXL/tree/main
All Tested Experiments List So Far
1) 16GB_Config.safetensors
2) 1e_04_bf16_128_rank-000170.safetensors
3) 1e_04_bf16_128_rank_full_bf16.safetensors
4) 1e_04_fp16_128_rank-000180.safetensors
5) 1e_04_fp16_128_rank_full_fp16-000180.safetensors
6) 1e_04_fp8_32_rank-000170.safetensors
7) 1e_04_fp8_bf16_accelerate_32_rank-000160.safetensors
8) 1e_04_fp8_bf16_accelerate_full_bf16_32_rank-000160.safetensors
9) 1e_04_fp8_full_fp16_32_rank-000170.safetensors
10) 2e_04_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
11) 3e_04_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
12) 6e_05_best_additive_sigma.safetensors
13) 6e_05_best_additive_uniform.safetensors
14) 6e_05_best_max_grad_norm_0.safetensors
15) 6e_05_best_raw_sigma.safetensors
16) 6e_05_best_raw_sigmoid.safetensors
17) 6e_05_best_sigma_scaled_sigma.safetensors
18) 6e_05_best_with_reg_images.safetensors
19) 6e_05_bf16_128_rank_full_bf16.safetensors
20) 7e_05_bf16_128_rank_full_bf16.safetensors
21) 8e_05_bf16_128_rank_full_bf16.safetensors
22) 8e_05_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
23) 9e_05_bf16_128_rank_full_bf16.safetensors
24) 9e_05_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
25) Best_V4_T5_1e_05-000150.safetensors
26) Best_V4_T5_1e_05_TE_5e_05-000150.safetensors
27) Best_V4_T5_2e_05-000150.safetensors
28) Best_V4_T5_3e_05-000150.safetensors
29) Best_V4_T5_4e_05-000150.safetensors
30) Best_V4_T5_5e_05-000150.safetensors
31) Best_V4_T5_5e_05_With_Captions-000150.safetensors
32) Best_v1.safetensors
33) Best_v1_3e_5.safetensors
34) Best_v1_4e_5.safetensors
35) Best_v1_512px.safetensors
36) Best_v1_512px_Lowest_VRAM.safetensors
37) Best_v1_5e_5.safetensors
38) Best_v1_5e_5_max_grad_norm_0.safetensors
39) Best_v1_Lowest_VRAM.safetensors
40) Best_v1_T5_attention_mask.safetensors
41) Best_v1_cfg_3_5.safetensors
42) Best_v1_max_grad_norm_0.safetensors
43) Best_v1_reg_0_1_percent.safetensors
44) Best_v1_reg_prior_lost_0_3.safetensors
45) Best_v1_reg_prior_lost_0_6.safetensors
46) Best_v1_reg_prior_lost_1_0_apply_t5_attention_mask.safetensors
47) Best_v2.safetensors
48) Best_v2_1280.safetensors
49) Best_v2_1280_256rank.safetensors
50) Best_v2_1536.safetensors
51) Best_v2_1536_256rank.safetensors
52) Best_v2_2e_05_Clip_1e_04.safetensors
53) Best_v2_2e_05_Clip_3e_04.safetensors
54) Best_v2_2e_05_Clip_6e_05.safetensors
55) Best_v2_2e_05_Clip_8e_05.safetensors
56) Best_v2_4e_05_Clip_1e_04.safetensors
57) Best_v2_4e_05_Clip_3e_04.safetensors
58) Best_v2_4e_05_Clip_6e_05.safetensors
59) Best_v2_4e_05_Clip_8e_05.safetensors
60) Best_v2_5e_05_Clip_1e_05.safetensors
61) Best_v2_5e_05_Clip_5e_05.safetensors
62) Best_v2_FP8.safetensors
63) Best_v2_FP8_LowVRAM.safetensors
64) Best_v2_Flow_Shift_0_5.safetensors
65) Best_v2_Flow_Shift_1.safetensors
66) Best_v2_Rank16.safetensors
67) Best_v2_Rank256.safetensors
68) Best_v2_Rank32.safetensors
69) Best_v2_Rank4.safetensors
70) Best_v2_Rank64.safetensors
71) Best_v2_Rank8.safetensors
72) Best_v2_Shift_Sample_3e_04.safetensors
73) Best_v2_Shift_Sample_4e_04.safetensors
74) Best_v2_Shift_Sample_6e_04.safetensors
75) Best_v2_Shift_Sample_7e_04.safetensors
76) Best_v2_Split_QKV.safetensors
77) Best_v2_joycaption.safetensors
78) Best_v2_joycaption_Split_QKV.safetensors
79) Best_v2_only_OHWX.safetensors
80) Best_v2_shift.safetensors
81) Best_v2_shift_1e_05.safetensors
82) Best_v2_shift_2e_05.safetensors
83) Best_v2_shift_3e_05.safetensors
84) Best_v2_shift_4e_05.safetensors
85) Best_v2_shift_6e_05.safetensors
86) Best_v2_shift_9e_06.safetensors
87) Best_v2_shift_value_1.safetensors
88) Best_v3.safetensors
89) Best_v3_4e_05_TE_6e_04.safetensors
90) Best_v3_Clip_Skip_0.safetensors
91) Best_v3_Clip_Skip_2.safetensors
92) Best_v3_Discrete_Flow_1_0.safetensors
93) Best_v3_Discrete_Flow_3_0.safetensors
94) Best_v3_TE_0.safetensors
95) Best_v3_xFormers.safetensors
96) Best_v4.safetensors
97) Best_v4_Clip_Skip_1.safetensors
98) CivitAI_FLUX.safetensors
99) Rank_5_11700MB_14_62_Second_IT.safetensors
100) Rank_8_15300MB_4_53_Second_IT.safetensors
101) Rank_8_15400MB_3_81_Second_IT.safetensors
102) Rank_8_15450MB_3_80_Second_IT.safetensors
103) Rank_8_15600MB_4_89_Second_IT.safetensors
104) Rank_8_9500MB_10_00_Second_IT.safetensors
Comments
yes these are my personal trainings not open to download
Furkan Gözükara
2025-03-27 09:29:21 +0000 UTCHello brother, I can't download the configuration file on huggingface. I need the author's consent. My name is he, a purple avatar. Please agree to my download request.
锦涛 何
2025-03-27 09:01:40 +0000 UTCgo for rank 4 it should be really fast , load fp16 version of the model it will auto cast
Furkan Gözükara
2024-11-04 15:33:58 +0000 UTCWhich configuration would you recommend for training a character model on an RTX 4090 so that it doesn’t take all day? My model is FP8 PixelWave
starysmok
2024-11-04 15:13:34 +0000 UTCye these are quality ranks :D
Furkan Gözükara
2024-09-14 11:52:01 +0000 UTCoh ... now i get it ... those are your ranks ... not lora ranks :D
Raul Ciotescu
2024-09-14 07:47:16 +0000 UTCwhat would be the explanation behind rank 1 being the best? it trains less params then rank 2?
Raul Ciotescu
2024-09-14 07:44:04 +0000 UTCyes totally normal. currently i am getting 6.19 second it on 8x gpu with 4x gpu fast rank
Furkan Gözükara
2024-09-10 21:52:40 +0000 UTCI use your 4x_GPU_Rank 1, it shows 2500 steps, 193 epoch and 9.24s/it. I thougt it will be faster. But does it sound right for you?
Lukas
2024-09-10 17:15:33 +0000 UTCThanks! I will share my results
Lukas
2024-09-10 11:52:42 +0000 UTCfor runpod go with Rank 1 - it is best - if it be slow go with rank 2
Furkan Gözükara
2024-09-10 11:46:50 +0000 UTCSo which config ist the best for style? I will give it a try in runpod
Lukas
2024-09-10 11:32:09 +0000 UTCYou are welcome
Furkan Gözükara
2024-08-27 03:03:07 +0000 UTCThank you
Mark Sutherland
2024-08-27 03:01:21 +0000 UTCkohya gui doesn't know automatically split so we set it manually. when you pay attention to the cmd window you will see it shows total epoch. it wasnt exactly 200 in my case but it was 188 total epoch on CMD when i set 50 for 15 images. probably due to i dont have 16 images :) or 20 - not dividable to 4
Furkan Gözükara
2024-08-26 23:21:07 +0000 UTCThanks for the update... Dividing the number of epochs by the number of gpu's was the key piece of info that I needed. I assumed that was what I needed to do but I wasn't sure and didn't want to waste time if it was going to cause it to be undertrained.
juggzz143
2024-08-26 23:06:17 +0000 UTCjust added configs you get effective 1.4 second / it check the updated post : https://www.patreon.com/posts/110293257 you can even get better with batch size 2 and 4x but i dont suggest due to quality loss
Furkan Gözükara
2024-08-26 22:55:02 +0000 UTCgive me few minutes to add
Furkan Gözükara
2024-08-26 21:49:10 +0000 UTCYes pls I am currently getting 3.85s/it on my 4090 but 9.10s/it on Massed Compute with 4 A6000s.
juggzz143
2024-08-26 21:45:00 +0000 UTCgood idea i can share massed compute max speed
Furkan Gözükara
2024-08-26 21:24:03 +0000 UTCCan you share the settings to make kohya_ss train faster with multiple gpu's? I am currently training with 4 A6000's on Massed Compute but it seems slower than my actual 4090.
juggzz143
2024-08-26 21:15:25 +0000 UTCit will help me for sure. after lora hopefully full fine tuning / dreambooth next
Furkan Gözükara
2024-08-26 20:59:46 +0000 UTCyes i have : FLUX Auto Model Downloader for Windows, RunPod and Massed Compute https://www.patreon.com/posts/110323512 zip file also has auto downloader for training needed models : https://www.patreon.com/posts/kohya-flux-lora-110293257
Furkan Gözükara
2024-08-26 20:59:28 +0000 UTCDo you have a one click installer for flux which downloads the models etc cheers
Mark Sutherland
2024-08-26 20:51:34 +0000 UTCThanks for all of your work and research. Incredibly helpful! Does your LORA research help you determine what DB fine tuning training will work best?
Ec Jep
2024-08-26 20:48:59 +0000 UTC
