Training a FLUX LoRA with 256 Images Experiments - Full Workflow and Conclusions

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI  GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

Details

Main FLUX training thread that you need to follow : https://www.patreon.com/posts/110879657

• I have been working on training a FLUX LoRA with 256 images due to pressure of the Reddit community.

• No matter how many times I told them that when you add expressions the model will generate perfectly, they don't believe :)

• So I used my not ready dataset, picked 256 images to show how it can perfectly generate expressions and different perspectives.

• I didn't caption them only ohwx man since captioning reduces likeliness and doesn't bring improvements.

• I have done total 3 different trainings.

• First test done on 8x A6000 GPU with using 4x_GPU_Rank_1_FAST_Lower_Quality config file shared here up to 200 epochs : https://www.patreon.com/posts/kohya-flux-lora-110879657

• I did set the LR 2x since it was 8x GPU instead of 4x so it was 0.0002

• This caused overtraining after 40 epoch and since LR was too big, it didn't learn details very well

• However it even learnt my broken teeth :D https://www.reddit.com/r/StableDiffusion/comments/1fe6ar3/i_am_continuing_flux_lora_training_with_256/

• The grid of first experiment is here : 256 images training 8x GPU.jfif

• Then i did another training with using Rank_1_T5_XXL_39200MB_9_46_Second_IT.json on a single L40S GPU. This time I trained up to 40 epochs and this was undertrained, the LR was not big enough

• You can see the grid here : 256 images training 1x GPU.jfif

• As a third test I have done another training with 8x A6000 - this time I have used exactly same LR as 4x_GPU_Rank_1_SLOW_Better_Quality.json

• The only difference was I have enabled T5 XXL training as well - makes minimal difference, used same LR for Clip L training

• I have made the comparison as can be seen below

  • 256 images training 8x GPU lower LR and T5 Attention and T5 Training.jfif

  • 256 images training 8x GPU lower LR and T5 Attention and T5 Training - Test 2 Grid.jfif

  • 256 images training 8x GPU lower LR and T5 Attention and T5 Training - Test 3 Grid.jfif

• After doing careful analysis I decided that 80 epoch is best

• 80 epoch means that 80 * 256 = 20480 steps but since it was 8x GPU it was total 80 x 256 / 8 = 2560 steps

Conclusions

• Instead of 4 GPUs if you use 8 GPUs do not double learning rate, keep 0.0001 - double overtrains quickly

• 256 images were too much since i had too many repeating clothing and especially backgrounds, so it overwritten the knowledge of the model a lot, thus overfit

  • So you should collect a perfect dataset that has different expressions, perspectives, distances, clothings but do not have similar or repeating ones as i did

  • Make sure that images sharpness, focus and lightning are as much as possibly perfect

• It perfectly learnt every detail of the body and expressions that exists in the training dataset

• The quality is astonishing for example below image is raw output

• It can do the expressions that exists in the dataset perfectly as can be seen below

• It can do distant shots very well as can be seen below all are raw images

• It even learnt my broken teeth or forehead veins perfectly - learns even minor details

• In the below given distant shot example the body shape pretty accurate, currently i am muscular and have some extra fat too :)

• Since i had different hair lengths and styles in the training dataset, it generates mixed hairs sometimes or generate one of the style randomly

• The studio shot images I got are just too amazing that no studio can take such shots that is the realism and quality

• If you don't provide expression in the prompt, it sometimes generates perfectly fitting expression according to the scene and sometimes doesn't

• If you generate images with big resolution like 1280x1280 or 1536x1536, you will get vertical or horizontal noticeable lines - looks bad

• I shared 2x upscaled 18 example images on this Reddit post, please check all and also give an upvote I appreciate it very much : https://www.reddit.com/r/StableDiffusion/comments/1ffwvpo/tried_expressions_with_flux_lora_training_with_my/

• When compared to 15 images training, the realism improved significantly (below grid)

  • 256 images training 8x GPU lower LR and T5 Attention and T5 Training - Test 3 Grid.jfif

  • The head shape, body proportions, head position, perspectives, all are much better, only the environment is more overfit since I had to do more steps

  • Since i did more steps and got more overfit, i used Claude 3.5 (like ChatGPT) to add more details and description of backgrounds and environments to the prompts

  • Main FLUX training zip file updated and new prompts added https://www.patreon.com/posts/110879657