Furkan Gözükara

FLUX De-Distilled and Anti-Bleeding Fine-Tuning / DreamBooth & LoRA Training Experiments - Also Testing CFG Impact for Stylized Images

Added 2024-10-31 12:28:00 +0000 UTC

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

=======

To understand this post, you need have fully watched this tutorial : https://youtu.be/FvpWy1x5etM
You need to have fully read this post : https://www.patreon.com/posts/112099700
When training with de-distilled models like (consolidated_s6700.safetensors or VerusVision_1.0b_Transformer_fp16.safetensors), what you need to change is from our FLUX Fine-Tuning or LoRA training configs is only single parameter which is Guidance Scale
You need to change it to 3.5 from 1
If you are training community Fine-Tuned models like PixelWave which is not De-Distilled, use and train it same way as training base FLUX Dev model
I have tested 1, 2.5 and 3.5 and 3.5 yields best performance as can be seen in attached grids below on De-Distilled model

Moreover, when doing inference, we set regular CFG to 3.5 like in Stable Diffusion models instead of setting as 1 for regular FLUX DEV models

As tested in below grids, CFG 3.5 yields best quality compared to 2.5, 3.0 for de-distilled models

Using Higher CFG on FLUX DEV for Stylized Images

Another finding of myself is that training on regular FLUX DEV model as usual but when doing inference, not using CFG Scale 1 but using higher CFG Scale when generating stylized images like anime or painting
As can be seen in grids, this is either hit or miss.
Again CFG 3.5 looks like a good value to test

Complete Test Results and Findings

I have used 28 images selected from sub-set of my 256 images to train myself
I have used Dwayne Johnson 28 images dataset fully shared here : https://www.patreon.com/posts/114972274
I have tested with the following models
FLUX DEV - base model - CFG 1
- https://huggingface.co/black-forest-labs/FLUX.1-dev
Verus Vision 1.0b - de-distilled model - CFG 3.5
- https://huggingface.co/SG161222/Verus_Vision_1.0b
PixelWave v03 - Fine-Tuned model - CFG 1
- https://civitai.com/models/141592/pixelwave?modelVersionId=992642
FLUX DEV-De-Distilled - CFG 3.5
- https://huggingface.co/nyanko7/flux-dev-de-distill
You can download all above models from below repository
- https://huggingface.co/OwlMaster/SD3New/tree/main

Conclusions

PixelWave is overfit and bad quality model for realism training
Verus Vision not working better than DEV-De-Distilled - don't see reason to use it
FLUX DEV-De-Distilled still not fixes the bleeding / mixing problem when training multiple-concept sadly
Moreover, still your training overwrites your trained subject's class, such as if you are training a man, all man becomes your trained man, you captioned or not doesn't matter since FLUX has internal auto captioning alike mechanism - image embeddings used
Or lets say I trained myself and Dwayne Johnson in the same training
- My images were captioned as only ohwx
- Dwayne Johnson images were captioned only as bbuk
- The mixing / bleeding issue still happens at DEV-De-Distilled model
- Looking at the full grids will give you exact idea
Yet DEV-De-Distilled model has lesser degree of bleed / mix problem compared to FLUX DEV model
DEV-De-Distilled is a good model as a standalone single concept training and you can see grids to decide yourself
When you train without class token, e.g. man, when you generate images sometimes it generates feminine image even if you are training a man, thus class token usage really important
- For example on some images when generating with only ohwx or bbuk, it was generating woman or woman like images - the trainings were also made without class token to reduce bleeding / mixing problem
- However, even though I didn't use class token, still it did bleed due to internal captioning system of FLUX and FLUX being originally distilled model
- Sadly de-distilled models not reached full model level yet
- Hopefully I will fully research SD 3.5 to see if it works better for multi-concept and how to train it like FLUX

Model Meanings

28_img_me_FLUX_De_Distilled
- Trained only with My 28 images (subset of 256 images used in FLUX Fine Tuning / DreamBooth training) - CFG 3.5
- Trained model is : https://huggingface.co/nyanko7/flux-dev-de-distill
- Caption : ohwx man
28_img_me_PixelWave03
- Trained only with My 28 images (subset of 256 images used in FLUX Fine Tuning / DreamBooth training) - CFG 1
- Trained model is : https://civitai.com/models/141592/pixelwave?modelVersionId=992642
- Caption : ohwx man
28_img_me_Raw_FLUX
- Trained only with My 28 images (subset of 256 images used in FLUX Fine Tuning / DreamBooth training) - CFG 1
- Trained model is : https://huggingface.co/black-forest-labs/FLUX.1-dev
- Caption : ohwx man
Me_and_Rock_FLUX_De_Distilled
- Trained with my 28 images and I have used Dwayne Johnson 28 images
- Caption : ohwx - my images
- Caption : bbuk - Dwayne Johnson images
- Dataset fully shared here : https://www.patreon.com/posts/114972274
- Trained model is : https://huggingface.co/nyanko7/flux-dev-de-distill
- CFG 3.5
Me_and_Rock_FLUX_Raw
- Trained with my 28 images and I have used Dwayne Johnson 28 images
- Caption : ohwx - my images
- Caption : bbuk - Dwayne Johnson images
- Dataset fully shared here : https://www.patreon.com/posts/114972274
- CFG 3.5
- Trained model is : https://huggingface.co/black-forest-labs/FLUX.1-dev