Multi-GPU FLUX Full Fine Tuning Experiments and Requirements on RunPod and Conclusions - Used 2x A100 - 80 GB GPUs
Added 2024-09-21 15:31:55 +0000 UTCPatreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.
Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord
Please also Star, Watch and Fork our Stable Diffusion & Generative AI GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)
=======
I have done an extensive multi-GPU FLUX Full Fine Tuning / DreamBooth training experimentation on RunPod by using 2x A100 - 80 GB GPUs (PCIe) since this was commonly asked of me.
Image 1
Image 1 shows that only first part of installation of Kohya GUI took 30 minutes on a such powerful machine on a very expensive Secure Cloud pod - 3.28 USD per hour
There was also part 2, so just installation took super time
On Massed Compute, it would take like 2-3 minutes
This is why I suggest you to use Massed Compute over RunPod, RunPod machines have terrible hard disk speeds and they are like lottery to get good ones
Image 2, 3 and 4
Image 2 shows speed of our very best config FLUX Fine Tuning training shared below when doing 2x Multi GPU training
Used config name is : Quality_1_27500MB_6_26_Second_IT.json
Image 3 shows VRAM usage of this config when doing 2x Multi GPU training
Image 4 shows the GPUs of the Pod
Image 5 and 6
Image 5 shows speed of our very best config FLUX Fine Tuning training shared below when doing a single GPU training
Used config name is : Quality_1_27500MB_6_26_Second_IT.json
Image 6 shows this setup used VRAM amount
Image 7 and 8
Image 7 shows speed of our very best config FLUX Fine Tuning training shared below when doing a single GPU training and Gradient Checkpointing is disabled
Used config name is : Quality_1_27500MB_6_26_Second_IT.json
Image 8 shows this setup used VRAM amount
Image 9 and 10
Image 9 shows speed of our very best config FLUX Fine Tuning training shared below when doing 2x Multi GPU training - this time Fused Backward Pass is disabled
Used config name is : Quality_1_27500MB_6_26_Second_IT.json
Image 10 shows this setup used VRAM amount
Image 11
Image 2 shows speed of our very best config FLUX Fine Tuning training shared below when doing 2x Multi GPU training on a different Pod
Used config name is : Quality_1_27500MB_6_26_Second_IT.json
Conclusions
For multi-GPU FLUX Fine Tuning, you have to use at least 80 GB GPUs
When doing multi-GPU FLUX Fine Tuning, Fused Backward Pass brings 0 VRAM usage improvements but slows down the training - I am going to report this
With A100 GPU, you are able to reach 2.89 second / it - probably it will get better as you do more steps
With 2x A100 GPU, you are able to reach 4.1 second / it - effective speed 2.05 second / it
The speed gain is 0.75 / 2.9 = 26% - so 2x GPU training totally doesn't worth it at the moment
If speed drop stays same due to multi-GPU overhead, 8x A100 may be beneficial but you have to experiment it properly and calculate speed gain
Currently single L40S would be way cheaper and faster
When doing multi-GPU FLUX LoRA training, we almost gain linear speed increase - I have tested with 8x RTX A6000 : https://www.patreon.com/posts/110879657
As shown in Image 11, there is also a chance that you will get a way worse performing pod
With same 2x A100 GPU and no visible difference, that random pod performed 1/4 speed of another same config pod - terrible terrible speed
Comments
you are welcome. i am glad you solved. so sorry for late reply
Furkan Gözükara
2024-10-05 00:45:04 +0000 UTCOk I just searched the Patreon posts and found the V3 best configs and found the optional 22gb vram files. Awesome. Thanks anyway.
Dinh Vu
2024-10-04 06:50:54 +0000 UTCHello Furkan, I've an important question regarding Dreambooth training using a 4090 for FLUX. I used the 23gb config you've provided but it takes way too long to train (600 hours for 20k steps). When I switched to the 15.5gb config, it runs at 108 hours. Still I think this is longer than usually expected (because I am not using the full vram of the 4090). Is there a way to slightly adjust the config to use 22gb of VRAM instead of 23? This will guarantee that my video card will work, because when I check my task manager, I always only have 22.5 out of the 24gb vram available for use. (it is a secondary card so no monitors or apps use the vram, I conserve the full 22.5gb available for only training in this case). Please let me know how I can adjust vram usage slightly on the 23gb config. This is for kohya flux Dreambooth training. Thank you Furkan.
Dinh Vu
2024-10-04 06:45:30 +0000 UTCMassed Compute is 100% not like you said. they are official partner of NVIDIA therefore they are not even allowed to use RTX 4090 and alike consumer GPUs
Furkan Gözükara
2024-09-22 15:00:18 +0000 UTCyes single L40S is best one at the moment
Furkan Gözükara
2024-09-22 14:59:51 +0000 UTCRegarding the above article, I can confirm via WarAnakin who uses Kohya_ss professionally and creates the Crystal Clear LoRA's on Civitai uses a single L40S for training.
21machines
2024-09-22 12:57:19 +0000 UTCThe problem with Massed Compute is it looks like I'm renting someone's Linux desktop in a third world country. I don't trust them. For me it's been very slow and difficult to connect Backblaze. I'll give it a second look but Vast.ai and RunPod are my current favorites. You get what you pay for.
21machines
2024-09-22 12:35:56 +0000 UTC
