I Am Researching Multi-GPU Training Speed Difference
Added 2024-07-23 00:13:46 +0000 UTCRented this machine on Massed Compute to test Stable Diffusion XL training speed difference with Kohya SS GUI on a 8x A100 (SXM4) machine and see the difference of multi-GPU training SXM vs PCIe linked GPUs
Hopefully will update this post as I get more info.
Update 1
Single GPU works with Batch size 7 but more than 1 GPU fails
Reported Kohya : https://github.com/kohya-ss/sd-scripts/issues/1434#issuecomment-2244049699
Single GPU training speed as below with batch size 7
Comments
yes it speed ups but the thing is we don't get linear speed up. we get like 25% speed increase due to communication delay with regular PCIe
Furkan Gözükara
2024-07-23 14:45:03 +0000 UTCHave you tested whether interconnecting two A6000 Nvidia graphics cards can speed up training?
楠 陈
2024-07-23 14:02:04 +0000 UTCye but which settings? there is no info regarding that. lets see if Kohya can comment : https://github.com/kohya-ss/sd-scripts/issues/1434#issuecomment-2244049699
Furkan Gözükara
2024-07-23 01:11:16 +0000 UTCDid you adjust the Accelerator settings to accommodate the extra gpus because I think the settings in Koyha only work if it has previously been setup in there
Michael
2024-07-23 01:06:44 +0000 UTC
