SDXL Kohya LoRA Training With 12 GB VRAM Having GPUs - Tested On RTX 3060
Added 2023-07-28 21:01:19 +0000 UTCJoin discord and tell me your discord username to get a special rank : SECourses Discord
Upgrade Kohya to latest version. Open a CMD and do a git pull.
Kohya repo : https://github.com/bmaltais/kohya_ss
SDXL training video : https://youtu.be/AY6DMBCIZ3A
SDXL training github file : click
I tested on my second GPU - RTX 3060 (12 GB). However since it is my second GPU it has 0 VRAM usage. Therefore I was able to use 96 Network Rank (Dimension). The more Network Rank (Dimension) means it will be able to learn more information.
But since you will use it on your main GPU there will be already some VRAM usage. Therefore start with Network Rank (Dimension) 8 and then increase until you get error as I have shown in video.
Sorry that it doesn't have audio because I am on vacation. Therefore I recorded via remote connection to my main computer.
Training command (training command.txt) with 32 Network Rank
32 Rank uses 11.5 GB VRAM
accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --pretrained_model_name_or_path="F:/0 models/sd_xl_base_1.0.safetensors" --train_data_dir="F:/kohya sdxl tutorial files\img" --reg_data_dir="F:/kohya sdxl tutorial files\reg" --resolution="1024,1024" --output_dir="F:/kohya sdxl tutorial files\model" --logging_dir="F:/kohya sdxl tutorial files\log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=32 --output_name="tutorial_video" --lr_scheduler_num_cycles="10" --no_half_vae --full_bf16 --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="5200" --save_every_n_epochs="1" --mixed_precision="bf16" --save_precision="bf16" --cache_latents --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale --noise_offset=0.0
Training json file : test_video_lowVRAM.json
Comments
yes once it uses shared vram it reduces speed significantly sadly
Furkan Gözükara
2024-01-06 12:27:48 +0000 UTCYou're right, it uses 0.2 on the shared ... i didnt see it
Anonyme pas trop anonyme
2024-01-06 11:33:41 +0000 UTCi think it still used shared VRAM. currently I am testing OneTrainer and it uses lesser VRAM than Kohya. but i need to find good hyper parameters
Furkan Gözükara
2024-01-06 11:23:22 +0000 UTCInformation for people who wants to try : 11 training images - same parameters as the .json with 96 NETWORK RANK - before starting i had 0.8GB/12GB VRAM. It is working right now and i have 11.7 GB VRAM/12 GB VRAM. On a 3060(12GB), it take about 6H30 to complete.
Anonyme pas trop anonyme
2024-01-06 08:56:58 +0000 UTCi agree
Furkan Gözükara
2023-08-25 19:48:42 +0000 UTCI played around with it for a couple of hours. You can actually get some small improvements out of the refiner, by seeting a very small denoise and a smaller cfg and 5 steps or lower. Doesn't make a big difference though. We'll have to wait until we find out how to train a refiner lora.
Amazing Asians
2023-08-25 16:55:31 +0000 UTCThat is correct. Refiner training is something I am waiting as well. Sadly none of the scripts supporting it as far as I know
Furkan Gözükara
2023-08-25 15:33:37 +0000 UTCI'm using a SDXL Lora like in your youtube tutorial and it works great. Thank you so much! But I would like to train a refiner Lora if that is possible. If i use the standard refiner model without my base lora, the results are great, but the ohwx man is changed to a random guy. If I add the base_lora after the refiner, the results just look terrible. Will you make a tutorial about how to train a refiner Lora? If not, do you have any tip for me so i can do it myself?
Amazing Asians
2023-08-25 15:30:43 +0000 UTCvery nice
Furkan Gözükara
2023-08-24 12:17:58 +0000 UTCI made it: https://huggingface.co/alessandro893/sdxl_base_1.0_FP32
Alexander Kolesnikov
2023-08-19 12:39:24 +0000 UTCSorry for late reply patreon didn't give me notifications. I am glad you solved
Furkan Gözükara
2023-08-15 01:42:13 +0000 UTCIt ended up working when I used reg images, somehow.
Cryptosai
2023-08-13 09:05:55 +0000 UTCYou are welcome and thank you so much for the comment.
Furkan Gözükara
2023-08-12 20:25:23 +0000 UTCI tested it with my 3060 and it worked. It took a long time but it was worth it, the quality of SDXL is amazing. Thanks for the tutorial!
San Milano
2023-08-12 19:45:08 +0000 UTCI have the most up to date drivers, are you aware of anyone that has had better success with 40 series GPUs and what drivers they might be using?
Cryptosai
2023-08-11 12:17:45 +0000 UTCHello. that is not normal. not 50 hours or that error. i suspect it is due to nvidia driver. regularization will not make a difference. i made a whole tutorial for 12gb vram cards have you watched it? : https://youtu.be/sBFGitIvD2A I suspect drivers because RTX 4090 owners are barely getting RTX 3090 performance and even lower
Furkan Gözükara
2023-08-11 12:05:37 +0000 UTCWhen testing on my 4070 TI, the process is estimated to take 50h and then stops with RuntimeError: CUDA error: unspecified launch failure after 1h and a half of running. The differences, in my case, is that I did not use any regularization images, I used 30 images as training samples, and the images had .txt files with descriptions (as that is the way I used to train in SD 1.5) Edit: For the record, my VRAM usage when nothing is running is 0.4, and I do have the latest Kohya version
Cryptosai
2023-08-11 11:25:03 +0000 UTCyes this happens when overtrained. you should do x/y/z checkpoint comparison. moreover improving the training dataset really really important. i have recorded a new video and explaining both of these very well. hopefully coming today. still editing . and thank you so much for your Patreon support. sorry for late reply
Furkan Gözükara
2023-08-08 14:29:36 +0000 UTCHello ty for everything you do, new patreon. I was able to train the SDXL lora with gtx1080ti but when i prompt "photo of ohwx man +lora" the resemblance is good, the moment I start adding to the prompt it goes away and never resembles. Any suggestions? TY
PhilipRikoZen
2023-08-08 11:00:07 +0000 UTCyes definitely dont use any instead of male. alternatively you can generate your own class images from text2img tab of automatic1111. hopefully i will show in upcoming video
Furkan Gözükara
2023-08-05 12:03:06 +0000 UTCHey thanks for your answer. So it is better to use no regularization instead of the male one in my case? When nothing is running I use 0,5 gig excactly
Clive Oven
2023-08-05 12:01:02 +0000 UTCyesterday one another patreon member were having same issue. out of vram error. it turns out he wasn't using the latest version. can update your kohya to latest with git pull? also what is your VRAM usage when nothing is running? try to get it below 500 MB it helps a lot and regularization pictures do matter. for woman you need woman dataset. hopefully i will release it
Furkan Gözükara
2023-08-05 09:45:21 +0000 UTCEven on a 4070 (12gig ram) it seems not to work. But I have 50 pictures, maybe reduce the pictures in the folder? What is the optimal picture number anyhow for SDXL? And about the 1024x regulation pictures, does it matter if the subject is a woman? Because they all show a man. Sorry for the stupid question haha. Anyway keep up the good work. You are amazing.
Clive Oven
2023-08-05 09:40:16 +0000 UTCyou are welcome.
Furkan Gözükara
2023-08-04 19:24:22 +0000 UTCthank you very very much!
Andrex Selivanov
2023-08-04 19:21:41 +0000 UTCnot yet sorry about delay. i will message you hopefully in 30 minutes.
Furkan Gözükara
2023-08-04 17:34:03 +0000 UTCyes, did you see it?
Andrex Selivanov
2023-08-04 16:07:21 +0000 UTCcan you message me from discord? i can connect your pc and try myself.
Furkan Gözükara
2023-08-04 15:24:30 +0000 UTCTrying 8, 7, 6, 5, 4, ... the same results (( , maybe 2 monitors 4k eating vram?
Andrex Selivanov
2023-08-04 15:23:29 +0000 UTCok you need to try with rank 16 probably. 0.6 is a lot usage :/ have you tried rank 16 and rank 8?
Furkan Gözükara
2023-08-04 11:33:41 +0000 UTCyou are welcome
Furkan Gözükara
2023-08-04 11:26:37 +0000 UTC0,6 - 0,9 gb i think so CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 12.00 GiB total capacity; 11.10 GiB already allocated; 0 bytes free; 11.34 GiB reserved in total by PyTorch)
Andrex Selivanov
2023-08-04 11:09:22 +0000 UTCThank you!
Alexander Kolesnikov
2023-08-04 06:33:57 +0000 UTCI made 2 topics for you : https://discuss.huggingface.co/t/how-can-we-convert-sdxl-diffusers-to-fp32-safetensors/49394/1 https://github.com/kohya-ss/sd-scripts/issues/710
Furkan Gözükara
2023-08-03 21:29:11 +0000 UTChello. can you check how much vram your system is using when you dont do training? i did a lot of training recently on rtx 3060
Furkan Gözükara
2023-08-03 20:51:43 +0000 UTCunfortunately not work for me (( 3060 12GB (OutOfMemoryError: CUDA out of memory.)
Andrex Selivanov
2023-08-03 20:40:09 +0000 UTCI will check and let you know hopefully
Furkan Gözükara
2023-08-03 19:05:30 +0000 UTCYes I mean diffusers. I did not find any scripts for sdxl converting in Kohya repo, but I know that the answer should be somewhere in this script: https://github.com/kohya-ss/sd-scripts/blob/sdxl/sdxl_train.py
Alexander Kolesnikov
2023-08-03 19:03:03 +0000 UTCyou mean diffusers i believe? kohya has such interface but i haven't tested yet. did you test?
Furkan Gözükara
2023-08-03 18:30:41 +0000 UTCit shouldnt matter. your mac should upcast it into fp32 while working. bu8t i dont have a mac so cant test. training fp32 will consume more vram. but if you can afford such training you can do training with fp32. may yield slightly better results . so sorry for late reply
Furkan Gözükara
2023-08-03 18:28:54 +0000 UTCOne more question - do you know how to convert sdxl raw model to .safetensors? I want to make sdxl-1.0 float32.
Alexander Kolesnikov
2023-08-03 12:28:06 +0000 UTCHello, I'm Mac user, mac works only with float32, should I need to train full precision fp32 model or it's fine to use fp16 model and upcast it to fp32? I also noticed that sdxl 1.0 is fp16 model, while sdxl 0.9 is fp32, which is better for training fp32? I tried training both - lora and dreambooth, both in fp32, and lora have much better result than dreambooth. (I used runpod RTX A6000).
Alexander Kolesnikov
2023-08-03 07:34:35 +0000 UTCdreambooth of realistic vision for face is still better than SDXL LoRA. I think we need refiner training as well. I am working on even a better workflow for SDXL
Furkan Gözükara
2023-08-02 21:03:48 +0000 UTCWhat have you discovered with the SDXL LORA training image quality vs the dreambooth training you have done recently? My latest RV5.1 & RV4.0 based dreambooth training have yielded amazing results.
Ec Jep
2023-08-02 20:14:31 +0000 UTC10 gb pretty low :/ i find that 32 rank is decent but uses 11.5 GB vram on my RTX 3060 - this is for SDXL 1024x1024
Furkan Gözükara
2023-08-01 19:00:45 +0000 UTCWhat rank would you recommend for a 3080fe with 10gb vram?
Virtamouse
2023-08-01 18:46:31 +0000 UTC
