Furkan Gözükara : A Kaggle Notebook to Train Stable Diffusion 1.5 and XL (SDXL) on a Free Kaggle Account with 2x Dual T4 GPU for free by using Kohya GUI

Hi multiple gpu explained in this tutorial : https://youtu.be/-uhL2nW7Ddw?si=QbHh9xafLXnJ1twt By the way it is very at the limit of gpu mem capacity so you may get OOM

Furkan Gözükara

2025-04-29 07:51:14 +0000 UTC

Hi, In the tutorial, I could not find any information on how to set up multiple GPUs, and when I select the multi-gpu with 2 process option on Gradio UI, I get an error. Do you have any videos in which you describe how to use the two Kaggle GPUs?

B. P.

2025-04-28 23:20:53 +0000 UTC

replied from discord

Furkan Gözükara

2024-11-25 21:45:29 +0000 UTC

Hi Furkan... how to use a different base model other than SDXL base? do I have to editi the json file and use that instead?

Art

2024-11-25 16:51:55 +0000 UTC

you need to use our config. lots of features doesnt exists on the kaggle since it is free. in your case memory_efficient_attention_forward you can already use 2x gpu i tested with sdxl works great

Furkan Gözükara

2024-11-13 13:15:11 +0000 UTC

Mr. Furkan Hello I wanted to get information about something. I am applying exactly what is in this guide in Kaggle. When the Kohya screen opens, it says “runwayml/stable-diffusion-v1-5” in the Pretrained model name or path section in the Model section. We are uploading 1024 images. should I change this part. how should I follow a path. because in this case I get an error. Also, we need to do any operation to run the 2nd gpu in kaggle. Also Fluxgym I think it works in kaggle. Are you planning to do a training on this...? Hata da bu şekilde: 2024-11-13 08:55:54 INFO epoch is incremented. ?]8;id=833029;file:///kaggle/working/kohya_ss/sd-scripts/library/train_util.py?\train_util.py?]8;;?\:?]8;id=559721;file:///kaggle/working/kohya_ss/sd-scripts/library/train_util.py#715?\715?]8;;?\ current_epoch: 0, epoch: 1 Traceback (most recent call last): File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 559, in train(args) File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 379, in train noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward return model_forward(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in call return convert_to_fp32(self.model_forward(*args, **kwargs)) File "/opt/conda/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast return func(*args, **kwargs) File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 1589, in forward sample, res_samples = downsample_block( File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 1024, in forward hidden_states = attn(hidden_states, encoder_hidden_states=encoder_hidden_states).sample File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 935, in forward hidden_states = block(hidden_states, context=encoder_hidden_states, timestep=timestep) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 857, in forward hidden_states = self.attn1(norm_hidden_states) + hidden_states File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 640, in forward return self.forward_memory_efficient_xformers(hidden_states, context, mask) File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 703, in forward_memory_efficient_xformers out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None) # 最適なのを選んでくれる File "/opt/conda/lib/python3.10/site-packages/xformers/ops/fmha/init.py", line 276, in memory_efficient_attention return _memory_efficient_attention( File "/opt/conda/lib/python3.10/site-packages/xformers/ops/fmha/init.py", line 403, in _memory_efficient_attention return _fMHA.apply( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply return super().apply(*args, **kwargs) # type: ignore[misc] File "/opt/conda/lib/python3.10/site-packages/xformers/ops/fmha/init.py", line 74, in forward out, op_ctx = _memory_efficient_attention_forward_requires_grad( File "/opt/conda/lib/python3.10/site-packages/xformers/ops/fmha/init.py", line 428, in _memory_efficient_attention_forward_requires_grad op = _dispatch_fw(inp, True) File "/opt/conda/lib/python3.10/site-packages/xformers/ops/fmha/dispatch.py", line 119, in _dispatch_fw return _run_priority_list( File "/opt/conda/lib/python3.10/site-packages/xformers/ops/fmha/dispatch.py", line 55, in _run_priority_list raise NotImplementedError(msg) NotImplementedError: No operator found for `memory_efficient_attention_forward` with inputs: query : shape=(1, 4096, 8, 40) (torch.float16) key : shape=(1, 4096, 8, 40) (torch.float16) value : shape=(1, 4096, 8, 40) (torch.float16) attn_bias : p : 0.0 `flshattF@2.5.6-pt` is not supported because: xFormers wasn't build with CUDA support requires device with capability > (8, 0) but your GPU has capability (7, 5) (too old) `cutlassF` is not supported because: xFormers wasn't build with CUDA support operator wasn't built - see `python -m xformers.info` for more info `smallkF` is not supported because: max(query.shape[-1] != value.shape[-1]) > 32 xFormers wasn't build with CUDA support dtype=torch.float16 (supported: {torch.float32}) operator wasn't built - see `python -m xformers.info` for more info unsupported embed per head: 40 steps: 0%| | 0/13000 [00:00 sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3.10', '/kaggle/working/kohya_ss/sd-scripts/train_db.py', '--config_file', '/kaggle/temp/models/model/config_dreambooth-20241113-084719.toml']' returned non-zero exit status 1. 08:55:57-339689 INFO Training has ended.

Bayram TATAR

2024-11-12 20:28:50 +0000 UTC

i dont have any mac sadly

Furkan Gözükara

2024-11-08 20:28:37 +0000 UTC

please also try mlflux, it runs on M1 chip

Rasika Singal

2024-11-08 01:52:33 +0000 UTC

Fp8 is better than Q8? I'm currently testing Lora models I trained on Replicate with the dev model on Kaggle with your SwarmUi notebook and it seems to be good so far

AiInfluence

2024-10-26 11:55:06 +0000 UTC

yes i think wont work good on Q8 GGUF. would work on FP8 though. Still need to be tested

Furkan Gözükara

2024-10-26 11:52:56 +0000 UTC

Also one last question, if I train my flux loras using your tutorials on the original dev model (23Gb) and then use these Loras on Q8 GGUF version will it have less quality?

AiInfluence

2024-10-26 11:51:34 +0000 UTC

sadly not. FLUX needs BF16 training, FP16 it didnt learn. But I didn't test fine tuning can be tested however still your session time probably wouldnt be sufficient

Furkan Gözükara

2024-10-26 11:49:46 +0000 UTC

Can this be done with Flux models aswell?

AiInfluence

2024-10-26 11:39:19 +0000 UTC

i tested and notebook works. just use the models in the configs . or properly download your custom model into notebook and give its path : https://youtu.be/X5WVZ0NMaTg

Furkan Gözükara

2024-09-04 11:55:12 +0000 UTC

if you update the notebook, please notice me. (ಥ﹏ಥ)

대훈 조

2024-09-04 05:40:13 +0000 UTC

runwayml deleted their repos :/ i need to update the notebook give me some time

Furkan Gözükara

2024-08-30 10:56:55 +0000 UTC

I have problem. ERROR model is not found as a file or ?]8;id=133710;file:///kaggle/working/kohya_ss/sd-scripts/library/train_util.py?\train_util.py?]8;;?\:?]8;id=672108;file:///kaggle/working/kohya_ss/sd-scripts/library/train_util.py#4789?\4789?]8;;?\ in Hugging Face, perhaps file name is wrong? / 指定したモデル名のファイル、またはHugging Faceのモデルが見つかりません。ファイル名が誤っているかもしれません: runwayml/stable-diffusion-v1-5 what's happen? how i solve this problem?

대훈 조

2024-08-30 07:51:18 +0000 UTC

great

Furkan Gözükara

2024-08-19 14:36:05 +0000 UTC

Thanks for your reply. I watched your video again and realized I forgot to execute the cell underneath the Ngrok cell before clicking the link. Now it works. I wanted to use the LORA merge funtion. My workaround was to copy the Loras into Input dataset, then create /outputs folder and copy loras from input to outputs folder. Now i can select them in the ui. For the finished lora i had to put a path (outputs folder) and filename incl. saferensors extension into the field to make it safe the result.

Oliver Koch

2024-08-19 14:20:47 +0000 UTC

i just made a fresh install no issues. perhaps it was temporary or you are out of your monthly ngrok bandwith. try with a fresh token

Furkan Gözükara

2024-08-19 12:20:37 +0000 UTC

I cannot connect to ngrok. Getting ERR_NGROK_8012. Ngrok works with your other notebooks but not with this one.

Oliver Koch

2024-08-19 10:09:05 +0000 UTC

latest kohya config for SD 1.5 is here : https://www.patreon.com/posts/very-best-kohya-97379147

Furkan Gözükara

2024-07-21 10:04:31 +0000 UTC

want try to run a 1.5 dreambooth trainign via Kohya, what are the latest Json settign files that should i be using...

eduardo

2024-07-21 01:53:57 +0000 UTC

I suggest you to use BF16 for training because our parameters are researched for BF16. FP16 or FP32 will work worse.

Furkan Gözükara

2024-07-15 22:22:43 +0000 UTC

with a 4090 NVIDIA and 64 gb RAM, do you recomend to use bf16 with experimental too in your configuration?

agustin perez

2024-07-15 11:05:26 +0000 UTC

for 200 images i would do this. train up to 50 epoch and save every 5 epochs and compare later. if be still undertrained you can train 50 more epochs.

Furkan Gözükara

2024-07-06 22:08:05 +0000 UTC

So what about ~200 images in set and without regularization images? Which formula to use in this scenario with this config?

George Gostyshev

2024-07-06 19:12:07 +0000 UTC

your image is broken. open a thread here and share your error and image : https://github.com/kohya-ss/sd-scripts/issues

Furkan Gözükara

2024-04-11 10:33:33 +0000 UTC

bmaltais fixing errors. he did lots of improvements in but in dev branch waiting him to commit . and i told you, your image is broken 1 (1)_resized.png

Furkan Gözükara

2024-04-11 10:32:48 +0000 UTC

he replied "bmaltais commented 9 hours ago Not sure what cause this error… it is caused by the training script and I don’t think I can do anything about it. You might want to open an issue directly on the as-scripts repo."

huiminh ding

2024-04-11 07:51:12 +0000 UTC

Traceback (most recent call last): File "/kaggle/working/kohya_ss/sd-scripts/sdxl_train_network.py", line 185, in trainer.train(args) File "/kaggle/working/kohya_ss/sd-scripts/train_network.py", line 272, in train train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process) File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 2080, in cache_latents dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process) File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 1023, in cache_latents cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.random_crop) File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 2428, in cache_batch_latents raise RuntimeError(f"NaN detected in latents: {info.absolute_path}") RuntimeError: NaN detected in latents: /kaggle/working/results/img/25_ohwx tanglaoya/1 (1)_resized.png [2024-04-11 07:13:14,425] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1115) of binary: /opt/conda/bin/python Traceback (most recent call last): File "/opt/conda/bin/accelerate", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command multi_gpu_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher distrib_run.run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /kaggle/working/kohya_ss/sd-scripts/sdxl_train_network.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-04-11_07:13:14 host : 6220ab6f6ca3 rank : 0 (local_rank: 0) exitcode : 1 (pid: 1115) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html, not sure what caused this errors, but several people using different ways to train, come across the same bug, but repo owner claimed it was not his problem

huiminh ding

2024-04-11 07:47:32 +0000 UTC

Yok OneTrainer ve Kohya kalitesi neredeyse aynı. önemli olan doğru ayarlarda eğitmek

Furkan Gözükara

2024-04-11 01:44:14 +0000 UTC

One trainer düşük kalitede mi üretiyor ? Lora için en iyi seçenek kohya_ss midir ?

cenk

2024-04-09 17:39:46 +0000 UTC

LoRA için yok çünkü düşük kalite. ama Fine tuning bugün video yayınladım 2 saat :) https://www.youtube.com/watch?v=0t5l6CP9eBg&lc=UgxoXVvh2CicH0J2PwJ4AaABAg

Furkan Gözükara

2024-04-09 17:36:07 +0000 UTC

One trainer için lora eğitim örneğiniz var mıdır ?

cenk

2024-04-09 16:21:21 +0000 UTC

that means you are clicking inaccurate button and not loading it accurately. gui changed but still working. if you show me how you are doing in discord i can tell your error

Furkan Gözükara

2024-04-09 10:11:17 +0000 UTC

what error are you having?

Furkan Gözükara

2024-04-09 10:10:32 +0000 UTC

yes seeming unable to train lora too,

huiminh ding

2024-04-09 05:55:34 +0000 UTC

Hi, I did use the preparation tab. I followed the 12 March 2024 Update completely. When I load the /kaggle/working/Kaggle_SDXL_DreamBooth_Best.json configuration file successfully, it doesnt change any of the fields in the model tab like in your video. The model is still set to runwayml/stable-diffusion-v1-5 and the "Image folder (containing training images subfolders)" field (which isnt in the version of the GUI demonstrated in your video) also doesn't get populated. As I noted I have tried setting the model and image folder fields in the model tab manually but it gives me the No [] were found in train_data_dir can't train...

onlycasual1

2024-04-09 00:35:42 +0000 UTC

please use the dataset preparation tab of the gradio. hopefully i will make a new updated video soon

Furkan Gözükara

2024-04-08 21:10:48 +0000 UTC

Hi, I'm having some issues starting the training. I have tried to follow this video as closely as possible: https://www.youtube.com/watch?v=16-b1AjvyBE The GUI has changed a bit since then and I can't get it to work due to this error: 18:17:57-749055 INFO Start training Dreambooth... 18:17:57-750671 INFO Validating model file or folder path stabilityai/stable-diffusion-xl-base-1.0 existence... 18:17:57-752778 INFO ...valid 18:17:57-753913 INFO Validating output_dir path /kaggle/temp/models existence... 18:17:57-755276 INFO ...valid 18:17:57-756431 INFO Validating train_data_dir path /kaggle/input/training-images existence... 18:17:57-758032 INFO ...valid 18:17:57-759210 INFO Validating reg_data_dir path /kaggle/working/outputs/reg existence... 18:17:57-760806 INFO ...valid 18:17:57-761954 INFO Validating logging_dir path /kaggle/working/outputs/log existence... 18:17:57-763374 INFO ...valid 18:17:57-765033 INFO log_tracker_config not specified, skipping validation 18:17:57-766726 INFO resume not specified, skipping validation 18:17:57-768008 INFO vae not specified, skipping validation 18:17:57-769217 INFO dataset_config not specified, skipping validation 18:17:57-770736 INFO Headless mode, skipping verification if model already exist... if model already exist it will be overwritten... 18:17:57-775088 INFO No [] were found in train_data_dir can't train... The only difference I could spot was that in GUI launched from the latest notebook under the "Model" section there is a field called "Image folder (containing training images subfolders)" which is in addition to the "Training images (directory containing the training images)" field under Dataset preparation. I have tried setting them both to /kaggle/input/training-images which results in the above error. Would appreciate any help here.

onlycasual1

2024-04-08 18:23:02 +0000 UTC

for sd 1.5 200 repeats may overtrain but if working fine great. because it would mean 400 repeats on 1 gpu

Furkan Gözükara

2024-03-14 15:57:58 +0000 UTC

I inserted the regularization folder, but something went wrong... I'll try again, but what means when you say: "you should make half number of steps. because it uses 2 gpu"... in the video, for dreambooth sdxl training I saw you show 100 repeats for training images... (the SD 1.5 is 200 repeats and it works very fine)

Art

2024-03-14 15:40:51 +0000 UTC

hi that would mean regularization images folder is missing. have you set it? hopefully i will make updated tutorial. i am waiting kohya to finish newest interface. by the way, on kaggle, you should make half number of steps. because it uses 2 gpu.

Furkan Gözükara

2024-03-14 14:39:26 +0000 UTC

Hi, Furkan... after some succesful 1.5 models, I wanted to try SDXL on one of my best dataset; I noticed that with new configuration jsons, the step calculation is (2000 / 1 / 1 * 1 * 1) = 2000 (i'm using 20 training images) and not (2000 / 1 / 1 * 1 * 2) = 4000 like the last time I used it... Is that ok, or there is some mistake in the parameters?

Art

2024-03-14 12:55:01 +0000 UTC

fixed

Furkan Gözükara

2024-03-12 01:18:36 +0000 UTC

simplified the thread. please watch this video : https://youtu.be/16-b1AjvyBE

Furkan Gözükara

2024-03-12 01:18:28 +0000 UTC

it is working for me. which one is broken?

Furkan Gözükara

2024-03-12 01:15:49 +0000 UTC

Thanks. The Kaggle dreambooth link is broken though.

BecauseReasons

2024-03-12 00:08:53 +0000 UTC

hello. hopefully tomorrow will fix sorry for delay. kohya did a lot of updates

Furkan Gözükara

2024-03-11 03:51:51 +0000 UTC

error, comes out 07:48:02-762329 INFO Version: v23.0.1 07:48:02-769461 INFO nVidia toolkit detected 07:48:04-589434 INFO Torch 2.1.2+cu118 07:48:04-630076 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700 07:48:04-654485 INFO Torch detected GPU: Tesla T4 VRAM 15102 Arch (7, 5) Cores 40 07:48:04-714962 INFO Submodule initialized and updated. 07:48:04-716439 INFO Verifying modules installation status from /kaggle/working/kohya_ss/requirements_linux.txt... 07:48:04-720915 INFO Installing package: torch==2.1.2+cu118 torchvision==0.16.2+cu118 xformers==0.0.23.post1+cu118 --extra-index-url https://download.pytorch.org/whl/cu118 07:48:17-363068 INFO Verifying modules installation status from requirements.txt... ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /kaggle/working/kohya_ss/kohya_gui.py:7 in │ │ │ │ 6 from textual_inversion_gui import ti_tab │ │ ❱ 7 from library.utilities import utilities_tab │ │ 8 from lora_gui import lora_tab │ ╰──────────────────────────────────────────────────────────────────────────────╯ ModuleNotFoundError: No module named 'library.utilities'

huiminh ding

2024-03-10 07:53:05 +0000 UTC

no when using reg images always use same dimension as training images. so all are 768x768 for sd 1.5

Furkan Gözükara

2024-02-10 15:37:06 +0000 UTC

thaank you! so: 1024x1024 training images, 768x768 reg images, steps calculation + 1...

Art

2024-02-10 15:35:27 +0000 UTC

i prefer 768x768 . steps calculation and everything is totally same as sdxl. just use the sd 1.5 config

Furkan Gözükara

2024-02-10 15:24:42 +0000 UTC

Hello, Furkan... I want to try another 1.5 training, but it's not clear what training images do I have to use: it's better to train 1024x1024 or 512x512 images? The saving steps calculation is the sama of the sdxl workflow?

Art

2024-02-10 12:30:59 +0000 UTC

you are welcome

Furkan Gözükara

2024-02-08 18:31:59 +0000 UTC

Great I try again ;) Very thank you for your help ;)

GuiGui

2024-02-08 17:32:40 +0000 UTC

it is because of gradio update. developer has to be fix but i just made a hack fix. you can download v15

Furkan Gözükara

2024-02-07 21:23:39 +0000 UTC

Hello, new in this field but already passionate I start to test various things by following your tutorials which are geniaux!!!! I have a problem on this tuto: Everything works fine until this error message. Can you help me? venv folder does not exist. Not activating... 16:58:24-113538 INFO Version: v22.6.0 16:58:24-120372 INFO nVidia toolkit detected 16:58:28-413512 INFO Torch 2.0.1+cu118 16:58:28-490005 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700 16:58:28-519266 INFO Torch detected GPU: Tesla T4 VRAM 15102 Arch (7, 5) Cores 40 16:58:28-521168 INFO Verifying modules installation status from /kaggle/working/kohya_ss/requirements_linux.txt... 16:58:28-525550 INFO Verifying modules installation status from requirements.txt... Traceback (most recent call last): File "/kaggle/working/kohya_ss/kohya_gui.py", line 1, in import gradio as gr File "/opt/conda/lib/python3.10/site-packages/gradio/init.py", line 3, in import gradio.components as components File "/opt/conda/lib/python3.10/site-packages/gradio/components/init.py", line 1, in from gradio.components.annotated_image import AnnotatedImage File "/opt/conda/lib/python3.10/site-packages/gradio/components/annotated_image.py", line 8, in from gradio_client.documentation import document, set_documentation_group ImportError: cannot import name 'set_documentation_group' from 'gradio_client.documentation' (/opt/conda/lib/python3.10/site-packages/gradio_client/documentation.py)

GuiGui

2024-02-07 17:20:51 +0000 UTC

Use RunPod and you will have no issues for quality and also extracting a lora. we have auto install update scripts for Automatic1111 on RunPod. LoRA extraction currently uses too much RAM or VRAM therefore I also just opened an issue on GUI github to reduce : https://github.com/bmaltais/kohya_ss/issues/1933

Furkan Gözükara

2024-02-05 12:01:31 +0000 UTC

hello everyone (and Furkan in particular); after having made a series of models with Kohya+Kaggle, I make some considerations: - the notebook works very well, and the materials provided by Furkan are very valid (I'm thinking of the reg images and the json) - among the 5 checkpoints generated, the best (due to similarity of the subject) are always the third or fourth (but I would say more the third) - I tested checkpoints on Automatic1111 on both Kaggle and Colab Pro: the outputs on Kaggle are much superior; and I'm wondering if I can get better quality on Runpod, so I might consider switching from Colab to Runpod - this notebook by Khoya (but also reading elsewhere) does not extract a LORA from a checkpoint; it's absolutely impossible to do this, and it seems to be a Khoya problem for some time now: can you tell me an effective method to correctly extract a LORA from a checkpoint? I greet everyone and await your feedback!

Art

2024-02-05 07:44:08 +0000 UTC

i use sdxl 1.0 base for sdxl training

Furkan Gözükara

2024-02-04 15:45:35 +0000 UTC

Thank you Furkan! Can you say to me also what SDXL model base is used by Kohya in the Kaggle notebook of this thread?

Art

2024-02-04 09:22:57 +0000 UTC

the kohya uses hyper realism v3 by default. that is the great sd 1.5 model

Furkan Gözükara

2024-02-02 15:14:21 +0000 UTC

I trained with the workflow you updated on the last version of the kaggle/Kohya notebook... I assumed it was the base version of SD1.5... do I have to use another one? and which one? Can you give links of the base models used for training both SDXL and SD1.5, so that I can upload them together with checkpoints of myself? Thank you...

Art

2024-02-02 14:20:44 +0000 UTC

did you train yourself on SD 1.5 pruned?

Furkan Gözükara

2024-02-02 14:06:54 +0000 UTC

yes: now I explain in details: yesterday i trained myself with the SD1.5 workflow, then I tested and found one that worked best; today I launched the kohya kaggle notebook and ran it uploading in the kaggle/working directory the ckeckpoint of myself, the v1-5-pruned.safetensors (from HF), then setted the lora extractions according to your image, end extracted in the directory of kohya model (copying the path from the lora training script in the notebook); the result is a file of 600 md called "model" without extension; I renamed it putting the extension .safetensors and testd in Automatic1111, but that lora doesn't generate nothing but random images (houses, persons, objects)... hot to resolve this?

Art

2024-02-02 14:02:44 +0000 UTC

you need to use the model you trained yourself on as base model. whichever the model you used for training that model. by the way lets say you trained yourself on model A, then extract lora and used that lora on model B may not work very well

Furkan Gözükara

2024-02-02 13:38:24 +0000 UTC

already tried with a SD1.5 checkpoint (uploaded in the working folder of kohya): it saves a 600 mb model (file without extension), but it doesn't generate images of myself (I was the subject of a training of yesterday: the safetensor checkpoint works well on automatic1111); where am I wrong? Do I need also the path to safetensor base model to input in the field "Stable Diffusion Base Model"? The chekpoints of the sdxl training I'm doing right now have to be retrieved from the /kaggle/temp/models repository one by one?

Art

2024-02-02 13:34:00 +0000 UTC

no you have to manually extract LoRA. use Kohya GUI SS with these settings : https://cdn-uploads.huggingface.co/production/uploads/6345bd89fe134dfd7a0dba40/NtToBFK2uumY_YMHOZ7MW.png

Furkan Gözükara

2024-02-02 12:27:26 +0000 UTC

Hi, Doctor, running the lora cell after the 6 safetensors generation, it will extract the loras from checkpoints? can you explain this step? And do you have right parameters to do the same with SD1.5 generated models?

Art

2024-02-02 12:16:49 +0000 UTC

Well what is expected is, it gets overfit so more resemblance but lesser generalization. that is weird

Furkan Gözükara

2024-02-01 00:06:10 +0000 UTC

I'm noticing something strange... after 5 complete training sessions (and a couple of failures), I notice that out of the 6 safetensors files generated, the ones that most resemble the person are the ones in the middle of the training (the third or fourth generated) ...does this happen to you too? do you have an explanation for this?

Art

2024-01-31 14:58:36 +0000 UTC

thank you...

Art

2024-01-30 22:13:13 +0000 UTC

Please watch these 2 videos : https://youtu.be/dpM02YMj8FY https://youtu.be/16-b1AjvyBE

Furkan Gözükara

2024-01-30 11:18:15 +0000 UTC

hi... I uploaded my checkpoint and yhe sdxl 1 in the root of the working folder, and ran the gui, the i made settings and tried to output lora in the same folder o the uploaded checkpoints, the output folder of the temp folder of the six checkpoints of the normal workflow and any other folder of the kohya instaaltion but none of them were writeable... Maybe I have to run all the cells of the normal workflow to get the lora in the temp folder?

Art

2024-01-30 06:46:14 +0000 UTC

hi... I uploaded my checkpoint and yhe sdxl 1 in the root of the working folder, and ran the gui, the i made settings and tried to output lora in the same folder o the uploaded checkpoints, the output folder of the temp folder of the six checkpoints of the normal workflow and any other folder of the kohya instaaltion but none of them were writeable... Maybe I have to run all the cells of the normal workflow to get the lora in the temp folder?

Art

2024-01-30 06:45:14 +0000 UTC

you must have given incorrect folder path. what folder path you given as output?

Furkan Gözükara

2024-01-29 21:21:46 +0000 UTC

tried, but not able to get an output folder for lora... uploaded checkpoint (mine and sdxl 19 but setting an output folder fom the web ui retturns a non writeable folder on kaggle notebook... maybe it's an easy fix to do, but I'm really a beginner... any help?

Art

2024-01-29 21:04:36 +0000 UTC

yep you can do. for SDXL it may fail due to RAM limitation though. give it a try

Furkan Gözükara

2024-01-29 14:30:31 +0000 UTC

can I do this with the Kaggle/Khoya notebook I used to generate my checkpoint?

Art

2024-01-29 14:16:19 +0000 UTC

hello here : https://cdn-uploads.huggingface.co/production/uploads/6345bd89fe134dfd7a0dba40/TT9D-nOtJqop9nozTGZOA.png

Furkan Gözükara

2024-01-29 13:57:51 +0000 UTC

thank you... but it seems I'm a little too dumb to find that resource... can you point me to that one, please?

Art

2024-01-29 13:51:20 +0000 UTC

You can extract as we have shown in discord channel #tips info check out there screenshot

Furkan Gözükara

2024-01-29 13:03:51 +0000 UTC

hi... yesterday I got my favourite model finally... it's a checkpoint, and I sould like to extract a lora finetuned version from that one: can you suggest how to achieve that ising a kaggle notebook?

Art

2024-01-29 06:44:56 +0000 UTC

great

Furkan Gözükara

2024-01-25 01:05:40 +0000 UTC

I launched the checkpoints one by one, use them individually and manually compared the results. Problem solved! Again, Thank you for your help!

Terresa P

2024-01-24 23:57:32 +0000 UTC

Well it shouldn't crush interesting.

Furkan Gözükara

2024-01-24 21:58:44 +0000 UTC

Hi Dr. Gozukara, Thank you for your instruction! It was my bad. What a silly mistake! Now it worked. For Automatic1111 with kaggle, I found checkpoint comparison via x/y plot computational extremely expensive. The system crushed several times. I will try it again. Do you have any thoughts on that?

Terresa P

2024-01-24 05:31:09 +0000 UTC

your links are incorrect. you see it downloaded only 36kb. here an accurate link : https://huggingface.co/Terresa/SDXL_training/resolve/main/My_DB_Kaggle-step00002084.safetensors

Furkan Gözükara

2024-01-24 00:57:51 +0000 UTC

Thank you, Dr. Gozukara! I screen recorded my whole process of launching Automatic1111 with Kaggle and edited a 3 minutes video. I pointed out all the problems happened during the process. Here is the video link on Hugging face: https://huggingface.co/datasets/Terresa/video/tree/main. Please check it! Thank you!

Terresa P

2024-01-23 18:59:09 +0000 UTC

I just tested on my computer and it works. tested step00002084. can you record a video and how you are trying to use it on kaggle?

Furkan Gözükara

2024-01-23 14:29:26 +0000 UTC

Thank you, Dr.! Here is the link on my Hugging face. I make it public and you should be able to download it: https://huggingface.co/Terresa/SDXL_training/tree/main

Terresa P

2024-01-23 03:18:30 +0000 UTC

there must be an error somewhere. you can send me model and i can try locally to see if model is accurately trained or not

Furkan Gözükara

2024-01-23 02:46:17 +0000 UTC

I do not have local powerful GPU, that is why I chose to run Automatic1111 and train models with Kaggle. On Kaggle, I use GPU T4*2, which is fine for both model training and Automatic1111. The problem is I cannot launch the models from checkpoints in Automatic1111. it just does not respond. I also tried comfyUI with Google colab. I did not work either. Here is the error I got from ComfyUI: Error occurred when executing CheckpointLoaderSimple: Error while deserializing header: HeaderTooLarge. So I got error from ComfyUI. For Automatic1111, no errors, but I could not get it responded when tying to launch my models from checkpoint.

Terresa P

2024-01-23 02:33:21 +0000 UTC

What error are you getting? it should work. also what is your current GPU can you let me know?

Furkan Gözükara

2024-01-23 02:13:02 +0000 UTC

Hello Dr. Gozukara, Thank you for your detailed SDXL_Dreambooth training tutorial! I trained one model yesterday. It took me around 5 hours and I ended up getting 6 . safetensors files(around 6G for each) and uploaded to my hugging face account. Now I have trouble running them. I do not have powerful GPU, so I decided to use Kaggle to run Automatic1111. Again I followed your tutorial and Kaggle script. I used wget command to pull my 6 models from huggingface website ,and after that it appeared in my /kaggle/working/models folder. The problem comes after I launched the Automatic1111: I could not get my model from the checkpoint, even though they are all in there. I can get stable-diffusion-xl-base-1.0 running but not my fine-tuned ones. Please help me. Thank you!

Terresa P

2024-01-22 23:42:12 +0000 UTC

it is not an error. open this link : https://a2a1-104-197-102-215.ngrok-free.app dont click visit site start kohya gui and then click visit site and it will work

Furkan Gözükara

2024-01-15 00:48:23 +0000 UTC

I get as far as starting the ngrok session, but get an error. Can anyone explain this error: * ngrok tunnel "https://a2a1-104-197-102-215.ngrok-free.app" -> "http://127.0.0.1:7860/" * Serving Flask app 'main' * Debug mode: off Address already in use Port 5000 is in use by another program. Either identify and stop that program, or start the server with a different port

Rick B

2024-01-14 23:42:44 +0000 UTC

great

Furkan Gözükara

2024-01-10 08:56:50 +0000 UTC

I tried it again, and it worked. Thanks!

Safiya Dubois

2024-01-10 08:50:45 +0000 UTC

great ty

Furkan Gözükara

2024-01-04 22:41:43 +0000 UTC

Thanks a lot Dr. I will keep you updated as i try this!

Anonyme pas trop anonyme

2024-01-04 22:41:24 +0000 UTC

Hello. This is a good question. Actually I am doing a research right now similar to this for a company. i don't have data yet so can't answer. but i assume that if you can manually generate your dataset you can fine tune model for that purpose. like generate certain type of donuts and train model to generate such

Furkan Gözükara

2024-01-04 22:40:19 +0000 UTC

Hello Dr. Gozukara. This message is not about the post but it is a question. I'm new to Patreon and i was wondering somethig by watching all the tutorials. Have or will you make a video of training with a Ai generated dataset ? Can AI produce its own photorealistic models (ofc we do the training) ? Thanks for your answer and have a great day !

Anonyme pas trop anonyme

2024-01-04 22:38:18 +0000 UTC

thanks for the trick

Furkan Gözükara

2024-01-03 22:47:27 +0000 UTC

one strange trick I found is keeping the CFG high at low number of steps somehow helps, whereas for turbo you'd expect CFG 1-2 to be best

Jim Winkens

2024-01-03 17:21:26 +0000 UTC

yes you can fix. you need to select fp16 not bf16. change all bf16 to fp16

Furkan Gözükara

2024-01-03 10:01:50 +0000 UTC

I have tried twice to train lora and got the same error I am seeing others are having. AssertionError: full_bf16 requires mixed precision='bf16' / full_bf16を使う場合はmixed_precision='bf16'を指定してください。 Traceback (most recent call last): File "/kaggle/working/kohya_ss/./sdxl_train_network.py", line 189, in trainer.train(args) File "/kaggle/working/kohya_ss/train_network.py", line 234, in train model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator) File "/kaggle/working/kohya_ss/./sdxl_train_network.py", line 47, in load_target_model ) = sdxl_train_util.load_target_model(args, accelerator, sdxl_model_util.MODEL_VERSION_SDXL_BASE_V1_0, weight_dtype) File "/kaggle/working/kohya_ss/library/sdxl_train_util.py", line 21, in load_target_model model_dtype = match_mixed_precision(args, weight_dtype) # prepare fp16/bf16 File "/kaggle/working/kohya_ss/library/sdxl_train_util.py", line 167, in match_mixed_precision weight_dtype == torch.bfloat16 AssertionError: full_bf16 requires mixed precision='bf16' / full_bf16を使う場合はmixed_precision='bf16'を指定してください。 Traceback (most recent call last): File "/opt/conda/bin/accelerate", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command multi_gpu_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher distrib_run.run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ ./sdxl_train_network.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-01-03_08:36:15 host : d32f44ffac1b rank : 1 (local_rank: 1) exitcode : 1 (pid: 1068) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-01-03_08:36:15 host : d32f44ffac1b rank : 0 (local_rank: 0) exitcode : 1 (pid: 1067) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html Is there a fix for this? I followed the tutorial exactly both times and watched the updated section with additional parameters. I deleted the notebook and started from scratch.

Safiya Dubois

2024-01-03 08:48:40 +0000 UTC

it is low quality so i haven't tested it yet. useful for companies

Furkan Gözükara

2024-01-02 23:34:39 +0000 UTC

do you have any advice on how to train dreambooth / lora with turbo models? it seems like it doesnt work well out of the box, giving low quality results

Jim Winkens

2024-01-02 23:34:04 +0000 UTC

hello. After this happens please start gui again and start training again. when it first time caches the images it leaves some VRAM left over. second time starting no more caching so it starts training

Furkan Gözükara

2023-12-31 13:36:00 +0000 UTC

Hey! I keep getting this error. Any suggestions? running training / 学習開始 num examples / サンプル数: 1300 num batches per epoch / 1epochのバッチ数: 1300 num epochs / epoch数: 2 batch size per device / バッチサイズ: 1 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 2600 steps: 0%| | 0/2600 [00:00 train(args) File "/kaggle/working/kohya_ss/./sdxl_train.py", line 512, in train encoder_hidden_states1, encoder_hidden_states2, pool2 = train_util.get_hidden_states_sdxl( File "/kaggle/working/kohya_ss/library/train_util.py", line 4100, in get_hidden_states_sdxl enc_out = text_encoder2(input_ids2, output_hidden_states=True, return_dict=True) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1230, in forward text_outputs = self.text_model( File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward encoder_outputs = self.encoder( File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward layer_outputs = encoder_layer( File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 393, in forward hidden_states = self.mlp(hidden_states) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 350, in forward hidden_states = self.fc2(hidden_states) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 14.76 GiB total capacity; 12.34 GiB already allocated; 3.75 MiB free; 13.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF steps: 0%| | 3/2600 [00:19<4:47:58, 6.65s/it, avr_loss=0.176] Traceback (most recent call last): File "/opt/conda/bin/accelerate", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command multi_gpu_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher distrib_run.run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ ./sdxl_train.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-12-31_12:57:20 host : a9b77c1db37b rank : 0 (local_rank: 0) exitcode : 1 (pid: 1046) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================

Denys Yarovyi

2023-12-31 13:34:48 +0000 UTC

you are welcome. sadly yes some custom models are causing RAM error.

Furkan Gözükara

2023-12-31 13:30:24 +0000 UTC

hello. this happens when something went wrong in installation. turn off session. start again from beginning. also the order matters. the newest video is here : https://youtu.be/16-b1AjvyBE

Furkan Gözükara

2023-12-31 12:23:13 +0000 UTC

i am trying to run the code for the token, but it keeps giving me this error: ModuleNotFoundError Traceback (most recent call last) Cell In[1], line 8 5 import threading 7 from flask import Flask ----> 8 from pyngrok import ngrok, conf 10 conf.get_default().auth_token = "2aAM5cepa9mI4BDrXweLQbWfmjM_7oBKhHhjoAQGSn29zPMn8" 12 os.environ["FLASK_ENV"] = "development" ModuleNotFoundError: No module named 'pyngrok' what should i do?

Kashif Salman

2023-12-31 06:22:35 +0000 UTC

I was able to wget the the model into the kaggle working directory and tell the gui where to find it, but I get out an out of memory error before any checkpoints are saved. Training with the sdxl base model still works fine, so maybe the custom one is just too big. Thanks for the extra help!

JB

2023-12-30 20:04:57 +0000 UTC

you can get them into kaggle working directory with wget command. then give their kaggle path. do that before starting the gui

Furkan Gözükara

2023-12-29 08:42:49 +0000 UTC

I was able to successfully complete a training using RealVisXL_V3.0. Thanks for the excellent instructions! I tried again using a model I uploaded to huggingface myself, and the training failed because it couldn't find the model. I think it's because the my folder is missing a model_index.json file but I'm not sure. How can we do this with models we have downloaded elsewhere, but aren't already on huggingface the way RealVisXL models are?

JB

2023-12-28 23:13:09 +0000 UTC

did you made internet on? that is settings on kaggle. i have no error

Furkan Gözükara

2023-12-25 19:45:45 +0000 UTC

testing right now

Furkan Gözükara

2023-12-25 19:44:17 +0000 UTC

I get this with V12 right in the first step: Err:1 http://packages.cloud.google.com/apt gcsfuse-focal InRelease Temporary failure resolving 'packages.cloud.google.com' Err:2 http://archive.ubuntu.com/ubuntu focal InRelease Temporary failure resolving 'archive.ubuntu.com' Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease Temporary failure resolving 'security.ubuntu.com' Err:4 https://packages.cloud.google.com/apt cloud-sdk InRelease Temporary failure resolving 'packages.cloud.google.com' Err:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease Temporary failure resolving 'archive.ubuntu.com' Err:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease Temporary failure resolving 'archive.ubuntu.com'

Tibor Martini

2023-12-25 19:26:14 +0000 UTC

great you founded. sorry for late reply

Furkan Gözükara

2023-12-22 23:20:38 +0000 UTC

That is accurate. That is why Kaggle is weaker than 24 GB GPU with BF16

Furkan Gözükara

2023-12-22 23:20:29 +0000 UTC

you answered my question in the later part of the video.

Ec Jep

2023-12-22 17:04:58 +0000 UTC

Quick question: in this kaggle json file you use fp16 and xformers where in other SOTA json setups you use bf16 and no xformers? I assume that is due to kaggle gpu's and available ram? With my 4090 24gb gpu, I am still using bf16 and no xformers for excellent training so I just wanted to check with you. Also, do you prefer questions here or in discord?

Ec Jep

2023-12-22 16:50:09 +0000 UTC

great so it was somehow image error

Furkan Gözükara

2023-12-22 10:48:10 +0000 UTC

I converted all the images to jpg again and now its working like normal!! :D

Lena Davis

2023-12-22 10:11:41 +0000 UTC

Yes I am using bucketing. The resolution is all over the place raning from about 800 to 1400 pixels. Yes I will try fixing the images, hopefully it works. Thanks!

Lena Davis

2023-12-19 16:08:20 +0000 UTC

hello. what are the resolution of images? are you using bucketing? this more looks like images are corrupted. change all images into different format such as JPEG and try again

Furkan Gözükara

2023-12-16 22:15:43 +0000 UTC

Hey I am getting this error since yesterday. Before everything worked fine. I have used the same model, same setting etc. Just different training dataset now. I have the 2x T4 selected and attempted a Lora training. Also whats weird is that the 2x T4 GPUs don't show any sign of life in the resource monitor tab. VRAM is at 0 bytes all the time and for usage its the same... loading image sizes. 2%|█ | 1/40 [00:00<00:01, 22.09it/s] Traceback (most recent call last): File "/kaggle/working/kohya_ss/./sdxl_train_network.py", line 185, in trainer.train(args) File "/kaggle/working/kohya_ss/train_network.py", line 192, in train train_dataset_group = config_util.generate_dataset_group_by_blueprint(blueprint.dataset_group) File "/kaggle/working/kohya_ss/library/config_util.py", line 495, in generate_dataset_group_by_blueprint dataset.make_buckets() File "/kaggle/working/kohya_ss/library/train_util.py", line 763, in make_buckets info.image_size = self.get_image_size(info.absolute_path) File "/kaggle/working/kohya_ss/library/train_util.py", line 996, in get_image_size image = Image.open(image_path) File "/opt/conda/lib/python3.10/site-packages/PIL/Image.py", line 3298, in open raise UnidentifiedImageError(msg) PIL.UnidentifiedImageError: cannot identify image file '/kaggle/working/Lora/img/20_teveo leggings leggings/Teveo_Leggings_10).png' Traceback (most recent call last): File "/opt/conda/bin/accelerate", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command multi_gpu_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher distrib_run.run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ ./sdxl_train_network.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2023-12-16_09:37:01 host : 66e4585c1664 rank : 1 (local_rank: 1) exitcode : 1 (pid: 1040) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-12-16_09:37:01 host : 66e4585c1664 rank : 0 (local_rank: 0) exitcode : 1 (pid: 1039) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================

Lena Davis

2023-12-16 09:40:22 +0000 UTC

this doesnt show error reason. if caching takes more than 30 min click start training again after loading config

Furkan Gözükara

2023-12-10 14:38:09 +0000 UTC

File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command multi_gpu_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher distrib_run.run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ===================================================== ./sdxl_train.py FAILED ----------------------------------------------------- Failures: ----------------------------------------------------- Root Cause (first observed failure): [0]: time : 2023-12-10_10:31:23 host : 3e6bcd590114 rank : 1 (local_rank: 1) exitcode : -6 (pid: 1051) error_file: traceback : Signal 6 (SIGABRT) received by PID 1051 ===================================================== looks like I have some issue with John Ceed

yuxiang chen

2023-12-10 11:15:23 +0000 UTC

OK, thank you! I'll try again!

Alex

2023-12-06 12:01:23 +0000 UTC

i would say start fresh and do lora. it should work. looks like for some reason it failed to use CUDA device. make sure that T4 GPUs selected

Furkan Gözükara

2023-12-06 11:16:00 +0000 UTC

yep it will work

Furkan Gözükara

2023-12-06 11:15:09 +0000 UTC

Hi! I have successfully trained Dreambooth, but I didn't like the results. Now I'm trying to train LORA, they make the settings right, but when I turn on the start of training, I get this error. epoch 1/16 Traceback (most recent call last): File "/kaggle/working/kohya_ss/./sdxl_train_network.py", line 185, in trainer.train(args) File "/kaggle/working/kohya_ss/train_network.py", line 825, in train accelerator.backward(loss) File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 1983, in backward self.scaler.scale(loss).backward(kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward torch.autograd.backward( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED Traceback (most recent call last): File "/kaggle/working/kohya_ss/./sdxl_train_network.py", line 185, in trainer.train(args) File "/kaggle/working/kohya_ss/train_network.py", line 825, in train accelerator.backward(loss) File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 1983, in backward self.scaler.scale(loss).backward(kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward torch.autograd.backward( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED steps: 0%| | 0/9600 [00:06 sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command multi_gpu_launcher(args) File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher distrib_run.run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ ./sdxl_train_network.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2023-12-06_08:36:19 host : 5fdccbe9f1b0 rank : 1 (local_rank: 1) exitcode : 1 (pid: 1136) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-12-06_08:36:19 host : 5fdccbe9f1b0 rank : 0 (local_rank: 0) exitcode : 1 (pid: 1135) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================

Alex

2023-12-06 08:48:59 +0000 UTC

Would this work with juggernaut XL model?

Jorge Reverte Sevillano

2023-12-06 06:59:12 +0000 UTC

the order like this. put token. start ngrok get link. dont click visit site. then start kohya gui. once it started click visit site

Furkan Gözükara

2023-12-05 22:41:42 +0000 UTC

hi I am having issues running the flask/ngrok cell where it says ## first put your ngrok token to the below and then run this code ## it will give a link like this at below : https://2fc5-34-134-226-xxx.ngrok-free.app ## open it and then run web ui and once web ui started that link will start working. I have put my auth token in there and it runs without error, but when I visit site from the 'ttps://2fc5-34-134-226-xxx.ngrok-free.app' I just get a message saying 'Hello from Colab!' from the video tutorial it seems I should be getting some web based UI with dreambooth on it... what am I missing?

James

2023-12-05 22:34:23 +0000 UTC

those are not errors. make a video and send me how you are failing.

Furkan Gözükara

2023-11-29 13:13:35 +0000 UTC

then i was told scip and numpy is incompatible

Juan Chen

2023-11-29 13:08:50 +0000 UTC

well you are not giving any information that can let us to understand your problem. please join discord and message from there in the channel

Furkan Gözükara

2023-11-29 11:33:25 +0000 UTC

i tried to install version 1.16.5, more errors come out, please double check

Juan Chen

2023-11-29 10:12:48 +0000 UTC

you should be able to. reduce repeating count

Furkan Gözükara

2023-11-27 21:40:01 +0000 UTC

I can't use more than 13 training photos. error

John Ceed

2023-11-27 20:25:07 +0000 UTC

help! ( error training File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ===================================================== ./sdxl_train.py FAILED ----------------------------------------------------- Failures: ----------------------------------------------------- Root Cause (first observed failure): [0]: time : 2023-11-27_19:08:29 host : c7aaca832a42 rank : 1 (local_rank: 1) exitcode : -6 (pid: 1053) error_file: traceback : Signal 6 (SIGABRT) received by PID 1053 =====================================================

John Ceed

2023-11-27 19:14:14 +0000 UTC

you are welcome

Furkan Gözükara

2023-11-26 09:22:56 +0000 UTC

Thank you very much Furkan

DAVID PEREZ

2023-11-26 02:38:11 +0000 UTC

I made a full video of checkpoint training please watch it carefully : https://youtu.be/16-b1AjvyBE

Furkan Gözükara

2023-11-26 00:07:12 +0000 UTC

possible to show me settings for checkpoint training on kaggle this weekend?

Juan Chen

2023-11-25 21:29:26 +0000 UTC

yep almost done uploading

Furkan Gözükara

2023-11-23 18:58:58 +0000 UTC

Great News a tutorial would be awesome thank you Furkan

DAVID PEREZ

2023-11-23 18:19:32 +0000 UTC

please wait tutorial i am editing it

Furkan Gözükara

2023-11-22 12:44:24 +0000 UTC

ValueError: Pipeline expected {'scheduler', 'vae', 'tokenizer_2', 'text_encoder_2', 'tokenizer', 'unet', 'text_encoder'}, but only {'scheduler', 'vae', 'tokenizer_2', 'text_encoder_2', 'tokenizer', 'unet'} were passed. Traceback (most recent call last):

Juan Chen

2023-11-22 11:52:29 +0000 UTC

we how to use tutorial here and we have kaggle notebook too : https://youtu.be/EEV8RPohsbw - https://youtu.be/_xVq23d2pgE - if you need any other help let me know. by the way you don't have to kill kaggle Kohya GUI anymore. you can run from GUI - https://www.patreon.com/posts/kohya-sdxl-lora-88397937 - hopefully i will make a new video after Kohya GUI updated into master

Furkan Gözükara

2023-11-09 17:49:28 +0000 UTC

Could you make a quick video tutorial for SDXL dreambooth training using Kaggle, or GitHub tutorial or a notebook update, please Furkan

DAVID PEREZ

2023-11-09 17:32:29 +0000 UTC

please make a video of how you are installing and trying and send me. i am waiting

Furkan Gözükara

2023-10-31 18:31:44 +0000 UTC

Ok. Thanks for the response. {edit} It seems to be working now. :)

Mister Story

2023-10-31 18:22:09 +0000 UTC

please use kohya-sdxl-lora-training-on-a-free-kaggle-notebook_v7.ipynb and watch this : https://youtu.be/_xVq23d2pgE

Furkan Gözükara

2023-10-27 23:30:10 +0000 UTC

I am having this same problem. I can't view or open the link you provided above on huggingface. I also downloaded the ngrok link and entered my ngrok token into the script. I still don't get a gradio link or ngrok link.

Mister Story

2023-10-27 19:13:12 +0000 UTC

hello it is on my todo list as next hopefully. will also make an auto installer hopefully

Furkan Gözükara

2023-10-25 21:00:38 +0000 UTC

hey, can i install TensorRT on runpod in automatic 1111

Joel Maynard

2023-10-25 20:37:24 +0000 UTC

hello. it is on my todo list. sorry for delay

Furkan Gözükara

2023-10-22 15:32:00 +0000 UTC

Thank you for the this huge Update Furkan! Please, could you make an updated github text guide or updated short video tutorial for best setting to train in SD 1.5 models please

DAVID PEREZ

2023-10-22 04:18:25 +0000 UTC

yes it is mandatory. also full fp16 is mandatory as well and xformers too

Furkan Gözükara

2023-10-20 20:29:51 +0000 UTC

seems like the issue was that somehow --gradient_checkpointing was dropped from my list of params. I'm trying again with it

Greg Rami

2023-10-20 19:58:05 +0000 UTC

Hi, as I'm running the training I got the following error: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76 GiB total capacity; 13.17 GiB already allocated; 7.75 MiB free; 13.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF I'm using more training images (16) so I have 6400 training steps. But this failed very early on

Greg Rami

2023-10-20 16:40:54 +0000 UTC

I just updated to v7. it is working. please try and let me know

Furkan Gözükara

2023-10-17 11:18:47 +0000 UTC

v6 comes out? i just came across with 03:36:11-458929 INFO nVidia toolkit detected 03:36:13-934002 INFO Torch 2.0.1+cu118 03:36:14-033892 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700 03:36:14-055546 INFO Torch detected GPU: Tesla T4 VRAM 15110 Arch (7, 5) Cores 40 03:36:14-056912 INFO Verifying modules instalation status from /kaggle/working/kohya_ss/requirements_linux.txt... 03:36:14-060349 INFO Verifying modules instalation status from requirements.txt... ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /kaggle/working/kohya_ss/kohya_gui.py:13 in │ │ │ │ 12 from library.custom_logging import setup_logging │ │ ❱ 13 from library.localization_ext import add_javascript │ │ 14 │ see kaggle notebook appears not working again it worked well yesterday

Juan Chen

2023-10-17 09:25:00 +0000 UTC

just fixes with recent updates

Furkan Gözükara

2023-10-12 21:32:28 +0000 UTC

Thank you Furkan, is working now, i hope i can test it today!

DAVID PEREZ

2023-10-12 21:30:16 +0000 UTC

hello please download v5 updated. refresh this page

Furkan Gözükara

2023-10-12 21:19:11 +0000 UTC

Hello when i try to download the v4 file gives this error in a new web browser tab: {"errors":[{"code":902,"code_name":"AttachmentNotFound","detail":"Attachment with id 16135838 was not found.","id":"abb94cdb-b94c-5571-80d5-e685c3b63769","status":"404","title":"Attachment was not found."}]} could you fix it please

DAVID PEREZ

2023-10-12 21:17:03 +0000 UTC

please show screenshot. you can write in our discord much easier : https://discord.gg/Pdjp7f5r

Furkan Gözükara

2023-10-11 20:49:25 +0000 UTC

I watched 3 of your videos about this (including the short update video for the latest update on the 9th of Oct). I'm following you step by step but the very first command gives me the same error I don't know how to fix it. The error says: N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details. Do you want to accept these changes and continue updating from this repository? [y/N] How can I say "yes" to this? It adds ^C in the end and then continues if I hit the "play" button for that cell. Please help! I already spent hours on this.

Mr. GT

2023-10-11 17:33:14 +0000 UTC

you have 2 issues 1: your command is wrong. you are using DreamBooth tab not LoRA 2: your images are corrupted. can you do this? install paint . net . it is a free open source tool. open everyone of your images. save as png with paint . net. give them shorter names and try again please.

Furkan Gözükara

2023-10-10 10:54:36 +0000 UTC

Hi, thanks for doing this tutorial! I am unfortunately unable to get this to run. 1. I am unable to get past this point without pressing the "play" button again (It does ^c which gets past the message): N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details. Do you want to accept these changes and continue updating from this repository? [y/N] 2. The Kohya GUI looks different than yours and is missing a few Parameters (v.22.0.1): a. Lora Type dropdown b. Text Encoder Rate c. Unet learning rate d. Network Rank 3. It fails when caching latents: RuntimeError: NaN detected in latents: /kaggle/working/results/img/25_ohwx woman/00100lrPORTRAIT_00100_BURST20200125175208649_COVER.jpg Here's my output: accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train.py" --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --train_data_dir="/kaggle/working/results/img" --reg_data_dir="/kaggle/working/results/reg" --resolution="1024,1024" --output_dir="/kaggle/working/results/model" --logging_dir="/kaggle/working/results/log" --save_model_as=safetensors --output_name="test_lora_1" --lr_scheduler_num_cycles="8" --max_data_loader_n_workers="0" --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="4400" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --full_fp16 --xformers --bucket_no_upscale --noise_offset=0.0 --lowram

Chad Lumley

2023-10-10 01:37:59 +0000 UTC

hello new video is here : https://youtu.be/_xVq23d2pgE

Furkan Gözükara

2023-10-08 22:41:58 +0000 UTC

hello. i am recording a video for how to use new version. give me like 2 hours please. thank you

Furkan Gözükara

2023-10-08 20:58:39 +0000 UTC

Hi, thanks for the support, i try to follow this guide: https://www.youtube.com/watch?v=JF2P7BIUpIU&ab_channel=SECourses but i don't understand when started the gui because when i try to navigate http://127.0.0.1:7860/ i received this error: dial tcp 127.0.0.1:7860: connect: connection refused Can you help me please?

Silverio Giancristofaro

2023-10-08 20:55:48 +0000 UTC

if you use share it won't work. kaggle banned it. here watch this to learn how to use ngrok : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/2023-09-23%2014-35-29.mkv

Furkan Gözükara

2023-10-04 21:23:05 +0000 UTC

New code is not working, it just makes a private url (not a public one) and when I try to open the local one it says " cant connect to the server" Even I tried changing !bash gui.sh --headless to !bash gui.sh --share --headless

JimmyHD

2023-10-04 21:17:07 +0000 UTC

very nice

Furkan Gözükara

2023-10-03 11:10:33 +0000 UTC

8, anyways I just got it to work and generate images in comfyUI, lovely!

Holeinone

2023-10-03 11:10:04 +0000 UTC

really hard to find. probably you can generate them with using sdxl

Furkan Gözükara

2023-10-03 00:12:29 +0000 UTC

a question, i see you use many regularization images. and this is optional. But if I want to train a cartoon animal, where to find regularization images for this kind? please let me know. Thanks.

Juan Chen

2023-10-02 23:48:42 +0000 UTC

how much vram it has? try loading with --medvram

Furkan Gözükara

2023-10-02 14:27:51 +0000 UTC

yeah can´t load the auto1111 sdxl model due to RTX 4060 laptop card. I will try running training again !

Holeinone

2023-10-02 13:43:52 +0000 UTC

yes you can use. you must have a config error. i used them on auto1111. you can try with auto1111 and see if error is from comfyui or from generated lora file

Furkan Gözükara

2023-10-02 12:28:28 +0000 UTC

Is it possible to use this lora with comfyui? When running this in comfyui (using the lora created via kaggle) it just resuts in a black image

Holeinone

2023-10-02 12:20:52 +0000 UTC

it is because of vram bottleneck. 10 gb is only suitable for SD 1.5 LoRA

Furkan Gözükara

2023-09-24 17:03:14 +0000 UTC

got it , thanks ! I am using a 3080 (10gb vram ) and with these settings, It will take ~3 days to complete. Not sure if this is normal or not with my card but feels excessive.

Ernesto Ponce

2023-09-24 14:49:32 +0000 UTC

i think you can use custom models. i tested and it worked. such as juggernaut xl

Furkan Gözükara

2023-09-24 12:12:31 +0000 UTC

can we use custom models already or the ram issue in koyha is still happening ?

Ernesto Ponce

2023-09-24 00:57:53 +0000 UTC

ngrok added to the notebook. we now use ngrok tunneling to reach locally runing automatic1111 since public gradio share is banned by google colab and kaggle.

Furkan Gözükara

2023-09-21 11:45:02 +0000 UTC

Sorry, new tier for you, what do you mean in this entry when you say using "ngrok"?. Is the kaggle option still working?

Salvador Robles

2023-09-21 11:17:08 +0000 UTC

sorry i mistakenly deleted your comment :/ as you said higher DIM will go out of RAM. so above 64 causes RAM error. thanks for letting us know

Furkan Gözükara

2023-09-19 12:39:24 +0000 UTC

thanks for letting us know. true as you go higher DIM it will use more RAM

Furkan Gözükara

2023-09-19 12:38:34 +0000 UTC

thanks for letting us know. true as you go higher DIM it will use more RAM

Furkan Gözükara

2023-09-19 12:38:27 +0000 UTC

"#$"#$ I managed to turn off the book before downloading all the checkpoints and did not have file persistence.. got two of them at least. Oh well. I'll restart another one later.

Mikael Svenson

2023-09-18 15:37:43 +0000 UTC

very nice speed for free. don't forget to do x/y/z checkpoint comparison

Furkan Gözükara

2023-09-18 11:29:22 +0000 UTC

Getting 4.32s/it when using dimension=64. GPU seems to be at 12.4/14GB per GPU. Let's see what the results are once complete - if my training set was good :)

Mikael Svenson

2023-09-18 08:33:25 +0000 UTC

Hmm I see where is the problem. I need to apply the updates of the 4th of September. Thanks! I will let you if it works :)

Quentin Guittard

2023-09-18 02:14:48 +0000 UTC

!accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --train_data_dir="/kaggle/working/results/img" --reg_data_dir="/kaggle/working/results/reg" --resolution="1024,1024" --output_dir="/kaggle/working/results/model" --logging_dir="/kaggle/working/results/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=32 --output_name="kaggle_test_1" --lr_scheduler_num_cycles="8" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="4000" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale --noise_offset=0.0 --lowram

Quentin Guittard

2023-09-18 02:11:38 +0000 UTC

hello. can you copy paste me your run command? did you add --full_fp16 ?

Furkan Gözükara

2023-09-17 21:56:40 +0000 UTC

Hi, I got memory error message when running my training command. "Your notebook tried to allocate more memory than is available. It has restarted." My training dataset is 10 1024*1024 images, and I followed the tutorial to free the RAM but it reaches the 13Go Ram limit. How can I continue to optimize my training, review the parameter or reduce the number of images or both? Without reducing the quality result to much.

Quentin Guittard

2023-09-17 21:51:00 +0000 UTC

how did you do it can you show as screenshot? this is tested today and working. after ngrok you need to start kohya

Furkan Gözükara

2023-09-17 00:08:57 +0000 UTC

failed to launch

John Ceed

2023-09-16 19:24:51 +0000 UTC

yes you can give lora weight as the last checkpoint. message me from discord i will show you screenshot

Furkan Gözükara

2023-09-16 13:53:40 +0000 UTC

My notebook was supposed to run for 16h, but they stop it after 12h on the free version. Is there a way to feed our existing safetensors files and continue the process from there?

Níckolas

2023-09-16 11:30:10 +0000 UTC

hello. i made it work. now working. redownload

Furkan Gözükara

2023-09-15 21:26:26 +0000 UTC

Hello. I made a solution and updated notebook. Now working. redownload new one

Furkan Gözükara

2023-09-15 21:26:14 +0000 UTC

added to the very top. maybe i can make a short tutorial for how to prepare dataset and command in your pc and run on kaggle.

Furkan Gözükara

2023-09-15 12:23:01 +0000 UTC

you are welcome. sorry about this :/

Furkan Gözükara

2023-09-15 12:21:11 +0000 UTC

First of all, thanks for the hard work you put into this. It would be nice if you could make an update on the post about this. I was trying to make it work for several hours yesterday.

Níckolas

2023-09-15 12:15:55 +0000 UTC

ok,thanks;)

迪吴

2023-09-15 09:38:31 +0000 UTC

ye Google don't allows web uis anymore. now you need to prepare dataset manually and execute training command. you can also prepare them in your computer and upload and use there with correct path > https://twitter.com/GozukaraFurkan/status/1702476057880756733

Furkan Gözükara

2023-09-15 09:03:10 +0000 UTC

hi~thanks for you notebook~ I run it success yesterday, but when i run webui command in kaggle today,it killing the server,what i do it yesterday is ok,does anyone know what happened? "Keyboard interruption in main thread... closing server."

迪吴

2023-09-15 04:28:11 +0000 UTC

Because lora is optimized version of this :) this is full training while Lora is only partial

Furkan Gözükara

2023-09-13 16:53:24 +0000 UTC

Why do you think it is much better than Lora? You mean specifically for non human characters?I’ve got so much to learn. :)

Paula Kühn

2023-09-13 16:49:09 +0000 UTC

yes you can. moreover i have completed my SDXL DreamBooth workflow. About to publish on Patreon. check it out once published. much better than LoRA

Furkan Gözükara

2023-09-13 13:37:13 +0000 UTC

hey there! first of all: thanks for your great work! I have a question: I want to train a little clay figurine, and i am following your lora kaggle tutorial, i trained it succesfully with the previous sd version, without regularization images, can i skip it here as well?

Paula Kühn

2023-09-13 11:36:39 +0000 UTC

I made it working. shared also on Discord. easier there to see. https://i.ibb.co/yQPmRwP/image.png

Furkan Gözükara

2023-09-07 10:41:30 +0000 UTC

I made it working. shared also on Discord. easier there to see. https://i.ibb.co/yQPmRwP/image.png

Furkan Gözükara

2023-09-07 10:41:26 +0000 UTC

It is working. Here the change you need to make. I also shared on Discord easier to see there : https://i.ibb.co/yQPmRwP/image.png

Furkan Gözükara

2023-09-07 10:40:52 +0000 UTC

I am going to test now.

Furkan Gözükara

2023-09-07 09:58:23 +0000 UTC

try this repo name. hugging face repo of realistic vision XL : SG161222/RealVisXL_V1.0 - i will test now

Furkan Gözükara

2023-09-07 09:58:10 +0000 UTC

try this repo name. hugging face repo of realistic vision XL : SG161222/RealVisXL_V1.0 - i will test now

Furkan Gözükara

2023-09-07 09:57:53 +0000 UTC

How to train Lora for other large models? I don't want to use sdxl0.9vae, for example, I want to use this to train Lora, https://civitai.com/models/139562?modelVersionId=154590 What should we do?

wzgrx

2023-09-07 09:51:32 +0000 UTC

import network module: networks.lora create LoRA network. base dim (rank): 32, alpha: 1.0 neuron dropout: p=None, rank dropout: p=None, module dropout: p=None create LoRA for Text Encoder: 264 modules. create LoRA for U-Net: 722 modules. enable LoRA for text encoder enable LoRA for U-Net use Adafactor optimizer | {'relative_step': True} relative_step is true / relative_stepがtrueです learning rate is used as initial_lr / 指定したlearning rateはinitial_lrとして使用されます unet_lr and text_encoder_lr are ignored / unet_lrとtext_encoder_lrは無視されます use adafactor_scheduler / スケジューラにadafactor_schedulerを使用します override steps. steps for 10 epochs is / 指定エポックまでのステップ数: 1840 running training / 学習開始 num train images * repeats / 学習画像の数×繰り返し回数: 275 num reg images / 正則化画像の数: 4318 num batches per epoch / 1epochのバッチ数: 184 num epochs / epoch数: 10 batch size per device / バッチサイズ: 3 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 1840 steps: 1%|█ | 1/1840 2.97s/it, loss=nan

ko samuel

2023-09-07 04:02:25 +0000 UTC

It seems to loss nan when I train my lora on the special edition SDXL model. How can I solve this problem. The SDXL basemodel I choose is https://civitai.com/models/139562/realvisxl-v10

ko samuel

2023-09-07 04:02:21 +0000 UTC

Miguel you are missing --full_fp16

Furkan Gözükara

2023-09-05 21:11:33 +0000 UTC

accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --train_data_dir="/kaggle/working/results/img" --reg_data_dir="/kaggle/working/results/reg" --resolution="1024,1024" --output_dir="/kaggle/working/results/model" --logging_dir="/kaggle/working/results/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=32 --output_name="kaggle_mjero_1" --lr_scheduler_num_cycles="8" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="4800" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale --noise_offset=0.0 --lowram

Miguel Jeronimo

2023-09-05 20:59:44 +0000 UTC

please show your full command here. a lot of people already did successful training. something must be different . you can also message in discord much better

Furkan Gözükara

2023-09-05 19:32:51 +0000 UTC

I tried add the new parameters, but still getting the error: Your notebook tried to allocate more memory than is available. It has restarted.

Miguel Jeronimo

2023-09-05 19:31:05 +0000 UTC

You need to reinstall. It is fast. I will also hopefully update notebook for even more performance if I can make. Automatic1111 notebook also will get sdxl controlnet support. Working on it

Furkan Gözükara

2023-09-05 16:00:50 +0000 UTC

If I install everything specified in the tutorial, will Kaggle store it for next time or do I have to install everything again?

San Milano

2023-09-05 15:50:48 +0000 UTC

I just tested and if i turn off browser kaggle stops

Furkan Gözükara

2023-09-05 13:16:00 +0000 UTC

I am starting a test right now for you

Furkan Gözükara

2023-09-05 09:36:20 +0000 UTC

I have a question regarding the usage of kaggle notebooks. Is it possible to run the notebook on kaggle, turn off the own computer and then come back to kaggle after the estimated training time for downloading the created loras? When I tried this, all data was deleted and the training has stopped.

Gerenier

2023-09-05 08:39:02 +0000 UTC

thank you updated and fixed

Furkan Gözükara

2023-09-01 15:26:50 +0000 UTC

please make changes to the code: !wget https://www.pokemonpets.com/woman_3786_imgs_1024x1024px.zip,man code is repetitive

Rasika Singal

2023-09-01 03:33:09 +0000 UTC

A Kaggle Notebook to Train Stable Diffusion 1.5 and XL (SDXL) on a Free Kaggle Account with 2x Dual T4 GPU for free by using Kohya GUI

27 May 2025 Update

13 March 2025 Update

22 February 2024 Update

30 January 2024 Update

Tutorial link for this notebook file : https://www.youtube.com/watch?v=16-b1AjvyBE

Comments

Hi multiple gpu explained in this tutorial : https://youtu.be/-uhL2nW7Ddw?si=QbHh9xafLXnJ1twt By the way it is very at the limit of gpu mem capacity so you may get OOM

Hi, In the tutorial, I could not find any information on how to set up multiple GPUs, and when I select the multi-gpu with 2 process option on Gradio UI, I get an error. Do you have any videos in which you describe how to use the two Kaggle GPUs?

replied from discord

Hi Furkan... how to use a different base model other than SDXL base? do I have to editi the json file and use that instead?

you need to use our config. lots of features doesnt exists on the kaggle since it is free. in your case memory_efficient_attention_forward you can already use 2x gpu i tested with sdxl works great

i dont have any mac sadly

please also try mlflux, it runs on M1 chip

Fp8 is better than Q8? I'm currently testing Lora models I trained on Replicate with the dev model on Kaggle with your SwarmUi notebook and it seems to be good so far

yes i think wont work good on Q8 GGUF. would work on FP8 though. Still need to be tested

Also one last question, if I train my flux loras using your tutorials on the original dev model (23Gb) and then use these Loras on Q8 GGUF version will it have less quality?

sadly not. FLUX needs BF16 training, FP16 it didnt learn. But I didn't test fine tuning can be tested however still your session time probably wouldnt be sufficient

Can this be done with Flux models aswell?

i tested and notebook works. just use the models in the configs . or properly download your custom model into notebook and give its path : https://youtu.be/X5WVZ0NMaTg

if you update the notebook, please notice me. (ಥ﹏ಥ)

runwayml deleted their repos :/ i need to update the notebook give me some time

great

i just made a fresh install no issues. perhaps it was temporary or you are out of your monthly ngrok bandwith. try with a fresh token

I cannot connect to ngrok. Getting ERR_NGROK_8012. Ngrok works with your other notebooks but not with this one.

latest kohya config for SD 1.5 is here : https://www.patreon.com/posts/very-best-kohya-97379147

want try to run a 1.5 dreambooth trainign via Kohya, what are the latest Json settign files that should i be using...

I suggest you to use BF16 for training because our parameters are researched for BF16. FP16 or FP32 will work worse.

with a 4090 NVIDIA and 64 gb RAM, do you recomend to use bf16 with experimental too in your configuration?

for 200 images i would do this. train up to 50 epoch and save every 5 epochs and compare later. if be still undertrained you can train 50 more epochs.

So what about ~200 images in set and without regularization images? Which formula to use in this scenario with this config?

your image is broken. open a thread here and share your error and image : https://github.com/kohya-ss/sd-scripts/issues

bmaltais fixing errors. he did lots of improvements in but in dev branch waiting him to commit . and i told you, your image is broken 1 (1)_resized.png

he replied "bmaltais commented 9 hours ago Not sure what cause this error… it is caused by the training script and I don’t think I can do anything about it. You might want to open an issue directly on the as-scripts repo."

Yok OneTrainer ve Kohya kalitesi neredeyse aynı. önemli olan doğru ayarlarda eğitmek

One trainer düşük kalitede mi üretiyor ? Lora için en iyi seçenek kohya_ss midir ?

LoRA için yok çünkü düşük kalite. ama Fine tuning bugün video yayınladım 2 saat :) https://www.youtube.com/watch?v=0t5l6CP9eBg&lc=UgxoXVvh2CicH0J2PwJ4AaABAg

One trainer için lora eğitim örneğiniz var mıdır ?

that means you are clicking inaccurate button and not loading it accurately. gui changed but still working. if you show me how you are doing in discord i can tell your error

what error are you having?

yes seeming unable to train lora too,

please use the dataset preparation tab of the gradio. hopefully i will make a new updated video soon

for sd 1.5 200 repeats may overtrain but if working fine great. because it would mean 400 repeats on 1 gpu

hi that would mean regularization images folder is missing. have you set it? hopefully i will make updated tutorial. i am waiting kohya to finish newest interface. by the way, on kaggle, you should make half number of steps. because it uses 2 gpu.

fixed

simplified the thread. please watch this video : https://youtu.be/16-b1AjvyBE

it is working for me. which one is broken?

Thanks. The Kaggle dreambooth link is broken though.

hello. hopefully tomorrow will fix sorry for delay. kohya did a lot of updates

no when using reg images always use same dimension as training images. so all are 768x768 for sd 1.5

thaank you! so: 1024x1024 training images, 768x768 reg images, steps calculation + 1...

i prefer 768x768 . steps calculation and everything is totally same as sdxl. just use the sd 1.5 config

Hello, Furkan... I want to try another 1.5 training, but it's not clear what training images do I have to use: it's better to train 1024x1024 or 512x512 images? The saving steps calculation is the sama of the sdxl workflow?

you are welcome

Great I try again ;) Very thank you for your help ;)

it is because of gradio update. developer has to be fix but i just made a hack fix. you can download v15

i use sdxl 1.0 base for sdxl training

Thank you Furkan! Can you say to me also what SDXL model base is used by Kohya in the Kaggle notebook of this thread?

the kohya uses hyper realism v3 by default. that is the great sd 1.5 model

did you train yourself on SD 1.5 pruned?

you need to use the model you trained yourself on as base model. whichever the model you used for training that model. by the way lets say you trained yourself on model A, then extract lora and used that lora on model B may not work very well

no you have to manually extract LoRA. use Kohya GUI SS with these settings : https://cdn-uploads.huggingface.co/production/uploads/6345bd89fe134dfd7a0dba40/NtToBFK2uumY_YMHOZ7MW.png

Hi, Doctor, running the lora cell after the 6 safetensors generation, it will extract the loras from checkpoints? can you explain this step? And do you have right parameters to do the same with SD1.5 generated models?

Well what is expected is, it gets overfit so more resemblance but lesser generalization. that is weird

thank you...

Please watch these 2 videos : https://youtu.be/dpM02YMj8FY https://youtu.be/16-b1AjvyBE

you must have given incorrect folder path. what folder path you given as output?

tried, but not able to get an output folder for lora... uploaded checkpoint (mine and sdxl 19 but setting an output folder fom the web ui retturns a non writeable folder on kaggle notebook... maybe it's an easy fix to do, but I'm really a beginner... any help?

yep you can do. for SDXL it may fail due to RAM limitation though. give it a try

can I do this with the Kaggle/Khoya notebook I used to generate my checkpoint?

hello here : https://cdn-uploads.huggingface.co/production/uploads/6345bd89fe134dfd7a0dba40/TT9D-nOtJqop9nozTGZOA.png

thank you... but it seems I'm a little too dumb to find that resource... can you point me to that one, please?

You can extract as we have shown in discord channel #tips info check out there screenshot

hi... yesterday I got my favourite model finally... it's a checkpoint, and I sould like to extract a lora finetuned version from that one: can you suggest how to achieve that ising a kaggle notebook?

great

I launched the checkpoints one by one, use them individually and manually compared the results. Problem solved! Again, Thank you for your help!

Well it shouldn't crush interesting.

Hi Dr. Gozukara, Thank you for your instruction! It was my bad. What a silly mistake! Now it worked. For Automatic1111 with kaggle, I found checkpoint comparison via x/y plot computational extremely expensive. The system crushed several times. I will try it again. Do you have any thoughts on that?

your links are incorrect. you see it downloaded only 36kb. here an accurate link : https://huggingface.co/Terresa/SDXL_training/resolve/main/My_DB_Kaggle-step00002084.safetensors

I just tested on my computer and it works. tested step00002084. can you record a video and how you are trying to use it on kaggle?