Massive 4K Resolution Woman & Man Class Ground Truth Stable Diffusion Regularization Images Dataset
Added 2023-08-15 00:37:09 +0000 UTCJoin discord and tell me your discord username to get a special rank : SECourses Discord
27 November 2023 Huge Update
- The dataset is improved and expanded to 5200 images for both Woman and Man dataset
- The cropping and resize scripts are further improved and all images are processed again
- Moreover all images are now sorted according to the face quality of the images
- So the training scripts will use the very best ones
- Naming is made starting from man_10001 or woman_10001 so when training script starts using reg images, the very best ones will be used
- Please re-download all of the new images for best quality
- Total time took is over 10 full days to prepare all reg images
- Both woman and man new datasets are added to the resources below
20 September 2023 Massive Update
- All of the images are reprocessed with a newer face detection algorithm RetinaFace
- RetinaFace is much better to detect and focus faces but it is really really slow
- Newest processing scripts are shared here (YOLO V7 cropper and RetinaFace resizer) : https://www.patreon.com/posts/sota-subject-and-88391247
- So with this update the datasets are much better. Please redownload them before using
- Processing all of the datasets took like 6 days with 13900K CPU and 3090 TI
How To Download All On A RunPod Or A Unix System
- download_man_reg_imgs.sh file will download and automatically extract 512x512, 768x768 and 1024x1024 man images. You can edit the file and add other resolutions if you need.
- download_woman_reg_imgs.sh file will download and automatically extract 512x512, 768x768 and 1024x1024 woman images. You can edit the file and add other resolutions if you need.
- These files can be used for Unix and possibly for MacOs systems as well. Don't forget to comment (put # beginning of a link) the links that you don't want to download and change folder paths if you wish.
- Upload into workspace folder of RunPod and execute below command
- cd /workspace
- chmod +x download_man_reg_imgs.sh
- ./download_woman_reg_imgs.sh
- cd /workspace
- chmod +x download_woman_reg_imgs.sh
- ./download_woman_reg_imgs.sh
How Datasets Are Prepared
I have gathered 40k images for woman and man class from unsplash . com. So total gathered images count is above 80k.
They are all real images. 0 AI image are used.
Then I post processed them with several AI models to clean the dataset. At the end, finally I checked each one of the images manually. Whole process took about 70 (for woman) + 70 (for man) hours.
The final output is 5200 perfect images for woman and 5200 for man. Minimum resolution of images are above 1536 x 1536 pixels and max resolution is up to 14999 x 9999 pixels.
The raw images and exact resolution having images are shared below. If you also need any other specific resolution let me know and hopefully I will update this post.
To use them on Windows you only need to extract zip images. If you can't make it install Winrar from https://www.rarlab.com/
Man Dataset
- Raw dataset is between 4 megapixels to 100 megapixels
- man_5200_imgs_raw (org resolutions - no cropped).zip - 15.3 GB
- man_5200_imgs_512x512.zip - 924 MB - 512x512 pixels
- man_5200_imgs_768x768.zip - 1.97 GB - 768x768 pixels
- man_5200_imgs_1024x1024.zip - 3.4 GB - 1024x1024 pixels
- man_5200_imgs_768x512.zip - 1.3 GB - 768x512 pixels
- man_5200_imgs_512x768.zip - 1.37 GB - 512x768 pixels
- man_5200_imgs_1024x768.zip - 2.52 GB - 1024x768 pixels
- man_5200_imgs_768x1024.zip - 2.63 GB - 768x1024 pixels
- man_5200_imgs_1536x640.zip - 2.99 GB - 1536x640 pixels
- man_5200_imgs_1216x832.zip - 3.19 GB - 1216x832 pixels
- man_5200_imgs_1344x768.zip - 3.2 GB - 1344x768 pixels
- man_5200_imgs_1368x768.zip - 3.26 GB - 1368x768 pixels
- man_5200_imgs_1152x896.zip - 3.28 GB - 1152x896 pixels
- man_5200_imgs_640x1536 - 3.32 GB - 640x1536 pixels
- man_5200_imgs_832x1216 - 3.35 GB - 832x1216 pixels
- man_5200_imgs_896x1152 - 3.4 GB - 896x1152 pixels
- man_5200_imgs_768x1344 - 3.44 GB - 768x1344 pixels
- man_5200_imgs_768x1368 - 3.5 GB - 768x1368 pixels
- man_5200_imgs_1280x1024 - 4.14 GB - 1280x1024 pixels
- man_5200_imgs_1024x1280 - 4.26 GB - 1024x1280 pixels
- man_5200_imgs_1536x1024 - 4.86 GB - 1536x1024 pixels
- man_5200_imgs_1024x1536 - 5.11 GB - 1024x1536 pixels
- man_5200_imgs_1536x1280 - 6.12 GB - 1536x1280 pixels
- man_5200_imgs_1280x1536 - 6.26 GB - 1280x1536 pixels
- man_5200_imgs_1536x1536 - 7.38 GB - 1536x1536 pixels
Woman Dataset
- woman_5200_imgs_raw_dataset (org resolutions not cropped).zip - 14 GB
- woman_5200_imgs_512x512.zip - 955 MB - 512x512 pixels
- woman_5200_imgs_768x768.zip - 2.03 GB - 768x768 pixels
- woman_5200_imgs_1024x1024.zip - 3.49 GB - 1024x1024 pixels
- woman_5200_imgs_1536x1536.zip - 7.49 GB - 1536x1536 pixels
- woman_5200_imgs_768x512.zip - 1.34 GB
- woman_5200_imgs_512x768.zip - 1.42 GB
- woman_5200_imgs_1024x768.zip - 2.6 GB
- woman_5200_imgs_768x1024.zip - 2.71 GB
- woman_5200_imgs_1536x640.zip - 3.07 GB
- woman_5200_imgs_640x1536.zip - 3.37 GB
- woman_5200_imgs_1216x832.zip - 2.58 GB
- woman_5200_imgs_832x1216.zip - 3.44 GB
- woman_5200_imgs_1344x768.zip - 3.29 GB
- woman_5200_imgs_768x1344.zip - 3.52 GB
- woman_5200_imgs_1368x768.zip - 3.34 GB
- woman_5200_imgs_768x1368.zip - 3.58 GB
- woman_5200_imgs_1152x896.zip - 3.37 GB
- woman_5200_imgs_896x1152.zip - 3.49 GB
- woman_5200_imgs_1024x1280.zip - 4.36 GB
- woman_5200_imgs_1024x1536.zip - 5.21 GB
- woman_5200_imgs_1280x1024.zip - 4.24 GB
- woman_5200_imgs_1280x1536.zip - 6.38 GB
- woman_5200_imgs_1536x1024.zip - 4.97 GB
- woman_5200_imgs_1536x1280.zip - 6.24 GB
How To Use On RunPod Or Other Cloud or Linux
To use these files unrunpod
First you need to install 7zip
- yes | apt-get install p7zip-full
Then download them with wget. Copy their link with right click and copy link then as below
wget
e.g. man:
- wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/man_5200_imgs_512x512.zip
- or another one
- wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/man_5200_imgs_1024x1024.zip
Then use below command to extract them
- 7z x man_5200_imgs_512x512.zip
- or another one
- 7z x man_5200_imgs_1024x1024.zip
e.g. woman :
- wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/woman_5200_imgs_512x512.zip
- or another one
- wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/woman_5200_imgs_1024x1024.zip
Then use below command to extract them
- 7z x woman_5200_imgs_512x512.zip
- or another one
- 7z x woman_5200_imgs_1024x1024.zip
Comments
(if i do everything like in the tutorial video)
A.S.
2024-09-03 17:15:19 +0000 UTCWhen training on Runpod should i make a new folder called "reg images" or whatever and upload all the reg images into it , then put that folder s path into the webUI then how much repeats should i give it? for the regularization images tab in the webui basic is 1 repeats should i leave that and proceed?
A.S.
2024-09-03 16:38:14 +0000 UTCye paint net should open perfectly. other files working for you? like 1024x1024?
Furkan Gözükara
2024-08-23 18:54:32 +0000 UTCpaint.net for example
eyal giron
2024-08-23 18:48:48 +0000 UTCwell works at me something must be wrong at your operating system. what do you use to open them? which software?
Furkan Gözükara
2024-08-23 17:57:38 +0000 UTCyea
eyal giron
2024-08-23 17:28:04 +0000 UTCi just downloaded and extracted and no issues. did you download fully and extracted? they are over 10 gb each one
Furkan Gözükara
2024-08-23 16:54:28 +0000 UTCboth
eyal giron
2024-08-23 16:16:00 +0000 UTCwoman or man dataset?
Furkan Gözükara
2024-08-23 15:33:01 +0000 UTCI get weird noise in all the raw original-resolution images https://imgur.com/a/G7kxfnO
eyal giron
2024-08-23 15:25:57 +0000 UTCyep i feel you :)
Furkan Gözükara
2024-08-08 03:04:38 +0000 UTCThank you sooo much for including women in the regularization images! It was so challenging to generate these before because of the nsfw results that come up. This saves me SO much time!!
Kasey Lx
2024-08-07 20:03:51 +0000 UTCsadly they are not trainable. they will fail. you need full model. lighting and turbo models are different
Furkan Gözükara
2024-08-04 17:16:12 +0000 UTCYes, It work quite well after all. I have another question that could I use a sdxl turbo or sdxl lightning as a base model while following your tutorial video?
The Strategist
2024-08-04 16:28:07 +0000 UTChi you can use. we have very mixed dataset. but an Asian collected dataset probably would even help more.
Furkan Gözükara
2024-08-04 14:20:49 +0000 UTCHi. Could I use it to train Asian face?
The Strategist
2024-08-04 04:01:31 +0000 UTCyou are welcome. yes this literally took several weeks. processing times are also huge :D
Furkan Gözükara
2024-06-11 00:53:13 +0000 UTCThank you very much for taking the time to curate such an enormous data set for us! :D
reaper557
2024-06-11 00:21:54 +0000 UTCthanks
Furkan Gözükara
2024-05-15 21:48:24 +0000 UTCSorry for the late response. I actually went ahead of the training and deleted the one corrupted image. I hope it didn't mess up a few things. I'll re-download the new version and let you know how it is next time I'll train a subject
GeeMan64
2024-05-15 19:15:46 +0000 UTCI think i deleted the corrupt one and added 5 more images. Reuploaded. Please test again and let me know
Furkan Gözükara
2024-05-15 14:25:04 +0000 UTCdo you know which image file is broken? i keep testing all images with python libraries and it says all valid.
Furkan Gözükara
2024-05-15 13:02:35 +0000 UTCthank you so much. ye i made them JPEG since original images were JPEG. but quality of JPEG is 100%. I am going to eliminate corrupted images and reupload with exact accurate number of images for 768x768 now
Furkan Gözükara
2024-05-15 12:44:20 +0000 UTCHi, within the 768x768 women regularization images there is one corrupted image that derailed the initial caching phase before the training. There is also the fact that the images collected aren't exactly 5200, but rather 5196. Don't know how much difference that would make it, perhaps a slight miscalculation dividing the reg images by my training images. Lastly I was going to point out how the images are in JPEG format instead of PNG but a previous person just above me mentioned this one. Any chance of a possible fix to these things? You do great work.
GeeMan64
2024-05-15 07:58:30 +0000 UTCHello you are right. Thanks for info I will fix hopefully
Furkan Gözükara
2024-02-29 17:17:15 +0000 UTCHellow I think there is a mistake in: How To Download All On A RunPod Or A Unix System. It should be: cd /workspace chmod +x download_man_reg_imgs.sh ./download_man_reg_imgs.sh
Franco Acosta Diaz
2024-02-29 17:11:04 +0000 UTCMerhaba yorumunuzu yeni fark ettim. Nasıl bir çoğaltma? Epey sayıda kaliteli resim var elimizde.
Furkan Gözükara
2024-02-15 22:34:30 +0000 UTCyes i can reprocess all images and replace with PNG. but PNG will take more space. but surely possible
Furkan Gözükara
2024-02-15 22:33:18 +0000 UTCCan you place the images in 768x768 in png format, since jpeg is a format that introduces compression artifacts in the images?
daniel mendoza
2024-02-15 21:11:46 +0000 UTCyes you can take half half that makes sense in that case
Furkan Gözükara
2024-01-31 23:52:52 +0000 UTCIf I prefer to use the class of "Person" should I just make a folder w/ 1024x1024 of Man + Women, or maybe half/half? Thoughts?
John Dopamine
2024-01-31 21:52:22 +0000 UTCawesome
Furkan Gözükara
2024-01-31 13:08:53 +0000 UTCI must admint I'm not a youtube tutorial watcher - but your video is great! And the answer was there :)
kaosnews
2024-01-31 09:05:51 +0000 UTCplease use dataset preparation feature to not make any mistakes. have you watched my tutorials? https://youtu.be/sBFGitIvD2A?si=x_6nDqIgHjvGUol5
Furkan Gözükara
2024-01-30 16:38:23 +0000 UTCI can't find the answer - but I manually extracted the images in a folder. Do I need to rename the folder (for Koyha) like this? woman_5200_imgs_768x768/1_images/..... or like woman_5200_imgs_768x768/1_woman/.....
kaosnews
2024-01-30 16:22:58 +0000 UTCye that makes sense. i will look for this
Furkan Gözükara
2024-01-29 01:18:04 +0000 UTCRight.. That is good idea for creating with realistic model, with ADetailer and high res fix... I can help with generating, if you just find optimal settings, model etc.
Nenad Kuzmanovic
2024-01-29 01:14:48 +0000 UTCThe problem is collecting boy or girl images. It would take so much time to collect all even if possible. Maybe I can use a realistic model to generate artificial high quality ones
Furkan Gözükara
2024-01-29 00:12:09 +0000 UTCCan you add reg images of boy and girl? I know u have daughter and you said one day you will do it :-) For start you don't have to make them in all resolutions, 1024x1024 will do the work...
Nenad Kuzmanovic
2024-01-28 23:47:48 +0000 UTCFurkan bey merhaba, öncelikle fotoğraf arşivi için teşekkür ederim. Arkadaşın sorduğu soruyla alakalı olarak mevcut fotoğraflar üzerinde veri çoğalta teknikleri uygulansa bu durum eğitimlerde işe yarar mı? Avrupa Bilim ve Teknoloji Dergisinde karlanma, gölgelendirme, yüksek gama, kontrast ayarlarını yükseltme, arttırılmış parlaklık, azaltılmış parlaklık, yansıtma gibi yöntemleri uygulanarak veri çoğaltma işlemi yapıldığını okumuştum. YOLOv4 algoritması kullanılarak yapılan eğitimlerde nesne tespiti testlerinde bu yöntemlerle çoğaltılan verilerin sonucu olumlu etkilediği gözlemlenmiş.
önal baş
2024-01-13 22:59:42 +0000 UTCPlease read repeat logic here : https://github.com/kohya-ss/sd-scripts/issues/640
Furkan Gözükara
2024-01-10 00:36:09 +0000 UTCQ1: When you say make them repeat 52 what do you mean? Do you mean create copies of the 5200 dataset images? So lets say I just copy them all and have 10400 dataset images? Q2: And then by making the epoch like 23 do you mean change the Save Model frequency (Epochs) from 10 to 23?
Alan Johnson
2024-01-09 22:07:52 +0000 UTCyes you can use 1024x1024. you dont need multiple ones just use the one same as your training images. i find it is best for sdxl.
Furkan Gözükara
2024-01-08 02:05:43 +0000 UTCVery hard work you did! Very good! Regarding the resolutions, I can pick one pack, like 1024x1024, and use it in my training, right? I don't need to use 2 or 3 packs with different resolutions in my training, right? Is there a resolution you think works best to have better results? PS: I'm still learning, I'm not an expert in fine-tuning or training SD models. :)
AI Squad
2024-01-08 00:47:56 +0000 UTChello. you can do this. lets say you have 100 images. then make repeat 52 and make epoch like 2 3
Furkan Gözükara
2024-01-07 23:31:13 +0000 UTCAny chance we could get more pictures, I've been getting FANTASTIC results with 40 high resolution pictures (I dabble in photography) and I'd like to bump it up to 40+ but I'm hitting a limit on the dataset images, i.e need more.
Alan Johnson
2024-01-07 23:26:28 +0000 UTCgood observation. yes we are focusing the human in this cropping. this is because the stable diffusion when training reduces the image dimension to 1/8. so we are already training with low resolution. therefore we want to capture as much as details of the subject. so focused subject better
Furkan Gözükara
2023-12-27 22:38:51 +0000 UTCHello, and thank you for this wonderful resource. If I may offer an observation and pose a question. The images skew heavily towards close-up and medium shots of faces and faces and torso for square crops. And all images tend to be perfectly centered on the subject. If I were to guess, this is a bias from the cropping automation that likely identifies the subject and crops accordingly. Now a question (technically two questions): Is there a technical reason why such perfect subject cropping is beneficial to training? Would you recommend albumentations or similar techniques for either regularization or training to avoid perfect cropping/framing for 'better' results? Thank you again for all your wonderful resources and knowledge.
Andre Molnar
2023-12-27 20:58:32 +0000 UTCadd all. the script will use as many as necessary. formula is repeating * number_of_training_images
Furkan Gözükara
2023-12-13 09:42:24 +0000 UTChow many reg images do i need to add?
Javi dltr
2023-12-13 01:56:29 +0000 UTCwhat model are you trying? also looks like settings error
Furkan Gözükara
2023-11-11 11:52:04 +0000 UTCI've reinstalled and fixed the error, but now the following appears: File "C:\Users\Ester\Desktop\SD DREAMBOOTH\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 717, in forward causal_attention_mask = self._build_causal_attention_mask( File "C:\Users\Ester\Desktop\SD DREAMBOOTH\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 760, in _build_causal_attention_mask mask.triu_(1) # zero out the lower diagonal RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'
Pc 0005
2023-11-11 07:39:03 +0000 UTCI am getting this error: huggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'C:\Users\Ester\Desktop\stable-diffusion-webui\models\dreambooth\Monica5\working\tokenizer'. 0%| | 0/3823 [00:00
Pc 0005
2023-11-11 06:26:59 +0000 UTCawesome ty
Furkan Gözükara
2023-11-06 22:37:27 +0000 UTCI used your resizer and everything's working as it should. Thanks a lot for your work!
Naiad's Nest
2023-11-06 21:49:20 +0000 UTCI can add 1368x768 . 1824x1024 would be probably too big. give me some time
Furkan Gözükara
2023-10-26 20:44:59 +0000 UTCI'm planning to generate imgs at 16:9 ar, so I guess I should set the training at 1824x1024 or 1368x768. if you have a man dataset available in any of these resolutions, that would be awesome. thanks a bunch in advance, appreciate your help!
Naiad's Nest
2023-10-26 19:31:38 +0000 UTCI would use closest one. Also I can upload more specific resolutions for you. What you need? Alternatively you can use raw images dataset and let script to auto resize
Furkan Gözükara
2023-10-25 23:44:47 +0000 UTCamazing dataset! what about when I'm using several different image resolutions in one training and using bucketing? which set of regularization imgs should I use then? thank you in advance for your response.
Naiad's Nest
2023-10-25 21:52:45 +0000 UTCaccurate it also works great
Furkan Gözükara
2023-10-15 16:32:23 +0000 UTCi think you can recommend birme.net/ to resize images faster brother! thanks!
JBO Solutions
2023-10-15 05:42:59 +0000 UTCthank you so much. this took days to prepare. another amazing resource coming soon hopefully
Furkan Gözükara
2023-09-30 11:30:16 +0000 UTCWow! You did really good job here! The more I dig into Stable Diffusion, the more I find your resources useful. Thanks :)
Tomas Docka
2023-09-30 11:23:31 +0000 UTCI would do 2 trainings after fixing the training images. do training with captions and without caption with only rare tokens
Furkan Gözükara
2023-09-08 22:31:20 +0000 UTCyes you also need to resize your training images. i suggest you to crop them perfectly and make them all 1024x1024
Furkan Gözükara
2023-09-08 22:30:59 +0000 UTCI also included CLIP captioning text files, like 'xxx a woman wearing a furry hat' for example, where 'xxx' is my girlfriend's name.
Paul Fidika
2023-09-08 22:22:33 +0000 UTCI used the settings from the recommend config.json file you included; I only reduced the number of epochs from 8 to 3 so it took less time. I have 20 images taken on an iphone; not cropped or resized in any way. I was referencing the 1024 x 1024 female regularization folder. Should all training images be cropped / resized to be of uniform size (like 1024 x 1024) and then match the regularization set?
Paul Fidika
2023-09-08 22:21:50 +0000 UTCi think it is about your training images dataset. also which settings did you use for the training? you can message me more details from discord
Furkan Gözükara
2023-09-08 20:40:21 +0000 UTCUnfortunately after training I'm ending up with very distorted and blurred faces (and some of the 'rainbowy stretch' distortion that seems to be characteristic of a problematic latent space). Any idea what I might be doing wrong? I didn't crop or resize my training images. Is that critical?
Paul Fidika
2023-09-08 20:38:28 +0000 UTCfixed the link. thank you so much for bringing into my attention
Furkan Gözükara
2023-09-04 08:40:15 +0000 UTCthis link "man_4330_imgs_raw (org resolutions - no cropped).zip - 12.8 GB" is the same download as "woman_3847_imgs_raw (org resolutions - no cropped).zip - 9.6 GB"
omv421
2023-09-04 02:58:57 +0000 UTCyes we use these images to improve flexibility of a person training. so sorry for late reply patreon didn't give me notification of your question
Furkan Gözükara
2023-09-01 09:27:57 +0000 UTCno each one has different links. they are all shared in the post. you should download the specific size you are going to do training or you can download and resize yourself.
Furkan Gözükara
2023-09-01 09:26:56 +0000 UTCthe woman and man ones, are the same links for the max res ones
Meito
2023-09-01 06:45:34 +0000 UTCif you can message me from discord i may help further. because i am not sure what you are trying to do :) it is really hard to communicate on Patreon
Furkan Gözükara
2023-08-28 22:30:36 +0000 UTCAmazing! I'm having good luck with the subject cropper, some great results. I was using some images taken with a cellphone and it looks like this script ignores exif data. Added to resize2.py ```python from PIL import ImageOps # ... # Resize image with best resampling filter (LANCZOS) image = image.resize(resolution, Image.LANCZOS) # transpose the image to the correct exif orientation image = ImageOps.exif_transpose(image) ``` If that helps anyone. Also bash for any of us linux nuts install.sh ```sh #!/bin/bash # If no permissions run chmod +x install.sh set -e # create and activate venv python3 -m venv venv source ./venv/bin/activate # install requirements pip install -r requirements.txt ``` run.sh ```sh #!/bin/bash # If no permissions run chmod +x install.sh set -e python3 resize2.py ```
Tim Forrest
2023-08-28 21:12:50 +0000 UTCThe scripts are released : https://www.patreon.com/posts/sota-subject-and-88391247
Furkan Gözükara
2023-08-28 19:52:49 +0000 UTCthese are higher than SDXL trained resolutions. you mean your own images are lower resolution? well if you upscale probably that will lower your quality.
Furkan Gözükara
2023-08-28 19:35:06 +0000 UTCThanks! I was also wondering about the resolutions for the script. These are lower than those that SDXL is trained on, would I need to upscale them to run XL Lora training correctly with kohya_ss?
Tim Forrest
2023-08-28 18:58:12 +0000 UTCI do not know if regularization images would give better results for a lora aimed at flexibility. Do you have a recommendation for a regularization dataset that would provide flexibility for a lora trained on an initial set of photographs?
Tim Forrest
2023-08-28 14:21:49 +0000 UTCyes i will share it. i was planning to share with a video but i will share right away hopefully today after several hours for you.
Furkan Gözükara
2023-08-28 14:19:53 +0000 UTCThis is great! Can you add the cropper script along with the versioned requirements.txt as a zip to the updates? I would love to run cropping on my own datasets and the cropper is a very cool tool!
Tim Forrest
2023-08-28 14:17:01 +0000 UTCwow that is terrible. rtx 4090 is the best gpu. with exact settings as mine you should get like very minimal 1.5 s/ it with 1024x1024. actually you should get much better but currently nvidia drivers are broken. i wonder what is the issue. if you can raise support to 25$ for only this month I can happily connect your PC and help you to figure out what is the core issue. by the way I have added man images as well to this post - fresh amazing ones
Furkan Gözükara
2023-08-28 09:08:48 +0000 UTCI am using a RTX4090 with 24GB VRAM, but I run into GPU Out of memory error when starting training. I am following your exact tutorial. It only works when I reduce the "Max image size" to 512, and not 1024. Any way to get this to work for 1024 as well?
epido
2023-08-28 07:36:44 +0000 UTCno you shouldnt make something like that. man images are here https://www.patreon.com/posts/4k-2700-real-84053021 and even better ones are coming very soon to this thread
Furkan Gözükara
2023-08-26 14:16:01 +0000 UTCHi there i couldn't find man images, can I use woman images to train a man , its a dumb question :)
MUNEER AHMED
2023-08-26 13:04:24 +0000 UTCHello again. All images are just updated. Can you redownload and let me know what you think. Thank you.
Furkan Gözükara
2023-08-22 19:04:03 +0000 UTCHello again. All images are just updated. Can you redownload and let me know what you think. Thank you.
Furkan Gözükara
2023-08-22 19:03:50 +0000 UTCi would resize myself. i will release an amazing resize script hopefully today. you can use it.
Furkan Gözükara
2023-08-21 14:23:31 +0000 UTCi just returned from family trip today. improved my scripts and processing again. i will update thread and notify everyone hopefully when it is done
Furkan Gözükara
2023-08-21 14:22:34 +0000 UTCYour findings were invaluable. I have much improved my scripts and processing again.
Furkan Gözükara
2023-08-21 14:21:57 +0000 UTCCan I ask if the image I trained does not need to be cropped? for example, i train my wife at a resolution of 2500x2500, the parameter I set is 768x768 in kohya to calculate it for me, right? I just need to choose 768x768 data size for the reg, right?
Đạt Nguyễn
2023-08-20 14:31:09 +0000 UTCThanks for letting me know will fix. I will also upload newer resized images in better quality. Hopefully today returning from my family trip. Sorry for the delay
Furkan Gözükara
2023-08-20 11:49:58 +0000 UTChas this been updated, along with the August 19 update?
Chester Ogilvie
2023-08-20 11:38:04 +0000 UTCFYI, the woman_3826_imgs_1024x1536px.zip link leads to woman_3826_imgs_1280x1536px.zip file
Chester Ogilvie
2023-08-20 11:37:03 +0000 UTCuse the resolution that you are going to train your images. lets say your are training your wife and your training images are 1024x1024. then only use that folder (woman_3826_imgs_1024x1024px.zip ) for training. raw images are only necessary if you want to crop for another specific resolution that you need.
Furkan Gözükara
2023-08-20 07:45:30 +0000 UTCdo i have to use woman_3847_imgs_raw (org resolutions - no cropped).zip to be reg or do i have to use all of them to make reg from org resolution _nocrop and other resolutions too, i mean i put them all in reg folder to can you make reg for train?
Đạt Nguyễn
2023-08-20 00:22:34 +0000 UTCthank you so much for understanding and support.
Furkan Gözükara
2023-08-18 22:17:51 +0000 UTCthank you so much. hopefully i will update resized files with even better cropping and resizing algorithm after monday. but raw dataset is perfect atm
Furkan Gözükara
2023-08-18 12:57:06 +0000 UTCAmazing work!
Gerenier
2023-08-18 09:34:37 +0000 UTCPlease don't excuse yourself, your job anyway is really welcome and great, I just want understand if I catched something wrong or it need to be like I saw. :)
Tapiocapioca
2023-08-17 11:34:09 +0000 UTCHello. You are right. When cropping and resizing it was mistake of automatic1111. When you look the raw dataset you will see original. I left home at the same day. Therefore, I will hopefully update resized images when I am back home. You will also notice in some cases some black area at the very bottom of the pictures. That is also mistake of automatic1111. Hopefully I will update this Monday with all much better face cropped and resized images. Sorry for the delay. I had to rush and left home that is why.
Furkan Gözükara
2023-08-17 09:22:02 +0000 UTCHello, I have some doubt about this package at the resolution 1024x1024, like the picture 00026-0-woman (1048) is a book. Other pictures cut a piece of the head like 00105-0-woman (1276) or 00319-0-woman (1809). Also I didn't understand, is full of woman with blanket on the head like this 00668-0-woman (2453) and hats, is this correct? Excuse me if I am wrong but in my mind I need to see the head and the body of the person.
Tapiocapioca
2023-08-17 09:10:04 +0000 UTCWell it depends. Do you want realistic flexibility or stylized flexibility? If you want stylized flexibility perhaps try without class images
Furkan Gözükara
2023-08-17 07:53:09 +0000 UTCFor training LoRA SDXL, if we want flexibility, are reg images necessary when training on a person?
Keith F
2023-08-17 01:23:13 +0000 UTCYou are welcome. If you make 100 repeating that means it will do actually 100 epoch in just a single epoch. So lets say you trained 3 epochs it will be actually 300 epochs. That is why.
Furkan Gözükara
2023-08-15 21:59:56 +0000 UTCThanks, this makes sense. The count of reg. photos doesn't matter, it's more of a "1 photo per iteration", so it's relative. I'm getting there, slowly understanding it all. (And yes, Kohya, using accelerate directly, sorry should have mentioned this.) I was doing 100 repeating every epoch, but I'll try the 20, yeah sometimes it's over trained and sometimes under, this will give me more stops to compare. Thanks!
Scrubbles
2023-08-15 21:57:45 +0000 UTC1 : it depends on your script. lets say you are using kohya then it will use repeating * number of training images you have. 1_woman means 1 time repeating of classification images so that is correct. 2 : for 50 training images make repeating 20 and save every epoch. then do a epoch comparison. you can train up to 8 epochs so it will be total 50 * 20 * 8 * 2 = 16000 steps. it may get overtrained but you can do checkpoint comparison to find sweet checkpoint . if you say that is too long reduce repeating count to 10 and do the same. it will be 8000 steps
Furkan Gözükara
2023-08-15 21:51:39 +0000 UTCIs there anything special we need to do beyond just adding using them as images? I named the directory 1_woman, should I use all the thousand of them or should it be dependent on how many training images I have? If I have 50 training images for example trained as 100_woman, should I use all of the regularization images or just pick a subset? To add, how many training steps do you recommend then? Or if there's already a guide on this subject, I haven't seen anything specific around regularization
Scrubbles
2023-08-15 20:43:51 +0000 UTCThank you. Hopefully I will even release a better one. Currently I am out of home sorry for the delay
Furkan Gözükara
2023-08-15 18:54:45 +0000 UTCThanks Furkan! I've used your man class images for several models, and I think they're the best yet.
Alex Voigt
2023-08-15 12:37:57 +0000 UTCyou are welcome. other resolutions are also uploaded and updated the post
Furkan Gözükara
2023-08-15 09:36:22 +0000 UTCExcellent, thank you very much sir
Dallin Mackay
2023-08-15 03:56:22 +0000 UTC