4K 2700 Real Class Images + Auto Cropping Script
Added 2023-06-04 15:18:52 +0000 UTCThis post is deprecated new Image Cropper here : https://www.patreon.com/posts/120352012
Use this updated dataset for both man and woman > https://www.patreon.com/posts/87700469
Join discord and tell me your discord username to get a special rank : SECourses Discord
19 August Update
download_man_reg_imgs.sh file will download all of the reg images and automatically extract them into the RunPod /workspace/regimages folder. That can be used for Unix and possibly for MacOs systems as well. Don't forget to comment the links that you don't want to download and change folder paths if you wish.
Upload into workspace folder of RunPod and execute below command
cd /workspace
chmod +x download_man_reg_imgs.sh
./download_man_reg_imgs.sh
10 July 2023 : face cropper added. it has different requirements.txt and cropper file.
The video for this post released : https://youtu.be/QTYX0tgA5ho
Please read carefully
Auto cropping script cropper.py will take your raw images and crop the subject (person) based on predefined aspect ratios with maximum efficiency. This tool is amazing to prepare training images for both classification and training.
Cropping script can be used for other objects as well such as cars. For cars here the code (line 46) : car = next((x for x in results.xyxy[0] if int(x[5]) == 2), None)
I have spent like a full day to code this script from scratch.
I am working on a new workflow to generate amazing quality realistic images. They will be beyond studio quality. For this task I needed real pictures. Therefore, I have prepared 2700 4K resolution real images for "man" class.
I have collected the images from https://unsplash.com/
They are free to use even for commercial purposes
Majority of the images had like 4000x6000 pixels original resolution.
Watch the above video to learn more about the dataset
I have manually picked the images. Images were portrait orientation. Which is the part of my new workflow. Hopefully I will make a new video where I will show my new amazing training workflow.
Then I used the attached script to extract subject into the following aspect ratios
(512, 512), (512, 768), (768, 512), (640, 960), (960, 640), (768, 1024), (1024, 768)
Then I used automatic1111 to resize them to these resolutions with focusing face. Because since the orientation was portrait, some of the images had to be cropped to be downscaled to this resolution
Below all of the images links. Each one is a zip file and the password of the zip file is:
secourses
Just plain secourses nothing else is the password
I also have uploaded the original raw images if anyone needs
These images can be used as classification / regularization images during training with DreamBooth or LoRA. They can be even used for fine-tuning training.
These images would likely to work best for realism training. For styling it may not work best. Need to be further tested
The class would be man for these images or photo of man
I use this Realistic vision (v5) model for realism (4 GB) : https://huggingface.co/SG161222/Realistic_Vision_V5.1_noVAE/resolve/main/Realistic_Vision_V5.1.safetensors
Raw images (7.54 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/raw_2735_imgs.zip
512x512 (0.91 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/512x512_2734_imgs.zip
512x768 (1.32 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/512x768_2734_imgs.zip
768x512 (1.25 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/768x512_2735_imgs.zip
768x768 (1.94 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/768x768_2734_imgs.zip
640x960 (1.99 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/640x960_2733_imgs.zip
960x640 (1.89 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/960x640_2735_imgs.zip
768x1024 (2.51 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/768x1024_2724_imgs.zip
1024x768 (2.43 GB) : https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/1024x768_2734_imgs.zip
1024x1024 (3.38 GB): https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/1024x1024_2734_imgs.zip
To use these files unrunpod
You need to install 7zip
yes | apt-get install p7zip-full
Download with
wget
e.g. :
wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/1024x768_2734_imgs.zip
or another one
wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/1024x1024_2734_imgs.zip
Then use this command to extract them
7z x 1024x768_2734_imgs.zip
or another one
7z x 1024x1024_2734_imgs.zip
It will ask you password
type
secourses
typing will be invisible
type again and it works
How to use face cropper
Download requirements_face.txt and cropper_face.py
Set your face crop percentage
Comments
This post is deprecated new Image Cropper here : https://www.patreon.com/posts/120352012
Furkan Gözükara
2025-05-13 11:05:02 +0000 UTCI have Nvidia Quadro RTX5000 with Turing arquitecture. I have the error " Unsupported CUDA architecture: sm75". Please help!
Jorge Fernández
2025-05-12 17:22:26 +0000 UTCthank you so much. i usually as reddit and gpt 4. hopefully i will release even better 2 scripts soon. 1 uses yolo v7 and another uses pil library to focus face and downscale. i am processing woman dataset right now. hopefully i will release new man dataset too
Furkan Gözükara
2023-08-22 12:00:28 +0000 UTCI just came here to drop a huge "good job" to you. I work as data engineer and just started having a look into the ML staff, and I know that what you have achived here is not simple and requires time. I wrote a stupid python script for face deetection using SDD_mobilenet_coco* and frozen_inference as dataset but even after playing around with the trashold value the result is always pretty bad compared to what you have done. might I ask you where can I find good staff in terms of code example and models for this kind of work ?
Gireiyu
2023-08-22 11:51:25 +0000 UTCyes some images won't be square. because this script will not crop subject. lets say you have a tall person. you can't crop it as square. you need to do another processing with automatic1111 and crop subjects. that way it will become exact resolution. if you message me on discord or if you watch video you will see.
Furkan Gözükara
2023-08-08 22:52:27 +0000 UTCI'm trying to crop to square only, but most of the outputs are not square. I've tried 512x512, 1x1, 1024x1024. All result in the same final size and aspect ratio. I also notice the problem files all have "Loops: 0" Final coords: (0, 21, 682, 1018), Final ratio: 0.6840521564694082, Loops: 0 The ones that do crop to square have Loops > 0 as in: Final coords: (30, 0, 710, 680), Final ratio: 1.0, Loops: 160
Mig Test
2023-08-08 22:26:07 +0000 UTChopefully it is on my todo list. sorry for the delay my vacation and other family stuff just ended
Furkan Gözükara
2023-08-04 11:35:44 +0000 UTCthank you so much for your comment. it is 100% related to the training dataset. if you can get better dataset it will become better. so what is better dataset? different clothing and different background having images. but they have to be high quality good focus and good lightning. moreover as you do a training with higher resolution you will get better results. at least 768x1024 on realistic vision latest version i suggest
Furkan Gözükara
2023-08-02 23:37:26 +0000 UTCThank you very much for your work and the information you've gathered. I'm impressed by all the details you've provided. I was able to train a model according to the tutorial but I still have a question. I have the impression that when I generate images I have a lot of faces that don't look like "me" sometimes they are perfect but only 1 time out of 10/15 images generated. Often the facial features are similar, but nothing more. What could be causing this, that my "selfie" dataset isn't providing enough information (about 10 images)? Is there a way to force greater precision on the use of the face? I've tried this on a dozen checkpoints, but the results are similar. and always used (subject man:1.1) in positive prompt. Thanks in advance for your answers.
Nicolas Vidmer
2023-08-02 22:31:46 +0000 UTChere example for you : for SDXL 1024x1024 - photo of a woman for sd 1.5 lets say realistic vision : 768x768 - photo of a woman if you want higher resolution calculate base resolution and use high res fix
Furkan Gözükara
2023-08-01 17:06:52 +0000 UTCSo just set a very simple prompt up? Ensure the image size is correct and leave it to generate a few thousand images? I'm happy to do that on my machine now if there are some suggestions for best prompt and settings
James R
2023-08-01 16:59:30 +0000 UTCthat would be better if you had collected them from unsplash. but that is hard without automation. for now you can generate them from text 2 image tab with high res fix. hopefully i will release a massive dataset for woman as well
Furkan Gözükara
2023-08-01 16:56:14 +0000 UTCIf I want to train for a woman do I just leave the Regularisation images option blank in kohya?
Virtamouse
2023-08-01 16:49:39 +0000 UTCHopefully it is coming. I am planning to work on that this week or next week
Furkan Gözükara
2023-08-01 13:07:09 +0000 UTCSee my comment below, but this is the bit I'm now planning on starting if nobody has done so already. With you class "man" already done, another of "woman" would be pretty much all we need?
James R
2023-08-01 12:05:36 +0000 UTCThis is fantastic... has anyone already started to compile a set with the class "woman" yet?
James R
2023-08-01 12:03:02 +0000 UTCAre you sure? Can you send me org image and cropped image please? You can private message. Sorry for late reply I didn't get notification
Furkan Gözükara
2023-07-28 09:03:54 +0000 UTCThanks for the info
Furkan Gözükara
2023-07-28 09:02:51 +0000 UTCYou are welcome. Latest version is v5 now :)
Furkan Gözükara
2023-07-28 09:02:35 +0000 UTCI use 7z x /workspace/1024x1024_2734_imgs.zip and it works
Filip A. M
2023-07-28 08:59:03 +0000 UTCcropper_face.py will crop only faces. someone requested so i added it too :)
Furkan Gözükara
2023-07-24 08:21:38 +0000 UTCHi Dr. I've noticed there are 2 py files attached to this post, what is the difference between cropper.py and cropper_face.py? I thought for prepare different aspect ratios we should use cropper.py?
Daniel Chen
2023-07-24 03:42:41 +0000 UTCThats not working. It keeps cropping humans.
Casper Smit
2023-07-21 11:06:42 +0000 UTCHello. Use torch 2.0.1 and cu118. Not anymore torch 1.13 suggested. Moreover Triton still not available on Windows. But it will work. Triton is just an optimization like xformers
Furkan Gözükara
2023-07-12 15:49:54 +0000 UTCInstall Kohya_SS Version: v21.8.2 Torch 1.13.1+cu117 ,Start training image, I get an error, A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton' , What can I Do? Fix this Error?
g Yu
2023-07-12 15:26:39 +0000 UTC100% sufficient. I used poco x3 pro which is probably much weaker than yours
Furkan Gözükara
2023-07-11 18:17:02 +0000 UTCUsing a iPhone 13 Max pro Camera is alright for training image of myself? What camera did you use for your latest video on making studio grade photos?
Brandon Diaz
2023-07-11 17:56:09 +0000 UTCRad. Thanks!
Kellan
2023-07-09 01:04:07 +0000 UTCv3 better. use it :) it just came after my tutorial
Furkan Gözükara
2023-07-08 23:04:49 +0000 UTCThoughts Realistic vision v2 vs v3? I noticed in your video you used v2 for training in dreambooth.
Kellan
2023-07-08 22:31:27 +0000 UTCthat is expected. for 1024x1024 your best option is 512x512 and then use high res fix. or collect images manually like from real images
Furkan Gözükara
2023-07-04 22:56:04 +0000 UTCThanks, I deleted a couple images that i thought would cause problems and it worked. I have another issue though, if I use max resolution of 1024 it generates two heads for some of the class images.
Mauricio Coronel
2023-07-04 21:47:09 +0000 UTCwow they didnt add exception handler. this means that the model didnt find face in the given image. this can happen but the developers should have put try catch. you need to remove that particular image that it cant find face in your given image
Furkan Gözükara
2023-07-03 07:50:32 +0000 UTCI get an error if I select auto focal point, when pre processing images. Preprocessing [Image 0/66]: 0%| | 0/66 [00:01 0 else [] File "C:\AI\stable-diffusion-webui\modules\textual_inversion\autocrop.py", line 150, in image_face_points faces = detector.detect(np.array(im)) cv2.error: OpenCV(4.8.0) D:\a\opencv-python\opencv-python\opencv\modules\dnn\src\net_impl.cpp:279: error: (-204:Requested object was not found) Layer with requested id=-1 not found in function 'cv::dnn::dnn4_v20230620::Net::Impl::getLayerData'
Mauricio Coronel
2023-07-03 04:07:27 +0000 UTCdid you make it work?
Furkan Gözükara
2023-06-24 23:49:41 +0000 UTCI can make a small video for you. But not today tomorrow hopefully. I am not at home :/
Furkan Gözükara
2023-06-23 21:33:34 +0000 UTCTo use these files unrunpod You need to install 7zip apt-get install p7zip-full Download with wget e.g. : wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/1024x768_2734_imgs.zip Then use this command to extract them 7z x https://1024x768_2734_imgs.zip It will ask you password type secourses typing will be invisible type again and it works
Furkan Gözükara
2023-06-23 21:33:07 +0000 UTCDid you follow the instructions? I unzipped several times. Check the description bottom part.
Furkan Gözükara
2023-06-23 21:32:24 +0000 UTCi cant unzip the file on jupyterlab
Rhkenigma
2023-06-23 21:23:34 +0000 UTCthank you for reply. i haven't tested stableTuner yet but i am planning. so far i am working with dreambooth extension of auto1111. it could be stableTuner bug
Furkan Gözükara
2023-06-15 15:53:44 +0000 UTCHello Dr, I have prepared some images in different aspects ratios with your amazing script and tried to feed those to StableTuner to train a dreambooth. My images sizes are: 1024x1024 768x768 512x512 960x640 768x512 Then in stableTuner I said training to be 1024 and asked to do image buckets but it seems it upscaled the smaller ones like for example the 768x512 and the others to make it 1024 or even higher for some reason. I am not sure if I am doing it right. I didn't find much info on how to use stableTuner with many aspect ratios. Do you use stableTuner for your trainings ? If not which other software would you recomend ? Are you going to make a video on how to train dreambooth using the different aspect ratios dataset ? kr,
Anduvo
2023-06-15 12:57:41 +0000 UTCamazing thanks for letting us know.
Furkan Gözükara
2023-06-12 00:03:02 +0000 UTCWaw it worked perfectly ! Such a great script, huge time saver. Thank you!
Anduvo
2023-06-11 23:12:31 +0000 UTCi think yes. can you try this? car = next((x for x in results.xyxy[0] if int(x[5]) == 2), None) - if you message me on patreon i can show
Furkan Gözükara
2023-06-11 22:22:28 +0000 UTCHi Doc, would it be possible to modify your cropping script to do the same for objects, like a car for example ?
Anduvo
2023-06-11 22:21:37 +0000 UTCthank you. here video : https://youtu.be/QTYX0tgA5ho
Furkan Gözükara
2023-06-04 23:16:04 +0000 UTCGreat stuff, can't wait for the video!
egormly
2023-06-04 19:46:25 +0000 UTC768x768 added upon your request
Furkan Gözükara
2023-06-04 18:40:26 +0000 UTCthat is really hard. i have written a simple script. for this script to work first i loaded the images manually on browser. then clicked download button to download loaded ones. took a while. there can be working ones but I couldn't find. If you become gold patreon i can connect your pc and install for you and show how works. it is a primitive script though. i coded it in .net core
Furkan Gözükara
2023-06-04 16:58:58 +0000 UTChow did you scrape images i dont see a bulk download
Kevin momplaisir
2023-06-04 16:56:04 +0000 UTCthis is man dataset unfortunately. but for woman, you can collect images and use the script. after collecting images rest so easy
Furkan Gözükara
2023-06-04 16:35:03 +0000 UTCwell i found that 768x1024 training produces super high quality without repetition. but i posted all resolutions. i will also upload 768x768 for you :) started preparing it
Furkan Gözükara
2023-06-04 16:34:43 +0000 UTCcan i use this for woman too
Kevin momplaisir
2023-06-04 16:20:20 +0000 UTCWhy 4K when all we can train (without a rediculous amount of VRAM) is 768x768? 🤔 Great job though. Can't believe how you have time for everything 😁👍
Kallamamran
2023-06-04 15:49:37 +0000 UTC
