Gender Classifier - Low Colors & Multiple Face Remover - Stable Diffusion Training Images Preprocessor
Added 2023-11-09 22:26:13 +0000 UTCJoin discord and tell me your discord username to get a special rank : SECourses Discord
This script uses Retina Face and a fine tuned gender-classification model from Hugging Face. Both are state of the art very high accuracy.
12 November 2023 whole post updated. Huge improvements. Now supports Retina Face GPU
Requirements:
- Make sure that you have git and Python 3.10.x installed. I used Python 3.10.11.
- May work with other Pythons too but I haven't tested
- Here a tutorial video : https://youtu.be/B5U7LJOvH6g
How To Use:
- Download gender_classifier.zip into any folder where you want to install
- Extract it. This file contains all 7 attached files so same.
- First run install.bat file. You need to double click and run only 1 time.
- It will generate a new VENV and install all necessary libraries.
- This will not break or affect your other Python apps and installations.
- Edit classify_and_process.py and change below folder paths.
- source_folder = r'G:\gender_classifier\blur_test'
- no_face_folder = r'G:\man dataset\no_face_detected'
- multi_face_folder = r'G:\man dataset\multi_faces_detected'
- blurry_folder = r'G:\man dataset\blurry_faces_detected'
- low_color_folder = r'G:\man dataset\low_colored_images'
- male_folder = r'G:\man dataset\male_new_images'
- female_folder = r'G:\man dataset\female_new_images'
- unknown_folder = r'G:\man dataset\unknown_images'
- log_file_path = 'logs.txt'
- DEBUG_IMAGES = True
- ENABLE_BLURRY_DETECTION = False
- DEBUG_FOLDER = 'debug_folder_path'
sort_sharp
This script will sort images based on their sharpness / blurness. Can be used to sort
- Edit sort_sharp.py and set below paths
- images_folder_path = r"G:\gender_classifier\blur_test"
- log_file_path_sharpness = r"G:\gender_classifier\sharpness_logs.txt"
- log_file_path = r"G:\gender_classifier\processing_log.txt"
- sorting_image_prefix="man"
You can change prefix to like woman, img, etc.
Also it will start with 10000 but if you have over 90000 images start from 100000
Hopefully a tutorial video coming soon.
Comments
thank you so much
Furkan Gözükara
2024-01-24 00:44:02 +0000 UTCyou're great, man...
Art
2024-01-23 20:36:14 +0000 UTCfixed the attachment linking issue.
Furkan Gözükara
2024-01-23 13:09:20 +0000 UTChi... tried to download, but it givrs back error: {"errors":[{"code":902,"code_name":"AttachmentNotFound","detail":"Attachment with id 16536026 was not found.","id":"adc2fd20-2528-5dc3-b3d4-c08ff1d10ebd","status":"404","title":"Attachment was not found."}]}
Art
2024-01-23 13:08:09 +0000 UTCit is not breaking the purpose. 0 faces still getting eliminated. what issue are you having right?
Furkan Gözükara
2023-12-24 20:33:28 +0000 UTCHow did you solve the initial problem?
Airpilot
2023-12-24 20:32:02 +0000 UTCdo you get any error while installing? what was your GPU?
Furkan Gözükara
2023-12-14 11:22:08 +0000 UTCcublasLt64_12.dll seems to be missing. I'm using CUDA 11.8.
mike oxmaul
2023-12-14 07:29:23 +0000 UTCi think you can use face similarity calculator for that : https://www.patreon.com/posts/sort-ai-images-82478694 https://www.youtube.com/watch?v=343I11mhnXs
Furkan Gözükara
2023-12-13 23:31:58 +0000 UTCHave you got a method of finding a specific face inside 100,000 images?
mike oxmaul
2023-12-13 12:28:15 +0000 UTCyes this is what i tell all the time. use python 3.10.x.
Furkan Gözükara
2023-12-13 12:26:33 +0000 UTCIf you can't install because of Torch issue - ensure your using Python 3.10.11 as specified in path. 3.12 didn't work.
mike oxmaul
2023-12-13 11:34:16 +0000 UTCplease ignore warning it should work 100%
Furkan Gözükara
2023-12-05 22:43:20 +0000 UTCWARNING:tensorflow:From A:\SD\gender_classifier\venv\Lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
Kevin O'Malley
2023-12-05 18:10:29 +0000 UTCgreat. if you find a better library let me know please.
Furkan Gözükara
2023-11-26 23:05:01 +0000 UTCit's ok I figured it out. However, I noted that it was not separating the 'no faces' properly. After digging into the code, I think this line face_count = 0 if not faces else len(faces) is the problem. Looking through the code for RetinaFace, it looks like it still returns a tuple when there are no faces, which still has a length. So instead, changing the code to face_count = 0 if not faces or not isinstance(faces, dict) else len(faces) seems to do the trick. This checks if the variable is indeed a dictionary, if it is not, then there were no faces found. I left the check to see if faces exists, though when debugging, it always seemed to have a length. It's entirely possible that this is specific to my machine; I have modified the venv as I'm playing with comparing the image to a reference image to find matches using facenet, though I might switch to another library as facenet is not producing amazing matches.
John Webb
2023-11-26 22:57:18 +0000 UTCthis is so weird. you are having problem when using classify_and_process_GPU.bat right? can you dump your venv pip freeze and email me or send me from discord? monstermmorpg@gmail.com
Furkan Gözükara
2023-11-25 11:06:46 +0000 UTCRunning this using the GPU, it just dumps the majority of the images into the unknown images folder. When I run it using CPU, it sorts them properly, though of course significantly slower - any idea what might be causing this? It's not a major problem of course, I'll just go and make a coffee and sort the laundry while it sorts though the 1500 images at what seems to be 1 per second. On a 4080 with the latest drivers. The sort_sharp script doesn't seem to have the same issue.
John Webb
2023-11-25 10:01:18 +0000 UTCyes i have script for that too : https://youtu.be/olX1mySE8HA https://www.patreon.com/posts/84292083 nswf remover source code.zip
Furkan Gözükara
2023-11-14 01:12:51 +0000 UTCis there a way to filter or remove 18+ photos ?
Hassan Alhassan
2023-11-13 19:08:42 +0000 UTClets try it out
Hassan Alhassan
2023-11-13 19:06:13 +0000 UTCI use this to sort downloaded images from internet. Yes for generalization images
Furkan Gözükara
2023-11-10 01:37:32 +0000 UTCWhat is this for? To generate generalisation images?
Siva
2023-11-10 01:32:37 +0000 UTC
