NokiMo
Furkan Gözükara
Furkan Gözükara

patreon


SOTA Image Captioning Scripts For Stable Diffusion: CogVLM, LLaVA, BLIP-2, Clip-Interrogator (115 Clip Vision Models + 5 Caption Models)

Added 2024-10-02 11:00:00 +0000 UTC

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI  GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

=======

Windows Requirements

2 October 2024 Update:

29 July 2024 Update:

19 July 2024 Update:

30 January 2025 Update

17 April 2024 Update:

20 February 2024 Update:

15 February 2024 Update:

12 February 2024 Update Massive Update:

8 February 2024 Update Update:

6 February 2024 Update Update:

5 February 2024 Update Massive Update:

3 February 2024 Update Massive Update:

13 January 2024 Update:

25 November 2023 Update:

20 October 2023 Update:

13 October 2023 Huge Update:

New tutorial video > https://youtu.be/PNA9p94JmtY

Please also upvote this Reddit thread I would appreciate very much

Old Video tutorial > https://youtu.be/V8iDW8iprqU

If you also upvote this Reddit thread I would appreciate very much

Requirements:

How To Install And Use

Use the .bat installer files for Windows and .sh installer files for RunPod. Each zip file has instructions for how to use on RunPod. Windows usage is so easy. Just run the .bat files.

How To Use Caption Scripts On RunPod

RunPod Tutorial Starts At Min 14 : https://www.youtube.com/watch?v=PNA9p94JmtY

RunPod referral link : https://bit.ly/RunPodIO

Select RunPod Fast Stable Diffusion template

Edit pod and expose HTTP ports and add 7861

After Install How To Run On RunPod

How to use RunPod and RunPodCTL tutorial >

https://youtu.be/QN1vdGhjcRc


Download the files from below attachments. captioners_clip_interrogator_v2 . zip contains all files as a zip.

Comments

no error it is running

Furkan Gözükara

2025-04-17 18:37:33 +0000 UTC

I get errors when installing Llava v5. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. llava 1.2.2.post1 requires gradio==4.16.0, but you have gradio 4.44.1 which is incompatible. llava 1.2.2.post1 requires gradio_client==0.8.1, but you have gradio-client 1.3.0 which is incompatible. llava 1.2.2.post1 requires httpx==0.24.0, but you have httpx 0.28.1 which is incompatible. llava 1.2.2.post1 requires torch==2.1.2, but you have torch 2.2.0+cu121 which is incompatible. llava 1.2.2.post1 requires torchvision==0.16.2, but you have torchvision 0.17.0+cu121 which is incompatible. and then Windows Run Part 1: LINK : fatal error LNK1181: cannot open input file 'aio.lib' test.c LINK : fatal error LNK1181: cannot open input file 'cufile.lib' [2025-04-17 11:11:34,174] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs. 2025-04-17 11:11:34 | INFO | controller | args: Namespace(host='127.0.0.1', port=10000, dispatch_method='shortest_queue') 2025-04-17 11:11:34 | INFO | controller | Init controller 2025-04-17 11:11:34 | ERROR | stderr | INFO: Started server process [13260] 2025-04-17 11:11:34 | ERROR | stderr | INFO: Waiting for application startup. 2025-04-17 11:11:34 | ERROR | stderr | INFO: Application startup complete. 2025-04-17 11:11:34 | ERROR | stderr | INFO: Uvicorn running on http://127.0.0.1:10000 (Press CTRL+C to quit)

Hockey

2025-04-17 09:12:09 +0000 UTC

amazing

Furkan Gözükara

2024-10-02 21:30:50 +0000 UTC

it works extremely well. It has been very helpful with accurately describing scenes and helps me write prompts better since I can follow how it prioritizes people, place, objects and scene descriptions

Ec Jep

2024-10-02 21:25:56 +0000 UTC

i think joy is best currently

Furkan Gözükara

2024-10-02 21:22:04 +0000 UTC

which captioning do you recommend? I recently used your post to install Joy and it works well. How are these different/better?

Ec Jep

2024-10-02 21:18:05 +0000 UTC

you are in inaccurate folder: /home/Ubuntu/Desktop/Cog/web_app_CogVLM.py' you have to be in after activating venv /home/Ubuntu/Desktop/Cog/CogVLM/web_app_CogVLM.py watch video and you will do it

Furkan Gözükara

2024-07-19 14:32:13 +0000 UTC

files are in /home/Ubuntu/desktop/cog/ CogVLM folder exists after runing the installer without error Your installer and start scripts are located in home/Ubuntu/desktop/cog/

mike oxmaul

2024-07-19 14:18:22 +0000 UTC

sorry. cannot see what i'm doing wrong here. Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$ cd CogVLM source venv/bin/activate cd .. chmod +x Mass_Compute_Start.sh ./Mass_Compute_Start.sh Please select an option: 1. Start As 4-bit Precision 2. Start As 8-bit Precision 3. Start As 16-bit Precision 4. Start As 32-bit Precision Enter your choice (1-4): 1 Please select an option: 1. Start Without Gradio Share 2. Start With Gradio Share Enter your choice (1-2): 1 python3: can't open file '/home/Ubuntu/Desktop/Cog/web_app_CogVLM.py': [Errno 2] No such file or directory Press enter to continue... (venv) Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$

mike oxmaul

2024-07-19 14:17:12 +0000 UTC

please watch between 6:20 and 9:20 of this video to learn how to install and run massed compute scripts : https://youtu.be/wG7oPp01COg

Furkan Gözükara

2024-07-19 13:49:11 +0000 UTC

(venv) Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$ chmod +x Mass_Compute_Start.sh (venv) Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$ ./Mass_Compute_Start.sh Please select an option: 1. Start As 4-bit Precision 2. Start As 8-bit Precision 3. Start As 16-bit Precision 4. Start As 32-bit Precision Enter your choice (1-4): 2 Please select an option: 1. Start Without Gradio Share 2. Start With Gradio Share Enter your choice (1-2): 1 python3: can't open file '/home/Ubuntu/Desktop/Cog/web_app_CogVLM.py': [Errno 2] No such file or directory

mike oxmaul

2024-07-19 13:45:01 +0000 UTC

thanks mate. trying now

mike oxmaul

2024-07-19 13:37:38 +0000 UTC

Yes follow the Massed_Compute_Instructions_READ.txt. i just tested on massed compute and works great

Furkan Gözükara

2024-07-19 13:19:08 +0000 UTC

fyi - you've got mass_compute_start and massed_compuete_install mass vs massed

mike oxmaul

2024-07-19 13:03:58 +0000 UTC

with V9 massed compute added. I am also testing right now. will update again if not works

Furkan Gözükara

2024-07-19 12:50:32 +0000 UTC

ok testing and fixing now

Furkan Gözükara

2024-07-19 12:44:52 +0000 UTC

its from your cogvlm_v8

mike oxmaul

2024-07-19 12:35:14 +0000 UTC

which script? i will update and fix it. massed compute uses python3 as command not python

Furkan Gözükara

2024-07-19 12:27:32 +0000 UTC

Currently using your Massed Compute VM

mike oxmaul

2024-07-19 12:26:44 +0000 UTC

That means you dont have python in your system. You have to install Python 3.10, CUDA 11.8, Nvidia drivers and C++ tools manually. Where do you use? Your local Ubuntu computer?

Furkan Gözükara

2024-07-19 12:25:55 +0000 UTC

install_Linux.sh: 9: python: not found Composing venv install_Linux.sh: 13: python: not found install_Linux.sh: 19: source: not found When running sh install_Linux.sh

mike oxmaul

2024-07-19 12:17:58 +0000 UTC

Ye that is a great gpu same as what I use. Follow video exactly same and reinstall and it should work

Furkan Gözükara

2024-06-28 11:06:39 +0000 UTC

I'll have a look at the YT vid. GPU is NVIDIA GeForce RTX 3090

Walker4k

2024-06-28 10:28:55 +0000 UTC

ok i did a fresh install and works for me. that means you have python cuda install error. here the video that you need to follow exactly : https://youtu.be/-NjNy7afOQ0 and what is your GPU model?

Furkan Gözükara

2024-06-27 22:15:01 +0000 UTC

installing right now to test

Furkan Gözükara

2024-06-27 22:00:53 +0000 UTC

Following error when attempting to use clip interrogator (ViT-L-14/openai + blip-large) - Traceback (most recent call last): File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 182, in convert_to_tensors tensor = as_tensor(value) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 141, in as_tensor return torch.tensor(value) RuntimeError: Could not infer dtype of numpy.float32 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\queueing.py", line 541, in process_events response = await route_utils.call_process_api( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api output = await app.get_blocks().process_api( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api result = await self.call_function( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function prediction = await anyio.to_thread.run_sync( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2177, in run_sync_in_worker_thread return await future File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 859, in run result = context.run(func, *args) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\utils.py", line 833, in wrapper response = f(*args, **kwargs) File "O:\captioning\captioners_clip_interrogator_v2\Clip_Interrogator.py", line 118, in image_to_prompt return ci.interrogate(image) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\clip_interrogator\clip_interrogator.py", line 244, in interrogate caption = caption or self.generate_caption(image) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\clip_interrogator\clip_interrogator.py", line 191, in generate_caption inputs = self.caption_processor(images=pil_image, return_tensors="pt").to(self.device) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\models\blip\processing_blip.py", line 103, in __call__ encoding_image_processor = self.image_processor(images, return_tensors=return_tensors) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\image_processing_utils.py", line 551, in __call__ return self.preprocess(images, **kwargs) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\models\blip\image_processing_blip.py", line 310, in preprocess encoded_outputs = BatchFeature(data={"pixel_values": images}, tensor_type=return_tensors) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 78, in __init__ self.convert_to_tensors(tensor_type=tensor_type) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 188, in convert_to_tensors raise ValueError( ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.

Walker4k

2024-06-27 11:48:33 +0000 UTC

thank you so much for such comment. yes i make venv for every installer to not break anything else :)

Furkan Gözükara

2024-06-11 18:59:11 +0000 UTC

Thank you very much for compiling these models; I am particularly grateful for the auto installs of the venv as every time I tried most manual installs, I would get stuck in a horrible mis-matched dependency loop. Every auto installer I have used from you has worked right out of the box, and you have helped safe me tremendous headaches. :D

reaper557

2024-06-11 15:07:43 +0000 UTC

updated to Kosmos-2_v6.zip let me know if that fix your error. also you can email me that particular image that gives error : monstermmorpg@gmail.com

Furkan Gözükara

2024-04-17 00:55:17 +0000 UTC

The above exception was the direct cause of the following exception: Traceback (most recent call last): File "W:\Kosmos\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "W:\Kosmos\venv\lib\site-packages\gradio\route_utils.py", line 231, in call_process_api output = await app.get_blocks().process_api( File "W:\Kosmos\venv\lib\site-packages\gradio\blocks.py", line 1594, in process_api result = await self.call_function( File "W:\Kosmos\venv\lib\site-packages\gradio\blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "W:\Kosmos\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "W:\Kosmos\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "W:\Kosmos\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run result = context.run(func, *args) File "W:\Kosmos\venv\lib\site-packages\gradio\utils.py", line 689, in wrapper response = f(*args, **kwargs) File "W:\Kosmos\web_app.py", line 204, in batch_caption_images caption = generate_predictions(img, text_input, is_batch=True) File "W:\Kosmos\web_app.py", line 265, in generate_predictions image_input.save(user_image_path) File "W:\Kosmos\venv\lib\site-packages\PIL\Image.py", line 2439, in save save_handler(self, fp, filename) File "W:\Kosmos\venv\lib\site-packages\PIL\JpegImagePlugin.py", line 653, in _save raise OSError(msg) from e OSError: cannot write mode RGBA as JPEG been getting this error a lot not sure if there is a way to get around it.

Steve

2024-04-16 19:11:53 +0000 UTC

it is running like this : run_pt1.bat , run_pt2.bat , run_pt3.bat run all these 3 with orders and wait each one

Furkan Gözükara

2024-03-26 12:22:45 +0000 UTC

Doc, I downloaded the LLaVA installer separately and there is no window .bat file, can I only run it on the runpod?

Additional Contributions

2024-03-26 10:25:04 +0000 UTC

Cool, I think I'll just use it on hugging face. Everything seems fine and there is no error, but the run_pt3 would just crash at a certain point when you run it. A tutorial comparing them would be great!

Erik

2024-03-01 11:20:22 +0000 UTC

each one has strengths and weaknesses. i plan to compare all later a time. kosmos 2 is working pretty fast with low VRAM. you can try. llava is working perfect. if you can show me entire process how you are running i can point your error. hopefully i will make a tutorial for it

Furkan Gözükara

2024-03-01 10:58:45 +0000 UTC

Which one of these is the best? Also, I installed LLaVA but the program would crash when I get to run_pt3

Erik

2024-03-01 07:00:13 +0000 UTC

updated v7 and fixed the issue. you can either manually downgrade the transformers library or do a fresh install

Furkan Gözükara

2024-02-23 09:30:10 +0000 UTC

same error working on it right now. weird

Furkan Gözükara

2024-02-23 09:25:16 +0000 UTC

Hi Doc, I had successfully installed CogVLM_v6 separately (2 days ago, I just needed CogVLM.As a visual designer, lengthy video tutorials are difficult for me, so I followed "runpod_instructions_READ" to perform the installation).but today it no longer works, I neither can open the 7861 port nor get the link after the installation is complete. Error message (KeyError: 'inv_freq')

design master

2024-02-23 08:24:30 +0000 UTC

i dont think so that error related to batch labeling.

Furkan Gözükara

2024-02-22 14:33:49 +0000 UTC

CogVLM_v6，After batch labeling, it cannot be used in dreamboth, prompt。Exception training model: ''NoneType' object is not subscriptable'.

楠 陈

2024-02-22 13:19:00 +0000 UTC

Especially for low VRAM gpus very best. I just added 4 bit 8 bit and 16 bit options

Furkan Gözükara

2024-02-16 14:25:44 +0000 UTC

Kosmos is IMHO the best captioner so far... generated captions requires just a bit of editing...

Nenad Kuzmanovic

2024-02-16 13:22:46 +0000 UTC

no they are all different captioners. you can test all and compare them. hopefully i am working on a massive tutorial for all

Furkan Gözükara

2024-02-11 19:36:53 +0000 UTC

interesting. i have rtx 3060 as my second gpu and when i set cuda visible devices to 1 it works directly and load model into that GPU.

Furkan Gözükara

2024-02-11 19:36:28 +0000 UTC

im a little confused, which one do i run for windows? i installed like it said in the clip interrogator folder, but i downloaded lava and qwen ect, does each do the same thing? running a 3090 and primarily use SDXL for my lora training with Kohya ss

OP-Cast

2024-02-11 18:39:52 +0000 UTC

I was able to resolve setting the gpu. I have a 3080ti as my primary and a 3090ti as my second. For some reason when selecting #6 (13b Model - Load in 16 bit - 24 GB VRAM) and running LLaVa, I receive a NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE. (error_code: 1). I ran pt1, pt2 then pt3 several times with the same error. When I choose any of the 7B models, it works fine. The 3090ti is at (0 usage) by the way. My monitors are connected to my 3080ti at (0.3 usage).

DaOldHeadz Music

2024-02-11 17:19:34 +0000 UTC

I just tested working on my second gpu RTX 3060 : https://pasteboard.co/xYUVhZaoZxoh.png

Furkan Gözükara

2024-02-11 11:15:00 +0000 UTC

you must have changed it inaccurately set CUDA_VISIBLE_DEVICES=1 will change it

Furkan Gözükara

2024-02-11 10:54:30 +0000 UTC

How do I use gpu1 instead of gpu0. I change set CUDA_VISIBLE_DEVICES=1 in run_pt3 but no luck. Thanks.

DaOldHeadz Music

2024-02-11 06:30:58 +0000 UTC

true. 4bit is slower but uses lesser vram.

Furkan Gözükara

2024-02-08 15:44:59 +0000 UTC

just fixed this with v3, it was encoding error thanks for letting me know

Furkan Gözükara

2024-02-08 15:44:34 +0000 UTC

CogLVM rules :-)

Nenad Kuzmanovic

2024-02-08 15:43:08 +0000 UTC

dmn then it is not dependable :/ thanks for letting me know

Furkan Gözükara

2024-02-08 15:41:09 +0000 UTC

Getting this warning when using 4bit. Is this normal behavior? UserWarning: Input type into Linear4bit is torch.float16, but bnb_4bit_compute_type=torch.float32 (default). This will lead to slow inference or training speed. warnings.warn(f'Input type into Linear4bit is torch.float16, but bnb_4bit_compute_type=torch.float32 (default). This will lead to slow inference or training speed.') Over 2 minutes and it never generated a caption. I'm on a 4070 8GB CogLVMv6 I should have pointed out.

leem0nchu

2024-02-08 15:11:09 +0000 UTC

hahha Qwen is so funny and ridiculous, here are some of it's captions: - Sorry, but I can't assist with that. - As an AI language model, I don't have access to images, but I can describe the scene you might be referring to. - As an AI language model, I don't have access to the image you are referring to, so I cannot provide a detailed description of it. However, if you could provide me with more information or context about the image, I would be happy to help you with your query.

Nenad Kuzmanovic

2024-02-08 10:04:36 +0000 UTC

I'm getting this error with batch captioning Qwen: Caption for callisto_ (1).png generated in 1.59 seconds. Estimated time to complete: 348.22 seconds. Processed 14/93. Caption for callisto_ (11).png generated in 6.52 seconds. Estimated time to complete: 356.07 seconds. Processed 15/93. Caption for callisto_ (13).png generated in 3.75 seconds. Estimated time to complete: 347.66 seconds. Traceback (most recent call last): File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\route_utils.py", line 230, in call_process_api output = await app.get_blocks().process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1590, in process_api result = await self.call_function( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run result = context.run(func, *args) File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\utils.py", line 678, in wrapper response = f(*args, **kwargs) File "H:\Qwen_captioning\Qwen-VL-Chat.py", line 88, in batch_caption_images caption_file.write(response) File "C:\python3109\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 215-217: character maps to Traceback (most recent call last): File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\route_utils.py", line 230, in call_process_api output = await app.get_blocks().process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1590, in process_api result = await self.call_function( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run result = context.run(func, *args) File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\utils.py", line 678, in wrapper response = f(*args, **kwargs) File "H:\Qwen_captioning\Qwen-VL-Chat.py", line 88, in batch_caption_images caption_file.write(response) File "C:\python3109\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 571-573: character maps to

Nenad Kuzmanovic

2024-02-08 09:25:41 +0000 UTC

you are welcome

Furkan Gözükara

2024-02-08 01:21:37 +0000 UTC

Awesome

leem0nchu

2024-02-08 01:18:30 +0000 UTC

to test those blip1 + clip visions they are inside : captioners_clip_interrogator_v2.zip

Furkan Gözükara

2024-02-08 01:17:49 +0000 UTC

I have added some newer stuff try them. especially blip + clip vision models uses really low VRAM if you have low VRAM.

Furkan Gözükara

2024-02-08 01:17:22 +0000 UTC

I haven’t. I guess I didn’t notice only so the sample scripts on the post a few days ago didn’t that showed 10.5gb vram as lowest

leem0nchu

2024-02-08 01:15:21 +0000 UTC

i am adding low vram options for every script are you testing them? for example 4 bit llava.

Furkan Gözükara

2024-02-08 01:11:14 +0000 UTC

Will there be similar options (I understand not so powerful) for lowvram?

leem0nchu

2024-02-08 00:59:23 +0000 UTC

You are right, but nothing is stopping you to add as much "terms" as you wish... I can do that, but if you can write short instruction which lines in script has to be changed...

Nenad Kuzmanovic

2024-02-07 01:08:17 +0000 UTC

Dots should be replaced with comas, not deleted ;-)

Nenad Kuzmanovic

2024-02-07 00:55:52 +0000 UTC

well sadly this really depends on the user specific input . but i may work on it

Furkan Gözükara

2024-02-06 22:04:43 +0000 UTC

nice info

Furkan Gözükara

2024-02-06 22:03:55 +0000 UTC

I'm testing CogVLM intensively and already i have some patterns for cleaning captions. Like this example, which occurs almost in every caption is: - change "there's a" to "with a" That change mainly concerns the adaptation of the sentence, in the sense of making it as concise as possible and corresponding to the format that SD understands best. Dots "." should be deleted also.

Nenad Kuzmanovic

2024-02-06 21:48:19 +0000 UTC

If you need any help for formating captions, just let me know. It would be great if that script for caption pruning can be integrated in your gradio UI, so we can easily change type of cleaning captions. I am constantly searching for the best caption tools and i am not satisfied with any of them, and if you manage to make it as i suggested, that WILL be gamechanger my friend...

Nenad Kuzmanovic

2024-02-06 15:47:17 +0000 UTC

yes i would also use kohya scripts to improve captions

Furkan Gözükara

2024-02-06 14:55:42 +0000 UTC

I think that can be fixed with custom python script, like in Kohyass/Finetune/clean captions and tags. In that way we can have separate script for style, concept etc.

Nenad Kuzmanovic

2024-02-06 14:55:09 +0000 UTC

thank you for bringing this to my attention. i just changed the default timeout to 60 seconds.

Furkan Gözükara

2024-02-06 14:27:39 +0000 UTC

it is true. they may add unnecessary words. i am also working on Qwen-VL captioning right now to add here

Furkan Gözükara

2024-02-06 14:16:21 +0000 UTC

I edited command and results are quite better: Question: just caption the image with details, colors, items and objects but do not add any description or comment. do not miss any item in the given image Answer: The image showcases a long hallway with white walls and doors. At the end of the hallway, there's a large, ominous portal or gateway. This gateway is surrounded by red, grotesque, and tentacle-like structures. Within the gateway, a massive, menacing face with hollow eyes and sharp teeth is visible. A woman in a long white dress stands in front of the gateway, seemingly confronting or observing the face. The floor is wet, possibly from a recent flood or rain, and there are puddles of water scattered around.

Nenad Kuzmanovic

2024-02-06 14:06:18 +0000 UTC

I've noticed that sentence construction all of VLM's is not very suitable for Stable diffusion training, cause it contains a lot of unnecessary words.. Example: The image showcases a futuristic or sci-fi setting with a central mechanical structure illuminated in green. Within this structure, there's a humanoid figure with a bald head, seemingly in a state of distress or unconsciousness. The environment appears to be a dimly lit room with large windows, possibly suggesting an industrial or laboratory setting. The art style leans towards realism with a touch of surrealism, given the juxtaposition of the mechanical structure and the human figure. Total number of tokens: 99 - 101 (depends of which model is used, gpt-4 says 101). So, undesirable words in this example are: The image showcases, or, Within this structure, possibly suggesting, The art style leans towards realism with a touch of surrealism, given the juxtaposition of the mechanical structure and the human figure. When those words are tokenized, multiple problems potentially arise: 1.Describing the style is not desirable when training the style, but also the character, because the model becomes inflexible and the transfer of the style present in the dataset is not achieved, and character (person) is best to train only with rare token trigger word. This is maybe good for Concept training, but for that we already have LORA and block weights. 2. The number of tokens is unnecessarily inflated, which can be a problem when generating images But this YET has to be tested, I am writing this from the experience I have with training models using filewords

Nenad Kuzmanovic

2024-02-06 13:18:40 +0000 UTC

It is working and yes, IT IS the best captioning model. Amazing how good it is

Nenad Kuzmanovic

2024-02-06 12:59:06 +0000 UTC

It is working now, thanks ;) Sometimes there is a problem with error about a network. For others - if u have lower GPU - you need to edit gradio_web_server.py "response = requests.post(worker_addr + "/worker_generate_stream", headers=headers, json=pload, stream=True, timeout=10)" - timeout from 10 to for example 500 and the error will not occur anymore :)

Joosheen

2024-02-06 11:44:16 +0000 UTC

you were right sorry about that. updated to latest version please download newest versions.

Furkan Gözükara

2024-02-06 00:28:17 +0000 UTC

you were right sorry about that. updated to latest version please download newest versions.

Furkan Gözükara

2024-02-06 00:28:14 +0000 UTC

For me llava doesn't work unfortunately :/ I tried to run 6gb vram model. I can open part 1 and part 2 but part 3 is crashing. Gradio is opening but I have several errors: "The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " and "No GPU found. A GPU is needed for quantization". Something is wrong with bitsandbytes?

Joosheen

2024-02-05 23:38:26 +0000 UTC

you see since you edited cuda to 118 you got the error below

Furkan Gözükara

2024-02-05 22:36:10 +0000 UTC

this is not supposed to happen because it is supposed to install torch 2.2.0 with cu121. but i see it installed 2.2.0 cu118. but my bat file containing this line pip3 install torch==2.2.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 you see it is installing cu121 also you should install python 3.10.11. but i see you have installed 3.10.9. i have shown this in the video if you want older cuda use this pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip install xformers==0.0.22

Furkan Gözükara

2024-02-05 22:35:32 +0000 UTC

Version 3 was ok, installed easily...

Nenad Kuzmanovic

2024-02-05 20:42:47 +0000 UTC

This last time, i went step by step and installed everything manualy. I edited links with cuda 118 (instead 121)...

Nenad Kuzmanovic

2024-02-05 19:56:17 +0000 UTC

Python is installed in root: C. I dont have antivirus, only Win 10 defender. But i have checked, Kohya installler has installed everything correctly, xformers, bitsandbytes etc....

Nenad Kuzmanovic

2024-02-05 19:54:33 +0000 UTC

ok, i will do it

Nenad Kuzmanovic

2024-02-05 19:52:25 +0000 UTC

Enter your choice (1-2): 1 WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.2.0+cu121 with CUDA 1201 (you have 2.2.0+cu118) Python 3.10.11 (you have 3.10.9) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details Traceback (most recent call last): File "H:\CogVLM\web_app_CogVLM.py", line 43, in model = AutoModelForCausalLM.from_pretrained( File "H:\CogVLM\CogVLM\venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 561, in from_pretrained return model_class.from_pretrained( File "H:\CogVLM\CogVLM\venv\lib\site-packages\transformers\modeling_utils.py", line 3032, in from_pretrained raise ImportError( ImportError: Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or `pip install bitsandbytes`. Press any key to continue . . .

Nenad Kuzmanovic

2024-02-05 19:47:40 +0000 UTC

ye 8 bit working nice

Furkan Gözükara

2024-02-05 19:45:10 +0000 UTC

i would do this. uninstall all python cuda. restart computer. install exactly as shown in this video into C drive directly : https://youtu.be/-NjNy7afOQ0 also if you have a antivirus that could be preventing the installation

Furkan Gözükara

2024-02-05 19:44:50 +0000 UTC

I have installed CUDA 11.8 but installer wants to install 12.1. Is that maybe a problem here?

Nenad Kuzmanovic

2024-02-05 19:24:37 +0000 UTC

8bit then

Nenad Kuzmanovic

2024-02-05 18:06:45 +0000 UTC

Hello. Download latest V5. it works with 4bit 8bit and 16bit. i have tested. 16 bit uses more than 30GB VRAM

Furkan Gözükara

2024-02-04 18:31:05 +0000 UTC

uploaded v4. but it requires more than 24gb so i couldn't test yet.

Furkan Gözükara

2024-02-04 14:10:03 +0000 UTC

CogVLM 16 bit.

Nenad Kuzmanovic

2024-02-04 14:05:20 +0000 UTC

you get this error at llava or CogVLM or? CogVLM 8 bit load not working. I am trying to fix. 4 bit load working fine

Furkan Gözükara

2024-02-04 13:19:48 +0000 UTC

Error processing scorn_ (99).png: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Nenad Kuzmanovic

2024-02-04 10:50:21 +0000 UTC

just added batch. I agree LLaVA better

Furkan Gözükara

2024-02-04 02:36:28 +0000 UTC

That is great, but i found it is not much better than LLAVA, to be honest

Nenad Kuzmanovic

2024-02-03 23:56:37 +0000 UTC

yep currently working on it

Furkan Gözükara

2024-02-03 23:46:56 +0000 UTC

Is it possible to add batch process for CogVLM_v2?

Nenad Kuzmanovic

2024-02-03 23:17:18 +0000 UTC

On average, it reads at 10-12 MB/s speed instead of 100. Sometimes it goes up to 20. As if the model loader reads the model slowly if it is not in the cache. I wrote a PM on Discord.

Mike Menders

2024-02-01 18:32:17 +0000 UTC

ye i also get apex message ignore it. what reading speed do you see on task manager when reading from the disk?

Furkan Gözükara

2024-02-01 17:42:11 +0000 UTC

3TB Toshiba, 7200 RPM with 64 MB cache, 6 Gbit/s, DT01ACA300 Python cache on this HDD (too large for System SSD), and symlinked. And I have this message when start CogVLM: Please 'pip install apex' Apex is installed in venv...

Mike Menders

2024-02-01 17:32:38 +0000 UTC

i think it is related to your disk. the model size is very big and your disk must be very slow

Furkan Gözükara

2024-02-01 17:30:14 +0000 UTC

After I restarted the machine and turned CogVLM back on, it says half an hour loading time! Is this normal? 07:48 - 25:24, 254.01s/it Loading the model is very slow on cold start. If it starts once, then it is fine, then I can turn it off and on, it starts fast (I guess from cache).

Mike Menders

2024-02-01 17:17:55 +0000 UTC

multiple gpu or more powerful gpu

Furkan Gözükara

2024-01-27 13:51:43 +0000 UTC

any way to speed up the LLava captioning? it takes a very long time to caption

Hassan Alhassan

2024-01-27 13:30:37 +0000 UTC

Hello. as I replied your private message it is system error. error of windows. Need more debugging to figure out the reason

Furkan Gözükara

2024-01-24 21:41:56 +0000 UTC

Hello! Why do I always get an error when starting the model in the third step when using LLaVA subtitles, such as part 3 - 7b model - 8bit - 8600 MB VRAM.bat, but my graphics card is 3090 and the video memory is enough, my Win11 system directly reports an error Restart, can LLaVA subtitles fail to win the system?

耀良 梁

2024-01-24 19:25:02 +0000 UTC

Hello. updated file names to be more clear you can redownload. the part is the model starting files such as 13b model - 8bit - 15470 MB VRAM.bat

Furkan Gözükara

2024-01-24 12:26:38 +0000 UTC

I tried to run "13b model - 8bit - 15470 MB VRAM.bat". It said "don't forget to run all run_pt1.bat run_pt2.bat run_pt3.bat" but there's no "run_pt3.bat". Is it a typo?

Thomas

2024-01-24 09:06:16 +0000 UTC

yes hopefully i will add that feature very soon just like llava

Furkan Gözükara

2024-01-16 12:09:43 +0000 UTC

yes hopefully i will add that feature very soon just like llava

Furkan Gözükara

2024-01-16 12:09:35 +0000 UTC

Can it be supported support batch inference

楠 陈

2024-01-16 06:01:31 +0000 UTC

CogVLM_v2 Can it support batch marking?

楠 陈

2024-01-16 05:39:22 +0000 UTC

Exactly! Thanks

Thomas

2024-01-13 15:15:51 +0000 UTC

thank you so much updated the file. you were right. you figured out from pip freeze i included?

Furkan Gözükara

2024-01-13 15:12:32 +0000 UTC

thanks testing right now to fix

Furkan Gözükara

2024-01-13 14:53:08 +0000 UTC

I only used the files inside your captioners_clip_interrogator.zip, it download the latest bitsandbyte which is currently 0.41.2.post2

Thomas

2024-01-13 14:45:50 +0000 UTC

did you upgrade it manually? which script you trying from this post?

Furkan Gözükara

2024-01-13 14:42:57 +0000 UTC

It worked only after I downgraded bitsandbyte to 0.41.1

Thomas

2024-01-13 13:47:21 +0000 UTC

did it still work? i think even though these warnings still should work . i also get a lot of warnings. these happens usually since we use Windows and libraries are optimized only for linux

Furkan Gözükara

2024-01-13 13:18:12 +0000 UTC

Hello! I can't use this tool because bitsandbytes throw an error : "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...". I do have the cuda toolkit installed.

Thomas

2024-01-13 13:09:32 +0000 UTC

this happens when you skip 1 step. have you seen this video? i have shown there https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images

Furkan Gözükara

2023-12-21 16:43:13 +0000 UTC

Hi! Great work and fantastic tutorial. I'm running into an issue however with the LLaVA chatbot. I installed it using your auto installer, but after loading the run bats and the model bat, I get an error "NETWORK ERROR DUE TO HIGH TRAFFIC, PLEASE REGENERATE OR REFRESH THIS PAGE." It seems to work for a second, and then fails. I try refreshing it a number of times and restarting, but no luck. Do you know how to fix this? Thanks again.

gawdman

2023-12-21 04:11:12 +0000 UTC

Batch Processing doesn't work with png files, I need to convert them to jpg and lose a bit of quality.

RayHell

2023-12-20 20:35:19 +0000 UTC

great solving

Furkan Gözükara

2023-12-15 22:09:15 +0000 UTC

Yep i just used fast stone to batch convert them to jpg

Meito

2023-12-15 21:55:23 +0000 UTC

i should test thanks for letting me know

Furkan Gözükara

2023-12-15 21:46:47 +0000 UTC

it seems png is affected, i converted them to jpg and it works now

Meito

2023-12-15 21:44:55 +0000 UTC

how to setup is shown here : https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images

Furkan Gözükara

2023-12-15 21:11:12 +0000 UTC

now this is interesting. you can write a simple python script to convert all your images into png.

Furkan Gözükara

2023-12-15 21:10:37 +0000 UTC

OSError: cannot write mode RGBA as JPEG when doing batch

Meito

2023-12-15 16:34:33 +0000 UTC

how do we setup llava, i'm having trouble running it. which bat files do we run?

Meito

2023-12-15 16:21:40 +0000 UTC

you are welcome. in such cases increasing virtual RAM may help

Furkan Gözükara

2023-12-14 11:17:53 +0000 UTC

thanks, that was the problem :)

Guillaume Bieler

2023-12-14 11:00:14 +0000 UTC

good it means it is starting loading. if failing could be your computer RAM not sufficient.

Furkan Gözükara

2023-12-13 23:31:02 +0000 UTC

Hello. Sadly I don't have information regarding ComfyUI

Furkan Gözükara

2023-12-13 23:30:41 +0000 UTC

Hi, I've had no problem using the app on Linux, but for some reason why I try using blip2-2.7b I get this: Loading caption model blip2-2.7b... Loading checkpoint shards: 0%| | 0/2 [00:00

Guillaume Bieler

2023-12-13 18:34:07 +0000 UTC

Im trying to use blip captioning to read an image to then use the prompt later in a custom workflow on ComfyUI, as far as iv seen using the WAS node suite I can BLIP Model Loader and BLIP Analyse Image, so apparently I just need add a BLIP model to the correct folder. Im curious if you have any insight into this or tips

Dungeon Master

2023-12-13 14:29:54 +0000 UTC

OK, thank you very much

楠 陈

2023-12-11 14:51:48 +0000 UTC

it shouldn't be that slow. i also have 3090 working really good. well i suggest you to use LLaVA or Blip2. i think those 2 are best right now for caption

Furkan Gözükara

2023-12-11 00:14:13 +0000 UTC

Hi. Two questions: 1. I'm trying to use ViT-bigG-14/laion2b_s39b_b160k and blip2-2.7b but it's taking up all GPU vram on my 3090 and taking 2hrs for a single image. Is this right? 2. What is the best CLIP model for sd 1.5?

Sarcasticest

2023-12-10 23:13:08 +0000 UTC

hello you were right. i found the error. now added download.py. it will download the gradio app instead of using .bat which was failing in some cases. you can directly run it with python download.py . redownload installer zip file

Furkan Gözükara

2023-12-10 14:48:44 +0000 UTC

yes this works on linux as well. use runpod installer

Furkan Gözükara

2023-12-10 14:00:21 +0000 UTC

Do you have a Linux batch processing script for LLava?

mypatreonemailacc

2023-12-10 13:22:49 +0000 UTC

there is checkout this tweet images : https://twitter.com/GozukaraFurkan/status/1728425482662592625

Furkan Gözükara

2023-12-09 21:44:05 +0000 UTC

true. thanks for comment

Furkan Gözükara

2023-12-09 21:43:28 +0000 UTC

There should be a path batch processing box at the bottom of the page. I can confirm that batch processing works.

DaOldHeadz Music

2023-12-09 17:14:36 +0000 UTC

Encountered the same problem, but using http://127.0.0.1:7860/ can indeed be used, but there is no place for batch operations and entering folder addresses.

楠 陈

2023-12-09 16:58:28 +0000 UTC

It has been automatically installed and enabled, but there is no batch folder input text box. Is it because the automatically installed file is not the latest version? Or you haven't updated it yet, and you have the same problem after reinstalling it several times.

楠 陈

2023-12-09 16:57:09 +0000 UTC

what have you installed so far? i have shown llava in this video : https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images hopefully i will make a full tutorial soon

Furkan Gözükara

2023-12-09 14:30:01 +0000 UTC

How to update and download

楠 陈

2023-12-09 14:24:41 +0000 UTC

yes my LLaVA gradio already supports batch subtitle given folder images

Furkan Gözükara

2023-12-09 12:49:45 +0000 UTC

I mean add LLaVA to batch subtitle file

楠 陈

2023-12-09 12:08:59 +0000 UTC

can you checkout this video? i have shown there quickly : https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images

Furkan Gözükara

2023-12-06 10:59:18 +0000 UTC

Hello. When trying to run the URL (http://127.0.0.1:40000) for LLaVA, I get this message from Chrome and Edge. I launched Part 1 then Part 2 then the Model (13b model - 8bit - 15470 MB VRAM). I have a 3090TI. { "detail": "Not Found" } Thanks...

DaOldHeadz Music

2023-12-06 08:53:20 +0000 UTC

i am not sure. i think could be but it already uses decent amount of VRAM. if you are meaning batch processing a folder my web app already have that. there is batch process folder input textbox. use it. hopefully will make a tutorial soon

Furkan Gözükara

2023-12-04 10:08:02 +0000 UTC

it works ignore it

Furkan Gözükara

2023-12-04 10:07:36 +0000 UTC

LLaVA Can it be batch processed?

楠 陈

2023-12-04 07:01:46 +0000 UTC

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. llava 1.1.3 requires bitsandbytes==0.41.0, but you have bitsandbytes 0.41.1 which is incompatible.

楠 陈

2023-12-04 03:52:49 +0000 UTC

1 : if you are doing fine tuning I would compare Blip 2 alone vs LLaVA 2 : for face i don't use any captions. just rare token + class. like ohwx man 3 : try answer 1

Furkan Gözükara

2023-12-03 18:06:25 +0000 UTC

for CPU sadly I don't know :/ but i think LLaVA is also great

Furkan Gözükara

2023-12-03 18:05:40 +0000 UTC

1. What's the reason we should use the CLIP model alongside the caption model? 2. Do you still believe that ViT-Big-GAN-14 combined with BLIP2-FLAN-T5 offers the best performance for captioning and training a face? 3. I used this combination, but I wasn't satisfied with the results. It added some unrelated and meaningless keywords.

So Sha

2023-12-03 03:20:01 +0000 UTC

Which captioning do you recommend for running on the CPU on a laptop? I'm using WD14 right now, which is really fast. Is there anything similar?

mypatreonemailacc

2023-11-29 17:27:24 +0000 UTC

I plan to do a full tutorial today. good reminding me. i should make that

Furkan Gözükara

2023-11-26 20:34:28 +0000 UTC

It would be soo helpfull if u make tutorial of how to change cache folder... Im struggling with free space on my system drive... There is explanation on Hugging face and i tried but something i did wrong and didt manage to make that work.

Nenad Kuzmanovic

2023-11-26 13:49:27 +0000 UTC

so easy. bat files are just pip commands. make your venv and execute them 1 by 1. also you can use runpod_install.sh file which is designed for linux

Furkan Gözükara

2023-11-20 23:30:34 +0000 UTC

How we can install it on ubuntu ?

So Sha

2023-11-20 21:43:57 +0000 UTC

i would try blip 2 + vit bigG 14 but that may not be sufficient. i plan to add llava to the collection as well hopefully soon

Furkan Gözükara

2023-11-14 12:36:40 +0000 UTC

Is there any suitable for analyzing images with facial expressions? For example, the character is opening his mouth, smiling, closing his mouth, being surprised...

Đạt Nguyễn

2023-11-14 10:37:29 +0000 UTC

i prefer currently rare token + class token and not using captions. so like ohwx man or ohwx car or ohwx woman etc. rare token is ohwx and class is the thing you are training.

Furkan Gözükara

2023-10-24 23:52:26 +0000 UTC

When training a model through Lora, I understood, just as you explained in the video, that there are two methods: instance prompts, class prompts, and the use of captions. Is my understanding correct? If it is, then if I do understand correctly, which of the two methods do you think is better, and could you explain the reasons for your preference?

준영 이

2023-10-24 15:19:44 +0000 UTC

i am experimenting with it the effect of captionings. hopefully will make a tutorial but not ready yet

Furkan Gözükara

2023-10-22 15:31:38 +0000 UTC

Is there a youtube video or something how to use this kind of caption for Lora?

준영 이

2023-10-22 05:02:54 +0000 UTC

thanks it may work. i usually make my auto installer

Furkan Gözükara

2023-10-21 20:42:15 +0000 UTC

I found this docker image that should be one-click for LLava in runpod: https://github.com/ashleykleynhans/llava-docker I have not tried it yet, but I will try it soon.

mypatreonemailacc

2023-10-21 17:34:40 +0000 UTC

:D

Furkan Gözükara

2023-10-19 21:15:09 +0000 UTC

Who cares about Windows, everybody uses Linux :)

mypatreonemailacc

2023-10-19 19:43:40 +0000 UTC

i am waiting them to add windows support. after that hopefully will bring it to the runpod and windows both

Furkan Gözükara

2023-10-19 10:38:47 +0000 UTC

yep working amazing

Furkan Gözükara

2023-10-19 10:38:24 +0000 UTC

Would it be possible to use llava on a runpod, maybe in batch processing?

Gerenier

2023-10-19 07:28:40 +0000 UTC

Works great with ViT-bigG-14/laion2b_s39b_b160k on 3090 Runpod.

Gerenier

2023-10-19 07:27:15 +0000 UTC

well i asked them to make it work on windows and they said yes. i am waiting them

Furkan Gözükara

2023-10-18 23:12:33 +0000 UTC

Llava is amazing when it comes to describing a picture: https://llava.hliu.cc It would be great to have a docker image or an installation guide for running it in batch mode. This could be a major game changer when captioning images for training SD models.

mypatreonemailacc

2023-10-18 21:18:50 +0000 UTC

ye pretty useful

Furkan Gözükara

2023-10-13 22:42:35 +0000 UTC

Hell yeah!

EnjoyerOfAll

2023-10-13 21:45:08 +0000 UTC

nice. i am still researching

Furkan Gözükara

2023-10-13 21:01:16 +0000 UTC

thanks but it is outdated not working

Furkan Gözükara

2023-10-13 20:42:29 +0000 UTC

I use WD14 for captioning my datasets for Lora training. If you observe that BLIP2 is better, then let us know!

mypatreonemailacc

2023-10-13 20:42:02 +0000 UTC

I think ViT-bigG-14 + blip2 is great

Furkan Gözükara

2023-10-13 20:41:54 +0000 UTC

If you have a Automatic1111 UI in runpod, you can use these plugins: https://github.com/Tps-F/sd-webui-blip2.git

mypatreonemailacc

2023-10-13 20:36:30 +0000 UTC

BLIP isn't that great, WD14 is already a massive improvement over BLIP. But WD14 doesn't really create full sentences, just a bucket list of what it sees.

mypatreonemailacc

2023-10-13 20:35:08 +0000 UTC

Gradio list 90 models . but i don't know which ones are lighter sadly :( i guess you have to test them

Furkan Gözükara

2023-10-13 18:58:52 +0000 UTC

Do you have the link to the gradio with lightweight models? I think anything less than 10GB should be runnable on a modern laptop CPU.

mypatreonemailacc

2023-10-13 18:32:40 +0000 UTC

Amazing. It is the most powerful one

Furkan Gözükara

2023-10-13 01:00:38 +0000 UTC

I tested blip2-flan-t5-xxl and it is amazing! It did a great job with images that were very complex

San Milano

2023-10-13 00:42:51 +0000 UTC

it is added check out instructions. hopefully a tutorial video coming too

Furkan Gözükara

2023-10-12 20:58:00 +0000 UTC

it uses a lot of VRAM. i think it may on ram. how much RAM you have? by the way I added another captioning gradio which is superb. it also has lightweight models

Furkan Gözükara

2023-10-12 20:57:48 +0000 UTC

Is it possible to run blip2 captioning on my laptop (CPU) ? I can run WD14 from the kohya repository on the cpu, and it's very fast.

mypatreonemailacc

2023-10-12 20:32:10 +0000 UTC

hello i am working on it too. thank you

Furkan Gözükara

2023-10-12 18:53:16 +0000 UTC

please could you create instructions for Runpod, thanks!

j

2023-10-12 17:29:17 +0000 UTC

not yet i am still in research

Furkan Gözükara

2023-10-12 15:24:32 +0000 UTC

Do you have any evidence of improvements in Dreambooth training of subjects or styles with captioning? Any comparisons?

Daniel Quandt

2023-10-12 15:20:43 +0000 UTC

hopefully this one coming today much stronger : https://twitter.com/GozukaraFurkan/status/1712093261249097924 i will update this post

Furkan Gözükara

2023-10-11 19:35:14 +0000 UTC

I tried the 8bit_precision_10_GB_VRAM script, but it's done a poor job: "a woman in a pink skirt and jacket posing"

Zsédely Ruben

2023-10-11 19:12:47 +0000 UTC

i am working on to it as well : https://github.com/haotian-liu/LLaVA/issues/521 also this is coming hopefully tomorrow : https://twitter.com/GozukaraFurkan/status/1711933282529452115

Furkan Gözükara

2023-10-11 03:30:00 +0000 UTC

Hi! This is "JohnDopamine" - Regarding that last question/answer: Would there be anyway (or benefit?) of trying to incorporate "llava" as a method to help caption? Hacksman had mentioned this online demo site: https://llava.hliu.cc/ - which works pretty well if you say "describe this image" and can follow up w/ questions etc. I'm not a coder so don't fully understand what is needed to get anything out of it (if it's even better than the models you have implemented). But I did see there was a commit an hour ago that said training code and dataset have been added. Maybe good news? Thanks for this either way ! code: https://github.com/haotian-liu/LLaVA

John Dopamine

2023-10-11 03:03:15 +0000 UTC

thank you

Furkan Gözükara

2023-10-11 02:51:38 +0000 UTC

JazakAllah 🙌🏽

diffusers

2023-10-11 02:19:08 +0000 UTC

yes they are definitely better. moreover i am working on adding ViT-bigG-14/laion2b_s39b_b160k too to the scripts. it is much different level. both can be compared used and tested

Furkan Gözükara

2023-10-11 02:12:38 +0000 UTC

Do you believe these models are better than Blip or WD14 ?

So Sha

2023-10-11 01:49:42 +0000 UTC

Windows tutorial video : https://youtu.be/V8iDW8iprqU

Furkan Gözükara

2023-10-11 01:05:47 +0000 UTC

yep totally. i may release hopefully tomorrow with a video

Furkan Gözükara

2023-10-11 00:52:36 +0000 UTC

possible to run this on runpod?

diffusers

2023-10-11 00:49:33 +0000 UTC

Related Creators

UmbralWhite (TheFrozenMoogle)

UmbralWhite (TheFrozenMoogle)

 patreon
reeze

reeze

 patreon
From Behind - AI IMAGES

From Behind - AI IMAGES

patreon
𝚎𝚟𝚎

𝚎𝚟𝚎

 gumroad
Mirrthful

Mirrthful

 patreon
ForbiddenWords ASMR

ForbiddenWords ASMR

 patreon
屍猫骸

屍猫骸

 fanbox
Jhon Larry

Jhon Larry

 patreon
Veil Art

Veil Art

 patreon
SwagFire Games

SwagFire Games

 patreon
Bread And Butter

Bread And Butter

 patreon
こまどりる

こまどりる

 fanbox
リュンタック

リュンタック

 fantia
VLVTsfm

VLVTsfm

 fanbox
WhiteMantis

WhiteMantis

 patreon
abisenlol

abisenlol

 patreon
るがつき

るがつき

 fanbox
KaliethVA 🔞🎙 - NSFW - Commissions Open

KaliethVA 🔞🎙 - NSFW - Commissions Open

 gumroad
EverEdge

EverEdge

 patreon
Pugilismx

Pugilismx

 fanbox
帝寒

帝寒

patreon
のおおお

のおおお

 fanbox
良之助@お仕事募集中

良之助@お仕事募集中

 fanbox
mario_reg

mario_reg

 patreon
WhiteSteel

WhiteSteel

 fanbox
NitaNi

NitaNi

 patreon
AurL

AurL

 gumroad
Autarch

Autarch

 patreon
Sakura Rabbit 樱花兔

Sakura Rabbit 樱花兔

 fanbox
DTZ is here!

DTZ is here!

 patreon
yonaki

yonaki

 fanbox
まかまち/makamati

まかまち/makamati

 fanbox
Saithorthepyro

Saithorthepyro

 patreon
saintbullart

saintbullart

 patreon
brinabel

brinabel

 patreon
Sunben4062

Sunben4062

 patreon
vfxtutors

vfxtutors

 gumroad
Igi

Igi

 patreon
dazfrog

dazfrog

 fanbox
Reina

Reina

 patreon
羽柴利緒

羽柴利緒

 fanbox
Cecils Pantyhose

Cecils Pantyhose

 patreon
toratora

toratora

 patreon
Siline Bacca

Siline Bacca

 gumroad
vegastarlight

vegastarlight

 patreon
たかつ

たかつ

 fanbox
Jess D. Astra

Jess D. Astra

 patreon
TomooInn

TomooInn

 patreon
cathycrues

cathycrues

 patreon
スティーリア

スティーリア

 fanbox