Most Powerful Vision Model CogVLM 2 Gradio APP Published, Caption Images and More, 1-Click to Install on Windows, RunPod and Massed Compute
Added 2025-01-17 13:53:57 +0000 UTCInfo
Latest installers : https://www.patreon.com/posts/120193330
Official repository of the APP : https://github.com/THUDM/CogVLM2
This app above is purely developed by me
CogVLM 2 is GPT4 level for vision tasks and it is open source and free
Check above screenshots to see how to use it
Currently the APP works amazing with 4-bit quantization very fast
I am searching to lower VRAM usage even further with like adding CPU-Offloading and other stuff if possible
Previously we were lacking Triton but it now works perfect
If you get Triton error make sure to delete your Triton cache after installing the app like below
C:\Users\Furkan\.triton
Instead of Furkan your username
The app has so many features so checkout the screenshots
1-Click installers will install into Python 3.10 VENV completely secure and isolated
Comments
hi do you have c++ tools installed? did you follow this tutorial step by step ? https://youtu.be/DrhUHnYfwC0 i plan to update installers hopefully soon
Furkan Gözükara
2025-03-26 23:49:50 +0000 UTCEncountered exception while importing triton: DLL load failed while importing libtriton: The dynamic link library (DLL) initialization routine failed. Error processing image: DLL load failed while importing libtriton: The dynamic link library (DLL) initialization routine failed.
楠 陈
2025-03-26 05:53:11 +0000 UTCyes this is a very big model. that is why it is slow. biggest image captioning model we have atm . about your last thing i dont know i am using gradio default image editor maybe they need to support ye i will check app title and i dont know your internet related question. and you have to use 4 bit on 24 gb gpu
Furkan Gözükara
2025-01-23 13:29:55 +0000 UTCI tried this yesterday and it worked but with 1 24 GB VRAM GPU it got really slow trying to use the non-quantized version. Maybe it takes about 40 GB of VRAM/RAM (based on the total size of the model files/shards it downloaded which were about 8x ~5 GB) which is over my card VRAM. Or maybe it was also because I also had other apps loaded at the same time or because I increased the amount of tokens to create. Maybe the menu where you select which version to load (unquantized or the different quantized versions) could say how much VRAM each uses. When I turned the router off after I'd pressed the caption button it said something like "Connection Error" in the UI. Why though if it's supposed to be running locally and the models used were already downloaded? Is it because it (or some part of gradio) is logging something and if so why? Or is it because in the code in the get_model section it says "trust_remote_code=True". Does it need that because the model is non-standard and does that make it need the internet even after the model is loaded? Is that a security risk when we can't see what remote code is being run? Also starting captioning it checks and downloads new versions of python files if they exist. Couldn't it just keep the versions that were already there unless you ask it to update them (eg. there's a warning that shows that says "Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.". I don't know how to pin a revision but it would probably be better if there was an option for whether to update. Also the browser tab for this shows "Gradio" as the title but that's the same title as the browser tab for Joy Caption Alpha2. If the browser tab was titled something like "CogVLM2 UI" it would be easier to distinguish each. I just tried the 8 bit quantized version and that also seems very slow (about 13 mins for 1 image) and using a lot of VRAM too. Maybe it's a much bigger model than Joycaption (maybe it's checking at a higher res too so could be more accurate eg. for text). With the 4 bit version it works reasonably fast (about 62-106 secs for creating with max of 2048 tokens, maybe depending what other things are running). So if it's more accurate than JoyCaption that could be used. Though at 2048 tokens max it only produces (in the tests I'd done at the time between 151 and 183 words, which is less than the 260 word count that JoyCaption can output (perplexity says 2048 tokens should be about 1500 words)). Maybe there's some cut off it's using (other than the 2048 token limit) that limits the number of words that might be able to be used. Though for small text in the image COGVLM2 seems much better than JoyCaption, since it described what was being said in 4 lines of text in an image (though it didn't quote it exactly first time, but if you ask it to quote it exactly in the prompt it sometimes does but didn't when I asked it for other more detailed info about the image too. Though asking it that also makes it hallucinate sentences of text that don't exist in the image sometimes.). Another thing that would be good if possible is if, when you want to put a different image into the UI when there's already one there, it would be good if you could just drag the new image over the old one without pressing x to close the original one. If you drag it without closing the original one it just shows the image in the browser not as part normal UI.
cool1
2025-01-23 09:50:17 +0000 UTCthanks for info
Furkan Gözükara
2025-01-21 14:24:34 +0000 UTCNice! I've noticed One trainer has added Hunyuan training but I'm not sure you can train video clips. Only images. :)
JamZam WamBam
2025-01-21 01:36:36 +0000 UTCI am gonna hopefully make the very best hunyun training on windows with so much easiness not WSL :D
Furkan Gözükara
2025-01-20 22:02:17 +0000 UTCtrue. also you can use our caption editor to replace those words
Furkan Gözükara
2025-01-18 19:07:34 +0000 UTCthis is more powerful and versatile but i would say it depends on dataset and usage case
Furkan Gözükara
2025-01-18 19:07:14 +0000 UTCHello, Is it better than Joy Caption? 😊
Ahmet Inceelli
2025-01-18 11:19:53 +0000 UTCIs this more accurate than the JoyCaption versions? One thing I noticed in the examples is, for photos it doesn't seem to say it's a photo. eg. it says "the image captures a snowy scene" or "the image features a man" but not "a photo/photograph of...". So maybe that could make the captions less specific. Though it does say for things like "digital artwork" for some other images.
cool1
2025-01-18 00:04:22 +0000 UTCThe captioner demo works great. They have the Models for 'CogVLM2-Video-Llama3-Chat' on Huggingface. https://huggingface.co/THUDM/cogvlm2-video-llama3-chat/tree/main
JamZam WamBam
2025-01-17 23:08:23 +0000 UTCThanks Dr Furkan. Im getting amazing loras results for Hunyuan video. You can train image and videos. I had to install the trainer through WSL Linnux sub system for windows which was a headache to figure out. On my RTX 3090 I'm able to get 1280x720, 5 seconds at 24fps in comfyui using the kijai nodes ;)
JamZam WamBam
2025-01-17 22:33:03 +0000 UTCYou are welcome. yes this is a huge upgrade
Furkan Gözükara
2025-01-17 21:47:51 +0000 UTCit doesnt have it yet. i plan to make Hunyuan training with images first then with video. try here and let me know if useful : http://cogvlm2-online.cogviewai.cn:7868/
Furkan Gözükara
2025-01-17 21:30:29 +0000 UTCIt says in the Repo that they released the CogVLM2-Video model. Does this include it. I'm busy training loras for Hunyuan video and a good video captioner would be great.
JamZam WamBam
2025-01-17 21:27:30 +0000 UTCI have been using Cog VLM 2 since your earlier releases, without proper triton compatibility on windows. Even though it was slow, I loved the results. This upgrade is very good news and I thank you for the upgrade
TD Film Studio
2025-01-17 18:17:04 +0000 UTCYou can use to ask anything about image.
Furkan Gözükara
2025-01-17 14:09:46 +0000 UTCSo, apart from captions what's the point of this?
BecauseReasons
2025-01-17 13:57:54 +0000 UTC
