ELLA - Making prompts better!
Added 2024-04-11 15:20:01 +0000 UTC
"The horizon shimmered with a kaleidoscope of hues, a huge towering, iridescent crystal structure erupts from the ground, their facets refracting the vibrant pink and indigo sky. Amidst this otherworldly landscape, a lone figure with his back to us stands wearing an old, tattered, black cloak and a sense of ethereal grace."
Video guide: https://youtu.be/_Pr7aFkkAvY
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
CLIP gets stuff wrong all the time! Ask for a blue bear with a red hat, and chances are you'll get a red bear with a blue hat. Not so much with ELLA! Now you can provide long and detailed prompts and actually get what you asked for. Just don't ask for perfect hands ;)
ComfyUI guide:
Install https://github.com/ExponentialML/ComfyUI_ELLA using ComfyUI manager
Models should be saved under your ComfyUI/models directory
The first model to download is the ELLA one, which you save to ComfyUI/models/ella
https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors
The second download is flan-t5-xl, which you save to ComfyUI/models/t5_model/flan-t5-xl
Original t5 = https://huggingface.co/google/flan-t5-xl/tree/main
Smaller t5 = https://huggingface.co/Kijai/flan-t5-xl-encoder-only-bf16/tree/main
Command to download all files:
git clone https://huggingface.co/Kijai/flan-t5-xl-encoder-only-bf16
(Optionally rename that directory to flan-t5-xl)
Original repo guide:
URL: https://github.com/TencentQQGYLab/ELLA
conda create --name ella python=3.10
conda activate ella
git clone https://github.com/TencentQQGYLab/ELLA.git
cd ELLA
pip install fire gradio==3.50.2 torch torchvision diffusers transformers accelerate sentencepiece
python inference.py test test ./ella-sd1.5-tsc-t5xl.safetensors
NB. Gradio 4.26.0 doesn't work due to deprecated functions
Links:
https://ella-diffusion.github.io/
https://github.com/TencentQQGYLab/ELLA
https://github.com/ExponentialML/ComfyUI_ELLA
https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors
https://huggingface.co/google/flan-t5-xl/tree/main
https://huggingface.co/Kijai/flan-t5-xl-encoder-only-bf16/tree/main
Comments
For the portable version, make sure you're using the portable environment for any manual installs.
Nerdy Rodent
2024-04-16 12:27:04 +0000 UTCOk after much redoing; I'm definitely having issues with the node: Error occurred when executing ELLATextEncode: "LayerNormKernelImpl" not implemented for 'Half' Thanks To your work I know it's the node or my setup, I'll chase it down in the Github... Thanks for all your work!
Oliver Baddeley
2024-04-13 15:48:47 +0000 UTCthere's the attachment.... Man I'm a blind rodent
Oliver Baddeley
2024-04-13 02:41:30 +0000 UTCUsing Ollama Vision Node with llava:7b to create descriptions of pics and trying to pipe into Ella prompts.... It's not working out so well.... Any chance your workflow for Ella Is available and maybe you can have one of these LLM picture description node tie into ella? Or am I just be a bloody hamster on a wheel? :P
Oliver Baddeley
2024-04-13 02:32:54 +0000 UTCNever seen that! I'd guess at ComfyUI needing an update?
Nerdy Rodent
2024-04-12 21:43:45 +0000 UTCError occurred when executing LoadElla: not a string
KING LIFER
2024-04-12 13:55:53 +0000 UTCComfyUI (portable)
Arjan Speelman
2024-04-12 12:51:22 +0000 UTCIs that in ComfyUI or their repo?
Nerdy Rodent
2024-04-12 12:49:55 +0000 UTCKeep getting a "T5Tokenizer requires the SentencePiece library" error, already did a pip install of sentencepiece and diffusers. Any ideas why I'm still getting this error? Using the smaller version of the model
Arjan Speelman
2024-04-11 17:44:00 +0000 UTC
