Innovate Futures @ Benji

Stable Diffusion ComfyUI Workflow - Using Multimodal Pipeline To Create AI Video

Added 2024-07-02 13:48:03 +0000 UTC

Video : https://youtu.be/EgM4OGRj2OQ

If you need to setup Local Ollama to host your local LLM with ComfyUI.

Here's previous tutorials :

1 - https://youtu.be/EQZWyn9eCFE

2 - https://youtu.be/yR2Y9G71w6E

We'll be exploring how to create stunning AI videos with the help of a multimodal pipeline.

Conceptual Diagram (Current Version)

As we all know, AI video models have been evolving rapidly, and now we have companies like Kling AI, Luma AI, and the latest Gen-3 AI video models from RunwayML. Although Runway Gen-3 is currently available only for text-to-videos, image-to-video capabilities are just around the corner.

In this workflow, I'll show you how to transform natural language content into Stable Diffusion Text Prompts, which can be used to generate images for each scene. These images serve as the initial keyframes for the AI video generator. We'll be using powerful models like LLaMA 3 fine-tuned SD Prompt large language models to transform the storyline into text prompts for image generation in Stable Diffusion.

But that's not all! We'll also explore the integration of ControlNet, a tool that allows you to generate specific actions and movements in your scenes. Whether you're using Luma AI, Kling AI, or RunwayML, once image-to-video capabilities are enabled, you can use these starting keyframes to generate captivating video scenes.

Throughout the video, I'll walk you through the different settings and custom nodes in ComfyUI, including the face detailer, upscalers, and reactor face swap. We'll also discuss the importance of scene descriptions and character styles in creating high-quality AI videos.

Goodies For Patreon Supporters: https://www.patreon.com/posts/create-story-llm-107317948