Innovate Futures @ Benji

MimicMotion_workflow_img2img-RestyleFrame-v1 (For Patreon Supporters)

Added 2024-07-05 15:43:02 +0000 UTC

This is a tutorial and workflow for Patreon Supporter and Above membership level.

Based on the Youtube video on the public : ComfyUI Create AI Influencers Dance Video Wtih Minic Motion. https://youtu.be/oQT5AwReRgM

We have talked about how to use the basic workflow from MimicMotion.

This Tutorial , we are going to more advance and customizable features, by using MimicMotion to create a character motion video.

So the workflow are divided in 5 groups.

Reference loader - Load in your reference video, it can be a normal video or a DWPose stick man video.

img2img Restyle - This group are using for create an image for MimicMotion Ref_Img.

It is based on an image frame of the Reference loader group output. For example, if you have a complete face-front, full body shot of a dancer from the reference video, you can set the Load Video frame cap and skip frames targeting to the timing of the reference video to capture the frame. (This group only take the first image from the image list.)

Then the rest of this group is just img2img features, you can configurate the Text Prompt, Sampler to create image that you prefer. Of course, if you have good reference image already then you can skip this part.

But one thing to do before you create an reference image for Mimic Motion. Is to ONLY enable Reference loader and img2img Restyle group in the Group Bypasser node. Until you get a good image you want, then we can move on to the next step.

Next step , Disable the img2img Restyle and Enable all other groups

Copy the generated image from img2img group, Copy to Clipspace

Then right click and choose Paste (Clipspace) , on the Load Image node in MinicMotion Output group.

In the MinicMotion Output group, we are ready to rock and roll.

You shouldn't have much setting need in this group, the only thing to be pay attention to is, Load Image for reference image, and Pose Images from Load Video.

For the reference pose image (the reference video), here is something you have to be aware of.

As I have a Note in the workflow "Use Img_ref_video_resized if you don't need SEG group

Otherwise , use Mask_Img_SEG_Obj"

So in what scenario do we need to use Mask_Img_SEG_Obj?

Answer is when there's multiple people on the screen in reference video.

As I try it, running this framework with multiple people on the screen. In some cases, a dancer as the main character on the front, and some people walking on the background.

It will also affect the DWPose to automatically get pose of those people into the stick man pose.

This kind of background will cause MimicMotion Pose node getting wrong stack of Pose array.

Therefore, we need a Segmentation group to take in before pass the reference video - image frames into the MimicMotion group.

I have tutorial and many of my AnimateDiff workflows have this group. You guys should be familiar with this one :)

So once we have segmented and get the only character we want. It will be safe.

Because MimicMotion are only take the DWPose stick man from the reference video image frames as the output motion guideline.

So the look and style of the reference video, it doesn't matter anymore, not like AnimateDiff V2V.

Lastly the MimicMotion group pass data to Faceswap Upscaler group , then the whole process is done.

I find out this framework take a lot less memory consuming than AnimateDiff.

I am able to generate a 18 second video, 1 take without setting frame cap. (And I am using GTX 4090) It can consumed over a thousand image frames without lagging or freeze my PC.

And next, what I might do for this workflow.