Innovate Futures @ Benji

ComfyUI Workflow SEG And Unsampling AnimateDiff - For Patreon Supporters

Added 2024-09-27 13:00:17 +0000 UTC

Tutorial Video : https://youtu.be/J7nuNjDwVx8

Previous Progress Of This Workflow : https://www.patreon.com/posts/coming-up-i-went-1127

This is another AnimateDiff workflow in ComfyUI.

The purpose of this workflow is to understand how to use unsampling steps for v2v animation, and style transfer of the selected Segment mask region.

Using Unsampling method in the first sampling , we are able to create more consistency objects in AnimateDiff. Similar effect as RAVE as what I talked about before, but Unsampling consume less GPU power.

Visual object tracking with Point Editor

We have a very good tool, Point Editor for SAM2. So without text prompting in Florence 2 or so , and hoping the BBOX can be highlight the object we want to track.

Now, we can have a more effective and intuitive way, by visually pointing the object we want to segment.

SamplerCustom , Unsampling and Add IPA for Sampling

We are basically leveraging the IPAdapter and Mask for the style transfer as usual in AnimateDiff workflow we have walkthrough before.

Second Sampling - Refine edges on the SEG object

Like I mentioned in the video (https://youtu.be/J7nuNjDwVx8), the Umsampling method create consistency based on source video, but it's appearing diffusion noise in some part of the Segment object. (appear some flicker blue dots noise)

Therefore, we use CannyEdge and Blur Mask outline to refine the animation and solve this problem.

Follow the Tutorial Video how we configurate before run the generation process. You will have no problem. But this workflow is kind of advance, you have to understand what is diffusion noise, and how those connections on the workflow works before you can get some good result.

Workflow file attached in this post below, if you have question , feel free to post in the comment section.

We all keep learning everyday.

And Have Fun!