Innovate Futures @ Benji

For Patreon Supporters - benji-LTX-Video V2V-v1 [Workflow]

Added 2024-11-27 16:05:26 +0000 UTC

Key Takeaways

Video-to-video transformation uses unsampling/resampling instead of traditional ControlNet methods
Higher sampling steps (50-100) produce better color and quality without significant time increase
Detailed text prompts are crucial for desired transformations
Works best with short video clips and clear reference footage

Technical Setup

Download LTX model weights (2B parameters, FP16) from HuggingFace
Install latest ComfyUI version
Add ComfyUI LTXtricks custom nodes (no additional requirements needed)
Set video frame parameters:
- Configure frame load cap
- Adjust skip frames if needed
- Set image resize to match LTX requirements

Optimization Tips

Sampling Steps:
- Use 50-100 steps for best quality
- Higher steps improve coloration and reduce deformation
- Generation time remains similar (∼30 seconds)
Prompting Strategy:
- Write detailed, descriptive prompts
- Maintain scene structure while changing specific elements
- Focus on key transformation elements (character, environment, style)
Quality Control:
- Check first 1-2 seconds for stability
- Use clear reference videos with minimal motion blur

Example Transformations Tested

Realistic to Anime:
- Woman in coffee shop → 2D anime style
- Maintains motion and composition
- Changes character style while preserving environment
Character Transformation:
- Dancer → Ghost with fire effects
- Beach selfie → Different person/outfit
- Human → Monkey/creature variations
Environment Changes:
- Office → Futuristic spaceship
- Modern room → Dark castle
- Studio → Supernatural setting