Regional LoRAs in ComfyUI
Added 2024-12-14 13:56:10 +0000 UTCVideo - https://youtu.be/dASMuSCCNSU
Ever wanted to use one LoRA for part of your image, and another totally different LoRA (or model) for another?
Well, if you've seen the recent ComfyUI blog post about some new nodes for adding LoRAs to conditioning areas, along with various other nodes too, then you'll know this is now a lot easier to do!
The blog quite is quite long and goes into some depth, so instead I figured I'd start off a lot more basic and kinda work up from there ;)
Example workflow combining Redux & using a style LoRA on just one part:
Example output (workflow 8):
Links
Workflows
Loads of workflows this time! They are therefore numbered this time, starting off with the most basic and working up to the more involved ones.
Workflow 1 is a very basic version of regional prompting using the new nodes, with workflow 8 mixing both Fluxtapoz regional conditioning via the model, along with a second set of regional conditioning using a LoRA, like the example above.
Do note that the more conditioning you have, the more it's going to have to compute - like when you add a second conditioning to Flux in the form of a negative prompt.
Adding a LoRA to an area also then increases the time required, though this can be reduced slightly through use of timesteps.
Comments
Yes, also a good option!
Nerdy Rodent
2025-01-29 17:40:01 +0000 UTCthanks but in this case ill create new Cond Set Props and Cond Set Combine to link 2 clips from 2 LoRAs to it? cuz i see Cond Set Props take only one Clip Text input
Hussein
2025-01-28 18:22:47 +0000 UTC4_Flux-Comfy-Regional-Basic-LoRA_v0.json is the simple LoRA example. All you need to do is add both the clip and the LoRA hook to each character prompt
Nerdy Rodent
2025-01-28 17:56:39 +0000 UTCHi thanks for your informative stuff, but what if i have background and and have 2 LoARs characters models and i want compose them with the background using mask and region flux workflow, which workflow from these can help in that??
Hussein
2025-01-28 02:35:18 +0000 UTCHi NerdyRodent. Thank you so much for these workflows and the tutorial. Fantastic work! I would like in the flux Lora workflow to apply a controlnet depth, to the overall image. However in Comfyui the Apply Flux Controlnet node has as an output “controlnet_condition”, and not “Conditioning”, hence I can’t find a way to make it work with the Sampler node group in your example. Any chance you could guide me?
Swettkittens123
2025-01-03 15:47:40 +0000 UTCLooks like the default node has now been updated :)
Nerdy Rodent
2025-01-02 15:24:24 +0000 UTCThank you! It really works, but it takes a long time. I opened an issue (#6295) to address this or to learn how to improve the efficiency. They have already seen it (#6296).
blynx
2024-12-31 17:38:10 +0000 UTCI'd be tempted to say that's a bug with the default node, as it's the only node causing issues. It probably didn't get updated like the rest. For now, any default node can just be replaced with the usual combine with mask. I've added a v1 for example 4 where the default node has been replaced with the standard masked combine.
Nerdy Rodent
2024-12-31 16:32:19 +0000 UTCFor me, this structure is no longer working. The sigma update directly affects the 'model options'. Error message: "ComfyUI\comfy\samplers.py", line 227, in _calc_cond_batch finalize_default_conds(model, hooked_to_run, default_conds, x_in, timestep) TypeError: finalize_default_conds() missing 1 required positional argument: 'model_options'." I fixed it by adding it to the hook kf, but I’m not sure if this is the best solution, especially in cases where I only use one sampler. It also results in many more nodes. :(
blynx
2024-12-31 15:04:24 +0000 UTCThe numbers show the order they appear in the video, starting off with a simple example. As the examples progress, you can see how easy it is to just add a lora to any existing workflow you may have, as well as combining it with areas from the "model" rather than "conditioning".
Nerdy Rodent
2024-12-31 14:58:03 +0000 UTCWhat is it that you think needs updating?
Nerdy Rodent
2024-12-31 14:54:59 +0000 UTChi. are you going to update the workflows after yesterday's update? Fix Hook Keyframe 'guarantee_steps' behavior and add 'sigmas' (#6273).
blynx
2024-12-31 14:30:36 +0000 UTCWhat are the difference to the workflows a few videos ago? I am getting so confused by this many workflows :D
Danny
2024-12-24 11:37:03 +0000 UTCAnd also thank you for NOT saying: Today's sponsor....manscape. Oh gooooooooood.
Asyouare
2024-12-20 13:46:38 +0000 UTC:) And thank you!
Nerdy Rodent
2024-12-20 10:58:09 +0000 UTCDear Nerdy, Thank you for the past year. The effort you have put into this for the benefit and enjoyment of all of us. I hope everything works out well for you as the year winds down and throught the next. But after that....kidding.
Asyouare
2024-12-20 07:22:51 +0000 UTC4 now has an SDXL example, as requested!
Nerdy Rodent
2024-12-18 12:11:11 +0000 UTCThat would be great
Clayfacer
2024-12-17 16:55:35 +0000 UTCYup! Or three, or four, or...
Nerdy Rodent
2024-12-16 23:17:29 +0000 UTCCan we got two different subjects in the same image with regional LORA?
Samuel
2024-12-16 19:24:41 +0000 UTCYup! You can just swap the model loader to whatever - SD1.5, Pixart, SDXL, etc :) If you need me to upload an SDXL example, I can do that
Nerdy Rodent
2024-12-16 18:20:27 +0000 UTCYup, unfortunately :/
Nerdy Rodent
2024-12-16 18:18:42 +0000 UTCHi NerdyRodent, do we still need to disable xformers for the flutapoz to work? Thanks
EruoAuri
2024-12-15 19:24:09 +0000 UTCDoes this work for sd15 and sdxl? Any plans for workflows for those older models?
Clayfacer
2024-12-15 04:50:43 +0000 UTC
