CTRLorALTer: Conditional LoRAdapter for Efficient Zero-Shot Control & Altering of T2I Models
AuthorsNick Stracke, Stefan Andreas Baumann, Josh Susskind, Miguel Angel Bautista Martin, Björn Ommer
AuthorsNick Stracke, Stefan Andreas Baumann, Josh Susskind, Miguel Angel Bautista Martin, Björn Ommer
Text-to-image generative models have become a prominent and powerful tool that excels at generating high-resolution realistic images. However, guiding the generative process of these models to consider detailed forms of conditioning reflecting style and/or structure information remains an open problem. In this paper, we present LoRAdapter, an approach that unifies both style and structure conditioning under the same formulation using a novel conditional LoRA block that enables zero-shot control. LoRAdapter is an efficient, powerful, and architecture- agnostic approach to condition text-to-image diffusion models, which enables fine-grained control conditioning during generation and outperforms recent state- of-the-art approaches.