Using Reference Frames to Create Complex AI Action SequencesUsing Reference Frames to Create Complex AI Action Sequences
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Wow, hey. It's me again. Surprise surprise! I want to talk about working from reference. There are some things in this world that no matter how hard you try a model is going to struggle with. Most of that is going to involve complicated motion (and in the case of Seedance, oh buddy, I dare you to ask it for a character to point with anything but their index finger or keep a thumb-p position). Most models work from fixed points. You tell the model where the foot goes, that's where the foot goes. And you can create some absolute nightmare fuel asking for things to happen when that foot is fixed in place. It's a good idea when you're forcing a model to turn away from the camera to specify '180 degree turn' or '360 degree turn' otherwise you're going to get a shrug and have wasted ten minutes generating. I love action sequences (surprise surprise, take a look at my portfolio). And one of the thing that took me maybe three hours of generation/editing was a jiu-jitsu turnaround into an armbar. Skip to around :55 to see when the jiu-jitsu starts happening. That was accomplished with a LOT of cheating and a few jiu-jitsu reference images functioning as reference frames (eg: image2 references the configuration of the subject's bodies in an ARMBAR; [then I defined what an armbar was/listed limb placement so I could refer to it in the future). The rest was combining shots with the model attempting to get the subjects INTO that configuration naturally (and a few cheats here and there to fake a full camera turnaround). These models can get GRANULAR if you let them, and the more you give it to work with (character references, reference poses, style references, lighting, previous frames) the more you can make them work for you.
Noel Dario's avatar
Complicated motion from reference is still where video models break. Fixed foot placement and default pointing poses match what I see when blocking shots before generation.
Steven's avatar
Oh absolutely, also if they have anything in their hands? I'm not sure why but so many models love filling hands with TWO things, and then teleporting that object to nowhere if they need to do something else. When working with seedance using time-codes helps a lot like: [00:00...
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started