Several AI art sites have started allowing the generation of short (5-10 second) videos in recent months. The word limiter in the Copilot version has made it… difficult to user for my purposes. The OpenArt version, however, has an extremely useful feature mitigating that. You give it a static image as a starting point, and then tell it what actions you want to happen. This means you can get the look of your character(s) just right thru traditional AI image generation, and then give that to OpenArt as a starting point and add in the motion. Below is the results from my initial experimentation with this new capability
I started off by experimenting with Jasper – the Gnome adventurer slated to work his way through BG3 later. I’ve seen some of the way motion can bring out personality in actDave’s blog, and I wanted to experiment with using that to introduce a new character. I left the model at the default “Kling 2.1” – I will experiment with some of the others later. So my baseline image for all of these will be my favorite render of Jasper:
First I wanted a simple introduction – the kind of short video you might see upon selecting a pre-gen character in an action game. It works, but its not stretching things very much
Kling 2.1: “Midget summons wispy blue magic while studying viewer”
Next I wanted one with a bit more of a story. This one is intended to show Jasper conversing with a squirrel as if it were a companion. And once again, just a few words were necessary to get a very satisfactory result.
Kling 2.1: “Midget kneels and converses with excited squirrel”
Here I wanted to try out a simple special effect – vanishing/turning invisible. And again I got an extremely satisfactory result with a very simple description.
Kling 2.1: “Midget vanishes in swirl of misty blue magic”
And now we start to get weird – as this “dagger” behaves more like a paper cutout than a real blade. Not sure if that is because there is no dagger in the original image.
Kling 2.1: “Midget tosses and catches a fine dagger”
This one has the “dagger” behave in similarly odd ways – but in a manner that supports my description of a fighting style leveraging illusion and slight of hand – so it worked for my purposes.
Kling 2.1: “Midget draws and flourishes a fine dagger”
Now lets try stretching things a bit more, by playing with Darius and company. This is a scene where I wanted Darius arguing with a mocking, hostile Guardian of the Bloom in a dream. The first take ends up with Darius doing all the arguing, while the Dryad appears to be ignoring him.
Kling 2.1: “Bard argues with mocking fey spirit in dream”
For the second take, we emphasize the Guardian a bit more. This seems to result in a better representation of the dynamic I wanted.
Kling 2.1: “bard reasons with a dryad in a dream as she mocks and belittles him”
And for a bit different take, we change the descriptors a bit more to emphasize their attitudes:
Kling 2.1: “thoughtful bard reasons with a dryad in a dream as she angrily mocks and belittles him”
Different, but not sure I didn’t like the previous one better. However the original image doesn’t really emphasize the emotions and I’m not sure how much that biases things.
Now lets push things a bit further and play with combat action… Here’s Kwan fighting with a ghost:
So we’ll describe the combat a bit and see what falls out.
Kling 2.1: “man with magic sword battles ghost”
That looks decent, but awful fuzzy with the details. What if we try the same thing with a different engine? We’ll try with “MiniMax Hailuo 2” – which claims “handles complex scenes with extreme physics and motion”.
MiniMax Hailuo 2: “man with magic sword battles ghost”
That’s certainly clearer. The only thing missing is better “on hit” effects – but it is a ghost. The “creme-de-la-creme” of video generation is supposed to be “Veo3” – but that costs a *lot* of credits. “Veo2” is much cheaper, so we’ll try that – again keeping the same prompt.
Veo 2: “man with magic sword battles ghost”
Well that was… utterly worthless. Especially for the cost. Not going try that again soon.
And now attempting something similar with Maja – back to “Kling 2.1” – because its cheap.
Kling 2.1: “man with magic sword battles ghost””woman dodges battling large owlbear”
The motion looks pretty good, but its again fuzzy on the details. We’ll try “MiniMax Hailuo 2” again – since that worked nicely last time.
MiniMax Hailuo 2: “man with magic sword battles ghost””woman dodges battling large owlbear”
Definitely better on the details – though it looks like the owlbear stabbed her in the back with his thumb pretty well. How about a more vivid fight description?
MiniMax Hailuo 2: “man with magic sword battles ghost””woman leaps back swinging axe as owlbear swipes at her”
So that…. got just plain weird. The first half was pretty good, and then it just lost spatial integrity altogether.
And now lets play around with magic using Angelique. First with healing.
MiniMax Hailuo 2: “woman lays her hands on a dying man to heal him with glowing magic. Her eyes blaze with a golden light.”
Not bad, but she doesn’t actually touch him and his visible injury doesn’t change.
MiniMax Hailuo 2: “woman lays her hands on a dying man to heal him with glowing magic. His wounds disappear and his eyes open, Her eyes blaze with a golden light.”
Better – but she still doesn’t touch him and his injury still doesn’t go away.
How about some water magic?
MiniMax Hailuo 2: “Sorceress summons huge tidal wave with wispy blue magic to wash away running kobolds.”
Interesting. A little too “cartoon-y”, but interesting. How about something more special-effects oriented?
MiniMax Hailuo 2: “Sorceress summons icy magic winds to freeze and blow back cultists”
Now that certainly looked better than the still shot!!
And to wrap up, how about a personality shot? I love the byplay between Maja and Kwan – who have very different outward personalities and love needling each about it. That can be hard to show in a still shot, but in video? Let’s see what happens.
Kling 2.1: “Man excitedly tells story while woman rolls her eyes and scoffs.”
Well that certainly didn’t get the relative emotions right. How about “Hailuo” with a little different description?
MiniMax Hailuo 02: “Korean man excitedly tells story. Maori woman rolls her eyes in disbelief.”
Much better. Much closer to the intended interaction.
Still lots to experiment with, and the long generation time (up to 2 minutes) and limited number of “credits per month” make it difficult to explore as quickly as I can with still images. But this has definitely got some potential!
Leave a Reply to ZenoCancel reply