EP 329 – The Future of Storytelling in Film: A Hybrid Approach of AI and Human Creativity – Jud Willmont and Jacky Lin

Read the best-effort transcript below (This technology is still not as good as they say it is…):

Michael Waitze 0:01
Hi. This is Michael Waitze, and welcome back to the Asia Tech Podcast. We are joined by Jud Willmont, the owner of and Producer and Director at Willmountain Films, and Jacky Lin, a Founding Partner at Creative Fitting. Jud and Jacky, thank you again, both so much for coming to the show. Let’s give our listeners a little bit of each of your backgrounds for some context. And Jacky, let’s start with you

Jacky Lin 0:23
Sure. Hi everyone. I’m Jacky Lin, and I’m the founding partner of creative fitting. And creative fitting is a actually a company, AI company, founded in 2001 and we are formed by a group of AI artists and create creative experts. We found it in Shanghai, and what we’re doing is we’re developing our own generative AI video models for high quality video generations. So this is basically my company, but actually I started my career actually in an audit firm in 2009 Yeah, like 15 years ago, I work in KPMG, Shanghai and New York office, then doing the auditing of those like the funds, real estate fund, alternative funds, PE funds. And then I joined a corporate private equities and doing global investments, invest in Brazil, in Europe, in the US Australians. And after that, I decided I wanted to start a startup company. And so I do that. In 2017 I started a co working space in Shanghai. We operate 20 locations, 20 communities. And during that time, I work with a lot of creative people, like designers, advertisements, self, media, Kol art exhibitions in my co working spaces, because they are, they are the biggest clients for co working spaces by then. But that started company. Actually, it’s really not a creative business. It’s really a real estate business, yeah. And luckily, luckily enough, I quit my I sell this company, and I quit the real estate business right before the pandemic, and I become the CFO of a US listed tech company six years ago. So where I met, where I met the founding partner of Creative Fittings, so that’s why I’m doing this whole AI tech company things until now.

Michael Waitze 2:21
I love it, pretty much it. I love it. I love it. Great timing on your part, by the way. Jud, how about you? I feel like you’ve been doing Willmountain Films like since the day you were born. But go ahead,

Jud Willmont 2:33
yeah, kind of, in a way, I, let’s see, I, I can’t. I’m a filmmaker, and I’ve been making films in China now since the late 90s. I moved to Shanghai in 1997 pretty close to after I graduated college. Will mountain films is named after my father’s company, which was will mountain press, and he used that to publish books. My interest in China, initially, you can say, was primarily inspired by my father. He was a scholar of Chinese philosophy, and also a practitioner of Chinese medicine, and I was always kind of interested in that. And he got me interested in things like Tai Chi and Shaolin martial arts and and that’s what kind of brought me to China, I suppose, but, but my more core, deeper passion is really film and storytelling. And I mean, I just, I just find it so thrilling, both to to make a film and share it with people, but also even just to watch other people’s films. And so I think I’ve spent my time here trying to figure out how I can do that. And I think will mountain films is the vehicle for for me to achieve that.

Michael Waitze 4:12
Is there something unique about living in China since 1997 I should tell you this, like my first time in Shanghai was 1991 I want to say maybe 1992 right? So you’ve seen but I hadn’t. I did not go back until 2017 so I missed, like all of those changes when they were voluminous, right? And massive, both economic and just structurally, in the whole country. Is there something unique about being there and telling stories in China? Maybe not necessarily from a Chinese perspective, but you know what I mean? Like being in that environment? That environment and then telling those stories?

Jud Willmont 4:46
Yeah, I think there. I mean, for me at least, I think there is, I remember the like, I remember the first time we stayed in some small, but also very kind of iconic little hotel over by the. And and it was high ceilings, hardwood floors. It was definitely something from the 30s. I think it was called the Astor Place. Everybody who was famous in the 30s had stayed there. And I was just looking out over what was then the early development of the lujad financial district. And it hardly had any buildings at all. It just had the one Pearl Tower and a few others. And it just, I remember feeling that this, this is what I think immigrants who landed in New York in the 20s and 30s were feeling it was the beginning of something big, the beginning of something new, yeah. And I don’t know for me, Shanghai has that history in it. It is, it is a Chinese city, but it’s unique in that it’s an international Chinese city, unlike any other city, and comparable to places like New York and Paris and London,

Michael Waitze 5:58
yeah. And I mean, if you look at its similarity actually, is that it sits on these rivers, right? And it because it was a trade gateway. It has very similar personalities, like you said, like New York, London and Paris, all which have rivers running through them because they were main gateways to the rest of the country. And if you and both of you know this, right, but historically, the Bund was the place where all the kind of international people not gathered, but kind of had their own little sections. I stayed at the Peace hotel, so kind of a similar experience to the Astor, I’m guessing. Wow. What an incredible experience. Sorry, go ahead.

Jud Willmont 6:34
And, yeah, no. And I think that people that have come since then, that I’ve, you know, had the pleasure to entertain and and review and re experience that that moment again through their eyes. Because I think that’s consistent. A lot of people have had that experience, and it just felt like there was this raw energy. People were friendly. People were coming from all over the place. You know, inter intercultural experiences have always been really interesting for me to want to have and to want to seek out. And Shanghai’s just been a great place where that’s going to happen, whether you’re looking for it or not.

Michael Waitze 7:18
Yeah, it’s a pretty dynamic place, and it kind of always has been, right? I think, I think if you actually ask people what HSBC, what HSBC stands for, none of them would know. But it’s like, it’s a hint, yeah, it’s a hint of the history of Shanghai and what it used to, what it used to be and what it is now, right? So, fair enough. I want you guys to talk to me about this new film you’ve made M, a D so mad, which I’m guessing is mutual assured discretion. And I want to talk about its entry into this global AI film marathon competition as well, which is held by what Shanghai film. Can you both talk about this? Maybe just give me some insight into this, like how it started, why you decided on this story. Just talk to me about this.

Jud Willmont 8:02
Okay, yeah, I’ll, I’ll tell it from my point of view, and then Jackie can share his point of view. Because, because, you know, in line with what I was just talking about, how, you know, it’s a great place to meet people and do new things. I mean, I met Jackie because we chose to make this film together. I found out about it through some friends and some people that I knew Who knew that I was interested in making films and working towards doing those sorts of things, and they shared this, this announcement with me, which was basically the announcement of an AI generated short film competition. It was pretty open in terms of what that meant. It could be full AI, or it could be like a hybrid. I think this, I think the requirement was 50% of the film had to be, had to involve AI in some way. And I had been curious about exploring AI specifically in the context of an animation project that I’ve been working on trying to develop for the past few years. And I remember a year ago we we started to see some really phenomenal breakthroughs with what the potential for that could be, but it still felt a little bit limited, and it wasn’t 100% where I thought it would need to be to really get into it, but I thought, if nothing else, it would at least be worth exploring and experimenting with. And then this AI short film competition came up, and it just seemed like, Oh, this is a good opportunity to test some things out. Let’s just have a try. And just by fortunate circumstances, we happened to meet some people from Creative fitting, and they were doing a demonstration of. What they were doing with AI, which was also animation related. And I just thought, oh, okay, this is it. This is the opportunity. Let’s go talk to them. So,

Michael Waitze 10:08
Jack, do you want to talk about what you were working on, the stuff that? Yeah. Jud, so excited. Yeah. So

Jacky Lin 10:14
this is a very interesting part, because when we talk about AI and AI videos, and most of most of us are from the technology perspective, who develops the foundation models like open AI, who like train their models and and from the technology perspective to create the best quality movies they believe they can do. From like those people are scientists, those peoples are algorithms. Are very good at mathematics. So when we talk about this AI films, basically, we have been making this AI videos for like, three years. And along this three years, what we found is the AI is progressing very fast. Every month we have new progresses, but still, the AI video is behind, still, behind the best quality of the human creative list. There’s no way that the AI created thing can be a film, for example, right? So when we do that, actually, I also received a notice from the Shanghai film group saying that there is a competition in AI, and which is the first time Shanghai film group is doing that. And even they will have a section of the AI film in their Shanghai International, international film festivals in June. So this is the very first time that Shanghai is doing that well. Well, at the same time, maybe two years ago last year, runway in is in the Silicon Valley is doing their first AI movie film very shortly, just like two years ago. I don’t think it’s a very long history, and most of the competitions, all of us are AI enthusiasms, AI technicians. So at that point of time, we wanted to do our we wanted to have our own team and to present what is the best videos that we can generate using our video models? And one day, we went to a seminar, actually a seminar. There’s nothing to do with AI, there’s a short drama seminar, right? So we went to a seminar, and we met Jud and was saying that, hey, why do we explore something that is a combination with the human creativities and AI, and we can do something really, really different. So I think, yeah, that will be very, very exciting idea for us as well, because that’s the opportunity to cooperate with the directors and filmmakers. And I think Jed will give us a lot more inputs, very unique inputs, to differentiate us from other AI competitors. So yeah, that was the time in last month we decided to form a team come together and make a film for this Shanghai Film Festival.

Michael Waitze 12:59
Can you talk a little bit more about the technology that runs this? I mean, your team developed this themselves, right before we get back into the storytelling part of it, I’m really interested in the tech that underlies all of this, right? And I do like this idea of where technicians, we’re scientists, we’re building this technology, but we need, kind of like a storyteller to be able to actually affect this thing in a way that’s really proper. But go ahead, I’m curious. Ahead, I’m curious about the tech as well.

Jacky Lin 13:24
Of course, the generative AI actually started, like just few years ago. We founded our company in 20 Oh, 2021, at that point of time, there’s no such generative AI video model, but there were generative AI language model, which is called GPT one and GPT two and only but, but that was not famous among the population, just the GPT one. GPT two is only famous among the scientists, only until 2022 when chart GPT has become a to see app, a famous app. Everybody is talking about the generated AI but by then, there’s no such thing as that, but we started to do our AI research in AI models to generate videos. So basically a video is series of pictures. So basically what we’re doing, we use the foundation of the diffusion models, and then we build our own network architectures. We do our pre training of the datas, and then we merge it into a models that can generate videos, like, frame by frame. And so the longest one,

Michael Waitze 14:33
sorry, does it generate the entire scene in the video, including, like, what the humans look like, what the background looks like with, you know, the swaying plants and all that stuff, it generates the entire thing. And it generates the whole video. Yeah,

Jacky Lin 14:44
so right now it’s because this is based on the like the big diffusion models, we can generate up to 10 seconds videos and but the but the unique part of it, we can keep the controllability blissing and consistent. See of the video clips so we can make like so this is good for Jud. So Jud wanted to do a storytelling video that needs to go beyond 10 seconds. You have to be like one minutes, 10 minutes, or even 100 minutes, right? So our own model is specifically built to make sure that the the clip by click consistency is high enough so the first in the people in the first clip will be the same people in the second clip, in the third clip, in the fourth clip. So goes all on and on, and that consistency is this is the own priority. And this is done by our design of the architecture and pre training of the data, because you can see a lot of the generative AI models doing now they do a fancy pictures, and then it kind of becomes blur, and then the scenes change a lot, change a bit, and even the clothes change a bit. And if you go beyond like 10 minutes, you can even even notice that even the human, the people change changes.

Michael Waitze 16:06
Well, yeah, and a lot of the stuff that you see getting generated, at least for the images you see people with like six fingers, or fingers missing, or just all that kind of discrepancies of what a real human should look like. So the fact that you can do videos that are consistent for more than 10 seconds is kind of amazing. Can we, yeah, Jud, can we talk about the story? Mad and again, I think this is uniquely Chinese, right? So this benefits from your being in China for what is that, almost 27 years now?

Jud Willmont 16:36
Yeah, yeah. Well, the so the competition required us to choose between either going for a fantasy genre, which would have been something based on the mythology in the world of nuja, which is a Chinese character that many that people are very, very familiar with, and there have been already several movies made on the ninja character, so that’s pretty well explored. And then the other option was to do a science fiction based on a writer, and all they told us was who the writer was, but they also said that only some of the writer’s IP would be available to work with, so we were able to read some of the writer’s previous works and kind of get a taste for what he was about. But we didn’t know if that would be the story that we would be adapting or not, but I kind of knew that I wanted to do a science fiction anyway. And again, like I said, it had a little bit to do. I did have some sort of like personal hidden agenda, which was, I did take this on as like an experimental opportunity to see if it could, if it could link in with what I had previously imagined for an animation project I had in mind, so So the science fiction fit right in with that. So I immediately knew I wanted to do a science fiction and I also immediately knew that I wanted to do more of like an animation, like like a flat 2d or a manga style animation, not like a Pixar 3d style animation. The the really cool animations that are out now they’re they’re really a hybrid anyway, they’re using 3d environments and they’re using 2d looking characters, but they’re really throwing all the tools at it to kind of make it look the way it looks. I think the Spider Man into the spider verse is a good example interesting of this hybrid technique. There’s a lot of other stuff that I’ve been interested in Japanese manga and anime since forever. So I kind of already knew I wanted it to be a little sci fi. I already knew I kind of wanted it to be a little manga. And so that’s kind of what I came to these guys with, was that in mind, without really knowing anything about what they would be able to do. I remember I knew a little bit about stable diffusion. And like, if you shoot actors, you can use stable diffusion to rotoscope them and turn them into animations. And I think that was like, the first thing I asked them, and they were like, no, no, no, no, we want to do complete, 100% generative from the script. And I was like, Okay, that’s cool. Let’s, let’s play around with that. And and then it turns out that it’s that there’s also a lot of art development involved in between as well. So it wasn’t just we write a script and then it automatically becomes an AI image. There’s, there’s, there’s illustration work that is also done in between. Is the collaboration like I was going to be able to get this look that I wanted. So I was kind. Of excited about that. So

Jacky Lin 20:01
yeah. And there in my Yeah, there in my desk, jump in. So

Michael Waitze 20:05
what you’re saying is you do some kind of, what I’ll call offline or kind of manual drawings, so that you can get a sense for what you want it to look like, what the what you want the characters look like. And then Jackie, you’re suggesting that you can then feed that into the generative AI, this diffusion model that you’ve created. And it can take that and say, we’ll make it look like this, but now we’ll make it move and give it life.

Jacky Lin 20:27
Yeah, basically it is, yeah, exactly right. There’s

Jud Willmont 20:33
some right, like, Jackie, you can talk to them about, like, the training that you had to do, you had to train it to, like, understand the characters. And we it was pretty ambitious. I think I gotta say it was like nine characters in our little formal film, which was, I think probably more than would have been ideal, like if we had just done like two or three characters, we could have had more time to train the AI with, with just those three characters, we probably could have gotten more movement or more interesting movements or more complicated movements out of them, but the story, you know, again, I think story is first for me, at least story still comes first. And when the competition finally kicked off and we had already decided to join forces and be a team, the the the books were released, I think three of Hershey’s books were released, and we were allowed to pick any one of them and do an adaptation, but the film could only be three minutes, so I had to. So I made my decision. I picked the the sea of wars, just because it felt like there was I didn’t have, I didn’t have the time to read all three of his books, so I just had to kind of pick one. They gave us a synopsis of each one. I read the synopsis, and the sea of wars just appealed more to me. It was there was Aliens, there were rebels, there was the Federation, there were caves. There was a base on the Moon. It was like, oh, yeah, this is it. This is what we’re gonna make.

Michael Waitze 22:08
So, Jackie, did you want to talk about this training?

Jacky Lin 22:11
Yeah, and bear in mind that this is a time limited competition. We received the topics, I think it’s later in end of April, right or, or fifth on, on, on all the first of May. And we have only three weeks to do all this stuff, all the designing, all this story writings, and even for the AI to generate all this screens and pictures and finish the all video. So this is a very time constrained competitions. So right now, actually, from the AI perspective, there are two type of pipelines that we’re thinking about, just compared to the traditional ones. So if you’re all human, traditional ones, what you do is you do your character designs, you do your background design, you do your story writing, you design everything, and then you ask, probably the the How to say, the human to draw every frame of this of of the enemies. So that takes a lot of time for for people to draw it. Story usually takes, like, not just story, from storyboarding to the actual enemies. Oh, the enemies you need to draw every, yeah, you need to draw the every frames in between, every cameras, every, every, every scenes, right? So that will be the most time consuming part of that. So right now there’s, there’s two ways of the AI can thinking of doing that. And the first way is, some of the AI are thinking of, they start with writing the story. They start with creating the designs by using AI, and then human do the adjustment after that. For example, human may have to do a lot of minor modification of each scene each each picture they generated. So ai do the design first, and then human just do the letter one, there’s a some of the other. AI are doing the way like it is, but from our perspective, we wanted to make the better movies. We have it turn around. We ask a human to do the created creations. We ask a human to do the designs for the storyboards, and then the AI just do the heavy work. The AI just fill in all these frames. They move those pictures, because that is the the work, which is, I would say, most tedious and hardworking. So we kind of collaborate with Jud in this, in this way, how do you

Michael Waitze 24:42
can ask a question, sorry, Joe, do you mind? How do you like what I spent a lot of time and what I do is simple, right? So just take it from a person who does very simple stuff using AI. I think I told you I created this background using mid journey. It’s very simple stuff, right? But learning, and I do writing with it as well, at least. Learning how to tell the model kind of what you want, at least for me, has become half the battle, right? You can’t just say, like, write a story about, you know, nuclear war. You’re not going to get anything great. It’s a lot of people confuse the fact that all you have to do is tell AI, like, write a story about nuclear war, and then it gives you, like, 17,000 pages of stuff. That’s not the way it works. But how do you tell the AI to like, here’s the scene, but here’s the scene from a different camera angle. Get the back of the head. You know what I mean, all these things you would do if you were a director and a producer with just physical people, like, please move over there a little bit. Like, how do you do that with the AI?

Jacky Lin 25:36
So right now, in real diffusion, which is the model we develop, we have those control options in terms of the cameras and also in terms of the movement of the people. So right now, our model is specialized on controlling the emotion of the people and the action of the people. So this is what we’re focusing on. The reason being that we prioritize the huge the characters movement. Because I think that’s more important in this point of time, to do a good drama, to do a good videos. Because right now we see a lot of very nice AI. They can produce a very, very grand picture of the of the of the movie. Movie feeling like you have a desert, a very, very large desert, a spacious like the universe is very big, very enormous. But in terms of storytelling, characters are more important. So we train we have our model. We have the control abilities to do the emotion controls to do this action control of the characters. This is the first part and second part. Our model also is a multi modality input. They can we can input languages, input Texas. We can also input pictures, when pictured in terms of scripts, the storyboard, so Jud can also draw something on a paper and showing that what is the the camera would be looking like, right? Um, just a simple stroke. And then we can fit into this model. And our model will generate a very approximate to that kind of, the kind of cameras for the whole for the whole clip, for the 10 second clips. So this is how we achieve controllability and consistencies.

Michael Waitze 27:23
Jud, were you blown away by this? Do you know what I mean? Because as a director and a producer, like all the things you have to do for a human like, Come on, just make this kind of expression and give me this kind of emotion with the multimodal controls that they have. Did it kind of change the way you think about directing? If you know what I mean?

Jud Willmont 27:39
Well, I I can say this about that, one of the more interesting aspects of the project was when we were, let’s you can say, looking at takes, for example. Can I ask you about, yeah, yeah. So it so, like, I mean, but also just so it’s clear like we did really design the storytelling upfront, you know, by humans. So the AI didn’t write the script. I wrote the script and then I did storyboard it. But I think, you know, the the thing that really we wanted to push and the thing that we really wanted to see was as Jackie was talking about the the ability to convey emotion, because that’s always the thing that you’re looking for as a director from your actors, and that’s the thing that you need to be able to do with animation, with your illustrations. And it’s and it’s one of the reasons why I’ve always been drawn to the rotoscoping treatment in animation, because you can actually shoot actors and and get their emotional reactions and then copy that into your your cartoon. And so this, for us, was kind of a test to see how much of that could we get with AI and and I think also what Jackie’s talking about with, like, a lot of the stuff that you see, it’s, it’s true, you get, like, these amazing rendered scenes, right? And it’s like, wow, that’s amazing. But it’s like, it’s almost just, like one frame from what could be an amazing film, right? If the rest of the film existed, but it doesn’t because, you know, they just did the one frame, and they could never do the next one or the one after that. It because, like, what Jackie was saying, with the consistency, it’s just not there. And so for us to maintain the consistency, we we trained it with the characters, and then we designed the scenes, and then in the scenes what we needed the it’s dialog driven. Most of the AI short films that we ended up watching, they’re all like some kind of voiceover. So, you know, you don’t actually have characters talking to each other, but that’s what we wanted to do, and that’s what I would. To do if I was going to make a feature length animation and use AI to do it. So I wanted to know how far we could go with this. So when we had characters talking to each other, and there’s like a heated moment, and the one character saying, We got to do this, we got to do this, and I’m looking at the takes, right? So they’re feeding the AI, the the prompt, and I’ll be honest that I there was a little bit of a black box treatment, like, when I made the decision to work with these guys, I I, I treated them like, the way you would treat a CGI studio, right? Like, if you’re going to go work with the CGI studio. You just need to know what you want. As a result, you do not need to learn how to like program Maya. That would be ridiculous. So,

Michael Waitze 30:52
yeah, I want to get on the plane. I want to get on the plane the AI

Jud Willmont 30:58
word, yeah. And I think that’s another interesting thing too. Is Like, there are going to be like experts who get really good at working with AI, and then there’s going to be creatives like myself who maybe aren’t going to learn all the AI tricks and trades myself, but I’m going to work with other people who are good at that. But going back to your question, and looking at the takes, I found this super fascinating, because you would get some really bizarre characterizations, like things that you couldn’t even imagine. And I remember at first, they were showing me some of the takes, and I was thinking to myself, ah, this isn’t that great. It’s a little static. I want a little more dynamism in this and and so I brought that up, and they said, Well, when we add more motion, would we give it more freedom to be more expressive? Sometimes it comes up with some crazy stuff. And I was like, oh, yeah, let’s have a look at that. And I loved it. I thought it was the coolest. And they were like, what really? That’s crazy. That’s not what I was like, Well, I don’t know what we want. Let’s see what it has to offer. And let’s, let’s be a little bit more open minded about it. And I think, yeah, I think that was a fun experience for all of us.

Jacky Lin 32:23
Yeah. Because, you know, every time AI generate a video, they got randomness. You got random stuff. And some of the stuff is good, and some of the stuff you may see, oh, this is this. Face is broken. You have six fingers and blah, blah, blah, you have all the things. So all this good takes, like we give to Jud. We we selected those good takes that we give to jet the thinking, Oh, this is how the good takes. These are the consistent ones. And InsurTech consistentness, we make sure, for example, the nodes are here are exactly or almost where they are. And then we get this Jud just saying that, Oh, no. Well, no to statics. So they look at some of the clips that we discarded. For example, one of the one of the clip, even the face, are all blur that you remember is bachelors, someone like turning his head, and he’s turning his head.

Jud Willmont 33:18
Yeah. Go. Ghost image of the face, like the features where they weren’t supposed to be as he turns his face and then it snaps back. That’s kind of cool. That’s kind of cool. I wouldn’t have imagined, yeah, that way, but why not? And it actually fit the mood of the scene. So, so it kind of generated a new idea for me, which was that, like, this isn’t just going to be a tool that you’re going to be able to control and and tell what you want, and as time goes on, certainly it’s going to get better at doing that, but it’s also going to be a tool that’s going to surprise you and and show you things that you hadn’t thought of and and that’s going to trigger new ideas, and it might even create a new esthetic.

Michael Waitze 34:11
Yeah, so here’s, here’s the I want to make kind of a general point. I want both of you to comment on this. You talked about this idea of hybrid, right? And I love this idea that you pointed it out on very specific I wrote the whole script. I conceptualized the whole thing. The AI did not do that. And the idea, the maybe the heavy lifting of maybe creating some of this stuff. But there’s also this hybrid experience that Jackie kind of touched on at the beginning of there are a bunch of scientists and sort of technologists that are working on something. You give them the idea. They go run it through the model and get it and get it. They go, did they go, Jud’s never going to want this thing because it’s too insane. And you’re sitting there going, I hope I get something insane. And that’s also a hybrid experience, right? Because you have to match what scientists want with what the creative wants. And yeah, when you were saying, like, the face was turning and then the. Snap in. The only thing that occurred in my brain was that’s kind of cool, but it also made me think of something else. Jud, and I want you to comment on this too. I don’t know what year it was, right, because I’m, I’m not that gigantic of a movie buff, although I do love movies, but I can imagine a director telling Sean Penn when he was playing I think the character’s name was pacole and Fast Times at Ridgemont High and he was, it was it was like method acting on steroids when you come into the room, but when you come into the room, right? I want you to do this thing, but I’m presuming that what he gave based on what the director wanted could have been two completely different things, and I’m not equating it with the spinning head and the snapping into place, but sometimes the director can go that’s even better than what I anticipated, but you have to see it to understand it. Does that make sense?

Jud Willmont 35:44
Oh, yeah. And that happens all the time, and that’s great when it happens. And that’s exactly what you want to happen when you’re working with actors. So in this case, it was almost like the AI was an actor, yeah, I think I even said it at the moment, I was like, let’s treat the AI like an artist, not not like a tool. Let’s treat the AI like an artist. It’s going to give us its own interpretation. Let’s be open to what that is.

Michael Waitze 36:15
So Jackie, did it take you and your team? And you only have three weeks which, is insanely intense, right? But to take your team a little bit of time to figure out, maybe we can lean a little bit over here, like maybe if we want crazy, we can, and we’re gonna get crazy. Let’s show crazy to Jud. You know what I mean? To take time for your team to figure out, let’s not filter actually, insanity. Do you know what I mean? Yeah,

Jacky Lin 36:36
actually, that is one of the takeaway from our team, and this is why I love to get Jud onto our team as well. Actually, during the three weeks, we upgraded our models in the middle of the in the in the middle of the week too, in order to have a large, larger actions and maybe blur actions. But that’s what the director Jud wants, and eventually, probably five or one out of five may have something very nice, very creative. And that’s something we have to do, and put it in front of the directors and for Judge to pick it from. It’s rather that we decided, Oh, this is not good. This is blur. The geometric is not right. The nose is not there. The nose is like one millimeters off the original design. So this is one of the take which we think that what is a very good movie, very good take. It’s not exactly that from the mathematics perspective, a very, very consistent wave. I mean, we also updated our definition of consistency before. We believe consistency means the people looks exactly it’s draw designed before, like from their eyes, the eyes and everything is designed before. But after that, after this competition within consistency means the audience can recognize the characters from the start to the end. That’s the most important consistency.

Michael Waitze 38:07
So do you think that the learnings go both ways, where, if the technical team, again, just to make a broad generalization, can now get a better understanding of what the director and the creative wants, that during this three week intense process, that as a storyteller, you can now get a better sense for what’s possible.

Jud Willmont 38:28
Oh yeah, for sure. And I think I, I remember, I remember reflecting on that too. I mean, like, you know, with this one, I kind of did go in with, like, a fairly fixed idea in terms of what was the genre I wanted and the overall look. And I think, you know, on a frame by frame basis, we definitely got that. It’s definitely like this 2d style, like animation, that I’m kind of into, but I also recognize that there’s a potential for AI to do stuff that is beyond what you can predict and beyond what you can imagine. And so we didn’t just include that in our allowance for how it rendered the characters in their expressions and interactions with each other. I actually designed a scene into the film that was intentional, and, and, and the purpose for that was to actually give AI the complete and full freedom to do something and kind of wow us a little bit. Yeah, and that was, we fed it. We fed it a single oil painting that that an artist painted by hand. But I but I wanted it to animate. I wanted it animated. And it was incredible what we got. And, I mean, it was still the same idea. You look at a bunch of. Different takes, and you pick the one that you like the best, but it was like, I couldn’t have even imagined that. That’s what it would have been, other than just some surreal, impressionistic idea that might have existed in my head, but that I could never have articulated. And so AI delivered that to us, and I think that, yeah, if I wanted to do more, you know, it depends what your goals are, but if I wanted to do something with AI again, like today or tomorrow, and I wasn’t trying to experiment with a specific kind of animated look, I might, I might try something completely different. I might try something extremely surreal, or, you know, experimental and and do more to lean into what AI is currently doing. So AI is good at a bunch of stuff right now, and that would be fun to play with. We wanted to see if we could make a how far we could push AI to do the thing that it’s not currently that good at right?

Michael Waitze 41:05
And I think that’s what Jackie’s implying when he says that during that three week period of time, they updated and changed their models to do some other things, because they were like, maybe we have another idea about the way this can work. And I’m I can only imagine. And Jackie said this at the beginning of the conversation, that the development of these models is just moving so quickly right now, right like the up into the right speed is just so fast that three months from now, or a year from now, the stuff that this, that this model can do, is just going to be, you know, stuff that you couldn’t even have conceived of today. And I presume as well. And Jackie tell me where I’m wrong here, that the more things you do then you feed back into the model that the model then learns like even that was possible. Now we can mix and match that with other things to get outputs that you hadn’t even imagined. Is that true as well?

Jacky Lin 41:49
Yeah, that’s definitely true. And also, for Jud, it’s a very clever strategy not to learn how to operate those AI tools at the very beginning, I think that’s very smart. Because I think eventually, because the AI tool is developing so fast. So maybe right now, we need some prompt engineering skill. Maybe we need those pre training skill. We need those SFT things, and these are all engineering nuances. So as soon as the AI video model is developing so fast, maybe in the next generation the model solve all of this. You don’t need such prompt engineering. You don’t need such SFT things, but still, you need just decision on what is good, what is the whole atheist things. So I think Jud is really, really smart on deciding how to cooperate with AI at this point of time, and from my from our company perspective, we actually wanted to support, like all this, creators that they have their atheism, they have their views on this, but we can do those hard work with the AI. And eventually, when AI abilities goes up. We still have the the the competitive advantage of making the good stuff still in hand. Yeah. So

Michael Waitze 43:07
I want to make this point, and then I want to ask you guys one more thing, and then I’ll let you go. But as I remember when GPT three came out, and it was sort of widespread, usage started happening at the end of 2023 people started using these llms. And I know that’s not what you’re doing. The diffusion models are different. But still, once it became like widely accepted, a lot of people started getting worried about this idea of this is just going to take the job of all the creative people that are writing and coming up with great ideas. And my contention at the time was always, we’re going to live in a hybrid world, right in the same way that, like, I can carry a package from here to Chiang Mai, but it feels really inefficient. It makes more sense either to give it to the airplane, which is just tech, right, or to give it to a car, and no one seems to complain about that. And that’s a hybrid experience. And I think what you guys are telling me as well is that without Jud, there’s no story, but to create the story in a way that, in the visual way that he wanted to become that using the AI makes it way more efficient, in the same way that putting that package on a plane to get to Chiang Mai is way more efficient than me carrying it. There. Is that fair, right?

Jacky Lin 44:12
I think I will talk about in two ways. And first of all is, you notice chatgpt, it’s very popular. But I also have another esthetics, because most of the people talked to charge gpts, at least 90% of the people, they do not have to use GPT correctly. The most conversation having will be, hey, what’s the weather? And then what’s the point? Tell me something. And then the conversation stopped at route three. So most of the people using GPT is like that. It’s like right now, at least at this point of time, 90% of the people are not really into this things. Yeah. So this is the first part, and the second part is, imagine when AI is strong enough to create a V. Video or create an article without any human effort. So imagine there is a super, super strong AI. So what would happen to that content? So everyone can just these kind of content. These type of videos will flush over the world, and the cost of making it will be zero. But at the same time, the value of that video would go to zero as well, exactly, because no one will ever look at like, Oh, this is good, but I have millions or billions of video like that. So why do I looking that exactly? I wanted to look something unique. And what makes it unique is what human comes in, and human have their ideas, and creative it comes in. So I think that’s this. The biggest value lies in the future. Videos, okay,

Michael Waitze 45:49
I think that’s a great way to end. If you guys don’t mind, I’d just like to thank both of you for doing this. I’d love to have you come back. Jud Willmont, the owner of and Producer and Director at Willmountain Films, and Jacky Lin, a Founding Partner at Creative Fitting. This was fascinating. I remember when this was presented to me as an opportunity. I’m like, I gotta get these guys on, and I want to say this too before I let you go. You can feel it in both of your voices. Like Jackie’s not just a technologist who’s like programming and crunching. He loves this stuff. I can feel it and judge the ultimate storyteller. You can feel it in what you’re saying too. Like I have this idea. And I want it represented this way, and I think when people listen to this recording, they’re going to get that feeling as well. I really want to thank both of you for coming and doing this

Jud Willmont 46:31
well. Thank you, Michael, for having us.

Jacky Lin 46:34
It was fun. Thank you, Michael.