Revolutionizing AI Video with Story Diffusion Deepswap
Introduction
Story Diffusion is an innovative open-source AI model that creates videos up to 30 seconds long, ensuring an exceptional level of character consistency and adherence to reality and physics. Unlike previous models, such as Sora, which struggled with issues like morphing and the sudden appearance of extra characters, Story Diffusion offers a significant advancement in character consistency.
Character Consistency and Realism
Story Diffusion excels not only in facial consistency but also in maintaining the integrity of clothing and body type. This advancement paves the way for creating believable characters that remain consistent between shots and scenes, enhancing the potential for AI-generated videos and AI comics with tools like deepswap.
The methodology involves creating a sequence of images that are consistent in terms of face and clothing. The model then predicts the movement between these images and animates them using a motion prediction model. Some of the videos produced by this model are as long as 23 seconds, with characters maintaining their appearance throughout the clip. Although there is slight jitteriness and most videos are square, these are minor issues compared to the improvements in character consistency and clarity.
Impressive Examples and Use Cases
One notable example features a female character riding a bike. She appears anatomically correct with minimal morphing or disfigurement, and her facial expressiveness is particularly impressive. No other AI video generator can produce videos as long and consistent as those from Story Diffusion, even when using deepswap technologies.
ByteDance, the company behind TikTok, is cited on the white paper for Story Diffusion. While the resolution of these videos is not specified in the white paper, previews are rendered at 832 pixels by 832, which can be upscaled to at least 2k definition with an AI upscaler. The lifelike movement and facial expressions of characters are particularly noteworthy.
Despite some issues, like hand animations not appearing natural when obscured by other objects, Story Diffusion can create animations that allow for more flexibility. For example, a bear character maintains consistent fur markings and eye color across different scenarios.
Comparison with Sora
A direct comparison between Story Diffusion and Sora shows that Story Diffusion performs remarkably well, even with significantly less compute power. Story Diffusion used only 8 GPUs for training, compared to Sora’s 10,000 GPUs, making it much more cost-effective to train and run.
Currently, Story Diffusion does not have a user-friendly interface and requires downloading and installation or running on a cloud server. It can be accessed via GitHub or a Hugging Face space for a demo, providing another platform for deepswap capabilities.
Multiple Characters and Comic Generation
Story Diffusion excels in maintaining multiple characters consistently across different scenes. For instance, an Asian man and a Shiba Inu are consistently rendered across various settings, with details like the dog’s collar and the man’s outfit remaining consistent.
One demonstrated use case is comic generation, where a comic strip of a girl interacting with a squirrel shows character consistency across different scenarios. However, some limitations remain, such as inconsistencies in the length of the girl’s tie and the squirrel’s facial markings.
Another example shows a real human being turned into a graphic novel character, maintaining consistency throughout different scenes. The model uses consistent self-attention to enhance the consistency of generated images, ensuring shared attributes or themes are visually coherent when viewed as a series.
Story Splitting and Animation
Story splitting is another technique used, where a story is broken down into multiple text prompts describing different parts of the narrative. These prompts are processed simultaneously to produce a sequence of images that maintain consistency in character and environment. The motion predictor model then predicts how these separate frames will animate, ensuring fluid and natural transitions.
Story Diffusion is also capable of creating effective and usable anime-style animations, opening up possibilities for full AI-generated films in these genres. It handles a diverse range of scenes well, such as a realistic tourist footage-like scene with animated elements like moving buses and walking individuals, while keeping static objects inanimate.
A simple South Park-like animation example shows characters marching across a scene with subtle animations and a moving background. This method ensures that videos generated from a series of images look fluid and natural, maintaining continuity in appearance and motion, especially when utilizing deepswap techniques.
AI video is making significant strides, and Story Diffusion represents a real evolution in character consistency and the creation of realistic and cohesive scenes. Embrace the future of AI-generated videos by exploring the capabilities of AiFaceSwap for your next viral-worthy video project. Create entertaining and realistic face swaps in just a click and join the revolution in AI technology today!
