How AI Motion Synthesis Tools Could Transform Animation

Over the last year, a type of generative AI tooling known as motion synthesis has begun to appear, compatible with existing animation software. As designed, these tools would be used by skilled animators to automate character motion performances in 3D animation.

Software applications built on these models will output editable 3D data rather than pixels, which can then be further manipulated inside of popular 3D animation software such as Maya, Blender and Unreal Engine.

Yet Catherine Hicks, a former Pixar directing animator who now serves as head of animation innovation at AI startup Cartwheel specializing in motion synthesis, said the capability most attractive to professional animators using Cartwheel was the ability to derive motion from a reference video. “Our tool derives specific motion — all these joint positions, their rotations, and trajectory and velocity in space — from a reference video [the animator uploads] and then applies it to the 3D character.”

Sources suggested that until systems improve, an animator could use motion synthesis to speed up blocking, a typical previsualization process in which animators create a first pass of the performance by roughly blocking out the main beats of character placement, poses and movements.

“For most animators, the first few weeks of working on their shot involves re-creating motion from a reference on their 3D puppet. The ability to take a reference video and get motion from it really speeds up your blocking process,” said Hicks, further suggesting speed gains could enable greater directorial flexibility. “I can bring that to the director probably two or three weeks early, and if they want something totally different, it’s OK.”

By automating motion, or even rigging, in a 3D animation process, the promise is that animators can instead focus on performance nuances (e.g., character expressions, refining movements) rather than the manual process of keyframing a character getting from point A to B frame by frame (e.g., a walk or run cycle or other more diverse and complex character poses and movements).

The main proposed benefit of these types of applications for professional animation is their controllability, giving animators the ability to make specific adjustments to 3D rigs in existing traditional 3D animation workflows. Furthermore, the generative output of these systems as it would likely be used by an animator in a professional production setting would not necessarily be new synthetic 3D assets, but rather, simply generating the motion applied to a studio’s or their own 3D characters.

Sources directly contrasted this with the overall lack of control achievable yet with video generation models. At present, this is still regarded among studios and VFX houses as one of the main limitations of video generators in professional production, although developers are hyperfocused on progressing features to offer much greater control over the output.

Controllability is one of the primary criteria for evaluating whether an AI tool is viable for production use, as VIP+ has written. Without such control, directors or animators are left trying to work with whatever the model outputs, which may not perfectly reflect the creative vision nor easily be changed, a state one source described as “vibe directing” or “vibe animating.”

“While they’re really interesting, it was pretty clear in 2024 that all of these big video models really lacked the control and editability that would be needed in order to make them production-ready. I was more interested in who’s starting to tackle controllable motion,” said Hicks. “There are a lot of different features we’re testing right now that will allow users to get really specific with their edits to character motion.”

In some cases, these tools would also be viewed as more usable in professional animation settings because they’re “clean,” meaning they’ve been trained on ethically sourced data rather than massive scrapes of unknown, unlicensed copyrighted or personal data like most of the large video generation models. For example, Cartwheel and Maya’s MotionMaker were trained on owned or licensed motion capture (mocap) data and animations created in-house.

Yet sources didn’t shy away from the reality of labor disruption and skills change in animation as usable AI-based tools arrive and offer high quality, edit controls and clean data. They signaled “generalist” skillsets would be prized rather than narrow specialization as they compared gen AI’s potential in animation to past disruptions, such as floors of ink-and-paint artists who were dispatched when Disney adapted Xerox, then subsequently by 3D animation disrupting cel animation and motion capture disrupting 3D animation.

“Animation has experienced this many times before,” said Hicks. “We’re at one of these places again where the tools are changing. With every new technology comes new jobs, and changes to existing jobs.”

VIP+ Unearths Gen AI Insights From All Angles — Pick a Story