--- title: "How AI Video Generation Works: From Noise to Motion" author: "Ankit Singh" published: "2026-11-15T06:18:30.799Z" updated: "2026-11-15T06:18:30.799Z" url: "https://zyvop.com/how-ai-video-generation-works-from-noise-to-motion-s8kfj" canonical: "https://zyvop.com/how-ai-video-generation-works-from-noise-to-motion-s8kfj" tags: ["machine learning","diffusion models","Generative AI","text to video","AI video generation"] --- # How AI Video Generation Works: From Noise to Motion The same noise-to-image idea, stretched across time, and why video is so much harder than pictures ![Cover image]() > **The short version:** An AI video generator is an image generator taught about time. It starts with static across a whole block of frames and denoises them together, guided by your prompt, so neighboring frames stay consistent. The recipe is the same as for images, but the cost, consistency, and physics problems are far bigger. A few seconds of video is dozens of images, and they all have to agree about who is in the shot, where the light comes from, and how things move. If you have read our guide to how AI generates images, you already know the core trick: start from noise and remove it step by step. This post shows how that trick is stretched across frames, what extra pieces video needs, and why it is still much harder than making a single picture. * * * ## The Pipeline at a Glance ```mermaid flowchart LR A["Your prompt"] --> B["Text encoder"] B --> C["Embeddings: meaning as numbers"] N["Random noise across all frames"] --> D["Denoising transformer: cleans every frame together, step by step"] C -->|"steers every step"| D D --> E["Video decoder"] E --> F["Finished clip, with synced sound in some models"] ``` The left half, turning your words into embeddings, works as it does for images. The rest of this post is about what changes. > **A note on sources:** The architecture details below come from published reports of open models such as Wan, Step-Video, and LTX-2. Most commercial systems do not publish their designs, so they may differ in the details. * * * ## 1\. From a Picture to a Block of Time A video is a stack of frames, so it has three dimensions: width, height, and time. Where an image generator denoises a flat grid, a video generator denoises a whole three-dimensional block at once. > **Analogy:** Think of a flipbook. Image generation draws one page. The naive way to make video is to draw every page separately, and you get flicker, because the face and the shadows change from page to page. A video model draws all the pages together, and each page can look at its neighbors while it is being drawn. That looking-at-neighbors is what keeps a character's face and a room's lighting steady across a clip. It is also the source of almost every difficulty later in this post. | | Image generator | Video generator | | --- | --- | --- | | **What gets denoised** | One compressed grid | A compressed block of frames | | **What must stay consistent** | Everything within one picture | Everything within each frame, and across all frames | | **Typical output** | One image | A clip of several seconds, sometimes with sound | * * * ## 2\. Compressing Time as Well as Space Image models avoid working on raw pixels by compressing images into a smaller _latent space_. Video models do the same with a **video VAE**, but it compresses across time too. In Alibaba's open Wan models: - Every four frames collapse into one in time. (The first frame is compressed in space only.) - Every 8 by 8 patch of pixels collapses into one position in space, with 16 numbers stored per position. The report for another model, Step-Video, notes that HunyuanVideo, CogVideoX, and Meta's Movie Gen use similar factors of 4×8×8 or 8×8×8. The VAE is also _causal_, meaning each moment depends only on earlier ones, which suits video. For a 5-second 720p clip, this turns about 224 million raw numbers into about 4.8 million, roughly 46 times fewer. * * * ## 3\. Cutting the Block Into Space-Time Tokens Next, a 3D convolution slices the compressed block into small patches, and each patch becomes a **token**, the unit a transformer works with. Wan uses patches one latent frame deep and 2 by 2 wide. A diffusion transformer then processes all of these tokens together using _attention_, which lets every token look at other tokens. Step-Video uses full 3D attention, so a token in frame 40 can be influenced directly by tokens in frame 3. That is the mechanism behind the flipbook analogy. Some designs save compute by attending across space and across time in separate steps instead. * * * ## 4\. Training and Generating: Same Recipe, More Data **Training.** Wan's published recipe starts with text-to-image pretraining, then moves to joint training on images and video in stages. Much of the effort goes into data. Wan's report devotes a section to dense video captions, because a model can only learn to follow a prompt about motion if the training clips are described in detail. The objective is a flow-matching-style one, a close cousin of the noise prediction described in our image post. **Generating.** The process mirrors images. Start with noise across the whole block, then denoise in steps, with your text embeddings steering each one. Wan 2.1's open-source default is 50 steps. Many tools also accept a starting image, or both a first and a last frame, and generate the motion in between. * * * ## 5\. Why Video Is So Much Harder ### The size problem Attention compares every token with every other token, so the work grows with the _square_ of the token count. Step-Video's report makes the same point about why compression matters so much. Here is a small calculator using Wan-style factors (8× in space, 4× in time, 2×2 patches): ```python def tokens(width, height, frames, space=8, time=4, patch=2): """Transformer tokens for a clip, using Wan-style compression factors.""" latent_frames = 1 + (frames - 1) // time # the first frame is only compressed in space return latent_frames * (width // space // patch) * (height // space // patch) image = tokens(1024, 1024, 1) # a single 1024x1024 image clip5 = tokens(1280, 720, 81) # 81 frames: about 5 seconds at 16 fps clip10 = tokens(1280, 720, 161) # about 10 seconds for name, n in [("image", image), ("5 s clip", clip5), ("10 s clip", clip10)]: print(f"{name:10} {n:>8,} tokens attention work vs image: {(n / image) ** 2:>7,.0f}x") ``` ```text image 4,096 tokens attention work vs image: 1x 5 s clip 75,600 tokens attention work vs image: 341x 10 s clip 147,600 tokens attention work vs image: 1,299x ``` A 5-second clip has about 18 times the tokens of a large image and, with full attention, roughly 340 times the attention work. Doubling the length nearly quadruples it again. And every one of the roughly 50 denoising steps repeats the whole job. This is an illustration, not a benchmark. Real systems use tricks: generating at lower resolution and upscaling, heavier compression, and distillation to cut the number of steps. One 2026 paper, FSVideo, compresses 64×64×4 and reports being 42 times faster than Wan2.1-14B, though that is the authors' own claim. Even so, cost is the central reason video is far more expensive than images. ### The consistency problem Across a clip, the model has to keep faces, clothing, objects, and lighting the same while everything moves. Errors that would be invisible in a single picture become flicker, morphing faces, or objects that appear and vanish. It is the price of working with time. ### The physics problem Looking real is not the same as obeying physics. A team including Google DeepMind researchers built the Physics-IQ benchmark, with 396 real videos across 66 scenarios, to test whether video models can predict how a scene continues. They found that the physical understanding of the models they tested (including Sora, Runway Gen-3, Pika 1.0, and VideoPoet) was severely limited and unrelated to how realistic the output looked. The paper appeared at WACV 2026 and notes that some test cases can already be solved. But it tested models from 2024, so treat it as evidence that realism and physics come apart, not as a verdict on today's models. ### The length problem There are two ways to get longer clips, and each fails differently: | | Whole-clip (bidirectional) | Chunk-by-chunk (autoregressive) | | --- | --- | --- | | **How it works** | Denoises all frames together within a fixed time window | Generates the next chunk conditioned on the frames before it | | **Strength** | Strong consistency inside the clip | Streaming, real-time output, and in principle unlimited length | | **Weakness** | The window is fixed, and cost balloons with length | Small errors compound | | **Examples** | A 2026 research paper groups Wan, Kling, Google, OpenAI, and Runway models here | Self-Forcing and Rolling Forcing (research) | The compounding in the second column is called **drift**. Research papers describe identity changes, structural degradation, over-sharpening, repeating motion, and a loss of global coherence over long horizons. One 2026 paper says global consistency beyond 10 to 20 seconds is still a struggle. Fixes are an active research area. Rolling Forcing's authors report real-time streaming at 16 frames per second on a single GPU, producing multi-minute videos with minimal error accumulation. That is a claim from research, not a feature you can buy today. * * * ## 6\. Adding Sound Early video models were silent. One approach was to generate the video first and then fill in audio afterward. Newer designs generate both together. LTX-2, an open model from Lightricks, uses two parallel transformer streams: a 14-billion-parameter one for video and a 5-billion-parameter one for audio. They are linked by attention in both directions, so sound and picture shape each other during denoising. Many leading commercial models now list native audio too, including Veo 3.1 and Kling 3.0. * * * ## 7\. Where the Field Stands This area moves monthly, and tool comparisons often disagree because their sources and tests differ, so treat this as a snapshot and not a ranking. | Model | Maker | Access | Notable | | --- | --- | --- | --- | | Veo 3.1 | Google | Closed | Native audio and 4K in tool comparisons | | Gemini Omni Flash | Google | Closed | Launched at Google I/O 2026 in the Gemini app, Flow, and YouTube Shorts. Accepts a wider range of inputs than Veo, including existing video, and supports conversational editing. Google had not disclosed its architecture at launch | | Kling 3.0 | Kuaishou | Closed | Native 4K and audio | | Seedance 2.5 | ByteDance | Closed | Generates picture and sound together in one pass | | Runway Gen-4.5 | Runway | Closed | Creative controls such as keyframes and video-to-video | | Wan 2.2 | Alibaba | Open weights | The latest open Wan release | | Wan 3.0 | Alibaba | API only | The open Wan line stops at 2.2 | | LTX-2 | Lightricks | Open weights | Joint audio-video generation | | Sora 2 | OpenAI | Discontinued | See below | **The Sora story.** OpenAI closed the Sora app and website on April 26, 2026, and shut down the Sora API on September 24, 2026, per OpenAI's own help page. Reports at the time linked the decision to computing costs and a shift in focus toward coding tools and business customers. According to The Decoder's reporting on OpenAI's announcement, Sora continues as a research project on world models. Whatever the full story, it illustrates the size problem from Section 5: serving video to millions of people is expensive. * * * ## Key Takeaways - Video diffusion is image diffusion over a block of space and time, with all frames denoised together so that neighbors agree. - Video VAEs compress time as well as space, and transformers work on space-time tokens. - Cost climbs steeply as clips get longer or sharper, which is why video is so much more expensive than images. - Realism is not physics, and long clips drift. Both are open research problems. - Sound is increasingly generated jointly with picture, not added afterward. If you want the foundations, start with _How AI Generates Images_. And because clips this convincing are getting easier to make, _How to Spot an AI-Generated Image (and What to Do When You Can't)_ covers how to check what you are looking at. * * * ## Sources and Further Reading - Wan Team (Alibaba), "Wan: Open and Advanced Large-Scale Video Generative Models" (arXiv, 2025): [https://arxiv.org/pdf/2503.20314v2]() - Step-Video-T2V Technical Report (arXiv, 2025): [https://arxiv.org/pdf/2502.10248]() - Lightricks, "LTX-2: Efficient Joint Audio-Visual Foundation Model" (arXiv, January 2026): [https://arxiv.org/html/2601.03233]() - Motamed et al., "Do generative video models understand physical principles?" (Physics-IQ, WACV 2026): [https://arxiv.org/abs/2501.09038]() - FSVideo (arXiv, 2026): [https://arxiv.org/pdf/2602.02092]() - HiAR, "Efficient Autoregressive Long Video Generation via Hierarchical Denoising" (arXiv, 2026): [https://arxiv.org/pdf/2603.08703v1]() - Context Forcing (arXiv, 2026): [https://arxiv.org/pdf/2602.06028]() - Rolling Forcing (arXiv, 2025): [https://arxiv.org/pdf/2509.25161v1]() - TokenTrim (arXiv, 2026): [https://arxiv.org/pdf/2602.00268]() - Engadget, "Google's Gemini Omni can generate 'anything from any input,' starting with video": [https://www.engadget.com/2176590/googles-gemini-omni-can-generate-anything-from-any-input-starting-with-video/]() - The Next Web, "Google launches Gemini Omni Flash": [https://thenextweb.com/news/google-gemini-omni-flash-video-model-io-2026]() - OpenAI, "What to know about the Sora discontinuation": [https://help.openai.com/articles/20001152-what-to-know-about-the-sora-discontinuation]() - The Decoder, "OpenAI sets two-stage Sora shutdown": [https://the-decoder.com/openai-sets-two-stage-sora-shutdown-with-app-closing-april-2026-and-api-following-in-september/]() - Gizmodo, "Is Sora Available in the UK in 2026?" (reasons reported for the shutdown): [https://gizmodo.com/best-vpn/access-sora-uk]() - Tool comparisons used for the model snapshot (secondary sources): [https://getleadwave.io/blog/best-ai-video-generators]() and [https://tech-insider.org/best-ai-video-generator-2026/]()