DGIST HSS118 · Future Literacy for the Age of Physical AI
Video Foundation Models
and World Models
How machines learn to see motion, and to imagine what comes next.
Hyeongmin Lee · SeoulTech, Dept. of Electronic Engineering · 2026-09-11
About
Hyeongmin Lee
Assistant Professor · SeoulTech EE
- 2018 – 2023 Yonsei University · Ph.D. · video frame interpolation
- 2021 Adobe Research · video stabilization
- 2023 – 2026 Twelve Labs · video foundation models
- 2026 – SeoulTech · ViViD Lab · motion and world models
One thread: motion.
Two stories
Video foundation models
how machines see motion
World models
how machines imagine what comes next
Story 1
Video Foundation Models
How machines see motion.
Foundation model
Learn once. Use everywhere.
The recipe
More data · more compute · bigger models
“general methods that leverage computation are ultimately the most effective”
Rich Sutton · The Bitter Lesson · 2019
The recipe behind language and image models · revisited at the end
Bigger works
Three image models. Simple objectives, lots of data, big models.
400M image–text pairs from the web
zero-shot ImageNet = a ResNet-50 trained on all 1.28M labels
hide 75% of the patches, reconstruct the rest
87.8% ImageNet top-1 with ViT-Huge
predict features, not pixels
no labels, no hand-crafted augmentations
Radford et al. 2021 · He et al. 2022 · Assran et al. 2023
Video: the same recipe, frames added
Each image model got a video version. Almost nothing else changed.
CLIP on every frame, then combine
hide 90–95% of the video, reconstruct
predict features across frames
Images worked. So: add frames.
Luo et al. 2021 · Tong et al. 2022 · Bardes et al. 2024
Add frames, literally
One image encoder per frame, then an aggregator. This is CLIP4Clip.
encoder
encoder
encoder
encoder
encoder
encoder
Just add frames.
The sealed rooms
What “add frames” means: one person, one frame, one room. Then a meeting.
What can this committee never know?
What is happening?
One frame. Type your answer in the chat.
What is there · one frame is enough
Up, or down?
One frame. Vote 1 or 2 in the chat.
How it changes · no single frame can tell
What is this?
One frame. Any guess?
words
shapes
Nothing is in any frame. Everything is in the motion between them.
Some things exist only in time.
SpookyBench · Upadhyay et al., Time Blindness: Why Video-Language Models Can’t See What Humans Can · CVPR 2026
Two layers of seeing
What is there
objects · scene · appearance · one frame
How it changes
motion · direction · things that exist only in time
Reverse a sunrise: a sunset. Same frames.
Forward = backward?
“Moving something up” videos (red) and the same videos played backwards (blue).
forward = backward · image recipe, frames added · same spot
two clouds · only when motion is trained for on purpose
Something-Something V2 · LDA projection · Lee et al., TWLV-I technical report, 2024
It works now. At a price.
V-JEPA 2 (Meta, 2025): predict what comes next, in feature space.
Up or down: solved. With 114 years of video.
Assran et al., V-JEPA 2, 2025
Nothing moves
An LED sign. Every light is fixed in place. You still see the words slide.
instant · effortless · cannot even be switched off
114 years of video · 60 GPU-years
Trivial for us. Expensive for machines.
Video frame interpolation
Frames in, more frames out: draw the frame in between. My problem in grad school.
Interpolation, on images
A missing pixel is filled from its neighbours. Nearest, bilinear, bicubic: one line of arithmetic.
Between two frames
Now do it in time: Frame A + Frame B → the frame in between. The neighbours moved.
two ghost balls
one ball, in between
Where did each pixel go?
Motion: where it came from
For every pixel of the new frame: find it in A, find it in B. The arrow is the motion.
Interpolation, in pixel space
The frame in between has to be generated from scratch: every pixel, decided again.
frame A
256 pixels, from scratch
frame B
AdaCoF (CVPR 2020, my paper) · a full neural network, 21.8M parameters, trained for days, to learn where each pixel came from
A heavy net, for a simple move.
What if video were primitives?
Like a vector image: things, with positions. Then the frame in between is arithmetic.
frame A
just move them
frame B
a direction, not yet a product · what should carry motion: pixels, or things?
Scale alone?
Solving the obvious with scale alone: is that the right posture?
Story 2
World Models
How machines imagine what comes next.
Easy is hard
Moravec's paradox: what is hard for us is easy for AI, and the other way round.
✓ Chess
Deep Blue · 1997
✓ Go
AlphaGo · 2016
✓ Bar exam
GPT-4 · 2023
✗ Folding laundry
still no robot at home
A three-year-old can. The chess champion of machines cannot.
They all lived inside a computer
A virtual world can be reset, fast-forwarded, and copied. The real one cannot.
practice is free
reset button · 100× speed · thousands of copies · millions of games in days
practice is expensive
1× speed · no reset · a fall breaks the robot · a million tries take a million times
The genius of the virtual world has to learn the real one.
images: Dreamer 4 (Google DeepMind, 2025) · ALOHA (Zhao et al., 2023)
Make the simulation real
If reality cannot be a game, build a game as real as reality. Not written by hand: learned from the world.
a world written by hand
gravity, collisions, blocks: rules typed in by programmers · fine for Minecraft · the real world has too many rules to type
a world learned from the world
watch how the world behaves (video, interaction) · learn its rules · simulate what comes next
A simulation as real as the world. But who could build one?
Then, video generation happened
Sora (OpenAI, 2024): a few lines of text, a realistic video. Could this be the simulator?
video: OpenAI · Video generation models as world simulators · 2024
Make it interactive: Genie
Genie 3 (DeepMind, 2025): a video generator that answers your key press. Watch: a world simulation, built from video generation.
change between two frames = action · learned from video, no labels · a world you can walk in, at last
video: Google DeepMind · Genie 3: A new frontier for world models · 2025
World model
Now the AI has a world inside. What I see + what I do → what happens next. Imagine first, then act.
Try it in your head first.
It works, in games
Dreamer 4 (Google DeepMind, 2025): an agent that practiced only inside its imagined Minecraft, then played the real game.
video: Google DeepMind · Hafner et al., Training Agents Inside of Scalable World Models · 2025
And into robots
DreamDojo (NVIDIA, 2026): the same idea for a real robot. A robot's dream: action in → future out.
video: NVIDIA GEAR · DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos · 2026
Look away, look back
Is the scene still the same?
① look left
② look right
③ a different house
The Nether challenge
AI Minecraft. No portal. Only lava texture.
obsidian portal → light it → walk through
make the screen look like lava → the model draws the Nether
watch: the world changes when the screen looks like lava
A plausible next frame is not a world.
No world, only weights
Where does the house live while you look away?
the whole world, compressed into weights · decoded again at every step
nothing in between persists
the house is stored once · a view is only a picture of it
look away, look back: unchanged, by construction
Plausible ≠ persistent.
An explicit world
3D Gaussian Splatting: the same garden from any view. Look away, look back: it is still there.
video: 3D Gaussian Splatting (Kerbl et al., 2023)
Implicit + Data vs Explicit + Prior
the world, stored in weights
+works now · scales with video · endless worlds
−no persistent state · consistency not guaranteed
the world, stored as structure
+consistent by construction · persistence for free
−slow now · interaction is hard · 4D data is scarce
one has data, the other has structure
The same shape, twice
Trivial for us. Expensive for machines.
The Bitter Lesson, misread
general methods that leverage computation win
“the ultimate reason for this is Moore’s law”
today’s method + more scale = everything, eventually
One more step, or a first step?
Horizontal: time invested. Vertical: technology level. Solid: walked. Dashed: how far it can go.
Know which step you are taking.
the legacy makes one of them cheap · the global optimum may sit on the other
Keep proposing new directions
Keep proposing new directions,
even the ones that do not work yet.
Local ≠ Global · you have the time
Materials for a first step
The untrodden direction needs materials. So the lab keeps several on the bench.
temporal semantics · representation learning · motion, cheaply
generative simulation of the visual world · a world that persists
frame interpolation · super-resolution · compression · stabilization
correspondence · optical flow · tracking
Gaussian splatting · reconstruction · dynamic scenes
different materials, one purpose: the first step
vivid.seoultech.ac.kr
Thank you
Trivial for us. Expensive for machines.
Keep proposing new directions.
Hyeongmin Lee · SeoulTech, Dept. of Electronic Engineering