Research Introduction
Computer Vision for Videos
— studying motion.
Hyeongmin Lee · SeoulTech, Dept. of Electronic Engineering
About
Hyeongmin Lee
Assistant Professor · SeoulTech EE
- 2018 – 2023 Yonsei University — Ph.D., Electrical & Electronic Engineering
- 2021 Adobe Research — Research intern · video stabilization
- 2023 – 2026 Twelve Labs — ML Research Scientist · video foundation models
- 2026 – SeoulTech — Assistant Professor · ViViD Lab
Sunset, or sunrise?
From a single frame, you cannot tell.
Direction of motion exists only between frames.
You can still call this “ice skating”.
The rink, the blade, the posture — the action category is in the frame.
Single-frame appearance is enough to pick the right class label. But recognizing the activity is not the same as understanding the motion.
Glasses going up, or coming down?
Same problem with a real photograph.
No model — and no human — can decide.
The intent, the before, the after — these all live in change, not in any single frame.
The thesis
Video understanding is not the same as image understanding.
Modern vision models are excellent at recognizing what appears in a frame. They are far less capable at reasoning about change across frames.
Like an LED sign.
Each LED is fixed. Lights flip on and off. The motion is invented by our eyes.
No LED moves. They blink in a pattern, and our visual system sews that pattern into a moving shape.
Most video models work the same way: each frame is a still image. The model detects which patches change between frames — but never models the motion itself. The "video" lives in our perception, not in the model's representation.
Today's video foundation models cannot tell direction apart.
LDA visualization of “moving something up” vs. its reversed video.
TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models · Twelve Labs, 2024 Technical report
UMT and InternVideo2 — strong on appearance — collapse forward and reverse motion into the same embedding cloud.
Where I work
Motion
Models are good at seeing things,
not at understanding motion.
For images, interpolation is a pixel problem.
Decide each missing pixel from the values of its neighbors.
Nearest-neighbor, bilinear, bicubic — all decided by the spatial neighborhood. The methods differ in smoothness, but the underlying problem is the same: interpolate pixels from pixels right next to them.
For video, neighboring pixels are the wrong neighbors.
Naively averaging two adjacent frames gives a ghosted double image.
The juggling balls in the input frames are at different positions. Treating "between" as a pixel average leaves both balls visible — neither was actually there at that moment.
What we actually want.
A single ball, at the position it would have occupied between the two frames.
To get here, the model has to know where each pixel in the new frame came from — i.e., it has to model the motion.
Find where each output pixel came from.
The core operation behind every video frame interpolation method.
For every pixel of the new frame, look at the two adjacent frames and figure out where that content was. The vector linking these locations is the motion. How a method defines this vector — single pixel, fixed window, learned sampling — determines what it can handle.
AdaCoF: Adaptive Collaboration of Flows
Video frame interpolation as motion modeling.
Prior VFI either referenced one location (flow-based) or a small fixed window (kernel-based). AdaCoF samples any location, any shape, with multiple pixels — a single operation that subsumes both.
Frame Interpolation
Synthesizing the frames between adjacent frames.
RIFE: Real-Time Intermediate Flow Estimation · Huang et al. ECCV 2022
Exploring Discontinuity for VFI
Real videos have logos, subtitles, UI — discontinuous motion is everywhere.
The problem
One of our solutions — Discontinuity Map
Two ideas to make any VFI network robust to discontinuity: Figure-Text Mixing (FTM) as augmentation, and a lightweight Discontinuity Map (D-map) module that decides per-pixel where to copy vs. interpolate.
Compressing 4D Gaussian Splatting with motion priors
When the video is a 3D scene moving over time, motion lives in the trajectories.
Stop treating each frame as a fresh pixel array. Represent the scene once as 3D points and move them through time.
We apply a wavelet transform on each point's trajectory and discard the noisy high-frequency detail — exploiting the inherent smoothness of real motion. Up to ~99% storage reduction at competitive fidelity.
Looking forward
From predicting frames
to predicting the world.
Each step has been about modeling motion at a finer level — between adjacent frames (VFI), across a continuous 3D scene (4DGS), and now in response to an agent's actions (world models).
World Models
Generative simulation of the visual world, conditioned on actions.
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos · Gao et al. 2026
Wrap-up
One thread:
studying how things change.
Thank you.