01
Extract style
Use a Consistent Style Descriptor (CSD) to obtain the reference style feature from one image.
Stylized video generation · Research project · 2025
FreeStyle-VidGen
Training-free stylized video generation via stochastic optimal control.
Silver Award, Tencent Rhino-Bird Science Talent Program for High School Students (top 3 awardees; 1 Gold and 2 Silver Awards granted in total)
Overview
One reference image.
No style-specific fine-tuning.
FreeStyle-VidGen integrates the stochastic optimal control (SOC) mechanism from RB-Modulation into AnimateDiff. A single style reference image and a text prompt are used to generate a temporally coherent video while preserving the prompt’s semantic content.
The key idea is to formulate reverse diffusion as a controllable stochastic process. Style matching is enforced through a terminal cost, temporal consistency through an adjacent-frame cost, and a control variable modifies the reverse-diffusion drift during sampling.
Method
Control the reverse diffusion trajectory
Style guidance is injected through optimization of the diffusion dynamics rather than style-specific model training.
02
Define costs
A terminal cost aligns generated frames to the reference style; a temporal cost penalizes style jumps between adjacent frames.
03
Solve control
A control variable u is optimized with a proximal-gradient formulation to balance style alignment and trajectory fidelity.
04
Generate video
The optimized control adjusts the reverse-diffusion drift at each denoising step, producing stylized video latents for decoding.
Qualitative results
Style references → generated videos
Animated examples showing the generated video results.
01
“A boat on the sea”
Two independent reference styles applied to the same content prompt.
Style reference A
Generated video A
Style reference B
Generated video B
02
“A girl turned”
Sketch and watercolor-like references produce distinct visual appearances.
Style reference A
Generated video A
Style reference B
Generated video B
Analysis
Quantitative result & ablation
LPIPS comparison and an ablation of the stochastic optimal control component.
Quantitative comparison
| Method | LPIPS ↓ | Inference time | VRAM |
|---|---|---|---|
| AnimateDiff | 0.64 ± 0.04 | 20 ± 2 | 5536 MB |
| AnimateDiff + LoRA | 0.60 ± 0.04 | 30 ± 3* | 6354 MB |
| FreeStyle-VidGen | 0.55 ± 0.03 | 25 ± 2 | 9231 MB |
* Fine-tuning time excluded.
SOC ablation
Limitations & future work
Base-model dependence
Final quality remains bounded by AnimateDiff’s content modeling and style-capture capability.
Temporal artifacts
The method inherits AnimateDiff’s temporal modeling limitations and can show frame-to-frame flicker in motion-heavy scenes.
Flow Matching
The current SOC design is tied to reverse-SDE diffusion dynamics and cannot be directly transferred to deterministic Flow Matching ODEs.
Next direction
Future work targets an ODE-compatible control strategy and temporal attention-feature aggregation, together with broader framework implementations.
Foundations