RIFE AI interpolation
RIFE is a Real-time Intermediate Flow Estimation algorithm based on a neural network named IFNet, that can directly estimate the intermediate flows from images. SVP can use RIFE engine in both transcoding and real-time playback modes.
Pros
- Very high quality of the intermediate frames
- Runs mostly on GPU
Cons
- Very demanding (comparing to SVP's interpolation method), real-time playback requires a decent video card
Required GPU for real-time playback
Estimated, also works on AMD and Apple Mx:
| Video size | Generic (any GPU) | TensorRT (NVidia only) |
|---|---|---|
| 720p @48 fps | >= RTX 2060 or M1 Max | 1660 (?) |
| 1080p @48 fps, 720p @60-72 fps | >= RTX 3070 | >= RTX 2060 |
| 1080 @60-72 fps | >= RTX 3080 | >= RTX 3060 |
| 4K @48 fps | RTX 4090 | >= RTX 4080 |
| 4K @60-72 fps | wait for RTX 5xxx… | RTX 4090 |
Installation
- Install an additional package via SVP menu → Utilities → Add. programs and features, called [DS_64][VPS_64] RIFE AI interpolation engine, restart SVP
- You’ll notice a new video profile called RIFE AI engine added.
TensorRT for NVidia RTX cards
TensorRT is a neural network engine heavily optimized for NVidia GPUs. It can be up to 100% faster than ncnn/Vulkan implementation, but there’re a few cons, for example very long optimization passes. To give it a try install an additional package called “TensorRT” via SVP menu → Utilities → Additional programs and features, then choose the TensorRT engine in the RIFE video profile. Only for Vapoursynth, so you have to switch to Vapoursynth Filter in MPC-HC (or any other DirectShow video player) first:- Close MPC-HC
- SVP menu → Utilities → Set environment variables for Vapoursynth → Continue
- Run MPC-HC, View → Options → External filters, uncheck “Avisynth Filter”, add “Vapoursynth Filter”, check it and set to “Prefer”
Everything is already installed EXCEPT Vulkan loader in Homebrew:
# Apple silicon CPU
/opt/homebrew/bin/brew install vulkan-loader molten-vk
# Intel CPU
/usr/local/bin/brew install vulkan-loader molten-vkIf everything is fine you’ll see a new RIFE AI engine video profile.
Usage
Transcoding:
- Open Transcoding panel
- Browse for a video file
- Choose a RIFE video profile
- Start transcoding!
Real-time playback (if you own a high-end video card), from more to less preferred methods:
- mpv or VLC
- DirectShow player (MPC-HC, etc) using Vapoursynth Filter: needs environment variables to be set: SVP menu → Utilities → Set environment variables for Vapoursynth
- DirectShow player (MPC-HC, etc) using Avisynth Filter: needs Avisynth 3.7.2; very slow initialization (10+ secs), be patient! TensorRT is not supported.
After playback started, switch to the RIFE video profile if needed — it can be in the Other profiles sub-menu depending on profile conditions currently set.
Note: when it runs the optimization pass, it’s OK for the command-line window to stay for several minutes. Don’t close it! Wait!
Advanced usage with profile conditions: for example, you may want to use RIFE for <= 720p in real time and for every video when transcoding.
Note: SVP provides only a few AI models, but you can easily put others into SVP 4\rife\models subfolder (on macOS: SVP 4 Mac.app\Contents\Resources\rife\models). For example, there’re a bunch of models to test on Github.
Adding TensorRT models
- Download ONNX models from there
- Put .onnx file(s) into SVP 4\rife\models\rife folder (or into SVP 4\rife\models\rife_v2 for “V2” models)
- Restart SVP
Adding ncnn/Vulkan models
- Download models from there
- Put the folder containing .bin and .param files into SVP 4\rife\models
- Restart SVP
Profile options
ncnn/Vulkan or TensorRT.
Defines interpolation quality, performance, memory usage and so on. Model 4.6 gives higher quality, but ~10-15% slower than 4.4. On the other hand, model 4.4 needs little bit less memory, so you can use it on a 8GB video cards for 4K interpolation.
If you have several video cards you can choose the one here.
Run several computation threads on the GPU for better utilization. You don’t need values higher that 3 even on a RTX 4090; and some GPUs can only accept 1 thread (e.g. Apple M1).
TensorRT only. Increase performance up to ~5% and lower memory usage in exchange for running a 5-10-minutes optimization pass in a command-line window for every new video resolution. Previous runs are cached. For example, you’re opening a 1080p video for the first time, you’ll wait 5+ mins. Every other 1080p videos will start instantly, but when the frame size changes due to, for example, black bars cut off, you’ll wait another 5+ mins.
Check if two adjacent frames are in different scenes or not, i.e. the frames are the same (but slightly moved) or completely different. If yes then there’s no interpolation between these frames. Higher values gives more interpolation artifacts when scene change was not detected. Lower values give more “stuttering” (i.e. repeated frames instead of interpolated) when there was a false detection.
Problems & Troubleshooting
Vulkan may not be installed and/or configured correctly. If you see an error like “no Vulkan devices found” or “invalid GPU device” — you may need to re-install graphics drivers OR choose the correct “GPU device” in the video profile.
In a real-time playback first-time initialization can be really slooow. Be patient.
Overall GPU usage in the Task Manager may not reflect the actual computing power in use. Switch to “Compute_1” (NVIDIA) graph.
B/W image or any other issue playing via Avisynth Filter: ensure you have Avisynth+ 3.7.2 installed!
macOS: it doesn’t work with Intel IGP at all.
macOS: threads count > 1 doesn’t work for Apple M1.
TensorRT: doesn’t work in MPC-HC. Ensure you switched video filter to Vapoursynth Filter!
RIFE profile won’t select by default: ensure you switched video filter to Vapoursynth Filter!
TensorRT: it still doesn’t work even with Vapoursynth filter! Ensure you modified the environment variables, restart MPC-HC. Check with double-clicking the “Vapoursynth Filter”, if everything is OK then the properties window must open.
TensorRT: optimization pass gives an error. Most likely you don’t have enough RAM on the video cards. <=1440p needs 6-8 GB of RAM, 4K wants at least 10 GB. Lower memory requirements by:
- Choosing AI model 4.4 instead of 4.6
- Turning on “Performance boost”