RIFE AI interpolation

From SmoothVideo Project
Jump to navigation Jump to search

RIFE is a Real-time Intermediate Flow Estimation algorithm based on a neural network named IFNet, that can directly estimate the intermediate flows from images.

SVP can use RIFE engine in both transcoding and real-time playback modes.

Credits:

Pros:

  • very high quality of the intermediate frames
  • runs mostly on GPU

Cons:

  • very demanding (comparing to SVP's interpolation method), real-time playback requires a decent video card.

Required GPU for a real-time playback (estimated, also works on AMD and Apple Mx):

Video size Generic (any GPU) TensorRT (NVidia only)
720p @48 fps >= RTX 2060 or M1 Max 1660 (?)
1080p @48 fps, 720p @60-72 fps >= RTX 3070 >= RTX 2060
1080 @60-72 fps >= RTX 3080 >= RTX 3060
4K @48 fps RTX 4090 >= RTX 4080
4K @60-72 fps wait for RTX 5xxx... RTX 4090
  • Apple M1 can only do 576p @48 fps.


Installation - Windows

  • Install an additional package via SVP menu -> Utilities -> Add. programs and features, called [DS_64][VPS_64] RIFE AI interpolation engine, restart SVP
  • You'll notice a new video profile called RIFE AI engine added.

Svp-rife.png

TensortRT for NVidia RTX cards

TensorRT is a neural network engine heavily optimized for NVidia GPUs. It can be up to 100% faster than ncnn/Vulkan implementation, but there're a few cons, for example a very long optimization passes.

To give it a try install an additional package called "TensorRT" via SVP menu -> Utilities -> Additional programs and features, then choose the TensorRT engine in the RIFE video profile.

Only for Vapoursynth, so you have to switch to Vapoursynth Filter in MPC-HC (or any other DirectShow video player) first:

  • close MPC-HC
  • SVP menu -> Utilities -> Set environment variables for Vapoursynth -> Continue
  • run MPC-HC, View -> Options -> External filters, uncheck "Avisynth Filter", add "Vapoursynth Filter", check it and set to "Prefer"

Note: when it runs the optimization pass, it's OK for the command-line window to stay for 4-5 or even 8 mins. Don't close it! Wait!

Installation - macOS

  • Everything is already installed EXCEPT Vulkan loader in Homebrew:
# Apple silicon CPU
/opt/homebrew/bin/brew install vulkan-loader molten-vk
# Intel CPU
/usr/local/bin/brew install vulkan-loader molten-vk
  • If everything is fine you'll see a new RIFE AI engine video profile.

Usage

  • open Transcoding panel
  • browse for a video file
  • choose a RIFE video profile
  • start transcoding!

Alternatively if you own a high-end video card you may try it in a real-time playback. From more to less preferred methods:

  • mpv or VLC
  • DirectShow player (MPC-HC, etc) using Vapoursynth Filter: needs environment variables to be set: SVP menu -> Utilities -> Set environment variables for Vapoursynth
  • DirectShow player (MPC-HC, etc) using Avisynth Filter: needs Avisynth 3.7.2; very slow initialization (10+ secs), be patient! TensorRT is not supported.

After playback started, switch to the RIFE video profile if needed - it can be in the Other profiles sub-menu depending on profile conditions currently set.

Advanced usage with profile conditions: for example, you may want to use RIFE for <= 720p in a real time and for every video when transcoding.

Rife-720-code.png


Note that SVP provides only one AI model, but you can easily put others into SVP 4\rife\models subfolder (on macOS: SVP 4 Mac.app\Contents\Resources\rife\models). For example, there're a bunch of models to test on the Github. Restart SVP after that.

Profile options

  • Engine - ncnn/Vulkan or TensorRT.
  • AI model - defines interpolation quality, performance, memory usage and so on. Model 4.6 gives higher quality, but ~10-15% slower than 4.4. On the other hand, model 4.4 needs little bit less memory, so you can use it on a 8GB video cards for 4K interpolation.
  • GPU device - if you have several video cards you can choose the one here.
  • GPU threads - run several computation threads on the GPU for better utilization. You don't need values higher that 3 even on a RTX 4090; and some GPUs can only accept 1 thread (e.g. Apple M1).
  • Performance boost - TensorRT only. Increase performance up to ~5% and lower memory usage in exchange for running a 5-10-minutes optimization pass in a command-line window for _every_ new video resolution. Previous runs are cached. For example, you're opening a 1080p video for the first time, you'll wait 5+ mins. Every other 1080p videos will start instantly, but when the frame size changes due to, for example, black bars cut off, you'll wait another 5+ mins.
  • Scene change threshold - check if two adjacent frames are in different scenes or not, i.e. the frames are the same (but slightly moved) or completely different. If yes then there's no interpolation between these frames. Higher values gives more interpolation artifacts when scene change was not detected. Lower values give more "stuttering" (i.e. repeated frames instead of interpolated) when there was a false detection.

Additional AI models

SVP comes with a few AI models, but there're a lot more available on the Internet. Different models produce different interpolation quality with different performance. You can easily add new models into SVP by your own.

Adding TensorRT models

  1. download ONNX models from there
  2. put .onnx file(s) into SVP 4\rife\models\rife folder (or into SVP 4\rife\models\rife_v2 for "V2" models)
  3. restart SVP

Adding ncnn/Vulkan models

  1. download models from there
  2. put the folder containing .bun and .param files into SVP 4\rife\models
  3. restart SVP

Problems

  • Vulkan may not be installed and/or configured correctly. If you see an error like "no Vulkan devices found" or "invalid GPU device" - you may need to re-install graphics drivers OR choose the correct "GPU device" in the video profile.
  • In a real-time playback first-time initialization can be really slooow. Be patient.
  • Overall GPU usage in the Task Manager may not reflect the actual computing power in use. Switch to "Compute_1" (NVIDIA) graph.
  • B/W image or any other issue playing via Avisynth Filter: ensure you have Avisynth+ 3.7.2 installed!
  • macOS: it doesn't work with Intel IGP at all
  • macOS: threads count > 1 doesn't work for Apple M1
  • TensorRT: doesn't work in MPC-HC. Ensure you switched video filter to Vapoursynth Filter!
  • RIFE profile won't select by default: ensure you switched video filter to Vapoursynth Filter!
  • TensorRT: it still doesn't work even with Vapoursynth filter! Ensure you modified the environment variables, restart MPC-HC. Check with double-clicking the "Vapoursynth Filter", if everything is OK then the properties window must open.
  • TensorRT: optimization pass gives an error. Most likely you don't have enough RAM on the video cards. <=1440p needs 6-8 GB of RAM, 4K wants at least 10 GB. Lower memory requirements by:
    • choosing AI model 4.4 instead of 4.6
    • turning on "Performance boost"
Retrieved from "https://www.svp-team.com/w/index.php?title=RIFE_AI_interpolation&oldid=6314"