RIFE AI interpolation

From SmoothVideo Project
Jump to navigation Jump to search

RIFE is a Real-time Intermediate Flow Estimation algorithm based on a neural network named IFNet, that can directly estimate the intermediate flows from images.

SVP can use RIFE engine in both transcoding and real-time playback modes.

Credits:

Pros:

  • very high quality of the intermediate frames
  • runs completely on GPU (* almost)

Cons:

  • very demanding (comparing to SVP's interpolation method), real-time playback requires a decent video card.

Required GPU for a real-time playback (estimated, also works on AMD and Apple Mx):

  • 720p @48 fps - at least RTX 2060 or M1 Max
  • 1080p @48 fps, 720p @60-72 fps - at least RTX 3070
  • 1080 @60-72 fps - at least RTX 3080
  • 4K @48 fps - RTX 4090 (hopefully :D)
  • 4K @60+ fps - wait for RTX 5xxx...
  • Apple M1 can only do 576p @48 fps.

Installation - Windows

  • SVP 4.6.220 is required
  • Install an additional package via SVP menu -> Utilities -> Add. programs and features, called [DS_64][VPS_64] RIFE AI interpolation engine, restart SVP
  • You'll notice a new video profile called RIFE AI engine added.

Svp-rife.png

Installation - macOS

  • SVP 4.6.220 is required
  • Everything is already installed EXCEPT Vulkan loader in Homebrew:
# Apple silicon CPU
/opt/homebrew/bin/brew install vulkan-loader molten-vk
# Intel CPU
/usr/local/bin/brew install vulkan-loader molten-vk
  • If everything is fine you'll see a new RIFE AI engine video profile.

Usage

  • open Transcoding panel
  • browse for a video file
  • choose a RIFE video profile
  • start transcoding!

Alternatively if you own a high-end video card you may try it in a real-time playback. From more to less preferred methods:

  • mpv or VLC
  • DirectShow player (MPC-HC, etc) using Vapoursynth Filter: needs environment variables to be set: SVP menu -> Utilities -> Set environment variables for Vapoursynth
  • DirectShow player (MPC-HC, etc) using Avisynth Filter: needs Avisynth 3.7.2; very slow initialization (10+ secs), be patient!

After playback started, switch to the RIFE video profile if needed - it can be in the Other profiles sub-menu depending on profile conditions currently set.

Advanced usage with profile conditions: for example, you may want to use RIFE for <= 720p in a real time and for every video when transcoding.

Rife-720-code.png


Note that SVP provides only one AI model, but you can easily put others into SVP 4\rife\models subfolder (on macOS: SVP 4 Mac.app\Contents\Resources\rife\models). For example, there're a bunch of models to test on the Github. Restart SVP after that.

Problems

  • Vulkan may not be installed and/or configured correctly. If you see an error like "no Vulkan devices found" or "invalid GPU device" - you may need to re-install graphics drivers OR choose the correct "GPU device" in the video profile.
  • In a real-time playback first-time initialization can be really slooow. Be patient.
  • Overall GPU usage in the Task Manager may not reflect the actual computing power in use. Switch to "Compute_1" (NVIDIA) graph.
  • B/W image or any other issue playing via Avisynth Filter: ensure you have Avisynth+ 3.7.2 installed! Note that SVP installs AVS 3.5 by default.
  • macOS: it doesn't work with Intel IGP at all
  • macOS: threads count > 1 doesn't work for Apple M1