176 (edited by UHD 31-03-2022 23:23:23)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

So, if anyone had the knowledge and skills on how to run RIFE ncnn Vulkan and compare performance I would greatly appreciate it.

RIFE ncnn Vulkan versions for comparison:

1. Release 20210520: https://github.com/nihui/rife-ncnn-vulk … g/20210520
RIFE 3.1

This version is based on the current VapourSynth filter: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan used by SVP

2. Release 20220330: https://github.com/nihui/rife-ncnn-vulk … g/20220330
RIFE 4.0

It is important to use both versions to get an idea of how much the performance of the new version of the VapourSynth filter should increase by, which we request here: https://github.com/HomeOfVapourSynthEvo … /issues/14


The file is the comparisons: the same one on which tests have already been performed several times on this thread:

original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

So, if anyone had the knowledge and skills on how to run RIFE ncnn Vulkan and compare performance I would greatly appreciate it.

RIFE ncnn Vulkan versions for comparison:

1. Release 20210520: https://github.com/nihui/rife-ncnn-vulk … g/20210520
RIFE 3.1

This version is based on the current VapourSynth filter: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan used by SVP

2. Release 20220330: https://github.com/nihui/rife-ncnn-vulk … g/20220330
RIFE 4.0

It is important to use both versions to get an idea of how much the performance of the new version of the VapourSynth filter should increase by, which we request here: https://github.com/HomeOfVapourSynthEvo … /issues/14


The file is the comparisons: the same one on which tests have already been performed several times on this thread:

original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

From discord

https://i.imgur.com/do1Ih1d.png

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> original demo video from the creator of RIFE

RTX 2060

model 3.1
20210520 - 4:30
20220228 - 4:30
20220313 - 2:30
20220330 - 1:10

model 4.0
20220228 - 1:05
20220330 - 0:54

model 4.0 also loads CPU a lot (30% of 4800H in the Task manager)

179 (edited by UHD 02-04-2022 12:53:50)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> original demo video from the creator of RIFE

RTX 2060

model 3.1
20210520 - 4:30
20220228 - 4:30
20220313 - 2:30
20220330 - 1:10

model 4.0
20220228 - 1:05
20220330 - 0:54

model 4.0 also loads CPU a lot (30% of 4800H in the Task manager)

Many thanks to Chainik for the tests! Even more so, for doing more tests and capturing the differences between the different versions.

Amazing progress!

180

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik, is it possible to swap the model to 4.1 instead of 4.0 and do the test also on the latest model? I have made a request here: https://github.com/nihui/rife-ncnn-vulkan/issues/43
but we will probably wait a bit for an update.

181 (edited by UHD 03-04-2022 00:13:09)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik, the results you presented are very promising in terms of using a lighter version of RIFE together with SVP for real-time motion interpolation.

Now I am intrigued by an anomaly that has come to light through testing on this thread:

Flowframes:

lwk7454 wrote:

I've tried Vulkan implementation too, but it only has RIFE 3.1 instead of 3.8:
720p, FP16
FPS: 36.79
Compute_1: 98%
Miracles: 0%
720p, FP32.
FPS: 38.85
Compute_1: 98%
Cuda: 0%

Interesting to see Flowframes RIFE with Vulkan has much better performance than SVP.

https://www.svp-team.com/forum/viewtopi … 531#p79531


SVP:

lwk7454 wrote:

I've also tried TTA Disabled for comparison, just 1 test:
720p, FP32
FPS: 25.1
Compute_1: 100%
Cuda: 15%

https://www.svp-team.com/forum/viewtopi … 526#p79526


Flowframes probably uses RIFE ncnn Vulkan directly: https://github.com/nihui/rife-ncnn-vulkan

SVP on the other hand uses RIFE filter for VapourSynth, based on RIFE ncnn Vulkan: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan

I don't know if this is the reason for the difference of about 50% more performance of Flowframes vs SVP or maybe just some settings are chosen differently. Anyway I have a big request to you to check how it looks like for you. We already have the results of the test you did for the base RIFE ncnn Vulkan tool:

Chainik wrote:

RTX 2060

3.1 model
20210520 - 4:30

https://www.svp-team.com/forum/viewtopi … 140#p80140

If we had comparison data for the following 3 variants, it would be easier to predict where the difference comes from:

4:30 RTX 2060 model 3.1 RIFE ncnn Vulkan 20210520
?:?? RTX 2060 model 3.1 SVP+RIFE filter for VapourSynth, based on RIFE ncnn Vulkan
?:?? RTX 2060 model 3.1 Flowframes

The free version of Flowframes still probably uses the 3.1 RIFE model.

I know, in light of new testing this old model is now obsolete, but it is still the basis of the RIFE filter for VapourSynth (ncnn Vulkan). If it turns out that there are actually differences between the results of these 3 tests, I think you could figure out from the code where the difference arises in the software. If it's the RIFE filter for VapourSynth, then we could ask HolyWu to include possible changes in a new update: https://github.com/HomeOfVapourSynthEvo … /issues/14

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

VapourSynth-RIFE-ncnn-Vulkan just updated to the latest ncnn & model v4 support
now this version performance is much closer to the CUDA one

183 (edited by UHD 05-04-2022 16:09:43)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Thanks!!! Great news!!!

Chainik, could you check how much closer with SVP and the new VapourSynth-RIFE-ncnn-Vulkan filter?

You gave us the results for the base RIFE ncnn Vulkan 20220330:

Chainik wrote:

> original demo video from the creator of RIFE

RTX 2060

model 3.1
20210520 - 4:30
20220228 - 4:30
20220313 - 2:30
20220330 - 1:10

model 4.0
20220228 - 1:05
20220330 - 0:54

model 4.0 also loads CPU a lot (30% of 4800H in the Task manager)


Which gives us the following fps result:

original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms

25FPS*53.68=1342frames

1342frames*2/54s=49.7fps


Chainik, what results do you get out on your hardware in case of:

1. SVP & RIFE filter for VapourSynth (PyTorch CUDA) re-encoding with x2 interpolation; RIFE model: 4.0; scale=1.0; FP16 or FP32 (whichever is faster)

2. SVP & RIFE filter for VapourSynth (ncnn Vulkan) re-encoding with x2 interpolation; RIFE model: 4.0

I am very much asking you for these tests, please. It's important that we have a comparison of these two tests along with your earlier result, 54 seconds (49.7fps) on the same hardware.

Is there any difference when you change:

gpu_thread

What are the impressions with:

multiplier

set to 3 for the 4.0 model?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

SVP updated.

rtx 2060, 720p: ~70 fps with v4 model, ~35 fps with v3 model (was ~8 fps before)
real time: works in mpv but for some unknown reason doesn't work well in MPC-HC + Vapoursynth Filter (only gives 0.25 SVP index)

185 (edited by UHD 05-04-2022 19:47:11)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

SVP updated.

rtx 2060, 720p: ~70 fps with v4 model, ~35 fps with v3 model (was ~8 fps before)
real time: works in mpv but for some unknown reason doesn't work well in MPC-HC + Vapoursynth Filter (only gives 0.25 SVP index)

70fps SVP & RIFE filter for VapourSynth (PyTorch CUDA) or
SVP & RIFE filter for VapourSynth (ncnn Vulkan)?

Thanks for the tests and update smile

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> VapourSynth-RIFE-ncnn-Vulkan just updated to the latest ncnn & model v4 support

187 (edited by UHD 05-04-2022 22:40:10)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Thanks, this 70fps is not only hugely impressive compared to the previous 8fps, but also to the max results given here earlier:


720p - SVP & RIFE filter for VapourSynth (PyTorch CUDA) Model 4.0:

82,4fps NVIDIA GeForce RTX 3070 Ti - dlr5668
https://www.svp-team.com/forum/viewtopi … 691#p79691
91.3fps NVIDIA GeForce RTX 3080 Ti - Quaternions
https://www.svp-team.com/forum/viewtopi … 727#p79727


1080p - SVP & RIFE filter for VapourSynth (PyTorch CUDA) Model 4.0:

42.5fps NVIDIA GeForce RTX 3070 Ti - dlr5668
https://www.svp-team.com/forum/viewtopi … 699#p79699
45.0fps NVIDIA GeForce RTX 3080 Ti - Quaternions
https://www.svp-team.com/forum/viewtopi … 719#p79719

188 (edited by dlr5668 06-04-2022 18:22:53)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

SVP updated.

rtx 2060, 720p: ~70 fps with v4 model, ~35 fps with v3 model (was ~8 fps before)
real time: works in mpv but for some unknown reason doesn't work well in MPC-HC + Vapoursynth Filter (only gives 0.25 SVP index)

https://i.imgur.com/uvttbqJ.png

demo 1 720p file - v4 model - 3070ti from https://www.svp-team.com/forum/viewtopi … 721#p79721

189 (edited by dlr5668 06-04-2022 20:41:06)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Can we enable rife profile as default ? I want to use it for <1080p content. Regular rules dont work
https://i.imgur.com/riPvktY.png

190 (edited by UHD 06-04-2022 22:05:26)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

Can we enable rife profile as default ? I want to use it for <1080p content. Regular rules dont work

I join in with your request. I would also love to use such a setting in the future.

191 (edited by UHD 06-04-2022 22:43:01)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

https://i.imgur.com/uvttbqJ.png

demo 1 720p file - v4 model - 3070ti from https://www.svp-team.com/forum/viewtopi … 721#p79721

Thanks for the test! Amazing performance boost!

Interpolation x2 and 188.2fps means 94.1 original frames and 94.1 interpolated frames per second. This means that we can have real-time x4 interpolation without any problem - 25 original frames and 75 interpolated frames!!!!!

Am I counting correctly?

Dlr5668, have you tried x3 and x4 realtime interpolation with this new RIFE filter?
Does the "gpu_thread" setting affect fps, %GPU, VRAM?

192 (edited by dlr5668 07-04-2022 08:08:46)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:
dlr5668 wrote:

https://i.imgur.com/uvttbqJ.png

demo 1 720p file - v4 model - 3070ti from https://www.svp-team.com/forum/viewtopi … 721#p79721

Thanks for the test! Amazing performance boost!

Interpolation x2 and 188.2fps means 94.1 original frames and 94.1 interpolated frames per second. This means that we can have real-time x4 interpolation without any problem - 25 original frames and 75 interpolated frames!!!!!

Am I counting correctly?

Dlr5668, have you tried x3 and x4 realtime interpolation with this new RIFE filter?
Does the "gpu_thread" setting affect fps, %GPU, VRAM?

1)only x2 works
2)each thread takes 0.5 GB VRAM for 1080p. 1 gives - 50% fps, 2 and more - 100%

https://i.imgur.com/cNSKpM9.png

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

.

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> real time: works in mpv but for some unknown reason doesn't work well in MPC-HC + Vapoursynth Filter (only gives 0.25 SVP index)

updated: everything is good as long as you pay attention to the Vulkan device in use
in my case for some unknown reason when playing in MPC-HC there're TWO devices (id0 = AMD IGP, id1 = RTX 2060), while all other players (MPC-BE. mpv) can only see ONE device (id0 = RTX 2060)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

=== RIFE / PyTorch installation ===

0. Update SVP to the latest version

1. Dependencies installation:
--------------
download https://bootstrap.pypa.io/get-pip.py into SVP 4\mpv64

run cmd as Administrator
cd SVP 4\mpv64

run

python get-pip.py

run

python -m pip install torch --extra-index-url https://download.pytorch.org/whl/cu113
python -m pip install vsrife

will download 3+ GB and unpack it into 6+ GB of data big_smile

2. SVP modification
---------------
replace SVP 4\script\base.py and SVP 4\script\generate.js, restart SVP

add custom profile option: go to Application settings -> User defined options, fill in:
- Title: RIFE via CUDA
- Script name: rife_cuda
- Options scope: FRC profile
- Allowed values: ON or OFF
Press "Add option"

in the "RIFE AI engine" video profile set "RIFE via CUDA" to ON
when it's ON then "AI model" and "GPU threads" change nothing

Nice! Is it possible to now run RIFE at real time on a RTX 2070 Super & 5800X CPU?

I remember it was impossible in the past, but maybe with these versions it is?

PS: If I do, most content I watch is Bluray Remux 4K HDR & SDR content. So it will be very demanding, but has v4 of RIFE improved so much that it can now work real time with a 2070 Super?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> Is it possible to now run RIFE at real time on a RTX 2070 Super & 5800X CPU?

1080p - probably yes, you'd better try it yourself

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> Is it possible to now run RIFE at real time on a RTX 2070 Super & 5800X CPU?

1080p - probably yes, you'd better try it yourself

any progress on making rife profile use generic rules ?

198

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

1660 Ti OC under VGA in MPV player runs beautifully at 48 FPS. No soap effect on the edges of moving people and objects, it's amazing! The only thing that bothers me is that I can't get this new Rife option to work in PotPlayer. In PotPlayer (x64) so far all movies work in all options by SVP 4, so I don't understand why this one doesn't now? Until a solution is found, I'm watching 1080p movies in MPV. It's just that MPV is very clunky for me.

199 (edited by dlr5668 23-04-2022 11:24:49)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

function gen_vs()
{
    // RIFE override for low resolution
    // Make sure default profile for such content is set to Automatic and interpolation disabled
    // https://imgur.com/a/cx6cKo5
    if(media.src_fps < 51 && media.dst_h_par * media.src_fps <= 1.2 * 1080 * 30)
    {
        profile.rife = true;
        profile.rife_model = 9;
        profile.rife_threads = 2;
        profile.rife_gpu = 0;
    }

https://imgur.com/a/cx6cKo5

Post's attachments

generate.js 19.1 kb, 174 downloads since 2022-04-23 

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

### Environment ###
Windows 10
DDR4-2933 48GiB
Nvidia RTX2070 8GiB
Nvidia Driver 511.79
CUDA Toolkit 11.3
cuDNN v8.2.1 (June 7th, 2021), for CUDA 11.x

### Software ###
Python 3.10.4
VapourSynth R58-RC2
PyTorch 1.11.0 (CUDA 11.3)
vs_rife v2.0.0
VapourSynth-RIFE-ncnn-Vulkan r3 (model: 4.0)

### Tools & Seting ###
GPU-Z 2.45.0
VapourSynth Editor r19-mod-5-AC2
VapourSynth threads: core.num_threads = 4
Decoder: lsmas.LWLibavSource(format="yuv420p8", prefer_hw=3)
Video: demo.mp4 [720p]

### Result ###
1. RIFE filter for VapourSynth (PyTorch CUDA) - vs_rife v2.0.0
    Interpolation: x2
    RIFE model: 4.0
    scale: 1.0
    FP16: False
   
    FPS: 54.115
    CUDA: ~50%
    PerfCap: VRel, VOp, Pwr

2. RIFE filter for VapourSynth (PyTorch CUDA) - vs_rife v2.0.0
    Interpolation: x2
    RIFE model: 4.0
    scale: 0.5
    FP16: False
   
    FPS: 69.997
    CUDA: ~40%
    PerfCap: VRel, VOp

3. RIFE filter for VapourSynth (PyTorch CUDA) - vs_rife v2.0.0
    Interpolation: x2
    RIFE model: 4.0
    scale: 0.5
    FP16: True
   
    FPS: 70.936
    CUDA: ~32%
    PerfCap: VRel, VOp

4. RIFE filter for VapourSynth (ncnn Vulkan) - VapourSynth-RIFE-ncnn-Vulkan r3
    Interpolation: x2
    RIFE model: 4.0
    GPU thread: 1
    tta: False
    uhd: False
    sc: True
   
    FPS: 27.356
    CUDA: ~1%
    Compute_1: 30%
    PerfCap: Idle
   
5. RIFE filter for VapourSynth (ncnn Vulkan) - VapourSynth-RIFE-ncnn-Vulkan r3
    Interpolation: x2
    RIFE model: 4.0
    GPU thread: 2
    tta: False
    uhd: False
    sc: True
   
    FPS: 92.956
    CUDA: ~15%
    Compute_1: ~94%
    PerfCap: VRel, VOp, Pwr
   
6. RIFE filter for VapourSynth (ncnn Vulkan) - VapourSynth-RIFE-ncnn-Vulkan r3
    Interpolation: x2
    RIFE model: 4.0
    GPU thread: 2
    tta: False
    uhd: True
    sc: True
   
    FPS: 92.366
    CUDA: ~15%
    Compute_1: ~94%
    PerfCap: VRel, VOp, Pwr
   
7. RIFE filter for VapourSynth (ncnn Vulkan) - VapourSynth-RIFE-ncnn-Vulkan r3
    Interpolation: x2
    RIFE model: 4.0
    GPU thread: 2
    tta: False
    uhd: False
    sc: False
   
    FPS: 87.083
    CUDA: ~15%
    Compute_1: ~94%
    PerfCap: VRel, VOp, Pwr
   
8. RIFE filter for VapourSynth (ncnn Vulkan) - VapourSynth-RIFE-ncnn-Vulkan r3
    Interpolation: x2
    RIFE model: 4.0
    GPU thread: 3
    tta: False
    uhd: False
    sc: True
   
    FPS: 90.645
    CUDA: ~15%
    Compute_1: ~94%
    PerfCap: Idle