Topic: RIFE filter for VapourSynth - AI interpolation in SVP?

RIFE in SVP: https://www.svp-team.com/wiki/RIFE_AI_interpolation
-------------------------------

RIFE filter for VapourSynth: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan

SVP with real-time RIFE support could provide better results and less artifacts than the motion interpolation algorithms in the best TVs!

Question to developers: is it now possible to try to combine RIFE with SVP?

2 (edited by UHD 01-06-2021 22:16:25)

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

If anyone still doesn't know what RIFE is here are links to the source:

RIFE: https://github.com/hzwer/Arxiv2020-RIFE

RIFE ncnn Vulkan: https://github.com/nihui/rife-ncnn-vulkan

Also, you can check for yourself the motion interpolation effects of several AI-based methods, including RIFE, in a Windows environment using:

Flowframes: https://nmkd.itch.io/flowframes
https://github.com/n00mkrad/flowframes

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

how fast is it ?

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

laptop rtx 2060 --> 704*528 @30->60 --> 24 fps transcoding speed
1080p @24->48 --> 4 fps
(fp32 = false, tta = false)

is it supposed to use all the GPU? hmm because I only see up to 10% GPU load regardless of gpu_thread value

---
flowframes shows the same speed and gpu usage so lets assume this is "normal"

bottom line - I think we could integrate this for the SVPcode usage

5 (edited by UHD 02-06-2021 19:49:05)

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Chainik wrote:

laptop rtx 2060 --> 704*528 @30->60 --> 24 fps transcoding speed

I think more powerful PC dedicated graphics cards, especially the latest GeForce RTX 30 should perform better. However, even this setup gives hope that at least DVD-quality material can be interpolated in real time. At least for now, as the RIFE algorithms are constantly being optimised.

NVIDIA GeForce RTX 2060 Mobile - relative performance:
https://www.techpowerup.com/gpu-specs/g … bile.c3348

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Chainik wrote:

is it supposed to use all the GPU? hmm because I only see up to 10% GPU load regardless of gpu_thread value

No idea what this is all about. However, if 10% GPU usage produces these results then there is hope for an increase in fps as much as 10X!!! I have no idea about programming, but I think a programmer with a programmer will very quickly be able to come to ways to use the full power of the GPU.


It may sound strange in the context of what I am writing about and striving for, but at the moment I don't even have a way to check how Flowframes works. I was supposed to build a PC based on the AMD Ryzen Threadripper 3970X in early 2020, but had to postpone plans.... Now I know that a good GPU will still come in handy. Well... I'm looking forward, especially to SVP on this giant CPU and compare the effects with RIFE on a good GPU. For the moment I'm relying on the opinions of others, and they are extremely enthusiastic about RIFE.

7 (edited by UHD 02-06-2021 15:39:41)

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Chainik wrote:

bottom line - I think we could integrate this for the SVPcode usage

Is it possible to implement RIFE so that we can watch motion interpolated videos in real time without transcoding? I care about this the most, and probably not only me. We can transcode movies using Flowframes, however, every transcoding means loss of some quality, unless we use -x265-params-lossless=1, but then we need huge HDD space;-)

Yes, I know. For starters, materials in DVD resolution and below are only with a powerful GPU.

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

„Q: What's the difference between RIFE CUDA and RIFE NCNN? Which one should I use?
A: The results should be identical, however, RIFE-NCNN also runs on AMD cards, CUDA only on Nvidia. If you have an Nvidia card, use CUDA as it's faster.”
https://github.com/n00mkrad/flowframes

For now we have an attempt to create a universal version of RIFE filter for VapourSynth based RIFE ncnn Vulkan. If in the future there will be a RIFE filter for VapourSynth dedicated for CUDA then there is a chance for some performance increase. I also wonder if it will be possible to increase performance with Tensor Cores?

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

yeah, CUDA version is like 3 times faster than ncnn (in flowframes) and loads my GPU up to 50%

10

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Great!!!

Does that mean: laptop rtx 2060 --> 704*528 @30->60 --> 72 fps transcoding speed?

That would already make sense!

On the other hand RIFE filter for VapourSynth without CUDA optimization might not be enough.

Now the question is whether to watch motion interpolated videos in real time without transcoding using AI RIFE interpolation and SVP...

...RIFE filter for VapourSynth is necessary?

Or would RIFE alone be sufficient: https://github.com/hzwer/Arxiv2020-RIFE

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

> Great!!!

not great, because we definitely won't use 2.5 GB of dependencies required to run the "original" RIFE

there's a ncnn/CUDA (https://github.com/atanmarko/ncnn-with- … ocator.cpp) but it was last updated in January... so I dunno...

12

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

If I understand correctly RIFE filter for VapourSynth does not need these dependencies?

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

more like "RIFE over ncnn does not need these dependencies"
and the "official" ncnn only supports Vulkan as of now

14

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

That is, it would be best for the author of the RIFE filter for VapourSynth to implement ncnn-with-cuda in his code. That 3x performance gain is worth it. It gets a bit complicated.

15

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Still such a question arises: Flowframes performs 2 operations:

-interpolation of motion using, for example, RIFE algorithms
-transcoding to the resulting video file or image files (PNG).

I am curious how much time could be saved if this second operation was omitted, i.e. what I am aiming at: sending the finished result of motion interpolation directly to the monitor, without transcoding and saving on HDD/SSD?

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

> how much time could be saved if this second operation was omitted

zero big_smile encoding via NVENC costs nothing...
even if you want good old software encoder then it will be like 10 times faster than "rife over cuda"

17

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

So we have 3 options:

1. Implement the "original" RIFE in SVP - disadvantage: 2.5 GB of dependencies
2. Implement the RIFE filter for VapourSynth in SVP - disadvantage: worse performance
3. Ask the developer of the RIFE filter for VapourSynth to implement the code https://github.com/atanmarko/ncnn-with- … ocator.cpp

Alternatively there is a 4th option...

Chainik maybe you could look into whether you would be able to somehow combine the codes from point 3 above to speed up the RIFE filter for VapourSynth code to be able to watch motion interpolated videos in real time without transcoding using AI RIFE interpolation and SVP? Just as an experiment. You are a motion interpolation enthusiast, just like all of us on this forum. The difference, however, is that you have a knowledge of how it all works that we do not.

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

>  Ask the developer of the RIFE filter for VapourSynth to implement the code

actually there no changes needed in the "RIFE filter for VapourSynth" itslef
the problem is "ncnn with CUDA" is not completed yet, some functions necessary for RIFE are not implemented... and the last commit was on Jan 18.

> Implement the "original" RIFE in SVP - disadvantage: 2.5 GB of dependencies

... plus very long (re-)start time. a video player will hang for 5-10 secs on each seek, for example.

19

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Chainik wrote:

actually there no changes needed in the "RIFE filter for VapourSynth" itslef
the problem is "ncnn with CUDA" is not completed yet, some functions necessary for RIFE are not implemented... and the last commit was on Jan 18.

This is probably even better. NCNN has a much broader application than motion interpolation, so there's a good chance that someone will want to improve NCNN performance by optimizing it for CUDA. It's not clear if the lack of updates are some temporary issues, lack of times, or something out of the blue? On the other hand, it's very interesting how so many new tools dedicated to AI and using AI are being developed in just the last year of time!

20

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Chainik wrote:

... plus very long (re-)start time. a video player will hang for 5-10 secs on each seek, for example.

Are you using a fast NVMe SSD? Maybe it takes that long to load those 2.5 GB of dependencies each time? Have you checked what is the bottleneck in that 5-10 seconds: GPU, CPU, HDD/SDD?

Would it be possible to check if the same amount of time would be needed if you put all data with all dependencies in RAM Disk?

21 (edited by UHD 03-06-2021 17:23:39)

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

...and thank you very much for that:
https://github.com/nihui/rife-ncnn-vulkan/issues/22

I keep my fingers crossed!

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

https://www.svp-team.com/wiki/RIFE_AI_interpolation

23

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

I cannot believe it! You did it!

Tell me, did you somehow manage to get the performance of the CUDA?

Is it possible to get smooth x2 playback with RIFE on your GPU on some resolution? For example 480p? 720p?

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

> did you somehow manage to get the performance of the CUDA?

nope

> Is it possible to get smooth x2 playback with RIFE on your GPU on some resolution? For example 480p? 720p?

not a chance
---
240p works great, 360p is too much big_smile

25

Re: RIFE filter for VapourSynth - AI interpolation in SVP?

Chainik wrote:

240p works great, 360p is too much big_smile

That is what I was about to ask. Every GPU should have a resolution threshold at which it can handle and one where it can't.

Here is what we know:
NVIDIA GeForce RTX 2060 Mobile
240p works great
360p is too much

Can you share what your GPU, VRAM, CPU utilization was at 240p and 360p?