1 (edited by UHD 17-08-2021 15:34:05)

Topic: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

As we all probably already know, since June 8 SVP supports RIFE AI interpolation at BETA stage: https://www.svp-team.com/news/

For this interpolation SVP uses the RIFE filter for VapourSynth created by HolyWu:
https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan

20 days ago HolyWu wrote:

The algorithm is not fast enough for playback, so you probably wanna go with SVP. Anyway I will have another CUDA version which is 3 times faster than Vulkan.

https://github.com/HomeOfVapourSynthEvo … n/issues/5


And here it is: a brand new RIFE filter for VapourSynth which beats its predecessor in performance and eliminates several other drawbacks: https://github.com/HolyWu/vs-rife

Hence my big request to Chainik and MAG79 to add this new RIFE filter for VapourSynth to SVP: https://github.com/HolyWu/vs-rife

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

as discussed before, this will need A LOT of dependencies and have a huge loading time

we can't bundle PyTorch with SVP

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Yes. These will be the main disadvantages of this filter. I am, however, at the moment preparing a huge list of benefits of the new filter over the previous one. It is going to be really long posts. I hope that I will be able to convince you, but also at the same time encourage other users to the new filter. Please give me some time though, English is not my native language, so it will take me until late evening or maybe even tomorrow to write all the posts. I really care - I am a big fan of real-time motion interpolation.

But now I will briefly address the drawbacks. A LOT of dependencies and a huge loading time can probably be offset by using a fast NVMe SSD based on PCIe 4.0 x4 interface, for example: Seagate FireCuda 530 with 7300 MB/s sequential read. In a year's time we may have 2x faster drives with the widespread adoption of the PCIe 5.0 x4 interface, so maybe even 14600 MB/s!

But what is most important and what I value most about SVP is that it gives users choice. SVP gives countless configurations and choices for each user and thus leaves other programs like DmitriRender behind.

Please give users the choice to use the RIFE filter for VapourSynth based on ncnn or the RIFE filter for VapourSynth based on PyTorch in SVP.

And I'm back to preparing more posts....

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

we can't bundle PyTorch with SVP

I see, in that case, if someone wanted to use the RIFE filter for VapourSynth based on PyTorch in SVP there may be a message warning of all the consequences of downloading this package with instructions on how to do it.

I know that people with poor connections may have problems downloading, as well as old HDDs will take a long time to load data into RAM or VRAM, but let's give it a try.

5 (edited by UHD 17-08-2021 19:29:06)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

And now I will present below the main advantages of the new RIFE filter for VapourSynth over its predecessor:

1. The RIFE filter for VapourSynth based on PyTorch: https://github.com/HolyWu/vs-rife is at least 3 times faster in performance than the RIFE filter for VapourSynth based on ncnn: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan

Developer confirmation of the above filters:
https://github.com/HomeOfVapourSynthEvo … n/issues/5

SVP developer confirmation:
https://github.com/nihui/rife-ncnn-vulkan/issues/22

Confirmation by one more person:
https://github.com/nihui/rife-ncnn-vulkan/issues/26


This is obviously a major advantage, especially if we think about real-time RIFE AI interpolation. This new PyTorch-based filter is especially useful for SVP users, as SVP is the only program on the market that allows real-time motion interpolation using AI with a GUI, that is friendly and easy to use for any user.

If someone wants to transcode video using RIFE AI motion interpolation then of course they have a wider choice of tools, because in addition to SVP they can choose from Flowframes, DVDFab or Rife-App among others.  When transcoding, the speed does not matter so much, because whether the procedure takes 3 hours or 9 hours, you still have to wait patiently.

However, when we want to perform interpolation in real time the situation looks completely different. Here, either the interpolation works in time or it does not. Either it is possible to watch or it is not.

The situation is worse than when playing computer games, where for example we either have 120fps or 40fps. In the latter case it is still possible to play, although with much less comfort. However, gamers can spend a lot of money, time and overclocking to gain at least those extra 10% fps!

And we have the chance to increase the performance by 3 times!!!

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

assuming nobody really use TTA option...

1. Dependencies installation:
--------------
download https://bootstrap.pypa.io/get-pip.py into SVP 4\mpv64

run cmd as Administrator
cd SVP 4\mpv64

> python get-pip.py

edit python38._pth, add 3 lines:
.
lib
lib\site-packages

run
> python -m pip install torch==1.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
> python -m pip install vsrife

will download 3+ GB and unpack it into 6+ GB of data big_smile

2. SVP modification
---------------
replace script\base.py and script\generate.js, restart SVP

in the RIFE video profile set
- TTA = disabled -> use RFIE/ncnn/Vulkan
- TTA = enabled -> use RIFE/torch/CUDA

"scale: Controls the process resolution for optical flow model. Try scale=0.5 for 4K video."
"Math precision" = Single ----> scale=1.0
"Math precision" = Half ---> scale=0.5 - increases encoding speed by 2

"GPU device" -> choose device in case there're several CUDA devices in the system.

"GPU threads" does nothing.

AI model is "3.5" by default, no idea what is the difference hmm, but you can set 3.1, 3.5 or 3.8 in the base.py file.

Post's attachments

base.py 2.03 kb, 136 downloads since 2021-08-17 

generate.js 18.84 kb, 128 downloads since 2021-08-17 

7 (edited by UHD 17-08-2021 20:45:15)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Thank you very much!

I was just writing the 2nd point of my essay, but it looks like it's not necessary. OK, maybe I'll still finish it and paste it in here sometime. It is not important now!

Chainik, please write if you were able to interpolate in real time a higher resolution file using the 3.8 model than last time?

Chainik wrote:

GPU; VRAM; CPU utilization; SVP index

424*240 - 23% (65% compute_1); 0.9 GB; zero; 1.0
640*360 - 9% (100% compute_1); 0.9 GB; zero; 0.76

https://www.svp-team.com/forum/viewtopi … 96&p=2


3.8 model is the fastest!

Based on the evaluation of dozens of videos, the v3.8 model has achieved an acceleration effect of more than 2X while surpassing the effect of the RIFEv2.4 model.

https://github.com/hzwer/arXiv2020-RIFE/issues/176

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

mpv just hangs after switching to RIFE profile in a real time
no idea why

transcoding seems to work

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I hope that a solution can be found. It is the real-time interpolation that I care most about. Of course it could be tomorrow, the day after tomorrow next week, whenever you have some time to find the cause.

Thank you once again!

10 (edited by UHD 20-08-2021 19:37:58)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Congratulations on the release of SVP for Android today!
I hope this will be another successful milestone in SVP development, alongside the RIFE implementation.

Hopefully there will now be a little more time on finding a solution to the problem of mpv hanging after switching to RIFE profile in a real time. Or maybe something has already been done?


By the way I have another question:

In the original RIFE project "scale" and "math precision" are two separate parameters to choose from.

The first one is the 64th line:

https://github.com/hzwer/arXiv2020-RIFE … deo.py#L64

Whereas math precision is 62 line:

https://github.com/hzwer/arXiv2020-RIFE … deo.py#L62

Selecting half precision (FP16) increases the performance by up to 4x:

the fp16 speed is phenomenal (~x4)

https://github.com/hzwer/arXiv2020-RIFE/issues/188

Now my question, because I don't really understand this fragment:

Chainik wrote:

"scale: Controls the process resolution for optical flow model. Try scale=0.5 for 4K video."
"Math precision" = Single ----> scale=1.0
"Math precision" = Half ---> scale=0.5 - increases encoding speed by 2

What does the above setting refer to? I guess "scale", but then where do you set the calculation precision: full precision (FP32) and half precision (FP16)?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> I guess "scale"

yep

> but then where do you set the calculation precision

there's no "precision" param in the "vsrife" filter
https://github.com/HolyWu/vs-rife/blob/ … t__.py#L16

12 (edited by UHD 20-08-2021 20:19:21)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

That's not good...

the fp16 speed is phenomenal (~x4)

https://github.com/hzwer/arXiv2020-RIFE/issues/188


And was it possible to solve the problem of mpv hanging after switching to RIFE profile in a real time?

13 (edited by UHD 21-08-2021 17:23:16)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik, am I perhaps too impatient with asking about the problem of mpv hanging after switching to RIFE profile in a real time? Do you need a little more time to look for solutions?

At the moment we have 4 cases:

1. SVP+RIFE filter for VapourSynth (ncnn) - transcode very slowly
2. SVP+RIFE filter for VapourSynth (ncnn) - run very slowly in real time
3. SVP+RIFE filter for VapourSynth (PyTorch) - transcode 2-3 times faster
4. SVP+RIFE filter for VapourSynth (PyTorch) - mpv hangs



"start playback in mpv (preferably) or VLC, DirectShow players may also work via Vapoursynth Filter (VPSF)"

.https://www.svp-team.com/wiki/RIFE_AI_interpolation

Have you tried using VLC or any DirectShow player?
Have you tried using some very, very low resolution file?
Have you tried waiting a few minutes?

I'd love to help, but I'm neither a programmer nor have a proper graphics card.

Maybe someone else besides Chainik could test the latest RIFE filter for VapourSynth based on PyTorch and share their impressions on this thread?

If anyone hasn't tried it yet, here are the advantages of RIFE over MVTools2:
http://forum.doom9.net/showpost.php?p=1 … stcount=17
http://forum.doom9.net/showpost.php?p=1 … stcount=24

This is an improvement on an aspect that has been raised many times on this forum:
https://www.svp-team.com/forum/viewtopic.php?id=5500
https://www.svp-team.com/forum/viewtopic.php?id=6139
https://www.svp-team.com/forum/viewtopic.php?id=4382

14 (edited by UHD 30-08-2021 14:25:18)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik, did you manage to solve the problem of mpv hanging after switching to RIFE profile in a real time?

What about other players?

"start playback in mpv (preferably) or VLC, DirectShow players may also work via Vapoursynth Filter (VPSF)"

https://www.svp-team.com/wiki/RIFE_AI_interpolation

Have you tried using VLC or any other DirectShow player?
Have you tried using other some very, very low resolution file?
Have you tried waiting a few more minutes?
Have you checked, maybe hardware is the bottleneck? VRAM? RAM?

I'd love to help, but I'm neither a programmer nor have a proper graphics card currently.

15

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

there's no "precision" param in the "vsrife" filter
https://github.com/HolyWu/vs-rife/blob/ … t__.py#L16

Now it is!!!!!!!!!!!!!!

HolyWu just released a new version yesterday:
https://github.com/HolyWu/vs-rife/releases/tag/v1.1.0

I guess he reads this thread;-) If so, thanks a lot HolyWu!



Selecting half precision (FP16) increases the performance by up to 4x:

the fp16 speed is phenomenal (~x4)

https://github.com/hzwer/arXiv2020-RIFE/issues/188 


Chainik, this is yet another reason to take a second look at running this filter in SVP in real time. The speed should increase dramatically!

HolyWu is actively supporting the RIFE filter for VapourSynth based on PyTorch, maybe he will be able to help you over the implementation of this filter for real-time interpolation?

Chainik, you will know best what to ask. I can of course write a request myself, but apart from the fact that it doesn't work I won't be able to add anything else, because I simply don't know anything about it :-(

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

fp16 vs. fp32 --> 64 fps vs. 53 fps (704*528 video); 31 fps vs. 28 fps (720p) hmm

upgrade from ver. 1.0.0 (https://www.svp-team.com/forum/viewtopi … 023#p79023)

> python -m pip install --upgrade vsrife

replace SVP 4\script\base.py with the one attached

from SVP's "RIFE AI engine" video profile:
- "Test-Time Augmentation": Enabled - use Pytorch version, Disabled - use Vulkan version
- "AI model": does nothing with Pytorch version; generic/anime model in Vulkan version
- "Math precision": precision in both versions
- "GPU threads": Pytorch: 2 = "scale=1.0", >=4 = "scale=0.5"; Vulkan - GPU threads
- "GPU device": GPU device in both versions

Post's attachments

base.py 2.06 kb, 92 downloads since 2021-09-02 

17 (edited by UHD 02-09-2021 22:50:42)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Thank you very much for the upgrade and sharing the test results.


Chainik wrote:

fp16 vs. fp32 --> 64 fps vs. 53 fps (704*528 video); 31 fps vs. 28 fps (720p) hmm

Strange...
cheesywheesy claims x4 on 3090.
n00mkrad, creator of Flowframes, claims that fp16 does not work with newer models:
https://github.com/hzwer/arXiv2020-RIFE/issues/188

Maybe it's a matter of the interpolation model used, or the GPU? I unfortunately can't check this as I don't have a compatible graphics card yet. Maybe someone else could check the speed in SVP and in Flowframes?


Chainik wrote:

- "AI model": does nothing with Pytorch version;

Does this mean that you are using a default model in the Pytorch version? If so, this is version 3.5:
https://github.com/HolyWu/vs-rife/blob/ … t__.py#L10

If so, it is neither the latest model nor, unfortunately, the fastest.
3.8 model is the fastest!

Based on the evaluation of dozens of videos, the v3.8 model has achieved an acceleration effect of more than 2X while surpassing the effect of the RIFEv2.4 model.

https://github.com/hzwer/arXiv2020-RIFE/issues/176

new model (3.8). It sometimes give worse results, other times better. But it is a looot faster.

https://grisk.itch.io/rife-app/devlog/2 … 207-update


Chainik, could you take a second look at running this filter in SVP in real time.

Could you please try to use VLC or any other DirectShow player?
Could you please try to use other some very, very low resolution file?
Could you please try to wait a few more minutes?
Have you checked, maybe hardware is the bottleneck? VRAM? RAM?

I'd love to help, but I'm neither a programmer nor have a proper graphics card currently.

HolyWu is actively supporting the RIFE filter for VapourSynth based on PyTorch, maybe he will be able to help you over the implementation of this filter for real-time interpolation?

Chainik, you will know best what to ask. I can of course write a request myself, but apart from the fact that it doesn't work I won't be able to add anything else, because I simply don't know anything about it :-(

18 (edited by UHD 09-09-2021 15:16:06)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

HolyWu just released a new 1.2.0 version yesterday:
https://github.com/HolyWu/vs-rife/releases/tag/v1.2.0

Chainik, could you take a second look at running this filter in SVP in real time?

HolyWu is actively supporting the RIFE filter for VapourSynth based on PyTorch, maybe he will be able to help you over the implementation of this filter for real-time interpolation?

19 (edited by UHD 15-09-2021 20:24:23)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

RIFE filter for VapourSynth (PyTorch) is already completely capable of producing enough frames per second in HD files today to think about real-time interpolation. Below are examples of using RIFE in FlowFrames:

RTX 3080 - 1280x720p - 55.5 FPS
RTX 3090 - 1280x720p - 60 FPS

More examples here:
https://github.com/n00mkrad/flowframes/ … chmarks.md

It should be mentioned that these are examples from 8 months ago, and the latest 3.8 algorithm is much faster, as I wrote above.

Are there any people reading this thread who have already tried the RIFE filter for VapourSynth (PyTorch) implemented in SVP?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Hm, I thought we wont be able to watch HD real time before new RTX 6XXX series. I ll try with my 3070ti soon

21

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I would be very grateful if you could test SVP with RIFE filter for VapourSynth (PyTorch) on 3070ti.

I would be particularly interested in how the following 2 issues compare on the new Ampere microarchitecture:

1. Do you have the same problem as Chainik on the Turing microarchitecture when attempting real-time interpolation?

Chainik wrote:

mpv just hangs after switching to RIFE profile in a real time
no idea why

2. What is the performance when choosing half precision (FP16)? Here quite different results appear:

Chainik wrote:

fp16 vs. fp32 --> 64 fps vs. 53 fps (704*528 video); 31 fps vs. 28 fps (720p) hmm


the fp16 speed is phenomenal (~x4)

https://github.com/hzwer/arXiv2020-RIFE/issues/188

22 (edited by dlr5668 16-09-2021 14:12:41)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I followed original chainik post. I ll be uploading videos to https://www.playbook.com/s/vadash/wcVXb … Ky5jaQMkLs so we can compare
My hardware https://valid.x86.fr/4z07b1
Source will be Stargate.Atlantis.S05E01.Multi.1080p.BluRay.DTS-HDMA.5.1.H.264-CELESTIS-055 30 sec fragment
Encoding H265 max q

my gpu (2010 @ 1V +500 mem)
https://i.imgur.com/gshg3JZ.png

half 3.8
https://i.imgur.com/UPZoB7W.png

full 3.8
https://i.imgur.com/l91DsBP.png

half 3.5
https://i.imgur.com/MRQIrUN.png

I was changing this line and restarting svp, right ?

smooth = RIFE(input_m,model_ver=3.5,scale=1.0 if rife_precision else 0.5,device_index=rife_gpu)

720p test
https://i.imgur.com/U62Cbp5.png

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

ok, so 3070 Ti is the absolute minimum for 1080p real-time playback

what is GPU temperature after 1 hour of encoding? smile

> I was changing this line and restarting svp, right ?

yep

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

ok, so 3070 Ti is the absolute minimum for 1080p real-time playback

what is GPU temperature after 1 hour of encoding? smile

> I was changing this line and restarting svp, right ?

yep

I mine 24/7 on this PC when I am not using it so its 75c core / 96 mem junction with silent curve

25

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I cannot believe it!

1920x1080 23.976 x2 and 54,0fps with RIFE!!!

So, NVIDIA GeForce RTX 3070 Ti is capable of producing enough frames per second in 1080p files today to think about x2 real-time interpolation!!!

Please post more, you are doing a great job!

Now, the most important question: do you have the same problem as Chainik on the Turing microarchitecture when attempting real-time interpolation?

Chainik wrote:

mpv just hangs after switching to RIFE profile in a real time
no idea why

Thank you very much!!!