126 (edited by UHD 15-12-2021 01:29:32)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I also have a request to everyone posting on this thread and reading this thread.

I have written a request for help in optimizing RIFE for the latest graphics cards. You can find the details at this link: https://github.com/hzwer/arXiv2020-RIFE/issues/217

I would like as many people as possible to do a simple test, described below. You can share your results either directly on the RIFE project page here: https://github.com/hzwer/arXiv2020-RIFE/issues/217 or here, and I will then paste them together there.

With the test of course you will have to wait until the 4.0 model is working with SVP.


And here are the details of the proposed test:

Fixed test parameters:

SVP & RIFE filter for VapourSynth (PyTorch)
re-encoding with x2 interpolation
RIFE model: 4.0
scale=1.0

Variable test parameters:

Math precision: FP16 and FP32

Test results:

re-encoding speed [FPS]
CUDA utilisation [%]

Video file:

original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

127 (edited by dlr5668 15-12-2021 03:36:59)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I did clean install of SVP and didnt found

from vsrife import RIFE

in new generate.js. Anything I am missing ?

Usage should be still

from vsrife import RIFE
ret = RIFE(clip)

https://i.imgur.com/lWIcRq1.png

128 (edited by dlr5668 15-12-2021 03:57:56)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

ok, I fixed it. Full instruction

<... skipped ...>

3.8 model gives me 24 fps and this one 41
https://i.imgur.com/tkP4tFn.png

129 (edited by dlr5668 15-12-2021 04:16:18)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Demo file https://drive.google.com/file/d/1i3xlKb … sp=sharing with 3070 ti

https://i.imgur.com/wByQQxO.png

FP16 and FP32 produces almost same result

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

EZ to install guide
1. Stop svp and delete C:\Program Files (x86)\SVP 4
2. Download and unpack https://www.playbook.com/s/vadash/wcVXb … Ky5jaQMkLs

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Wanted to try RIFE for transcoding, I usually transcode with my own profile. But with RIFE profile, every time the transcode reaches 100% it gets stuck there and never finishes, any tips?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre
any tips?

check the log! big_smile

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Updated instructions there: https://www.svp-team.com/forum/viewtopi … 169#p80169

134 (edited by UHD 16-12-2021 01:33:14)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

3.8 model gives me 24 fps and this one 41
https://i.imgur.com/tkP4tFn.png


Many thanks for the tests! A huge improvement in the RIFE algorithm can be seen!

These are for comparison the results on a GeForce RTX 3090 graphics card done by: lwk7454:


lwk7454 wrote:

Parameters:
Test-Time Augmentation: Enabled [sets RIFE filter for VapourSynth (PyTorch)]
re-encoding with x2 interpolation
RIFE model: 3.8
scale=1.0
Encoder: NVIDIA NVENC H.264

720p, FP16
FPS: 63.5
Cuda: 56%

720p, FP32
FPS: 69.8
Cuda: 58%

1080p, FP16
FPS: 26.9
Cuda: 62%

1080p, FP32
FPS: 28.1
Cuda: 66%

https://www.svp-team.com/forum/viewtopi … 526#p79526

We can see how similar results the 3.8 model gave for GeForce RTX 3090 and GeForce RTX 3070 Ti graphics cards:

28.1 FPS vs. 24 FPS

and how huge the difference is with the 4.0 model: 40.9 FPS!!!!!


dlr5668, could you please specify the CUDA load for these tests. This is very important data to determine if the new model makes better use of the graphics cards capabilities or if there is still room for optimization, as I described here: https://github.com/hzwer/arXiv2020-RIFE/issues/217

By the way, thanks for checking that FP32 performance is close to FP16. However, if you would provide precise data for which precision this is, that would be additional information that I could paste in the link provided above.

It would be good if you could present data in the same range as lwk7454, it would be consistent with what I presented here https://github.com/hzwer/arXiv2020-RIFE/issues/217 and maybe the RIFE developer will come up with something extra to achieve 50FPS for 1080p, which would be satisfactory for real time interpolation!

135 (edited by UHD 16-12-2021 01:38:15)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

=== RIFE / PyTorch installation ===

Many thanks Chainik!

Could you find a moment of time and add a new parameter to the RIFE configuration in SVP:

https://github.com/HolyWu/vs-rife/blob/ … t__.py#L20

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I tried x3 with

        smooth = RIFE(input_m, 3)
        smooth = smooth.resize.Point(format=input_um.format.id, matrix_s="709")
        smooth = core.svp2.SmoothFps_RIFE(smooth,smoothfps_params,src=input_um,fps=src_fps*3)

but SVP crashed. Probably need to change it in some place else

137 (edited by dlr5668 16-12-2021 03:21:38)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Sadly cant measure CUDA

https://i.imgur.com/83Dd84a.png

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> Could you find a moment of time and add a new parameter to the RIFE configuration in SVP

yeah... but you do understand that x3 (or hypothetical x2.5) will be 2 times slower, and x4 will be 3 times slower, right?

139 (edited by dlr5668 16-12-2021 08:16:23)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Yep, sadly its not like svp. Calculate vectors, then any interpolation takes close to zero time

140 (edited by UHD 16-12-2021 15:30:53)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

Sadly cant measure CUDA

https://i.imgur.com/83Dd84a.png


Check out my post here: https://www.svp-team.com/forum/viewtopi … 493#p79493 and especially lwk7454's reply here: https://www.svp-team.com/forum/viewtopi … 497#p79497

Am I seeing correctly? Even 42.5fps manages to be achieved?

141 (edited by UHD 16-12-2021 14:25:57)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> Could you find a moment of time and add a new parameter to the RIFE configuration in SVP

yeah... but you do understand that x3 (or hypothetical x2.5) will be 2 times slower, and x4 will be 3 times slower, right?

Thanks smile

If I understand correctly only integer is possible:
https://github.com/HolyWu/vs-rife/blob/ … t__.py#L40

so x2, x3, x4, x5 should be possible, but not x2.5.

Also I understand the mechanism, that x3 is 2 new interpolated frames, so theoretically it means 2x more computing power or takes 2x longer than interpolating 1 frame with x2 setting.

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> but not x2.5

this line (https://github.com/HolyWu/vs-rife/blob/ … t__.py#L85)
"(n % multi) / multi" is a float value meaning the exact position [0;1] between source frames

so it's easy to make a x2.5-sequence like frame0, 0.4, 0.8, 1.2, 1.6, frame2, ...
still it will be 4 generated frames for every 2 source ones => two times slower than frame doubling

143 (edited by UHD 16-12-2021 15:17:48)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

If I understand what is written here correctly: https://arxiv.org/pdf/2011.06294.pdf

Position 0 is the first real frame.
Position 1 is the second real frame.

Before RIFEm and RIFE 4.0, only position 0.5 (x2) was possible.
To get positions 0.25 and 0.75 we had to repeat the process again.

Now it is possible to get any position between 0 and 1 the first time on the principle:
0, 0.5, 1 (x2)
0, 0.333, 0.667, 1 (x3)
0, 0.25, 0.5, 0.75, 1 (x4)
0, 0.2, 0.4, 0.6, 0.8, 1 (x5)
... and so on

I, for quality reasons would never use a method that omits real frame 1.
It is better to use: Custom Resolution Utility (CRU): https://www.monitortests.com/forum/Thre … tility-CRU

In anyway, trying to use x2.5 should generate an error report:
https://github.com/HolyWu/vs-rife/blob/ … t__.py#L40

144 (edited by UHD 16-12-2021 23:25:28)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Too bad I don't have a RIFE compatible graphics card yet, but I'll play around with theoretical calculations based on the dlr5668 result:

1920x1080 @23.976

42.5 fps: https://www.svp-team.com/forum/viewtopi … 699#p79699

so

21.25 new interpolated frames per sec.


90 minutes movie has
=90*60*23.976=129470.4 frames

re-encoding will take:

x5 (for 120Hz TV or monitor)
=4*129470.4/21.5=24088 seconds or 6h42m

x10
(for 240Hz TV or monitor)
=9*129470.4/21.5=54197 seconds or 15h4m

smile

145 (edited by UHD 16-12-2021 23:30:18)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Full list as per details in previous post:

x2    1h41m
x3    3h21m
x4    5h2m
x5    6h42m
x6    8h22m
x7    10h3m
x8    11h43m
x9    13h23m
x10  15h4m

Of course, it may be that 2 new interpolated frames at the x3 setting will not necessarily require 2x the time to interpolate 1 frame. It is possible that some synergy effect will take place and x1.9 time or 1.8 will be enough. This needs to be verified in a real test.

146 (edited by dlr5668 17-12-2021 00:41:35)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I also reuploaded image. Almost good to distribute with svp installer like regular RIFE wink You can remove all .lib and *train* files from pytorch + apply NTFS compress to reduce unpacked

https://i.imgur.com/gljSMfI.png

147 (edited by dlr5668 17-12-2021 00:55:11)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

Am I seeing correctly? Even 42.5fps manages to be achieved?

Yep. I use my PC as server for hyper-v machines so I cant change GPU driver or install CUDA toolkit

4.0 is also better under GPU load. 3.8 dropped my encode fps to 16 and 4.0 maintains 30-35 with less VRAM. Crazy magic

148 (edited by dlr5668 17-12-2021 00:53:34)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

re-encoding will take:

x5 (for 120Hz TV or monitor)
=4*129470.4/21.5=24088 seconds or 6h42m

x10
(for 240Hz TV or monitor)
=9*129470.4/21.5=54197 seconds or 15h4m

smile

Not worth it. Just switch TV to 50/100 Hz. Wish we could include step with

# "ReClock" replacement
video-sync=display-resample
video-sync-max-video-change=10

so it speedup 23.976 * 2 to 50 fps. Maybe Chainik will help ?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

3080 ti is very close to real time speed with cuda for 1080p
https://i.imgur.com/RasLLrw.png
4k runs at 11.6fps which is much faster than the 1.9fps I got without cuda
https://i.imgur.com/XQ0Iiux.png
but unfortunately the interpolated video has striped artifacts outside of a 2048x2048 area
https://i.imgur.com/wAGBCLK.jpg
All in all very cool, I will definitely be watching anything 720p with rife, which is only Star Trek in 480p right now.  I have already transcoded some episodes of 1080p shows, perhaps I will watched the rife'd video with the regular SVP filter on top

150

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> Could you find a moment of time and add a new parameter to the RIFE configuration in SVP

yeah... but you do understand that x3 (or hypothetical x2.5) will be 2 times slower, and x4 will be 3 times slower, right?

Have you succeeded in adding the following parameter yet?
https://github.com/HolyWu/vs-rife/blob/ … t__.py#L20