Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> python -m pip install numpy

252 (edited by UHD 11-06-2022 21:56:51)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Fourth-generation Tensor Cores - Peak FP16 using the Sparsity feature:

2000 TFLOPS - NVIDIA H100 SXM5 (528 Tensor Cores, 700W TDP)
1600 TFLOPS - NVIDIA H100 PCIe (456 Tensor Cores, 350W TDP)

Third-Generation Tensor Cores - Peak FP16 using the Sparsity feature:

624 TFLOPS - NVIDIA A100 (432 Tensor Cores, 400W TDP)
320 TFLOPS - NVIDIA GeForce RTX 3090 Ti (336 Tensor Cores, 450W TDP)
284 TFLOPS - NVIDIA GeForce RTX 3090 FE (328 Tensor Cores, 350W TDP)
273 TFLOPS - NVIDIA GeForce RTX 3080 Ti (320 Tensor Cores, 350W TDP)
244 TFLOPS - NVIDIA GeForce RTX 3080 12GB (280 Tensor Cores, 350W TDP)
238 TFLOPS - NVIDIA GeForce RTX 3080 10GB FE (272 Tensor Cores, 320W TDP)
174 TFLOPS - NVIDIA GeForce RTX 3070 Ti (192 Tensor Cores, 290W TDP)
162.6 TFLOPS - NVIDIA GeForce RTX 3070 FE (184 Tensor Cores, 220W TDP)
129.6 TFLOPS - NVIDIA GeForce RTX 3060 Ti (152 Tensor Cores, 200W TDP)
102.4 TFLOPS - NVIDIA GeForce RTX 3060 (112 Tensor Cores, 170W TDP)

Second-Generation Tensor Cores - Peak FP16:

130.5 TFLOPS - NVIDIA TITAN RTX (576 Tensor Cores, 280W TDP)
113.8 TFLOPS - NVIDIA GeForce RTX 2080 Ti FE (544 Tensor Cores, 260W TDP)
107.6 TFLOPS - NVIDIA GeForce RTX 2080 Ti (544 Tensor Cores, 250W TDP)
89.2 TFLOPS - NVIDIA GeForce RTX 2080 Super FE (384 Tensor Cores, 250W TDP)
84.8 TFLOPS - NVIDIA GeForce RTX 2080 FE (368 Tensor Cores, 225W TDP)
80.5 TFLOPS - NVIDIA GeForce RTX 2080 (368 Tensor Cores, 215W TDP)
72.5 TFLOPS - NVIDIA GeForce RTX 2070 Super FE (320 Tensor Cores, 215W TDP)
63 TFLOPS - NVIDIA GeForce RTX 2070 FE (288 Tensor Cores, 185W TDP)
59.7 TFLOPS - NVIDIA GeForce RTX 2070 (288 Tensor Cores, 175W TDP)
57.4 TFLOPS - NVIDIA GeForce RTX 2060 Super (272 Tensor Cores, 175W TDP)
51.6 TFLOPS - NVIDIA GeForce RTX 2060 (240 Tensor Cores, 160W TDP)

First-generation Tensor Cores - Peak FP16:

125 TFLOPS - NVIDIA Tesla V100 (640 Tensor Cores, 300W TDP)

---------------------------------------------------------
H100
https://resources.nvidia.com/en-us-tensor-core
A100, V100
https://images.nvidia.com/aem-dam/en-zz … epaper.pdf
RTX 3090 FE, RTX 3080 10GB FE, RTX 3070 FE, TITAN RTX, RTX 2080 Super FE, RTX 2070 Super FE
https://images.nvidia.com/aem-dam/en-zz … per-V1.pdf
RTX 2080 Ti FE, RTX 2080 Ti, RTX 2080 FE, RTX 2080, RTX 2070 FE, RTX 2070
https://images.nvidia.com/aem-dam/en-zz … epaper.pdf
RTX 3090 Ti
https://www.anandtech.com/show/17334/nv … lpowerful-
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3829
RTX 3080 Ti, RTX 3080 12GB, RTX 3070 Ti
https://www.anandtech.com/show/17204/nv … more-money
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3735
https://www.techpowerup.com/gpu-specs/g … 2-gb.c3834
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3675
RTX 3060 Ti, RTX 3060, RTX 2060
https://www.anandtech.com/show/16512/la … eam-at-329
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3681
https://www.techpowerup.com/gpu-specs/g … 3060.c3682
https://www.techpowerup.com/gpu-specs/g … 2060.c3310
RTX 2060 Super
https://www.anandtech.com/show/16290/la … re-for-400
https://www.techpowerup.com/gpu-specs/g … uper.c3441

253 (edited by UHD 11-06-2022 14:27:45)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

RIFE Model 4.0 Benchmarks


720p - x2 interpolation - RIFE filter for VapourSynth (ncnn Vulkan):

70fps - NVIDIA GeForce RTX 2060 Mobile - Chainik
https://www.svp-team.com/forum/viewtopi … 158#p80158
92.956fps - NVIDIA GeForce RTX 2070 - blackmickey1007
https://www.svp-team.com/forum/viewtopi … 219#p80219
188.2fps - NVIDIA GeForce RTX 3070 Ti - dlr5668
https://www.svp-team.com/forum/viewtopi … 163#p80163


1080p - x2 interpolation - RIFE filter for VapourSynth (ncnn Vulkan):

90fps - NVIDIA GeForce RTX 3070 Ti - dlr5668
https://www.svp-team.com/forum/viewtopi … 477#p80477

---------------------------------------------------------
720p is original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

1080p is an arbitrary file chosen

254 (edited by Enif 06-08-2022 17:35:10)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Hello!
Recently I got a new TV and it had interpolation techniques built into the software so I searched for the same thing for PC and I found this. I tried the standard algorithm and Nvidia optical flow but they were both artifacting so I tried RIFE. It has a lot less artifacts and the flow of the images are smooth I very like it. The only problem is my weak GPU. I have to downscale my source material for it to work realtime. My current setup is:

Source material:
8bit 1080p 23.976 fps 8Mb/s (anime)

Rife via Vulkan v4
Windows 10
MPC-BE x64 + LAV decoder (CUVID) + MadVR
GTX 1660 Super 6GB, 2100Mhz core, 7800Mhz memory

Downscale via Spline64 to 1280x720 (via avisynth)
Interpolate with rife (x2)
Upscale via MadVR Jinc back to 1920x1080 (1920x1078 actual)
GPU usage 71-74%
Power 100-110 W

I can also downscale by integer and use nearest neighbour for resizing for increased smoothness:
Downscale via Pointresize to 960x540
Interpolate with rife (x3)
Upscale via MadVR nearest neighbour
GPU usage 90-94%
Power 110-120W

Actually what I found strange was Rife with high GPU utilization was consuming less power than either NVOF or standard SVP algorithm, both used around 115-130W.
The quality suffers but the interpolation makes movement smooth and nice on those few scenes where there is camera movement. 540p was a bit too pixellated for my like so I remained at 720p interpolation with Rife.
I tried to increase the resolution but it would always make the video choppy and max out mx GPU utilization even when I only upped the resolution by a little like to 1408x792 (frequent dropped frames, GPU usage spiking, EDIT: after a minute of playback gpu usage normalizes and playback gets smooth with around 85% GPU usage EDIT2: it start spiking and dropping frames again even in not very busy scenes so I had to go back to 720p).
What do you think about my current settings? Do you think there is a better way of achieving real time interpolation with my computer and in better quality? I feel like x2 interpolation is smooth enough I don't need x3 but I would like to increase resolution or quality somehow if possible. I tried Cuda but it freezes my player on start, in mpv it worked but it's speed was half of vulkan v4.

255

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I hope someone can help you. I'm all for the highest quality, so for me it's either the right graphics card or encoding when it comes to RIFE. That's why I'm waiting patiently and raising funds for a suitable graphics card.

If you are happy with the interpolation on your TV, then why bother?

256 (edited by Enif 06-08-2022 17:36:33)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

If you are happy with the interpolation on your TV, then why bother?

I don't always watch content on the TV as it is shared with others. That is why I searched for a universal solution that I can use on PC. I don't want to pre-encode hundreds of episodes into higher framerate because I don't feel like making my my gpu work at 100% for weeks constantly and I don't want the size of my shows to extend and to amass terabytes of storage. That is why I seeked a real-time solution. But maybe you are right, I should just re-encode them with rife in full resolution, that would be the highest quality option. I can run transcoding with vulkan v4 at around 35 FPS and cuda pytorch at around 21-22 FPS at 1080p. Almost realtime but not quite. I just need 12 fps more.

257 (edited by UHD 06-08-2022 18:53:49)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I understand. I will say that I'm a bit surprised that you are achieving such high speeds for GTX 1660 Super and 1080p x2 RIFE. I wonder if you achieve the same speeds for real life movies. Anime mostly have duplicate frames.

What is your result for Rife via Vulkan v4 for this file:
original demo video from the creator of RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

I'm also curious to know what kind of TV you have and if you have compared the quality of TV interpolation with transcoded anime in full 1080p quality (RIFE) and good lossy or lossless compression.

And remember that you will always get a higher quality when interpolating with RIFE x3 than when interpolating with x2 - see Figure 6: https://arxiv.org/pdf/2204.03513.pdf

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Hello. It seems there's some problem in particular with SVP implementation of RIFE interpolation and/or VapourSynth - it runs out of CPU on the threads it uses for high FPS output.
(That at gpu_threads=8... could use 16 there but filter does not support it.)

720p 60 x2 works. So does 1080p 30 x2. 1080p 24 x3 runs into a CPU cap, getting 105 FPS output. 1080p 60 x2 same, getting 85 FPS output.
Encoding is *much* faster than this.

Hardware: GeForce 3080 (OC) and Ryzen 5950x.

Can anything be done to make it actually not run out of CPU?

259 (edited by dlr5668 07-08-2022 19:03:15)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Enif wrote:

Hello!
.

Sadly GTX wont be able to do realtime. Your best bet is to get 3060 for 1080p realtime x2. 4k is still not possible, maybe with 4090

260

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

What is your result for Rife via Vulkan v4 for this file:
original demo video from the creator of RIFE

Transcoding via SVP Manager with NVENC into H264 8Mb/s by the end it was working at 78 fps but it was climbing throughout the whole transcode. I am not sure if the fps counter in SVP Manager is accurate, is there any other way of keeping track of how fast my GPU is spitting out frames?

UHD wrote:

I'm also curious to know what kind of TV you have and if you have compared the quality of TV interpolation with transcoded anime in full 1080p quality (RIFE) and good lossy or lossless compression.

It is an LG Oled C1 48". It has a setting named TrueMotion and you can manually set de-judder and de-blur setting on a scale of 0 to 10. It seems like the video retains all of it's quality and the interpolation looks very smooth both in anime and movie footage. I would say it has less artifacts than RIFE x2 but on very few scenes (very choppy camera pans in anime) there is still some judder left. To be fair I believe on those scenes RIFE has problems as well but I haven't cross tested the same footage to compare them yet.

UHD wrote:

And remember that you will always get a higher quality when interpolating with RIFE x3 than when interpolating with x2 - see Figure 6: https://arxiv.org/pdf/2204.03513.pdf

With my current knowledge, I don't understand this figure but I will believe you. Still, compromising resolution for interpolation accuracy is a hard decision and I will have to test more to find the sweet spot.

Today I found out that if I run a very simple script in Vapoursynth filter instead of SVP manager, I get a lot more consistent performance and without frame drops or gpu usage spikes. Now I can run 1536x864 resolution real time without any problems.

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

It's really amazing reading all the high level technical discussions you guys are having about frame motion interpolation. Everyone on this thread is super smart, and it shows. Very impressive, keep up the good work!

I became interested in the SVP project when I made the bizarre discovery that AMD has decided to no longer support it's Fluid Motion hardware based frame motion interpolation on it's RX5000 and newer series of video cards, which was literally the only reason I was thinking of buying one. Yup, you read that right. They literally just dropped support for the one bonus feature, unique to AMD video cards, that Nvidia didn't have an in-house answer for.

Especially @UHD I want to thank you for your dedication to the thread. Your interest in motion interpolation burns with the white hot passion of a thousand burning suns. I further want to thank you for making me aware of Waifu2x-Extension-GUI by Aaron Feng, I had no idea this program existed, and the image upscaling on the REAL-ESRGAN-NCNN-VULKAN algorithm is just incredible. And above all, it's free!

Rock on, SVP! The world needs more smoothing.

262 (edited by UHD Yesterday 18:12:55)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

RunAwayFromTheSky wrote:

Especially @UHD I want to thank you for your dedication to the thread. Your interest in motion interpolation burns with the white hot passion of a thousand burning suns. I further want to thank you for making me aware of Waifu2x-Extension-GUI by Aaron Feng, I had no idea this program existed, and the image upscaling on the REAL-ESRGAN-NCNN-VULKAN algorithm is just incredible. And above all, it's free!

Thanks. I'm glad you learned something new thanks to this thread. That was exactly my goal.

I've always been interested in watching videos in a way that reflects reality as much as possible. That's what brought me to SVP long before I started posting here on the forum. It was SVP that allowed me to experience the magic of real-time interpolation even on very archaic hardware.

It wasn't until 2018 that I saw for the first time that even better quality could be achieved with AI: https://developer.nvidia.com/blog/trans … n-with-ai/
A year later, I was admiring on YouTube what DAIN could do, but I saw no hope that this technology could ever be used in real time.

It wasn't until RIFE that I came to believe and to this day I am still fascinated by it and share this fascination with others. However, I am still raising funds for a PC that will allow me to fully enjoy the possibilities of AI.

So I invite anyone interested, like me who does not yet have the right hardware, but even more so those who have the hardware and can share the results of their tests, to join the discussion.

I, for the time being, can at most deal with the theoretical side and that is what I am trying to do. I will soon have some very interesting information on what the possibilities currently are in the field:

- video frame interpolation
- video deblurring
- video super-resolution

Or maybe even all 3 in 1 wink

I don't want to write too many things at once so as not to create too much confusion. However, it will be worth following this and my other thread dedicated to IFRNet: https://www.svp-team.com/forum/viewtopic.php?id=6666 big_smile

263

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Enif wrote:

Transcoding via SVP Manager with NVENC into H264 8Mb/s by the end it was working at 78 fps but it was climbing throughout the whole transcode. I am not sure if the fps counter in SVP Manager is accurate, is there any other way of keeping track of how fast my GPU is spitting out frames?

Enif wrote:

It is an LG Oled C1 48". It has a setting named TrueMotion and you can manually set de-judder and de-blur setting on a scale of 0 to 10. It seems like the video retains all of it's quality and the interpolation looks very smooth both in anime and movie footage. I would say it has less artifacts than RIFE x2 but on very few scenes (very choppy camera pans in anime) there is still some judder left. To be fair I believe on those scenes RIFE has problems as well but I haven't cross tested the same footage to compare them yet.

Enif wrote:

Today I found out that if I run a very simple script in Vapoursynth filter instead of SVP manager, I get a lot more consistent performance and without frame drops or gpu usage spikes. Now I can run 1536x864 resolution real time without any problems.

Thanks for sharing your results. It's a bit surprising that a card without Tensor Cores gives better results than a card with Tensor Cores. It's a shame that so few people share their results, we could then try to find the reasons for this.

It's good that you also shared a comparison of RIFE with TV capabilities. That's something we're missing a bit here too.

As you can see, for now there is no any better solution to your problem. The easiest thing to do, of course, is to buy a better graphics card. However, that's the way it is in life that you have to make the decision yourself. Money spent on a new card will save transcoding time and disk space. And vice versa. The time needed for transcoding and disk space will save money. All intermediate solutions come with the acceptance of lower quality.