New RIFE filter - 3x faster AI interpolation possible in SVP!!! (Page 12 of 84)

Thanks so much

This is actually very good news for me! I wasn't expecting a performance increase, quite the opposite: I was worried that there might be worse performance, and then no real-time 4K 10-bit HDR.

It's probably only a matter of time before it becomes an official release.

Now the biggest concern still before me: will AviSynthPlus-RIFE work with 4K 10-bit HDR...?

Now this is my dream: real-time x2 interpolation of 4K 10-bit HDR video files using SVP with AviSynthPlus-RIFE.

I know it probably won't be that easy, it will require some time, but I think it might be of interest not only to people with the new 4090 card, because after all SVP can downscale source to, for example, 1080p while maintaining: 10-bit HDR: https://www.svp-team.com/wiki/Manual:Resizing

I also have a question if SVP working with AviSynthPlus-RIFE would be able to use RGB as the output format to a monitor? From what I understand AviSynth has more capabilities in this regard compared to VapourSynth.

RIFE requires RGB format with float sample type of 32 bit depth and the idea is not to go back to YUV at the end anymore, because any such conversion is lossy. The monitor needs RGB anyway.

Thank you very much for your answer, but unfortunately it doesn't explain anything to me.

I still don't have a suitable graphics card for vapoursynth-rife testing, so I have to rely on the knowledge of others here. Unfortunately, neither on this forum nor anywhere else have I found an answer as to how vapoursynth-rife handles 4K 10-bit HDR files.

My knowledge at this point is as follows:

1. Testing real-time interpolation for 4K 10-bit HDR files using the RIFE algorithm will only be possible after the release of the 4090 cards. Today, however, we can test 1080p 10-bit HDR files from re-encoded 4K 10-bit HDR files according to this thread: https://www.reddit.com/r/ffmpeg/comment … intaining/

2. SVP allows real-time interpolation for 4K 10-bit HDR files, but needs Avisynth Filter for this: https://www.svp-team.com/wiki/SVP:4K_and_HDR

3. Finally we have a port of the VapourSynth plugin RIFE to AviSynth+:
https://github.com/Asd-g/AviSynthPlus-RIFE hence my hope related to point 2 above that HDR can be pass through.

4. RIFE supports RGB format with float sample type of 32 bit depth: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan so I assume it could handle 10 bit colour depth.

5. This RIFE implementation: https://github.com/may-son/RIFE-FixDrop … ConvertFPS seems to confirm that handling 10-bit colour depth might be possible:

AI drop frame fixer and FPS converter
For image sequence from video. Not limited to 8 bits!

Chainik, I know my knowledge is poor. If you could explain what else is needed for real-time interpolation using SVP and RIFE algorithm of 4K 10-bit HDR files then please write.

I have set up an account on GitHub: https://github.com/AIVFI and I can ask someone to adjust the RIFE filter to adapt to 4K 10-bit HDR files. Just please write what I should ask, because I have no testing capabilities myself and I'm not a programmer, so probably a lot of things that are obvious to you might be not understandable to me.

> SVP allows real-time interpolation for 4K 10-bit HDR files, but needs Avisynth Filter for this

"Avisynth Filter" is a confusing naming cause there're _two_ different filters there, one for Avisynth and one for Vapoursynth
both supporting 10-bit and hdr

> It is said that the performance of Optical Flow is also improved by about 2.5 times

it only makes difference for interpolating 4k@60 to a higher rates

Thanks for your reply. I'm guessing you mean:
avisynth_filter: https://github.com/CrendKing/avisynth_filter

So if I understand correctly whichever combination:

AviSynthPlus-RIFE +avisynth_filter
or
VapourSynth-RIFE-ncnn-Vulkan +avisynth_filter

should be sufficient in SVP for real-time interpolation of 4K 10-bit HDR files using RIFE?

And how does real-time x2 RIFE interpolation of 1080p 10-bit HDR video files look like on your NVIDIA GeForce RTX 3070 Ti?

I found a publicly available demo that you can download for testing:

LG 4K HDR Demo - New York.ts
File size: 448 MiB
Duration: 1 min 13 sec
Overall bit rate: 51.4 Mbps
HDR format: SMPTE ST 2086, HDR10 compatible
Width: 3 840 pixels
Height: 2 160 pixels
Frame rate: 25.000 FPS
Color space: YUV
Chroma subsampling: 4:2:0
Bit depth: 10 bits
Direct link: https://drive.google.com/file/d/1dfR5TT … _bGfEXUvJ/
Source: http://hdr4k.blogspot.com/

The file of course needs to be re-encoded to 1080p 10-bit HDR for example as described in this thread:
https://www.reddit.com/r/ffmpeg/comment … intaining/

dlr5668 please write what you manage to do with this file. I am of course interested in real-time interpolation using the RIFE algorithm. Normal SVP interpolation is described here:

https://www.svp-team.com/wiki/SVP:4K_and_HDR

Good news!

Today AviSynthPlus-RIFE has been updated with all the newest RIFE models: https://github.com/Asd-g/AviSynthPlus-RIFE

These models were added earlier by styler00dollar to his version of VapourSynth-RIFE-ncnn-Vulkan:
https://github.com/styler00dollar/Vapou … cnn-Vulkan

Now both AviSynth as well as VapourSynth filters include all current RIFE models: v4.2, v4.3, v4.4, v4.5 and v4.6.

About the v4.0 and v4.1 models, their author hzwer wrote:

https://github.com/hzwer/Practical-RIFE

I would like to once again give a big thank you to everyone who shared the v4.0 test results on this forum.

Now, I would very much encourage everyone to share their observations on the new v4.2, v4.3, v4.4, v4.5 and v4.6 models compared to the old v4.0 model. Is there an improvement in quality and speed of interpolation?

What is worth noting is that styler00dollar has also added the latest models in their higher quality version, which of course also involves a longer inference time for a given model. What difference does this make?

styler00dollar has provided examples of RIFE AI inference times for the v4.0 model in various quality variants

fastmode False, ensemble True
fastmode True, ensemble False
fastmode True, ensemble True

here: https://github.com/styler00dollar/VSGAN-tensorrt-docker

You will also find there inference times for other AI-based interpolation methods. In addition to RIFE, there are also: Sepconv, FILM, eisai and... IFRNet, which I already mentioned here: https://www.svp-team.com/forum/viewtopic.php?id=6666 and... M2M, which I already mentioned here: https://www.svp-team.com/forum/viewtopi … 345#p80345

I am glad that we finally have a choice of a given RIFE model in the fastest version as well as in the highest quality version. It will be possible to boost the quality even more for files with lower resolutions or if we have graphics cards with high AI inference performance

Thanks! Looks like there were some significant changes done to these two models: https://github.com/styler00dollar/Vapou … n/issues/3
In any case, 10% is not much and it will all depend on which model gives the highest quality.

I already have a video card and have it for several years. Unfortunately my ATI Radeon HD 3450 graphics card does not support the Vulkan

NVIDIA GeForce RTX 4090 graphics cards are expected to start shipping on October 12, 2022, but I will still have to wait with my purchase for the availability of custom spreaders for passive cooling. Once they are available I will choose a particular model of graphics card based on the availability of the spreaders and other users' opinions about coil whine. In other words, it will still be a few months before I can start shopping. If I'm going to build a new HTPC for the next decade or so it needs to be done right. I'll miss my energy efficient ATI Radeon HD 3450 and its 25W TDP

Chainik, since it will still be a long time before I can share my test results on this forum, maybe you can tell me the big secret that nobody wants to talk about on this forum and nowhere else either:

Does real-time x2 RIFE interpolation of 1080p or 720p 10-bit HDR video files is possible without losing 10-bit colour depth and HDR?

I revealed myself with my secret graphics card

This is great news!

Until now, I thought there would be some problems based on statements from n00mkrad, who uses RIFE in his Flowframes:

https://github.com/n00mkrad/flowframes/ … -803571016

For some reason I can't download from this source. Hence I have provided an alternative source for the same demo in this post: https://www.svp-team.com/forum/viewtopi … 008#p81008

Thanks a lot for the precise data!

174 TFLOPS Peak FP16 using the Sparsity feature - NVIDIA GeForce RTX 3070 Ti

1321 TFLOPS Peak FP8 using the Sparsity feature - NVIDIA GeForce RTX 4090
so
660 TFLOPS Peak FP16 using the Sparsity feature - NVIDIA GeForce RTX 4090
so
3,79x
so
69 fps !!!

I'll have to go down to 250W TDP for passive cooling anyway, so there's a good chance of getting 50 fps.

Thanks for the test!

As I understand it this is the result after reencoding.
This is what I expected after reading this part of the tutorial:

https://www.svp-team.com/wiki/SVP:4K_and_HDR

I don't know if this is still valid. That's why in that post https://www.svp-team.com/forum/viewtopi … 008#p81008 I suggested downscaling to 1080p, checking without interpolation that everything is OK (10-bit, HDR) and only from that 1080p file trying RIFE interpolation in real time.

I think there will be a lot of people who will sacrifice the resolution reduction from 4K to 1080p if it helps to keep 10-bit HDR, chroma subsampling: 4:4:4 (1080p) and real-time RIFE interpolation. I am very curious to see if this is possible....

> Do you plan to update SVP with the newer models?

kind of... not exactly like this
there will be only ONE model bundled (*), plus you'll be able to add whatever model you want manually, by copying model's folder

(*) if you _know_ which one from 4.0 to 4.4 is the best in terms of overall quality - this is the right moment to tell

del