I found out what caused the problem. It was hardware acceleration in LAV Video Decoder:

Hardware acceleration: None
GPU load: ca. 65%
CPU load: ca. 30%
Works fine.

DXVA2 (native):
GPUload: ca. 70%
CPU load: ca. 30%
Works fine.

All remaining hardware decoders cause heavy stuttering (GPU loads of ca. 90%).

Only DXVA2 (native) is working without stuttering. It seems that in case of a sufficiently powerful CPU there is no benefit from hardware acceleration. To the contrary, it can even be detrimental. Moreover, for HDR in madVR "let madVR decide" seems to work best.

This is my hardware:
Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
128 GB RAM
NVIDIA GeForce RTX 3070
Samsung QE-65QN800A

Although I have followed the recommended settings (i.e. AviSynth and madVR) the image stutters and even freezes at 4K using MPC-BE (x64) with SVP, whilst it works fine with Full HD and SD resolutions. The CPU load is just around 8%. What could be the culprit?