(random SVP user who worked in technology for a while)
It is hard for me to imagine many other users wanting or needing that specific functionality (blacklist/whitelist SVP application based off of the file system folder location).
Wonder if there is a keyboard shortcut, or one could be set/added, to enable/disable SVP as a whole like you can do from the manager with the mouse.
Or you could kill the SVP manager process, do your work, then run it again (if you tried to automate what you're doing manually now).
I've messed around with a number of player/filter settings trying to remove all initial playback issues like I assume you've tried (MadVR had a "pause until it'd rendered all the frames first" versus always displaying new frames ASAP, etc). It was more an annoyance for me than anything, so I stopped poking/trying after a try or two.
It'd be annoying, but maybe if you played the video, paused, rewinded, and played again it wouldn't have any pauses or delays... Sort of pre-cache / pre-calculate the work the first time, then reuse what you need from RAM on any following attempts. If you can figure out how to do that without jumping between your mouse and keyboard it'll go a lot faster too.
Oh... since there are already other tools in the middle... wonder if you can do conditionals in avisynth or vapoursynth? Guessing probably not, but you won't know for sure unless you look for it.
Oh, you could have a different video player that isn't SVP aware. Like VLC. Not sure how to switch quickly between two default video players efficiently though. But SVP seems to need different workarounds and tweaks for different players, so if you just didn't apply one of them it wouldn't render using SVP.
Anyway, good luck fellow lazy person (doing way more work ahead of time so they can do zero work later).
. I'm pretty sure I hit the same issue adding that openCL/openGL compatibility layer a year or two ago, and had forgotten. I'd like to code up some protection in the GPU detection/selection feature.