I would avoid it, if you care at all about availability and downtime. The result will probably not be great, you need to ensure the server side gets enough resources under load, and setting it up may require constant restarts if things aren’t immediately working as expected.

Nonetheless, here is a link where someone did essentially exactly that on NixOS: https://astrid.tech/2022/09/22/0/nixos-gpu-vfio/

That’s exactly what it is. I haven’t looked into it too much, but as far as I know it’s main advantage is simplifying the setup process, which in turn reduces the chances of a misconfigured VPN.