For the Protocol Handler, I found this on github. https://github.com/stefansundin/vlc-protocol
I did my testing on windows using the supplied bat files. download the 3 bat files and copy them to the vlc install directory. on my machine with 64bit vlc, it was installed to C:\Program Files\VideoLAN\VLC if you install the 32bit version it may be different.
run the vlc-protocol-register.bat file as administrator to register the protocol handler. when you do this, it will send all calls with vlc:// to be handled by the vlc-protocol.bat file. that file just opens vlc to playback the file.
https://developer.shotgridsoftware.com/af0c94ce/ Create custom protocols