nikitazep Is it possible to install third-party js players such as plyr.io?
No. It's possible to code 3rd party players from the javascript source, but not sure what you mean "install". Files Gallery currently uses the browsers own video player, which has the same capabilities as any 3rd party player, except perhaps reading streaming formats like m3u playlist format. I plan to integrate a 3rd party video player, mostly only to create a common design across all browsers.
nikitazep Is it possible to transcode video formats using ffmpeg?
FFmpeg can do almost anything, but it's a command-line tool, not a PHP application. You can transcode a video from FFmpeg command-line, but how are you going to effectively do this from PHP/server? Even if Files Gallery triggers a transcode of a video, this will take a long time to complete before the video can be output. It certainly can't do it in real-time speed. And even if we did this, you would need to store the transcoded video somewhere (cache).
Why not transcode your videos up front with FFmpeg? This is how any video service like Youtube would do it ...