Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Subscribe to our newsletter

Advertise on LowEndTalk.com

Latest LowEndBox Offers

    FFmpeg Encoding over multiple VPS
    New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

    FFmpeg Encoding over multiple VPS

    noamannoaman Member
    edited November 2018 in General

    Hello

    Does any body has any idea how to distribute FFmpeg Encoding over multiple vps servers

    If any body has done it do comment

    Currently looking to convert h264 to h265 or vp9

    Edit: Its a live stream

    Comments

    • notgodnotgod Member
      edited November 2018

      yep, you can do
      B.mkv 1024M
      first, split it , B.mkv.01 B.mkv.10 / per 100M
      per VPS encoding one file

      merge after all is encoding completed

      Pingtest.eu : web ping test service an internet host from 60+ locations in worldwide
      StorageVPS.com : Large Storage VPS, 1000G disk just $20/month .....

    • @notgod said:
      yep, you can do
      B.mkv 1024M
      first, split it , B.mkv.01 B.mkv.10 / per 100M
      per VPS encoding one file

      merge after all is encoding completed

      Its a live stream

    • are you converting live feed from somewhere? not just playlist (Bench of vod) to live stream?

      signature for rent - ^_^

    • jcalebjcaleb Moderator

      you are using nginx rtmp?

    • PUSHR_VictorPUSHR_Victor Member, Provider

      You can not distribute it.

    • Nope. You will need a powerful server to convert a live stream. You cannot do that distributed. But if you encode in different resolutions you can distribute that load.

    • JackHJackH Moderator

      I'm sure it's possible but you'll be looking at some very custom scripts or some super advanced ffmpeg/avconv switch no one's ever used ever. :/
      Out of interest, why would you want to do it over several VMs? I'm sure it'd be cheaper to just get a more powerful machine... If it's load balancing you're worried about, you can rebroadcast with very little CPU from edge VMs

      NVMe KVM VPS in Amsterdam, Stockholm, Oslo, Vienna and LA ($2.50/1GB RAM/10GB NVMe/month) (AFF LINK)

    • There's a project that achieves something similar but for Plex servers. Might be a good starting point? https://github.com/wnielson/Plex-Remote-Transcoder

      Arcruy - Let Us Fill in The Blanks

    • Get a Hetzner or Netcup VPS (with dedicated cores) or a dedi.

    • mfsmfs Banned, Member

      @Wolveix said:
      There's a project that achieves something similar but for Plex servers. Might be a good starting point? https://github.com/wnielson/Plex-Remote-Transcoder

      From a quick glance it seems something to distribute transcoding jobs opportunistically in order to relieve a "master node" from transcoding burdens but it seems to me that each "transcoding node" is transcoding one thing and one thing only (e.g. a stream, like OP suggested). Correct me if I'm wrong obviously

      AC_Fan said: Hetzner or Netcup VPS (with dedicated cores)

      This, you can turn it off (if it's a stream there's a chance it's not a 24/7 stream) when you're done; dedi if it's something you'll run on the daily or on a predictable constant basis

      Thanked by 1Wolveix

      I've left LET since February 2019, account made inactive on request.

    • @JackHadrill said:
      I'm sure it's possible but you'll be looking at some very custom scripts or some super advanced ffmpeg/avconv switch no one's ever used ever. :/
      Out of interest, why would you want to do it over several VMs? I'm sure it'd be cheaper to just get a more powerful machine... If it's load balancing you're worried about, you can rebroadcast with very little CPU from edge VMs

      Because i have tons of Idling VPS

    • I read that you could do distributed computing using apache hadoop but not sure if even it works and not even how to set it up

      @Wolveix said:
      There's a project that achieves something similar but for Plex servers. Might be a good starting point? https://github.com/wnielson/Plex-Remote-Transcoder

      Will take a look

      @jcaleb said:
      you are using nginx rtmp?

      Yes

      @mrclown said:
      are you converting live feed from somewhere? not just playlist (Bench of vod) to live stream?

      Live stream only not VOD

    • WolveixWolveix Member
      edited November 2018

      @noaman said:

      @JackHadrill said:
      I'm sure it's possible but you'll be looking at some very custom scripts or some super advanced ffmpeg/avconv switch no one's ever used ever. :/
      Out of interest, why would you want to do it over several VMs? I'm sure it'd be cheaper to just get a more powerful machine... If it's load balancing you're worried about, you can rebroadcast with very little CPU from edge VMs

      Because i have tons of Idling VPS

      Why don't you cancel those VPS' and then buy a Hetzner auction dedi to save you a lot of hassle?

      Thanked by 2vimalware eol

      Arcruy - Let Us Fill in The Blanks

    • PUSHR_VictorPUSHR_Victor Member, Provider
      edited November 2018

      @JackHadrill said:
      I'm sure it's possible but you'll be looking at some very custom scripts or some super advanced ffmpeg/avconv switch no one's ever used ever. :/
      Out of interest, why would you want to do it over several VMs? I'm sure it'd be cheaper to just get a more powerful machine... If it's load balancing you're worried about, you can rebroadcast with very little CPU from edge VMs

      Actually, the real reason is keyframes and stiching the videos back after transcoding. One can never guarantee it will not be a mess.

      Thanked by 1vimalware
    • u need good dev to do that

    • Live stream usually needs low latency encoding. Videos have key frames inserted every 10 seconds or so. If you want to split streams into chunks you probably have to split in every 10 seconds. Considering those are VPSes, it might take an extra 30-40 seconds to encode that. That adds a lot of latency on it, leaving alone the development time on intercepting the video data and send to different encoders to encode, get them back, reassemble the stream, before you can push it to the public.

      If you are interested in studying video encoding technologies, it would be a fun project. But in practical, it's ready not an easy job to do.

    • I wonder too how many of those idle VPS will let you do something as cpu heavy as transcoding over any significant timespan. vp9 is very slow to encode even on a dedi.

      #lexit spread the word.

    • jcalebjcaleb Moderator

      why does it need to be distributed? Is it to avoid hammering cpu of a VPS?

    • @jcaleb said:
      why does it need to be distributed? Is it to avoid hammering cpu of a VPS?

      yap you got it right :-)

      It is going to be a maximum of one-two 1080p streams or in a few cases one UHD
      not more

      only one user...That is going to be me :-)

    • When you need low latency, you cannot do distributed. If latency or delay isn't your concern, there are some lib which allows u to do horizontal transcoding (which may or may not be free lib)

      video encoding is CPU heavy for most cases, I doubt ur VPS are efficient or able to tolerate like others said as above. Go for dedicated and save the cost of VPS for better and more stable stream if you are the only one user.

      Take a look at wowza cloud too if u want to compare $$$ by $$$.

      Thanked by 1Wolveix

      signature for rent - ^_^

    • CyberMondayCyberMonday Member
      edited November 2018

      ..just lower the priority and don't offer realtime transcoding?

      Own a piece of internets history.

    • It's a phase. Going from 40LEBs to under 8 takes pain, but the lesson usually sticks.

      getaDedi

      Tired of LET scams?
      A Moderated forum : https://talk.lowendspirit.com/

    • inklightinklight Member
      edited November 2018

      Why you need multiple VPS , did you have one or multiple sources .
      anyway I have live broadcast website and this is my ffmpeg settings

      ffmpeg  -thread_queue_size 32768 -re -i "http://sorce" -pix_fmt yuv420p -r 23.976 -vcodec libx264 -filter_complex " scale=480:360" -b:v 165k -break_non_keyframes 1 -profile:v high -keyint_min 24 -g 48 -x264opts no-scenecut -strict experimental -acodec aac -b:a 32k -map_metadata -1 -f flv rtmp://localhost/hls/live
      

      you can change bitrate of video/audio as you like same with picture size , but this is lowest picture quality I can broadcast on ,
      BTW the output are sent to Nginx RTMP server that running locally

      Edit== Don't ever try to encode on VP9 I think it's 50 times slower then H264 and 20 time slower then H 265 , Web browsers do not support H265 even it a lot faster then VP9 , but you can watch live H265 streaming on Desktop/phone using any video player
      here is H265 trans coding :-

      ffmpeg -thread_queue_size 32768 -re -i "http://sorce" -map 0 -s  854x480  -c:v libx265 -preset ultrafast  -vb 185k -c:a aac -b:a 32k -f ssegment -segment_list /yourdmain_path/live.m3u8 -segment_list_type hls -segment_list_size 10 -segment_list_flags +live -segment_time 10 -segment_wrap 7 /yourdmainpath/live%01d.ts
      

      last cool I didn't notice about emoji in this forum :blush: :sunglasses: :wink: :sweat_smile: :relaxed: :trollface: :love: sorry

      Thanked by 1gol3m
    • zllovesukizllovesuki Member
      edited November 2018

      The "cloud" was born with the promise of on demand computing yet somehow we are still circlejerking the idea of some unproven ideas and "technology"...

      1. Write a script that instantiate an instance on AWS/DigitalOcean/Linode or your choice of poison
      2. Run your script
      3. ???
      4. Remember to terminate it
      5. Profit

      Since you mention H264 to H265, it would be beneficial to test out AWS's GPU offering. Encoding H265 of any kind with pure CPU power is an absolute abomination to one's sanity.

      Now works for a public institution. Opinions expressed here are mine and mine alone.

    • 4th gen Intel/Xeon Haswell on up supports hw en-/decoding.

      ˙ɹǝuzʇǝɥ

    • jcalebjcaleb Moderator

      noaman said: yap you got it right :-)

      you can generate the stream using the correct codec and whatnot from your local. Or you can stream to a local machine and do the ffmpeg there to push to your remote server. That way your remote don't need to re-encode.

      Currently, I am pushing a livestream in correct format to a 128mb OVZ with 1 core. And it doesn't use up any memory or cpu.

    • It depends on the streaming format. For example, it's rather easy to distribute HLS pieces encoding while hard to distribute real live streams like RTSP/RTMP/HTTP.

      Why do you need it anyway? Distributed transcoding is usually implemented to encode much faster than the realtime. For live streams you need just real time encoding which any semi-modern CPU or GPU would handle.

    • @jcaleb said:

      noaman said: yap you got it right :-)

      you can generate the stream using the correct codec and whatnot from your local. Or you can stream to a local machine and do the ffmpeg there to push to your remote server. That way your remote don't need to re-encode.

      Currently, I am pushing a livestream in correct format to a 128mb OVZ with 1 core. And it doesn't use up any memory or cpu.

      I have a limited network connection the whole purpose of this is to able to encode h264 to h265 or vp9 and reduce bandwidth

      @ValdikSS said:
      It depends on the streaming format. For example, it's rather easy to distribute HLS pieces encoding while hard to distribute real live streams like RTSP/RTMP/HTTP.

      Why do you need it anyway? Distributed transcoding is usually implemented to encode much faster than the realtime. For live streams you need just real time encoding which any semi-modern CPU or GPU would handle.

      I think i might hit the VPS cpus limit

      The whole idea is to kill two birds with one stone

      1. Use the Idling vps for a purpose

      2. Decrease bandwidth usage

    • msg7086msg7086 Member
      edited November 2018

      Let's make the problem simple.

      The cost to develop such a software that allows you to do so can pay you a dedi-core VPS for 10 years.

      Also, UHD live stream costs a lot of resources. I have a few E5 dedi-servers that barely encode UHD video at 2fps at x265 moderate settings. If you set preset to fast, you'll probably get about 5 fps, at most. So for live streaming, you need at least 6 of them, so that's 36 physical cores we are talking about. In oversold VPS world, that's about 144 idling VPS if you only share the core with 3 neighbors.

      Oh right, for distributed encoding, you probably have to transfer raw video data between nodes. Raw data is calculated at pixels * frames. To reach 30 fps, It'll be 3840*2160*12(bits)*30 per second, which is about 2.8 Gbps networking bandwidth. For a 7200 seconds movie, It's 2.5 terabytes traffic, just to distribute the UHD raw video data to the VPSes.

      We do distributed encoding on our servers, but they sit in the datacenter LAN, and we only slice it per chapters, not per seconds. In this case, we only distribute the much smaller source video, not the raw data.

    • Video encoding, is all about money if you care about quality even just a little bit. If you have ever played fansubbing, you'd know that people pay thousands of dollars to get Dual E5 workstations just to be able to encode x265 at, like, 2-3fps, for UHD videos. A bit better on 1080p movies, but pretty hard to get real time speed. Larger companies that works with such contents would probably get their own farms, like couple hundred rack servers to do such work. It's not something a virmach idle VPS can handle, not even 10 of them, or 20 of them, or 50 of them.

    • ValdikSSValdikSS Member
      edited November 2018

      msg7086 said: Oh right, for distributed encoding, you probably have to transfer raw video data between nodes.

      You don't need to, just split the original compressed video by Group Of Frames (GOPs, a video chunks from between 2 keyframes).

      Take a look at proof of concept I made 5 years ago. There are also more professional software like lancoder and clustercode, as well as more simple like dve and klaxa's scripts.

      noaman said: Use the Idling vps for a purpose

      It's almost impossible to use idle VPSes for that, better forget about it. I would say it was more effective to get several free ARM baremetal machines from scaleway for 30 minutes for distributed encoding purpose than a bunch of low-end idle VPSes.

      Thanked by 1mfs
    • mfsmfs Banned, Member

      ValdikSS said: Take a look at proof of concept I made 5 years ago. There are also more professional software like lancoder and clustercode, as well as more simple like dve and klaxa's scripts.

      Leaving aside feasibility evaluations in OP's scenario, some of those software seem simply distributing encoding jobs to various nodes according to the 1 node = 1 video logic (seems the case of clustercode: "Each node encodes a video, enabling parallelization"); other products like nergdron's dve seem to propose an actual parallelization of a single video/source/stream (distributes chunks of a single stream). While it possibly can get the job done, and nicely, chances are that the result is sub-optimal as far as size is concerned; a single node capable to evaluate all chunks seems advantaged in this regard (especially if multiple passes are selected). Yet it could be very convenient to distribute chunks in some cases, ty for sharing.

      I've left LET since February 2019, account made inactive on request.

    • @ValdikSS said:

      msg7086 said: Oh right, for distributed encoding, you probably have to transfer raw video data between nodes.

      You don't need to, just split the original compressed video by Group Of Frames (GOPs, a video chunks from between 2 keyframes).

      I'm aware of GOP based splitting. But it's not always the case that you can split it into GOPs. For a bluray disc it's probably OK. But if the source file has Open-GOP then it screws up. x265 also defaults to open-gop thus you can't split those videos without corrupting the files.

    • @mfs said:
      Leaving aside feasibility evaluations in OP's scenario, some of those software seem simply distributing encoding jobs to various nodes according to the 1 node = 1 video logic (seems the case of clustercode: "Each node encodes a video, enabling parallelization"); other products like nergdron's dve seem to propose an actual parallelization of a single video/source/stream (distributes chunks of a single stream). While it possibly can get the job done, and nicely, chances are that the result is sub-optimal as far as size is concerned; a single node capable to evaluate all chunks seems advantaged in this regard (especially if multiple passes are selected). Yet it could be very convenient to distribute chunks in some cases, ty for sharing.

      This doesn't differs much if we're talking on encoding HLS stream chunks. They are already distinct files.

    • JanevskiJanevski Member
      edited November 2018



      PS: Get dedicated for cpu intensive stuff.

      You are dreaming. | And it's a nightmare. | THE SECRET THREAD | THE TRUTH | HAVES YOU SEEN THIS YURA?
      „Homo homini rattus.“ | It's not nightmare, it's reality, but it's still nightmare.

    • jcalebjcaleb Moderator

      noaman said: I have a limited network connection the whole purpose of this is to able to encode h264 to h265 or vp9 and reduce bandwidth

      I think it will conserve bandwidth actually

    • inklightinklight Member
      edited November 2018

      noaman said: I have a limited network connection the whole purpose of this is to able to encode h264 to h265 or vp9 and reduce bandwidth

      Man don't say it , how much limited bandwidth (1mb/s !) how about 250kb/s transcoding for SD and 600kb/s for HD
      use this code (you need to change bit rate only )

       ffmpeg -i "sorce" -vcodec libx264 -preset slow -break_non_keyframes 1 -profile:v high444 -x264-params "nal-hrd=cbr" -b:v 220k -af "aresample=async=1:min_hard_comp=0.100000:first_pts=0" -acodec aac -b:a 42k -map_metadata -1 -s 480x360  out.mp4  
      
       ffmpeg -i "sorce" -vcodec libx264 -preset slow -break_non_keyframes 1 -b:v 665k -profile:v high444 -x264-params "nal-hrd=cbr" -af "aresample=async=1:min_hard_comp=0.100000:first_pts=0" -acodec aac -b:a 64k -map_metadata -1 -s 854x480  out.mp4  
      
    Sign In or Register to comment.