Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Looking for Partners/Devs For CDN & other projects
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Looking for Partners/Devs For CDN & other projects

I'm looking out for any partners/employees/outsource companies that is interested to develop a simple CDN frontend + backend.

Main purpose of the CDN is to value add to existing cPanel hosting customers or for CDN customers with large bandwidth needs, not really targeted to sell CDN as a service to the general public. There's plenty of competition in the market already.

Current antique setup is basically central config repo + rsync cronjobs. Anycast network for both DNS + HTTP. It works well enough for just our own website + previously sponsored a couple of image hosting sites, but the CDN setup just isn't customizable (eg. able to select which locations to serve from) or scalable. I started with 6 nodes in 5 cities back in 2014, network has grown to >15 cities but I've not touched the existing VMs since 2015. Spent too much time growing the network. I manage SG.GS/AS24482 if anyone wants to know.

I'm open to options/discussion, if you already run a CDN and want to spin-off your software for my use case, I'm game. Otherwise a ground up build works for me too. Alternatively if you want to partner up to develop this further as a larger public offering it's also on the table. Budget/salary can be discussed in private.

As for other projects:

1) I have plenty of bandwidth in various cities but only small amount of infrastructure (1-2 Proxmox servers) locally in those cities. Most my resources (servers & storage) are in Singapore/Asia. Looking for bandwidth intensive projects which are not latency sensitive (one of the reasons I'm building the CDN) that can use up my excess bandwidth since it's already paid for. My purpose is not to recover costs but to generate traffic, so token sum or free is possible depending on how interested I am in your project. Any interested parties can share your plans here or contact me in private.

2) Looking for Network+Sysadmin Devops that is looking for work. Hoping for someone that can optimize the network by developing tools to automate latency testing, etc. There's a lot on my want-to-do but I only have 2 hands. Needs to be familiar with BGP, traceroute, etc.

Admins/mods if it's in the wrong section please help move it to a more appropriate section, thanks!

Comments

  • lentrolentro Member, Host Rep

    I thought CloudFlare already had a cPanel and Plesk plugin?

    That should save you some time as opposed to creating your own CDN.

  • KenshinKenshin Member

    @lentro said:
    I thought CloudFlare already had a cPanel and Plesk plugin?

    That should save you some time as opposed to creating your own CDN.

    Then customer will have to migrate DNS servers to CF, which is a pain especially for non-tech savvy customers. We're trying to integrate it seamlessly for existing customers since we already host the DNS and control the web servers.

  • LeviLevi Member

    @Kenshin said:

    @lentro said:
    I thought CloudFlare already had a cPanel and Plesk plugin?

    That should save you some time as opposed to creating your own CDN.

    Then customer will have to migrate DNS servers to CF, which is a pain especially for non-tech savvy customers. We're trying to integrate it seamlessly for existing customers since we already host the DNS and control the web servers.

    When 'non-tech' knows what is the meaning of word CDN, than he is more than tech savvy to switch dns. Especially when it's free... How can you compete with something that is a world class product and free on top of that?

  • KenshinKenshin Member

    @LTniger said:

    @Kenshin said:

    @lentro said:
    I thought CloudFlare already had a cPanel and Plesk plugin?

    That should save you some time as opposed to creating your own CDN.

    Then customer will have to migrate DNS servers to CF, which is a pain especially for non-tech savvy customers. We're trying to integrate it seamlessly for existing customers since we already host the DNS and control the web servers.

    When 'non-tech' knows what is the meaning of word CDN, than he is more than tech savvy to switch dns. Especially when it's free... How can you compete with something that is a world class product and free on top of that?

    Consider this from a hosting provider perspective. Your customer is a local mom and pop store with a simple wordpress website, they buy a <$10/month hosting package from you. Their visitors are mainly local, the hosting server is local and SSD storage. There's absolutely no reason to need a CDN.

    Then out of the blues they run a page load test from USA which is halfway around the globe from the server and it says their website is SLOW! Big company's site says they NEED a CDN to speed up their website. This is UNACCEPTABLE. Guess who they bitch and whine at? :smile:

    Now to the customer, it's the hosting provider's fault that the website is slow and not the hundreds of huge images they uploaded on their front page which load in seconds locally because 1G broadband and <4ms latency across within country. Big company's speed test says the website is slow, YOU MUST FIX IT NOW, IT'S ALL YOUR COMPANY'S FAULT. Trying to convince them that the page loads in seconds on their home PC in the same country is useless, asking them to reduce their number of images is asking for a good beating. As far as they are concerned, you are a huge liar and big company is god.

    This is a mom and pop shop, trying to get them to sign up to CF on their own is next to impossible and they'll likely lose the password and bug YOU for a reset instead of CF, plus remember they're pissed with you that their website is "slow". Doing it for them becomes a chore for the support team especially if the domain isn't under your control.

    When you get enough of such customers, the easiest way out is to give them what they want in the way that is the least annoying for the team. Internal CDN would solve this and we still retain full control because the next thing that's coming from the customer will be CF's 502 bad gateway error which specifically says it's the Host Error even though the your hosting servers are running just fine. With your own CDN you can troubleshoot both edge and origin.

    So yeah, it's not about competing with free, which is why I said I have no plans to fully commercial it at this point to compete with other providers in offering CDN services to the public. It's about offering a value added solution to existing customers for no costs that avoid customers deciding to take a more painful route that will likely turn around to bite us in the ass.

  • deankdeank Member, Troll

    Just one problem.

    When you load a site from the other side of world, sites generally load fast enough that your eyes won't notice. Ping might be horrible but that doesn't really have an effect that our eyes can tell.

    The era of a site taking 5 seconds to load is long gone unless something was horribly configed.

  • KenshinKenshin Member

    @deank said:
    Just one problem.

    When you load a site from the other side of world, sites generally load fast enough that your eyes won't notice. Ping might be horrible but that doesn't really have an effect that our eyes can tell.

    The era of a site taking 5 seconds to load is long gone unless something was horribly configed.

    Yep, that's the thing. Even 5 seconds is reasonable as long as the HTML loads quickly enough while the rest of the element loads so it doesn't feel as slow. But with page benchmarks it's all about hard numbers. I just tested one of my customer's sites, total content 23MB in 227 elements for their homepage alone. Took 13.5 seconds to load from USA. Locally on Chrome just 4 seconds. I can go all on about optimizing their website and how they don't have USA visitors but some customers just aren't going to listen.

  • BoltersdriveerBoltersdriveer Member, LIR

    deank said: When you load a site from the other side of world, sites generally load fast enough that your eyes won't notice. Ping might be horrible but that doesn't really have an effect that our eyes can tell.

    I can think of quite a few websites where there are issues with this.

    That aside, asking a customer to use an external service, especially one where you don't have control over and has no SLA for is a recipe for disaster, especially if you don't have control over the stack at that point. What if Cloudflare has some weird issue? Or, the more likely scenario given this is in APAC and SG, is what if Cloudflare decides not to send traffic to/from your website from the local POP and offloads it to LAX instead? (Happens a lot for free customers on Cloudflare) Customer will complain and it just ends up being more work for you again.

  • deankdeank Member, Troll
    edited May 2020

    Who in the WSS designs a website with 23mb worth of shit?

    And is the site for bots or humans?

  • KenshinKenshin Member
    edited May 2020

    @deank said:
    Who in the WSS designs a website with 23mb worth of shit?

    And is the site for bots or humans?

    Magento site, they sell products locally. Their largest element is a 3.3MB animated GIF 2200x1177 which is shrunk to 385x199 on the homepage via height/width in HTML. :wink:

  • deankdeank Member, Troll

    Oh, Gods. Oh, my dear Lord.

    You have one of those customers who are stuck in 18th century. I pity thee.

  • KenshinKenshin Member

    I have a LOT of customers who are like this. 3-5MB images shrunk down via width/height scattered throughout their site. The thing is that we have extremely fast broadband and 4G mobile locally. The country is so small that latency is really low, 22MB of elements loads in 2 seconds, the other 2 seconds is the main page HTML generation by Magento. As sinful as it is, the customers and visitors themselves don't feel it. But when a benchmark is performed then shit hits the fan and because the hosting provider is in most cases the single point of technical contact for everything website related that they can pick up the phone and scream at, we get the shortest end of the stick.

  • deankdeank Member, Troll

    I know educating them is not possible.

    So, good luck to ya. I have no solution, other than citing The end is nigh.

  • LeviLevi Member

    @deank said:
    I know educating them is not possible.

    So, good luck to ya. I have no solution, other than citing The end is nigh.

    Sometimes the nigh is the end.

  • lentrolentro Member, Host Rep

    @Kenshin said:

    This is a mom and pop shop, trying to get them to sign up to CF on their own is next to impossible and they'll likely lose the password and bug YOU for a reset instead of CF, plus remember they're pissed with you that their website is "slow". Doing it for them becomes a chore for the support team especially if the domain isn't under your control.

    I believe you can sign up to become a CF hosting partner.

    This PDF is old (and I don't know if they still have hosting partner solutions - if not, at least I signed up when it still existed), but check this out.

    https://www.cloudflare.com/media/downloads/CloudFlare-Training-Part-2-Partner-Benefits.pdf

    If a hosting partner controls the DNS, you can point a CNAME at CF, so your customers won't need to manage the DNS.

    You also have complete control over the domain name, and the customer does not have access to that, preventing the password issue listed above.

    OR

    You, if you control the domain names, can just point it at their NSes.

  • KenshinKenshin Member

    @lentro said:

    @Kenshin said:

    This is a mom and pop shop, trying to get them to sign up to CF on their own is next to impossible and they'll likely lose the password and bug YOU for a reset instead of CF, plus remember they're pissed with you that their website is "slow". Doing it for them becomes a chore for the support team especially if the domain isn't under your control.

    I believe you can sign up to become a CF hosting partner.

    This PDF is old (and I don't know if they still have hosting partner solutions - if not, at least I signed up when it still existed), but check this out.

    https://www.cloudflare.com/media/downloads/CloudFlare-Training-Part-2-Partner-Benefits.pdf

    If a hosting partner controls the DNS, you can point a CNAME at CF, so your customers won't need to manage the DNS.

    You also have complete control over the domain name, and the customer does not have access to that, preventing the password issue listed above.

    OR

    You, if you control the domain names, can just point it at their NSes.

    You can't do CNAME on the root domain (domain.com) as it already has NS records. The plan is to CDN the domain.com + www.domain.com. There are customers who install wordpress on the domain itself without www, so I can't do www CNAME and redirect domain.com -> www.domain.com.

    As you can see, implementing CF means playing by their rules and technical limitations. A lot easier to just do this whole thing internally. I don't need fancy features that the other CDNs have since this is only for internal use + a few private account, not for public signup at least on day 1.

  • lentrolentro Member, Host Rep

    @Kenshin

    Hmmm... I have been able to do CNAMEs on root domains before.

    But, do whatever you think is best. You certainly seem to have the drive to get things done. Best of luck to you, and I'm excited to see what you will create. :)

  • duckeeyuckduckeeyuck Member
    edited May 2020

    but if you got large files, how is a cdn supposed to help? by removing a few ms from latency? eh? because once files are cache'd by the browser.... then you don't need the cdn at all. Unless your host is not keeping up with the bandwidth demand.

    But srsly, why would downloading 5mb from my country be faster than 200ms from another part in the globe? If I can 50mbits to OVH at 200ms and 50mbits to LA at 20ms, then what's the difference lol, barely anything

    I think this would only be useful if you compress files cache'd by cdn.

    If w.e software you use to manage 'hosted users', if it uses nginx, have you tried simply enabling gzip compression?

    Idk if I'm getting anything wrong here.
    edit: I'm high

  • pbxpbx Member
    edited May 2020

    Did you try using mod_pagespeed or something similar? That could help improving the page size, without the need to do a lot of teaching... Seems like a cool project anyway, but as you seem to already have several locations and real use will in most cases be rather small as most of your customers' traffic is local, why do you want so many locations? SG + FRA or AMS + NYC, ASH or MIA would likely be enough to guarantee good performance in most of the world (add Sydney, Brisbane, LA and Sao Paulo if you want, but that seems clearly unneeded)...

  • OhJohnOhJohn Member

    If you are rather interested in a pull setup (with some automatic-pull twists) as opposed to a push setup (as it sounds with rsync) I may be of help as I set up my own cdn with about 20 pops right now as none of the cdns on the market could exactly offer what I need for a project. But I'm not the network knowlegde guy so I don't use any own-operated BGP and the like but rather public networks so far.

    But your setup so far sounds rather homebreed (which is not wrong - mine is homebreed as well) with a - as it seems - totally different setup.

  • KenshinKenshin Member
    edited May 2020

    @duckeeyuck said:
    but if you got large files, how is a cdn supposed to help? by removing a few ms from latency? eh? because once files are cache'd by the browser.... then you don't need the cdn at all. Unless your host is not keeping up with the bandwidth demand.

    But srsly, why would downloading 5mb from my country be faster than 200ms from another part in the globe? If I can 50mbits to OVH at 200ms and 50mbits to LA at 20ms, then what's the difference lol, barely anything

    I think this would only be useful if you compress files cache'd by cdn.

    If w.e software you use to manage 'hosted users', if it uses nginx, have you tried simply enabling gzip compression?

    Idk if I'm getting anything wrong here.
    edit: I'm high

    TCP doesn't work that way, the initial speed is slow due to TCP window size starting off small and it will only increase over time. So for files that can be downloaded in <1 minute, the less latency the faster the speed during the first 30 seconds. This is why I mentioned I can view the same site in 4s but in USA it takes 13.5s.

    @pbx said:
    Did you try using mod_pagespeed or something similar? That could help improving the page size, without the need to do a lot of teaching... Seems like a cool project anyway, but as you seem to already have several locations and real use will in most cases be rather small as most of your customers' traffic is local, why do you want so many locations? SG + FRA or AMS + NYC, ASH or MIA would likely be enough to guarantee good performance in most of the world (add Sydney, Brisbane, LA and Sao Paulo if you want, but that seems clearly unneeded)...

    Doesn't help with images, which is where the CDN comes into play. The current locations are sufficient, which is why the old setup has been there for ages, but the idea is to build a proper scalable setup that can add/remove edge servers easily and customize routing to some extent. The corner case locations is just to prevent the genius customers who select Australia/Sao Paulo as testing locations on the publicly available website speed tests.

    @OhJohn said:
    If you are rather interested in a pull setup (with some automatic-pull twists) as opposed to a push setup (as it sounds with rsync) I may be of help as I set up my own cdn with about 20 pops right now as none of the cdns on the market could exactly offer what I need for a project. But I'm not the network knowlegde guy so I don't use any own-operated BGP and the like but rather public networks so far.

    But your setup so far sounds rather homebreed (which is not wrong - mine is homebreed as well) with a - as it seems - totally different setup.

    The current setup is a content pull from origin. I only rsync the configs from a central server so I don't have to make config changes on 6 machines. But this is not scalable, the ideal setup is a proper database with all the necessary data then each edge server generate it's own config which is unique to the site. For example, DNS server in Sao Paulo may return a USA IP because this particular website is video heavy and I don't have enough bandwidth/storage in Sao Paulo where bandwidth costs are higher. I can't do that easily with the current setup.

    Also what doesn't the current CDNs in the market have that you need? PM me if it's not something you want to share publicly, quite curious to know.

  • 1q11q1 Member

    @Kenshin now it's 2023, is this still relevant? (building your own CDN)

  • @1q1 said:
    @Kenshin now it's 2023, is this still relevant? (building your own CDN)

    It is 2023 so why are you bumping old thread?

  • srch07srch07 Member

    And some one dug a post from graveyard. And without looking at the date, I read full thread.
    Give me back my time.

    Thanked by 21q1 vedran
  • 1q11q1 Member
    edited June 2023

    @Hotmarer said: It is 2023 so why are you bumping old thread?

    what's wrong with asking the state of the project? also I was a customer of his company

  • @Arkas necro?

    Thanked by 1Arkas
  • ArkasArkas Moderator

    @1q1 said: what's wrong with asking the state of the project? also I was a customer of his company

    Quote

    Please make a new thread and read the rules, again.

This discussion has been closed.