Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
IX traffic
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

IX traffic

Hi Folks

Want to ask, if any dedicated server providers offer separate network interface for within peer exchange they are connected to.

Let’s say, I want to install a CDN edge, with 2 network interfaces, one connected with transit provider, and another looking into exchange. And obviously I want different traffic plans, as IX traffic is almost free.

How is this usually handled as long as you are not Netflix willing to ship your own servers for colo.

Thanked by 1mrclown

Comments

  • @at0mic said:
    Hi Folks

    Want to ask, if any dedicated server providers offer separate network interface for within peer exchange they are connected to.

    Let’s say, I want to install a CDN edge, with 2 network interfaces, one connected with transit provider, and another looking into exchange. And obviously I want different traffic plans, as IX traffic is almost free.

    How is this usually handled as long as you are not Netflix willing to ship your own servers for colo.

    You send @clouvider a nicely worded message.

    Thanked by 1Clouvider

    I like my uptime down low and my servers all hacked. Can see me droppin' twenty-fours with a router in the rack.
    Ya like ya Switch-Ports hot and ya servers all hacked. If ya pings real high and ya networks pitch black.

  • @teamacc said:
    You send @clouvider a nicely worded message.

    Thanks! But it is UK/NL/FRA only, my question is are there any providers operating on IX level at all? Cause @Clouvider does not seem to offer that off the shelf

  • rubenruben Member, Provider

    at0mic said: @Clouvider does not seem to offer that off the shelf

    It's not that common that a customer needs this, so it's rarely "off the shelf". I also provide IXP Access in FRA, but only on a VM level.
    Many Dedicated Server providers (like @Zare, @Clouvider) offer this if you ask, as @teamacc said, nicely;)

    at0mic said: But it is UK/NL/FRA only,

    Which IXPs / Locations do you want? UK/NL/FRA gives you access to almost every Europen Networks.

    at0mic said: Netflix willing to ship your own servers for colo.

    Quite frankly, this is the way most companies do it. Just as you are more likely to find a 1HE Colo than a Dedi with an IXP connection.

    iFog GmbH - Webhosting, DNS Hosting, vServer, IP-Transit, IXP VMs, RIPE LIR, BGP VMs
    Tunnelbroker - Free BGP v6/v4 Tunnels

  • at0micat0mic Member
    edited December 2019

    Which IXPs / Locations do you want? UK/NL/FRA gives you access to almost every Europen Networks

    I’d say US coasts, Northern Europe, Russia.

    but only on a VM level.

    VMs are somewhat better, when you are starting/growing, but how much of the value can it be with minimal ram/disk space?
    Is there any approximate price range for such VMs? Are they cheaper than regular ones?

    Just as you are more likely to find a 1HE Colo than a Dedi with an IXP connection.

    Bet most of dedis are IX connected, as it’s money out of nothing.

    I get that most heavy players are using colo, but that’s lowendtalk, right? Ima gonna build cdn for pennies 🙂

  • hzrhzr Member, Moderator

    @ruben I don't think he's looking for an IX port + routeserver + BGP, i think he just wants a 2nd nic with cheaper traffic that's IX only.. .or for IX / transit billing buckets different while the upstream is on both.

  • hzrhzr Member, Moderator

    no, they will be more expensive... and a lot more complex.. this is not what you think

  • ClouviderClouvider Member, Provider

    Thanks @teamacc

    @at0mic it depends what you’re looking to achieve and your individual circumstances. To join an IX you need an ASN, a routable subnet, usually an incorporated business, and you need to run BGP. If you have all that, assuming your scale, you find a dedicated server provider who is a partner/reseller of an IX you’re liking to join, perhaps an IX Sales team will be able to direct you. For example, we are partner of LINX and can supply you access to LINX on a VLAN. We can then provide you a session with a full table on one vlan, billed separately, and a vlan to IX that you would handle, billed separately.

    If you need a hand in London, Amsterdam or Frankfurt, we’re your people and happy to help :-)

    Clouvider Limited - Leading Hosting & Connectivity Partner || Dedicated Server Sale from £45/m - Our Latest LET Offer

    Cloud Web Hosting | SSD & SAS HA OnApp VPS | US, UK, NL & DE Dedicated Servers | Network Services | Colocation | Managed Services

  • @hzr said:
    @ruben I don't think he's looking for an IX port + routeserver + BGP, i think he just wants a 2nd nic with cheaper traffic that's IX only.. .or for IX / transit billing buckets different while the upstream is on both.

    Basically yes, second IX nic. No need for upstream on IX nic though, down only.

    I suppose IX traffic is not “cheaper”, it should be free, no? Or almost free at least.

  • hzrhzr Member, Moderator

    at0mic said: I suppose IX traffic is not “cheaper”, it should be free, no? Or almost free at least.

    Cross connects (port from router to IX) can be $1k+ PER MONTH per port in the US. It's not free. (This doesn't include traffic or IX membership at all)

    This also helps upstream providers balance a bit, because imagine if you were the one paying by 95% instead of per-GB.

    Thanked by 1TimboJones
  • ClouviderClouvider Member, Provider

    @hzr said:
    @ruben I don't think he's looking for an IX port + routeserver + BGP, i think he just wants a 2nd nic with cheaper traffic that's IX only.. .or for IX / transit billing buckets different while the upstream is on both.

    In that case unless you have a very specific use case with a ton of traffic that goes that particular cheaper route it will be more expensive.

    First of all - complicated and so adds to costs.

    Second of all - I won’t charge you the same price for a full table if you explicitly intend to push only the most expensive traffic through it, and the remainder through cheaper option. You’ll pay more for the “remaining traffic” and less for the IX only traffic, with all complexity involved you’ll likely loose - not save.

    Thanked by 1SplitIce

    Clouvider Limited - Leading Hosting & Connectivity Partner || Dedicated Server Sale from £45/m - Our Latest LET Offer

    Cloud Web Hosting | SSD & SAS HA OnApp VPS | US, UK, NL & DE Dedicated Servers | Network Services | Colocation | Managed Services

  • @hzr said:
    no, they will be more expensive... and a lot more complex.. this is not what you think

    Can you please elaborate on that? I’ve checked couple of IXes, and it looks like fees are quite reasonable for 10G port unlimited traffic for colo.

  • hzrhzr Member, Moderator

    at0mic said: Can you please elaborate on that? I’ve checked couple of IXes, and it looks like fees are quite reasonable for 10G port unlimited traffic for colo.

    For example there's an IX in one of the DCs I use. The DC's cost to have a cable run from my rack to the IX router costs more than the 10G port per month. This doesn't include the IX cost.

  • @Clouvider said:

    @hzr said:
    @ruben I don't think he's looking for an IX port + routeserver + BGP, i think he just wants a 2nd nic with cheaper traffic that's IX only.. .or for IX / transit billing buckets different while the upstream is on both.

    In that case unless you have a very specific use case with a ton of traffic that goes that particular cheaper route it will be more expensive.

    First of all - complicated and so adds to costs.

    Second of all - I won’t charge you the same price for a full table if you explicitly intend to push only the most expensive traffic through it, and the remainder through cheaper option. You’ll pay more for the “remaining traffic” and less for the IX only traffic, with all complexity involved you’ll likely loose - not save.

    Serving video content is exactly the case you’ve described. Minimal transit, maximum IX. Guess in the result it’s going to be more profitable.
    Don’t quite get where the complexity comes from, adding a BGP entry? Almost sure you set up internal traffic routed to IX for all already set up servers.

  • at0micat0mic Member
    edited December 2019

    @hzr said:

    For example there's an IX in one of the DCs I use. The DC's cost to have a cable run from my rack to the IX router costs more than the 10G port per month. This doesn't include the IX cost.

    Wow, that’s tough! Does that mean, that it’s easier for them to route internal IX traffic on that 10G port for you, rather than letting you do it?

    Is this IX inter-connected with any other IXes?

  • trewqtrewq Administrator, Moderator, Provider

    Any provider worth working with will negotiate bandwidth pricing with you based on your transit/peering ratios. What you're trying to achieve here is possible but it doesn't sound like you're at the scale where it'd be worthwhile.

    Thanked by 1Clouvider
  • @trewq said:
    Any provider worth working with will negotiate bandwidth pricing with you based on your transit/peering ratios. What you're trying to achieve here is possible but it doesn't sound like you're at the scale where it'd be worthwhile.

    You are totally right, but I’ve worked in content delivery before and know how expensive hitting the roof can be. Especially with video.

    Getting to know the options does not seem to be worthless.

  • hzrhzr Member, Moderator

    The real tl;dr here is you need enough scale before it's even worth the engineer time to respond to a sales quote like this. Saving 10TB of bw is literal pennies.

    Thanked by 1Clouvider
  • hzrhzr Member, Moderator
    edited December 2019

    FYI: getting on an IX does not imply you will be able to send traffic over the IX. Larger players, home ISPs, etc will refuse this unless you meet minimum xx Gbps 24/7 traffic ratios over transit already AND can peer in multiple locations simultaneously meeting the same rules.

    This is normal:

    image

  • @hzr said:
    The real tl;dr here is you need enough scale before it's even worth the engineer time to respond to a sales quote like this. Saving 10TB of bw is literal pennies.

    Cannot agree completely, depends on the business type we are talking about. Sure thing, considering one specific traffic generator it does not make sense.

    But clearing things up, before re-selling traffic does not look meaningless.

    I sincerely appreciate your comments and explanations.

  • hzrhzr Member, Moderator

    at0mic said: But clearing things up, before re-selling traffic does not look meaningless.

    If you are willing to commit to 20-30 Gbps for a minimum of a year contract immediately I would bet people will reduce pricing based on your transit:peering ratios after measuring them over time. If this is for a single dedicated server, i don't think I would really find anyone that would do this.

  • PUSHR_VictorPUSHR_Victor Member, Provider

    @at0mic I am doing what you seem to be doing, and at some point I had the same idea and even reached out to providers to see how they will respond. I then figured out it won't work out because a provider has no motivation to offer such a thing. When they bill you for bandwidth and they give you a good price on that, they already count on a lot of your traffic going over IX links and not transit so they can make a healthy profit AND still give you good pricing. Asking them to separate your bandwidth and bill you lower for the part that goes over IXs will make no sense to them. To make this happen, you go out there, colo yourself, buy the equipment and stuff, and then either go with someone who can help you get a port at an exchange cheaper (thats where you can contact these providers again for help), or deal directly with the exchange.

  • FYI: getting on an IX does not imply you will be able to send traffic over the IX. Larger players, home ISPs, etc will refuse this unless you meet minimum xx Gbps 24/7 traffic ratios over transit already AND can peer in multiple locations simultaneously meeting the same rules.

    Are you talking about cross-connects with them?

    If you are willing to commit to 20-30 Gbps for a minimum of a year contract immediately I would bet people will reduce pricing based on your transit:peering ratios after measuring them over time. If this is for a single dedicated server, i don't think I would really find anyone that would do this.

    I see, very valid point. That’s why I thought some dedi providers operating on a scale might offer this option already.

    Thanks for clarification that it’s better to PM for such offers.

  • @PUSHR_Victor said:
    @at0mic I am doing what you seem to be doing, and at some point I had the same idea and even reached out to providers to see how they will respond. I then figured out it won't work out because a provider has no motivation to offer such a thing. When they bill you for bandwidth and they give you a good price on that, they already count on a lot of your traffic going over IX links and not transit so they can make a healthy profit AND still give you good pricing. Asking them to separate your bandwidth and bill you lower for the part that goes over IXs will make no sense to them. To make this happen, you go out there, colo yourself, buy the equipment and stuff, and then either go with someone who can help you get a port at an exchange cheaper (thats where you can contact these providers again for help), or deal directly with the exchange.

    Thanks! That makes A LOT of sense 👍🏻 And once you are at this point, you join the group of providers making a healthy profit with no intent to re-sell split traffic.

  • hzrhzr Member, Moderator
    edited December 2019

    at0mic said: Are you talking about cross-connects with them?

    No. I mean they will not accept your IX traffic or peer with you (the selective ones are often not on the routeservers either) - and require that traffic to them go over transit, unless you are willing to push xx Gbps + peer at ALL IXes you both are present at, with a minimum of some amount.

    And they require your own network topology to more or less be that you have your own private transport between locations - they expect the same routes to be announced from all locations consistently.

  • at0micat0mic Member
    edited December 2019

    @hzr said:

    at0mic said: Are you talking about cross-connects with them?

    No. I mean they will not accept your IX traffic or peer with you (the selective ones are often not on the routeservers either) - and require that traffic to them go over transit, unless you are willing to push xx Gbps + peer at ALL IXes you both are present at, with a minimum of some amount.

    And they require your own network topology to more or less be that you have your own private transport between locations - they expect the same routes to be announced from all locations consistently.

    I apologise for this bunch of dumb questions, but why would they prefer transit over IX, like the whole IX idea is about avoiding transit as much as possible, no?
    Won’t it affect their 95/5?
    What is the benefit of getting off the routeserver?

  • hzrhzr Member, Moderator
    edited December 2019

    at0mic said: I apologise for this bunch of dumb questions, but why would they prefer transit over IX, like the whole IX idea is about avoiding transit as much as possible, no? What is the benefit of getting off the routeserver?

    They do not want "anyone" to crap up their sessions. Or they are Comcast and want people to pay them both for sending and receiving the traffic. They will only accept your IX traffic directly if you can demonstrate 24/7 instant NOC response, consistent and always available routes, and they will kill your peering ports if you let them go past x% of port capacity (ie if you are sending 8 Gbps on a 10G port, they will stop accepting IX traffic from you until you upgrade to 20G, 100G, whatever, otherwise their users will get lossy at peak)

    They do not prefer, it is just policy that they do not peer with risky small players that might do dumb shit accidentally, or some other thing like that

    Won’t it affect their 95/5?

    They don't care, they do terabits of traffic, your 1G port is nothing

    Thanked by 1TimboJones
  • @hzr said:
    They do not want "anyone" to crap up their sessions. Or they are Comcast and want people to pay them both for sending and receiving the traffic. They will only accept your IX traffic directly if you can demonstrate 24/7 instant NOC response, consistent and always available routes, and they will kill your peering ports if you let them go past x% of port capacity (ie if you are sending 8 Gbps on a 10G port, they will stop accepting IX traffic from you until you upgrade to 20G, 100G, whatever, otherwise their users will get lossy at peak)

    They do not prefer, it is just policy that they do not peer with risky small players that might do dumb shit accidentally, or some other thing like that

    Won’t it affect their 95/5?

    They don't care, they do terabits of traffic, your 1G port is nothing

    Sounds pretty logical for companies, who are both transit providers and ISPs themselves. Accepting anything from IX would mean losses on transit side.

  • hzrhzr Member, Moderator
    edited December 2019

    at0mic said: Sounds pretty logical for companies, who are both transit providers and ISPs themselves. Accepting anything from IX would mean losses on transit side.

    It's also more that accepting peering with a tiny 1-2 person company for 1 Gbps is just a liability to them in terms of route leaks, accidental hijacks, unfiltered sessions, whatever other enterprisey stuff. The engineer time required to resolve any of these for 30 seconds completely outnumbers the cost savings of having 1-10G on an IX.

    Keep in mind at those scales, your 10G of transit probably costs nearly as much as 10G of IX

    If you're providing 100G+ of value to their customers, then they miiiiight possibly give half a shit.

  • What you want, usually only makes sense at a volume where you should already be considering to run your own network.

    If you just want to stream some gbps of video, get some cheap bandwidth from budget providers like OVH. If that bandwidth becomes some tens of gigabits per second, then you can start considering other options.

  • at0micat0mic Member
    edited December 2019

    @Nyr said:
    What you want, usually only makes sense at a volume where you should already be considering to run your own network.

    If you just want to stream some gbps of video, get some cheap bandwidth from budget providers like OVH. If that bandwidth becomes some tens of gigabits per second, then you can start considering other options.

    That’s what I was doing, and it sounds good on paper but in reality does not work that well.
    When you are streaming live video, 1gbps port can handle only 400 users watching stream in 720p and this port better be not shared.
    Secondly, streaming from OVH to US west coast is hardly an option as of hops and jitter, meaning you’ll never provide a decent stream quality without edges somewhere in LA for instance.
    And this traffic is highly unpredictable, cause for 1 stream you might have 500 viewers or 5000 easily. They also come with spikes, so you literally have no time to react to traffic clogs.
    Besides that, during just 2 hours 1000 customers will generate at least 2TB of traffic, and it is 1 stream only, having couple of them running in parallel will get you more. And this traffic spikes only during weekends, while servers being idle for the rest of the week.

    And using traditional CDN like Akamai is damn expensive.

    What I’m trying to say, is that volume and scalability issues in such situations aren’t something theoretical but a typical issue from the very beginning. You can’t even start that business without having an idea in mind how you scale from the very beginning.

  • @at0mic thanks for sharing all of this. Really interesting to read your special requirements, a network engineers dream :smile:

  • FHRFHR Member, Provider

    @at0mic said:

    And using traditional CDN like Akamai is damn expensive.

    BunnyCDN has got you covered

    SkylonHost - affordable hourly-billed KVM VPS in Prague, CZ!
    Featuring own high performance network AS202297 | RIPE NCC member | Contact us for IPs/ASNs

  • at0mic said: And using traditional CDN like Akamai is damn expensive.

    It can be expensive, but it's exactly what you are requesting: quality bandwidth with unlimited scalability and big but isolated peaks.

    Using budget dedicated servers, you can manage those 5k 720p streams with $1k/month which is very reasonable imho.

    A third option which I would use if I was desperate, would be to (ab)use DigitalOcean + Scaleway and hope that they don't ban me (which shouldn't happen with your numbers).

  • at0micat0mic Member
    edited December 2019

    @Nyr said:

    It can be expensive, but it's exactly what you are requesting: quality bandwidth with unlimited scalability and big but isolated peaks.

    True, you are absolutely right! If I were building a video product using VC money, I’d go this way! But building CDN is more challenging.
    >

    Using budget dedicated servers, you can manage those 5k 720p streams with $1k/month which is very reasonable imho.

    I managed to deliver such amount of traffic for less than $150

    A third option which I would use if I was desperate, would be to (ab)use DigitalOcean + Scaleway and hope that they don't ban me (which shouldn't happen with your numbers).

    That’s exactly what I did 🙂 abused DigitalOcean. I also tried using ultra-cheap VPSes as point edges, but it did not work out well.

    FYI: cheap $10 Atom dedi can pass through about 800mbps, limiting in CPU.

  • fpmagicfpmagic Member
    edited December 2019

    Its an odd case. Since your provider probably wants to load off as much of your traffic as possible on peering for their own cost savings, if you're using only transit in the a blend, they would be enticed to price you higher than a customer using their blend that includes peering.

    Thanked by 1Clouvider
  • @at0mic said:

    @trewq said:
    Any provider worth working with will negotiate bandwidth pricing with you based on your transit/peering ratios. What you're trying to achieve here is possible but it doesn't sound like you're at the scale where it'd be worthwhile.

    You are totally right, but I’ve worked in content delivery before and know how expensive hitting the roof can be. Especially with video.

    Getting to know the options does not seem to be worthless.

    You can contact www.first-root.com, they are connected to DEC-IX Düsseldorf, Hamburg and Munich and you might have the chance to connect to ECIX and similar as well.

  • It seems to me that your real problem is a classical peak problem that is, you need high bandwidth only about 40 hrs/mo (2 weekend days at 5 hrs and (min) 1 TB/hour) plus a scale problem that is, 40 TB are more than many provider give you in total per month yet it is way too small to play even in the medium league.

    About the only solution that is financially bearable (other than a CDN) I see for you is to find two (?) well connected providers, one in Europe and one in the USA who don't care much about how you use your bandwidth (e.g. in 4 "bursts" per month) and to rent a dedi with each that is well connected, e.g. 2 x 10 Gb (or even better).

    I think that your approach to go for connectivity/bandwidth directly, e.g. via IXs, won't work out because the players in that field are in another league and consider your business too small fries. If you had enough money you could become a mid-size player and then resell what you don't need. If ...

    Alternatively I'd look at BunnyCDN because they do have the critical mass needed -and- would be glad to serve you as a customer (I guess) which is the combination you need and which is hard to find.

    Thanks no.

  • Does cloudflare and their network that have reduced bandwidth fees between cloudflare providers able to help him here somehow? Or because this is live stream and not cached, it's minimal benefit if any?

    Or this is how he was able to use Digital Ocean for $150/month and serve 5K users?

  • @TimboJones thanks for pointing out on the Cloudflare offer. 1000 minutes for $1 seems quite suspicious though. That would be a damn good offer if it included 4K videos, but when it comes to 720p it is quite the average.

    Good thing about Cloudflare is they are present in many complicated regions, like Africa and South America. Having customers there it could be a very good fit.

    They also have a very nice (though a bit outdated) article about Transit/IX traffic distribution in different regions

    https://blog.cloudflare.com/bandwidth-costs-around-the-world/

    Talking about $150/month. I used to stream 3mbps 720p videos, and with a cost of $0.005 per GB of traffic it fits perfectly into the numbers I specified. Worth mentioning: half a cent per GB is not the cheapest offer on the market

  • Choosing the DC with good IX connection is right choice where they could basically helped you to connect. However there's still the cost involved. Quite many of them (especially in not so cheap internet bandwidth countries) sold them as "local bandwidth".

    If your budget is pretty low, just go decent CDN in low end range or with dedi and 10Gbps port. The more you know about your audience location, the more cost effective that will be.

    signature for rent - ^_^

  • PureVoltagePureVoltage Member, Provider

    We provide this normally for customers who do colocation in Seattle and New York.

    As said from someone else on here you will need your ASN and everything to get linked up with it. We typically will vlan off one of our ports from our router and connect it up to you and pass along a fee for it.

    This works quite well, we don't normally like doing this via dedicated servers itself as we have had a lot of scammers trying to get into exchanges.

    PureVoltage Colocation with 6 Global locations, Seattle, LA, New York, Dallas, Chicago, and Amsterdam

  • SplitIceSplitIce Member, Provider
    edited December 2019

    at0mic said: IX traffic is almost free.

    Traffic might be but the monthly connection cost can be quite high. Likely higher than a peak user will be using to saturate the connection for short windows.

    If your traffic requires the host to upgrade from say 10Gbps at IX to 100Gbps for your peaks that's cost that needs to be recouped. Hint: it's recouped against the bandwidth costs. Or more likely your prefix gets removed from the IX announcements.

    Honestly you might be better with a local node with good Transit, not IX. Transit isnt that expensive these days. And on a large network where you won't be massively changing the 95th it's not going to make much of a difference.

    If looking at the CDN route, you could perhaps reduce costs by serving major areas yourself and using a CDN service for the more difficult areas (i.e Asia & Africa).

    Also don't forget bandwidth and hardware costs will only be a small portion of your costs. Competent management will cost far more.

    Thanked by 1at0mic
    X4B - DDoS Protection: Affordable Anycast DDoS protection including Layer 7 mitigation with PoPs in the US, EU and Asia.
    Latest Offer: Black Friday 2019 Offer
  • anyNodeanyNode Member, Provider

    While IX traffic can often be cheaper than transit there is a lot of other factors.

    Cross connects, transport if in another building and then theres the IX port fee

    https://peering.exposed is a good look into the costs if port fees are publicly available

    Thanked by 2ma2t at0mic

    anyNode.netLA & Miami KVM & Miami Dedicated Servers

    http://twitter.com/anynode

  • vserversitevserversite Member, Provider

    Hi,

    we can provide you VMs and Dedicated with Access to KleyReX + LocIX Frankfurt VLAN. Also possible for dedicated Servers.

    https://vserver.site/de/bgpvm.html
    Feel free, to contact us.

Sign In or Register to comment.