Software routing performance

randvegeta

Hello.

I'm looking into building a number of backup routers using our surplus servers. Since they are backups, they will be idle most of the time, and so it's hard to justify spending thousands on proper routers.

So I'm talking about using x86_64 systems with decent network cards running Vyatta/VyOS.

As far as I am aware, Quagga does not utilize multiple cores for its routing protocol, and so table lookups may be slow as the routing tables are now over 500k!

I have been testing with a Dual Xeon L5520 server, and it seems to work fine under normal load. But if the number of packets were to rise, for example under even the most modest of DDoS attacks, I suspect it may just fall over.

If Quagga really only uses a single core for routing, it suggests that higher clock speed is more important, and so the L5520 is limited to 2.26Ghz. And i3-3240 has 3.4Ghz so in theory could probably do the same job about 50% faster. Any thoughts on this?

I was also thinking if the routing table was dramatically reduced, then the number of lookups would also be reduced, and therefore the number of CPU ticks would be much less. So in theory, if you were to build a static routing tables of say 200+ entries (all the routable /8s), or even just use a default route, then the work load would be reduced by upto 2,500 x.

So would this make any significant improvement in performance and ability to handle moderate DDoS attacks?

Baring in mind the amount of traffic I expect to be running on these routers to be around 100Mbit, and rarely more.

If anyone has experience setting up software routers in production, I would love to know how they get on!

Thanks.

timnboys

@randvegeta said:
Hello.

I'm looking into building a number of backup routers using our surplus servers. Since they are backups, they will be idle most of the time, and so it's hard to justify spending thousands on proper routers.

So I'm talking about using x86_64 systems with decent network cards running Vyatta/VyOS.

As far as I am aware, Quagga does not utilize multiple cores for its routing protocol, and so table lookups may be slow as the routing tables are now over 500k!

I have been testing with a Dual Xeon L5520 server, and it seems to work fine under normal load. But if the number of packets were to rise, for example under even the most modest of DDoS attacks, I suspect it may just fall over.

If Quagga really only uses a single core for routing, it suggests that higher clock speed is more important, and so the L5520 is limited to 2.26Ghz. And i3-3240 has 3.4Ghz so in theory could probably do the same job about 50% faster. Any thoughts on this?

I was also thinking if the routing table was dramatically reduced, then the number of lookups would also be reduced, and therefore the number of CPU ticks would be much less. So in theory, if you were to build a static routing tables of say 200+ entries (all the routable /8s), or even just use a default route, then the work load would be reduced by upto 2,500 x.

So would this make any significant improvement in performance and ability to handle moderate DDoS attacks?

Baring in mind the amount of traffic I expect to be running on these routers to be around 100Mbit, and rarely more.

If anyone has experience setting up software routers in production, I would love to know how they get on!

Thanks.

I have setup software routers before like untangle, sophos utm and other distro's including simplewall and others(but I threw simplewall out the window when finding it was open source packages that are free from linux but all they did is wrap it with a nice gui and put a very nice very expensive price tag on it)

Maounique

I have setup tons of software routers, from 386 SX 20 Mhz to Xeon E5520.
A routing table is now in memory, the search will not be done on disk, however, I also suspect they will fail during attacks, that is the whole point of the attack, to produce a failure of some sort, depends what will buckle first, the router, the bw, the NIC, etc.
Any Xeon starting with CoreDuo will be able to deal with 1 Gbps normal line, but they will buckle even with good NICs under DDoS.
10 Gbps will do with an E5520, up to 100k pps or so if the cards are good as they should in a server and minimal optimization is done.
Your problem, though, will be power usage, make sure to aggressively save power on the idle machines.

randvegeta

Maounique said: Your problem, though, will be power usage, make sure to aggressively save power on the idle machines.

I'm thinking about using my i3-3240 machines. They tend to idle at about 0.2amps (44 watts) so I'm not too concerned.

I don't expect the software to hold up to any REAL DDoS attack, but it would be nice if it could handle 200k pps or more.

The NICs I plan to use are pretty good, and I don't expect that to fall over, at least not before the BW pipe becomes saturated.

If you're getting 100k pps on an E5520, I imagine an i3-3240 would be at least 50% beter (so 150k pps). Still better than a Juniper J2320 (which is crap!).

I also realize that the routing table will be stored in RAM rather than HDD, but would a reduction in the routing table size mean an increase in routing capacity?

If the routing table is 500k lines long, and it takes 1 CPU cycle to go through each entry of the routing table, it may take upto 0.25ms to find that a match for the destination. Of course I don't know this to be the case, I am just assuming somthing like this is done, and if it is, then reducing the size of the routing tables (through use of much fewer static routers) should reduce the time to match and get the packet on its way to the next destination. No? Am I just talking crazy here?

guyz92

What about using pfsense?

It can also function as firewall too.

Microlinux

It's not routing table lookups that will kill you, it's interrupts. I run VyOS with multiple full BGP sessions pushing multiple gigabits of traffic on E3 systems, works like charm. 100k PPS is nothing, but a high rate DDoS will cripple just about any x86 based system.

Microlinux

This a traffic low point on one interface with netflow capture also running:

In: 362.1 Mbps, 66874 PPS, 0 errors
Out: 293.0 Mbps, 65614 PPS, 0 errors

top - 08:50:40 up 310 days,  7:14,  3 users,  load average: 0.38, 0.41, 0.42
Tasks: 100 total,   1 running,  94 sleeping,   0 stopped,   5 zombie
Cpu0  : 12.7%us,  0.6%sy,  0.0%ni, 66.0%id,  0.0%wa,  0.2%hi, 20.5%si,  0.0%st
Cpu1  :  7.1%us,  0.3%sy,  0.0%ni, 89.4%id,  0.0%wa,  0.0%hi,  3.1%si,  0.0%st
Cpu2  :  4.5%us,  0.2%sy,  0.0%ni, 92.1%id,  0.0%wa,  0.0%hi,  3.2%si,  0.0%st
Cpu3  :  4.5%us,  0.2%sy,  0.0%ni, 94.1%id,  0.0%wa,  0.0%hi,  1.2

rm_

Microlinux said: it's interrupts

Yeah and to deal with a high number of them you will need server-grade NICs (likely also means Intel ones) with multi-queue support, then manually assign queues to separate CPUs (real cores, not HT), similar to this: https://greenhost.nl/2013/04/10/multi-queue-network-interfaces-with-smp-on-linux/
That way it's very much possible to handle insane PPS in software with no problem.

dot_txt

Quagga itself does not do packet routing. That is done within the kernel itself. Quagga runs the routing protocols that inject routes into the Linux kernel. Your routing table lookups with a full set of routes should be quite fast no matter what cpu you use so long as it was made in the last 10 years. As for OS choice, vyos is probably the best prepackaged router platform out there right now.