Online.net packet loss?

smiba

(Please move if this is not the right category, unsure if Outages is only for confirmed issues)

Hey everyone,

Is anyone else suffering from (high) packet loss on their online.net servers? Both my server in AMS1 and DC3 are affected by noticeable packet loss. (This started prox. about 9 hours ago)

As for now the server in both DC's are suffering from about 6% packet loss (tested over 1000 pings) and it doesn't seem to happen at my server itself. Back when it started there was over 50% packet loss so the issue has become less worse over time.

Here is a quick MTR report made an hour ago:

mtr --report -c 100 [My Server in DC3]
Start: Sat Feb  9 09:00:22 2019
HOST: [Server from diff. provider]  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- [Server from diff. provider] 0.0%   100    0.6   0.8   0.5  15.6   1.5
  2.|-- 94.103.144.121               0.0%   100    2.7   8.1   2.5 185.7  27.4
  3.|-- vl4001.pcr1.asd-nh.nedzon    0.0%   100    2.7   8.8   2.7 193.7  28.9
  4.|-- online-1.telehouse-pa2.nl    0.0%   100   18.1  18.1  17.6  19.1   0.0
  5.|-- 45x-s32-1-a9k1.dc3.poneyt    7.0%   100   18.4  19.7  18.0  57.5   5.3
  6.|-- [My Server in DC3]           3.0%   100   17.6  17.6  17.5  19.0   0.1

mtr --report -c 100 [My Server in AMS1]
Start: Sat Feb  9 10:12:11 2019
HOST: [Server from diff. provider]  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- [Server from diff. provider] 0.0%   100    1.1   0.9   0.5  19.4   1.8
  2.|-- 94.103.144.121               0.0%   100    2.5   7.4   2.5 170.2  22.4
  3.|-- 9-1-25.ear3.Amsterdam1.Le    0.0%   100    2.8 169.9   2.7 597.7 249.2
  4.|-- xe-5-6.rt1.ams3.baseip.co    6.0%   100   16.4  16.6  16.2  19.2   0.3
  5.|-- 51.158.8.27                  3.0%   100   15.8  15.9  15.6  16.4   0.0
  6.|-- 195.154.2.129                7.0%   100   16.2  16.3  16.1  18.1   0.1
  7.|-- [My Server in AMS1]          3.0%   100   16.2  16.1  15.8  17.4   0.1

An iperf3 test to their own ping.online.net service also shows trouble (Taken from server in AMS1)

# iperf3  -c ping.online.net -p 5202
Connecting to host ping.online.net, port 5202
[  4] local [Server in AMS1] port 33816 connected to 62.210.18.40 port 5202
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  8.85 MBytes  74.2 Mbits/sec   14   31.1 KBytes       
[  4]   1.00-2.00   sec  2.67 MBytes  22.4 Mbits/sec    2   53.7 KBytes       
[  4]   2.00-3.00   sec  4.16 MBytes  34.9 Mbits/sec    3   41.0 KBytes       
[  4]   3.00-4.00   sec  2.92 MBytes  24.5 Mbits/sec    3   46.7 KBytes       
[  4]   4.00-5.00   sec  3.36 MBytes  28.1 Mbits/sec    3   45.2 KBytes       
[  4]   5.00-6.00   sec  3.29 MBytes  27.6 Mbits/sec    4   29.7 KBytes       
[  4]   6.00-7.00   sec  2.92 MBytes  24.5 Mbits/sec    3   33.9 KBytes       
[  4]   7.00-8.00   sec  1.80 MBytes  15.1 Mbits/sec    5   29.7 KBytes       
[  4]   8.00-9.00   sec  2.42 MBytes  20.3 Mbits/sec    3   45.2 KBytes       
[  4]   9.00-10.00  sec  2.73 MBytes  22.9 Mbits/sec    5   24.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  35.1 MBytes  29.5 Mbits/sec   45             sender
[  4]   0.00-10.00  sec  32.4 MBytes  27.2 Mbits/sec                  receiver

Just wondering if its just me, been trying to talk to support but with hours between each message while they just keep asking me to do more iperf or ping tests.

Their status pages don't list anything related to this issue.

(No online.net server? You can still run a ping test agains ping.online.net, for me it has about the same amount of packet loss from my home connection indicating its not an isolated issue)

Neoon

https://twitter.com/online_fr/status/1093904118991781891

smiba

Yeah I've seen, but the issues started only 9 hours ago. Before that near zero packet loss showing in my monitoring, while those cables definitely were cut already back then!

Also broken cables are unlikely to affect all DC's? I'd just expect lower throughput, higher ping rates or more hops in the route and not packet loss in the affected areas.
(Given their status page also says they've rerouted all traffic, so packet loss shouldn't happen in these amounts)

sgheghele

More infos in this thread:

https://twitter.com/scaleway/status/1093891295377002496

We're suffering from a large fiber cut on our network backbone causing partial connectivity loss to several servers in PAR1. Our Network Team is on it. We will keep you regularly updated.

rm_

You can see their network map at http://netmap.scaleway.com/

Currently many links are black (inactive), which causes the rest to go into overload.

lowenduser4531

my DC2

mtr --report -c 100 [Server in AMS1]
Start: 2019-02-09T11:31:16+0100
HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- [My server in DC2]      0.0%   100    0.3   0.3   0.3   0.8   0.1
  2.|-- 195.154.2.131              0.0%   100    0.6   0.4   0.4   0.8   0.1
  3.|-- a9k1-pni-th2.dc3.poneytel  0.0%   100    0.6   0.6   0.4   1.0   0.1
  4.|-- 195.154.2.104              0.0%   100   11.6  11.5  11.3  12.4   0.2
  5.|-- 51.158.8.27                0.0%   100   11.1  10.8  10.7  11.3   0.1
  6.|-- 195.154.2.129              0.0%   100   13.3  13.4  13.3  15.0   0.2
  7.|-- [Serverin AMS1]         0.0%   100   13.1  13.1  13.1  13.3   0.1

Janevski

@smiba said:
Yeah I've seen, but the issues started only 9 hours ago. Before that near zero packet loss showing in my monitoring, while those cables definitely were cut already back then!

Also broken cables are unlikely to affect all DC's? I'd just expect lower throughput, higher ping rates or more hops in the route and not packet loss in the affected areas.
(Given their status page also says they've rerouted all traffic, so packet loss shouldn't happen in these amounts)

Simba, it is i, Mufasa! I am talking to you from beyond the grave, via clouds.
I forgot what it was, but something about inner strength.
Keep it in mind.

k tnx bye

eol

Ancient wireless is the best.