All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Stockholm hosthatch - problems with peers
Hello, Does someone have a problem with hosthatch Stockholm servers? When I'm under vpn at this server I don't have access to many peers.
even openduckduckgo does not open (this is from server console):
ping duckduckgo.com
PING duckduckgo.com (52.142.124.215) 56(84) bytes of data.
ping duckduckgo.com
PING duckduckgo.com (52.142.124.215) 56(84) bytes of data.
And many other peers not accessible the same way. And this is from very beginning. When I discover this I idle the server for months and hope that everything will be solved, but still the same problems or even worse.
I read the thread about hosthatch and "change mtu" to fix this. Tried MTU 1450, 1420, 1400, but it doesn't help.
What can be done to fix this? Any other experiencing similar problems in this location?
Thanks
Comments
Open a ticket?
I don't think duckduckgo.com responds to pings at all: https://dnstools.ws/ping/duckduckgo.com/?proto=Ipv4
http/https works fine for me:
In ticket I received the answer that it's not possible to move it to other location to fix this problem...
We don't offer location migrations for any of the plans in our promotions, but if there are many destinations that are unreachable from our network, that is probably something we're interested in fixing
Don't know, but it feel like some kind of blocking/firewall/maybe ddos protection( ? @hosthatch ). Under this server decentralized apps lose there peers, tor lose there enter points time to time and as said some sites do not response.
I have several others servers at other providers and never experience something like this.
I too have a similar issue with my STO storage server when performing remote backups via Proxmox Backup Server.
I frequently get broken pipe errors as the connection times out, which leads to failed backups. I did raise a ticket and they told me that the issue was with my ISP. Now I'm thinking that it might not be an issue on my end.
@flagpic yes, it feel like some firewall block our connections (when there are a lot of them at a time)... not ISP, I have other servers and never expect the same as already notices.
And I tried to run decentralized apps without VPN (directly from server console) and expect the same problems. It's not our ISP
I have another storage VPS in LA from LetBox that I use for the same purpose and backups run fine on that one.
Out of the few thousand customers we have there, yes it might look like there is an issue because someone also saw a similar issue to what you saw.
I am making an assumption here, but we probably asked for (at the very least) mtrs and traceroute for troubleshooting. No provider can investigate based on “some websites don’t work”.
I am happy to look into your ticket if you can share the ticket #
I don’t see why an issue with your ISP would affect your server backups. Can you please give me the ticket # so I can make sure we didn’t miss anything?
Probably should have explained better as I'm backing up my local VMs to the storage VPS in Stockholm.
Here's the ticket number #235619
That's odd. I doubt very much that there's any connection, but I'm also in the middle of troubleshooting a super weird issue with backups that work fine with destinations in one location (London) but not with destinations in another (Chicago).
Are you able to share what happens? Do the backups get cut off at random times, or is there some repeatable pattern?
My assumption is that this has nothing to do with HostHatch. I already confirmed with them that they have no traffic filtering in place. I've been pursuing it with the backup software vendor (Veeam), who as of yet hasn't been able to nail down the cause.
I am not sure if it's related, but yesterday my HostHatch Los Angeles server was also unreachable from some networks, including HostHatch's own London IP range and some websites I tested. Running mtr on the LA server, I found the route stopped at the 3rd hop. (Now it gets routed via Cogent as of writing.) The partial downtime lasted 8 hours, from my smokeping graphs.
Since the servers were totally unreachable from and to the affected networks, and it had not resolved itself after a few hours, I thought it was something that required HostHatch’s attention ASAP. So, I opened an "emergency" ticket*, and attached the traceroute and details. Guess what, I received zero replies before I closed the ticket hours later when the connectivity restored.
I'd say HostHatch's support is just golden. Most decent providers would at least promptly (let's say, within an hour) reply with something like "we'll have a look", even if they can't check with their upstream ISP right away.
* Ticket number being 848420, in case anyone would deign to check.
Can you PM me your IP range? I have an idea that this is due them using rented IPs that do not have correct RPKI/signing set. Since some weeks multiple Tier1s enforce signed routes.
works today...
Yes. Done
It is pretty random but I noticed that it mostly happens on large VMs since the connection has to stay alive for a long period of time.
As you said, I too think that the issue might not be hosthatch related, but I'm stuck trying to diagnose the issue. This is what shows up on the server side .
and this on the client side
route: 153.92.126.0/24
origin: AS63473
ROA is there, but it is non-RPKI signed Legacy space - I have a route via Telia/Twelve99 to Portlane so i assume it is fine. Try MTR when it does not work and see if it goes until Portlane - If not it likely is RPKI rejected.
Solution: As it is Afrinic have the ISP login there and set the RPKI to AS63473.
Interesting. I'm using different backup software, but my experience is similar. Do you notice any other network issues at the same time the backup fails? For instance, are there any interruptions with your SSH connection to the destination server? Have you tried running a ping test between the source and destination to see if there is any packet loss at the time the backup fails?
In my case, there is no interruption of pings, but any active network sessions (such as SSH) get reset. Since this is seemingly triggered by the backup software, that's my main focus at this point.
Psychz had a problem in LA yesterday reaching certain networks - this is completely unrelated to any problems in Stockholm though.
You have to understand that we run 14 global locations, and people are going to have routing issues more often than a provider with 2 locations.
Yes, I think it is quite obvious from the ticket (even you said so yourself looking at the mtr) that the issue starts way before it hits our network. We confirmed it with GleSYS (our upstream in Stockholm) and they said the same thing. GleSYS response was: "Looking at this relatively long route and the list of players involved, I doubt theres much we can do. I think your customer might be more successful contacting his/hers provider."
You're welcome to share the information here if you feel we missed something.
My only issue with the OP is the generalized nature of this complaint - "some websites are not working"....unfortunately that is not enough to investigate anything. We need mtrs (like you provided in the ticket) to be able to look into problems like these.
Yes, sure - but you are comparing LA vs Stockholm which most likely has completely different routes to get to your country.
Could be related to RPKIs, but then there are large providers running their whole networks without signed RPKIs....the internet would break and not just for us if that was the case. Again though - we can keep going in circles or we can see mtrs to the unreachable destinations and hopefully resolve this
Yeah, right, I figured I also have to understand that in order to serve thousands of customers with a small margin, slow or unresponsive support has to be your work ethic.
On a serious note, that was the first time I tried to open an "emergency" ticket, and the experience was a dealbreaker for any serious use of servers from HostHatch for me.
Yes, I'm afraid emergency tickets for routing errors/requests are largely ignored for users from certain countries, from where we get a large number of "route optimization requests" that we can do nothing about.
Sorry to hear that is a dealbreaker for you, but I assure you routing errors like these are very rare.
I understand that people from "certain countries" are in the habit of opening emergency tickets for those issues, and that can be a problem for you. But that does not justify simply ignoring all those tickets. If anyone cared to read just past the title of my ticket, they'd find it was very specific and was about a real issue.
Sorry, but how can I trust 911 if they ever ignored my calls because people from my neighborhood are in the habit of making false reports? Yes, I believe that routing issues like this are very very rare. And your uptime and performance are just great, so I had not had any need to open an "emergency" ticket at all. That's also why I thought I still could reply on your support in case of issues like these.
The same thing happens to me from a HostHatch VPS in Los Angeles. I'm using Borgbackup over SSH, and backups fail because the connection gets reset. I have to retry one or two times to get it to work. Similarly, large downloads over HTTP will sometimes fail part way and I'll need to resume them. SSH connections drop, but pings seem fine and no pings are dropped when this happens. If I'm connected to SSH all day, the connection will drop multiple times per day (I use Eternal Terminal to make that less of a problem).
I haven't opened a ticket yet because so far I don't have a consistent way to reproduce the issue, as it just sporadically occurs. IPv4 seems to do it less frequently than IPv6 so maybe I'll just force my backups to happen over IPv4...
I haven't tested with an SSH connection but do you use pfsense as your firewall by any chance?
@hosthatch
I'm not saying that it is an issue with the STO vps. I was just pointing out things that can be used to diagnose the issue at hand. As another user also mentioned, the issue seems to be with the backup software or something else.
I have multiple VPS with you and have been a customer since 3+yrs and can honestly say that you are the best provider in terms of pricing and quality. Hope this continues, so I can get more servers from you!
No, all of my servers use the basic firewalld service. I don't think it has any effect on the issue I've experienced.
I just want to note that the SSH connection failures are a common issue i already had with Hosthatch Sweden years ago.
No idea if related.
Sorry, I haven't tested other destinations yet .. But for example connection to duckduckgo.com not working again. Worked yesterday.
Here is mtr:
153.92.126.1 0.0% 113 36.8 37.4 35.6 50.3 1.8
be-4-121.pe4.sto1.se.portlane.ne 0.0% 113 37.1 37.7 36.4 46.0 1.7
be-7.cr2.sto1.se.portlane.net 0.0% 113 37.5 39.1 36.5 129.4 8.9
as8075-20g-sk1.sthix.net 0.0% 113 42.6 38.8 36.6 129.1 8.8
ae26-0.ear01.sto30.ntwk.msn.net 0.0% 113 37.7 40.2 37.1 99.3 6.2
be-22-0.ibr01.sto30.ntwk.msn.net 0.0% 113 75.5 76.1 75.0 95.0 2.2
be-6-0.ibr01.mma01.ntwk.msn.net 0.0% 113 75.2 76.6 74.6 175.2 9.4
Ultra interesting case as you get a route out but the return path is insanely dumb apparently. Wish i had SSH to verify this. But cba to pay for a server at this time.