MXRoute Down?

sandanista

Anyone else having issues with MXRoute. It's been flawless since I signed up, but today:

An error occurred while sending mail. The mail server responded: Unexpected failure, please try later. Please verify that your email address is correct in your account settings and try again.

Can't receive or send.

jar

All of it, whole thing is down.

Actually just a repeat of this: https://status.mxroute.com/incident/38

sandanista

Thanks @jar, that was fast. Hope you get it sorted soon.

jar

@sandanista said:
Thanks @jar, that was fast. Hope you get it sorted soon.

All sorted

sandanista

That my man is the fastest fix ever. No doubt about the quality of your service. Thank you.

SplitIce

@jar check your inodes df -i. Your disk can be full because there are no remaining inodes as well as because there is no remaining free space.

Quite common on applications that create large numbers of small files (e.g mail servers).

LTniger

Probably deleted files but file descriptors was opened. Solution: reboot server to recalculate space.

jar

@SplitIce said:
@jar check your inodes df -i. Your disk can be full because there are no remaining inodes as well as because there is no remaining free space.

Quite common on applications that create large numbers of small files (e.g mail servers).

Aye. Was only 2% utilization on that. Disks seem healthy otherwise too. I'm missing something.

@LTniger said:
Probably deleted files but file descriptors was opened. Solution: reboot server to recalculate space.

That very well may be it.

randomq

Reserved space? Check dmesg for errors?

Inodes was a good guess, a lot of people miss that one!

yoursunny

@randomq said:
Inodes was a good guess, a lot of people miss that one!

I use FAT16 or NTFS file system, so that I don't need to worry about inodes.

sandanista

The message could not be sent because connecting to Outgoing server (SMTP) xxxxx.mxrouting.net failed. The server may be unavailable or is refusing SMTP connections. Please verify that your Outgoing server (SMTP) settings are correct and try again.

Anyone else having these issues? This is becoming more and more common just lately. Seems the service is gradually deteriorating?

TimboJones

@sandanista said:
The message could not be sent because connecting to Outgoing server (SMTP) xxxxx.mxrouting.net failed. The server may be unavailable or is refusing SMTP connections. Please verify that your Outgoing server (SMTP) settings are correct and try again.

Anyone else having these issues? This is becoming more and more common just lately. Seems the service is gradually deteriorating?

Contact them.no issues listed for 5 days. https://status.mxroute.com/

yoursunny

@sandanista said:
The message could not be sent because connecting to Outgoing server (SMTP) xxxxx.mxrouting.net failed. The server may be unavailable or is refusing SMTP connections. Please verify that your Outgoing server (SMTP) settings are correct and try again.

Anyone else having these issues? This is becoming more and more common just lately. Seems the service is gradually deteriorating?

Please attach tcpdump trace when it happens.

Also, MXroute enabled IPv6 last week.
If your IPv4 and IPv6 connections are from different providers, and one of them is unstable, it could cause intermittent errors.
Once again, such problem will show up in tcpdump.

No tcpdump trace => PMS.

sandanista

Ok, thanks guys. It is fine again now.

jar

Deployed new modular update system to work with DA’s custombuild, to update applications and re-apply customizations to them immediately, since the update system overwrites so many things.

https://github.com/mxroute/da_server_updates

This one in particular did it’s job fine on all but two servers:

https://github.com/mxroute/da_server_updates/blob/master/exim/update_exim.sh

Pixel and Blizzard servers ended up with an interesting problem: Exim was only listening on port 25. But since it was listening on 25, monitors didn’t trip (I generally expect up is up and down is down, never expected this result). I never figured out what happened exactly, I just ran the script again and it did it’s job. Would have loved to grab more data beforehand, but opted for fixing first. I’ll add in some checks, alerts, and quick capturing of relevant data for further analysis in case it happens again.

These scripts are only intended to be run while supervised for now anyway,

sandanista

Thanks @jar. Yes, I'm on blizzard, so I guess these issues were related. All good now anyway.