All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Host identification changed warning and I can't explain why
I recently encountered "Host identification changed warning" and I am genuinely curious how this is even possible. "Host identification changed warning" is the following message
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
==============deleted==============
Please contact your system administrator.
Add correct host key in =deleted= to get rid of this message.
Offending ECDSA key in =deleted=:51
ECDSA host key for =deleted= has changed and you have requested strict checking.
Host key verification failed.
Here is more context. My VPS suffered from some downtime. Then, when it recovered and I tried to ssh into it, I encountered this warning. I filed a ticket regarding the downtime and this warning to the host and I didn't hear a satisfactory explanation else than it is due to a power outage.
I tried to reason about this warning
1. I didn't reinstall OS.
2. The host might have reinstalled or recovered from extremely old backup before I reinstalled. => My system logs, .zsh_history, local files, and cron jobs are intact, including latest changes I remember from couple days ago.
3. It may be MITM attack? a. Is my personal computer under attack? => I have used GNU+Linux for years as my primary and only OS and I know the basic security hardening rules. The applications I install are very boring and I try to use flatpak or docker for any dubious or closed source applications. I see no signs of other suspicious change in my computer. I see the same host identification changed warning from three different computers I use.
4. It may be MITM attack? b. Is my ISP under attack? => This specific VPS is the only one causing this problem among 6 VPS's I currently manage. I run into this error no matter if I use VPN, coffee shop wifi, or from work.
5. It may be MITM attack? c. Is it my host messing with my VPS with malicious intent? => It's OpenVZ and afaik (I know almost nothing about VPS, though), the host can mess with my files without shutting down the system. But I think this is very, very, very unlikely. They are not a summer host.
6. My host might have misconfigured something in the VPS system to lead to this error? I have no single clue how this is even possible.
In the end, I am very lost. Google didn't give me a satisfactory answer. Is there anyone here who could give me any possible explanation about this warning? Thank you.
Comments
Are IP address and hostname same as before?
Have you inspected the previous (now mis-matched) entry on line 51 of your known_hosts file for possible clues?
Is your ssh daemon the same version? It's not uncommon for updates/upgrades to create a new host key if the existing one is decidedly insecure, or new protocols such as ed25519..
Thank you for suggesting more hypothesis.
Yes, it follows my local host file, not DNS. I double-checked with my personal wiki that the IP nor hostname has changed.
My previous entry of known_hosts file looks like a valid entry to me.
{deleted_correct_ip_port} ecdsa-sha2-nistp256 {some_hash_value}and they are exactly same across three different computers (one with different distro). And they look very similar format as other entries of different servers. I do not have enough expert knowledge to confidently say that this is the correct hash value, but with common-sense rules (from length or format), I see no problems.
My previous (now offending) host key was in
ecdsa-sha2-nistp256and the new host key is inecdsa-sha2-nistp256as well. I just checked dpkg log and I see no upgrade in sshd/openssh-related packages since last boot. It is now usingv7.9p1since February, which seems to be the most recent version.Not sure if you're our customer ... we had downtime in St. Petersburg last weekend and there were some issues with our cloud-init configuration.
If the guest VM started before metadata server, it may re-generate SSH host keys. We're aware of this issue and you can uninstall cloud-init to avoid this as we won't provide snapshot feature in the near future and cloud-init is unnecessary or useless after provision.
If you want a detailed explanation:
This bug was introduced by a workaround to fix root password not being saved properly.
The script (/usr/local/sbin/setrootpw) should only run during the first boot but if the server can't reach metadata server, it will run again and re-generate ssh host keys and set root password to default password.
No I am not.
Thanks a million. This seems to be the most plausible answer to my issue. A couple of issues happened with the same host due to misconfiguration of cloud-init and this seems to be yet another issue coming from cloud-init. Considering that this issue happened right after a reboot from power outage, this makes perfect sense. I will let my host know about it.
On successful connection to meta data server, the script should remove itself from being launched ever again.