Today, we (me and a colleague) had to fight with a potentially corrupted SSH key. Indeed, we got the following error message on an existing server.
identity_sign: private key ~/.ssh/id_rsa contents do not match public
When SSH produces such an error, the first reflex of most of us (me included) will just be to delete the existing keys and generate a new one. This time, we chose not to follow that path.
The reason is quite simple: For that specific server even if the SSH connection was broken at that time T, the dependency chain, usage, or even reliability of that single SSH private key was not clear enough to justify such a drastic solution.
In other words, those were our possibilities:
Choose the drastic path (delete, regenerate, share) but then potentially have to deal with a storm of new issues caused by the change.
Locate and fix the issue with what we had in hand and therefore reduce or completely avoid a storm of new issues.
Therefore, we forced ourselves into the deep hole of an unknown SSH error.
To locate the issue, the best path is to try to find out if the issue is
locally or remotely. That's where the verbose mode helps.
It turned out that it was happening locally because the error was popping out
just after the
We then tried to compare both files. It surprisingly turned out that it is as simple as the generation of an SSH Key.
ssh-keygen command has a
-l argument that can show us the fingerprint
of the given key file.
To quote the manual:
-f filename Specifies the filename of the key file. -l Show fingerprint of specified public key file. For RSA and DSA keys ssh-keygen tries to find the matching public key file and prints its fingerprint. If combined with -v, a visual ASCII art representation of the key is supplied with the fingerprint.
To give you an example:
$ ssh-keygen -l -f [key-file]
That's what we did and after about an hour of lookup, we were starting to doubt because we were getting the same fingerprint for both files. That implicitly meant that - if we choose to trust the output - the issue was maybe somewhere else.
In our doubt, we then decided to compare the sshd configuration with another
known server, tried to change some of the SSH configuration related to the key,
completely regenerate the
authorized_keys on the remote
machine and even tried to use the SSH key with another server.
But unfortunately, none of that worked.
Strangely enough, now that I'm back home, I'm able to demonstrate what we saw.
In the following example, the
example_key.pub file was manually altered after
Even if the public key file is altered I still get the same fingerprint.
$ ssh-keygen -l -f example_key 4096 SHA256:7ipvW4hTm/200wSm2tRvcjenq6TMIPd6IGQgXnbkD8M firstname.lastname@example.org (RSA)
$ ssh-keygen -l -f example_key.pub 4096 SHA256:7ipvW4hTm/200wSm2tRvcjenq6TMIPd6IGQgXnbkD8M email@example.com (RSA)
After doubting long enough, I came to the idea that:
- the fingerprint may not be correct.
- the public key is corrupted.
Therefore, I asked myself if it is possible to regenerate the public key from the private key?
That's what solved our problem as it turns out that the
let us do that through the
To quote the manual:
-y This option will read a private OpenSSH format file and print an OpenSSH public key to stdout.
Therefore, a simple:
$ ssh-keygen -y -f [ssh-private-key-file] > [ssh-private-key-file].pub
regenerate the public key from the private key.
We learned that:
we should first try to regenerate the public key from the private key instead of regenerating the whole pair.
ssh-keygencommand lets us do much more thing than just the generation of a key pair.
mastering or understanding some of the most useful arguments of the
ssh-keygencommand - mentioned in this post - may save us some time in the future.