Funilrys Reading time ~3 minutes

Today, we (me and a colleague) had to fight with a potentially corrupted SSH key. Indeed, we got the following error message on an existing server.

identity_sign: private key ~/.ssh/id_rsa contents do not match public

When SSH produces such an error, the first reflex of most of us (me included) will just be to delete the existing keys and generate a new one. This time, we chose not to follow that path.

The reason is quite simple: For that specific server even if the SSH connection was broken at that time T, the dependency chain, usage, or even reliability of that single SSH private key was not clear enough to justify such a drastic solution.

In other words, those were our possibilities:

  1. Choose the drastic path (delete, regenerate, share) but then potentially have to deal with a storm of new issues caused by the change.

  2. Locate and fix the issue with what we had in hand and therefore reduce or completely avoid a storm of new issues.

Therefore, we forced ourselves into the deep hole of an unknown SSH error.

To locate the issue, the best path is to try to find out if the issue is locally or remotely. That's where the verbose mode helps. It turned out that it was happening locally because the error was popping out just after the sign_and_send_pubkey step.

We then tried to compare both files. It surprisingly turned out that it is as simple as the generation of an SSH Key.

The ssh-keygen command has a -l argument that can show us the fingerprint of the given key file.

To quote the manual:

-f filename
    Specifies the filename of the key file.
-l  Show fingerprint of specified public key file.
    For RSA and DSA keys ssh-keygen tries to find the matching public key file and prints its fingerprint.
    If combined with -v, a visual ASCII art representation of the key is supplied with the fingerprint.

To give you an example:

$ ssh-keygen -l -f [key-file]

That's what we did and after about an hour of lookup, we were starting to doubt because we were getting the same fingerprint for both files. That implicitly meant that - if we choose to trust the output - the issue was maybe somewhere else.

In our doubt, we then decided to compare the sshd configuration with another known server, tried to change some of the SSH configuration related to the key, completely regenerate the known_hosts and authorized_keys on the remote machine and even tried to use the SSH key with another server. But unfortunately, none of that worked.

Side note

Strangely enough, now that I'm back home, I'm able to demonstrate what we saw. In the following example, the example_key.pub file was manually altered after generation. Even if the public key file is altered I still get the same fingerprint.

Private key:

$ ssh-keygen  -l -f example_key
4096 SHA256:7ipvW4hTm/200wSm2tRvcjenq6TMIPd6IGQgXnbkD8M me@example.org (RSA)

Public key:

$ ssh-keygen  -l -f example_key.pub
4096 SHA256:7ipvW4hTm/200wSm2tRvcjenq6TMIPd6IGQgXnbkD8M me@example.org (RSA)

After doubting long enough, I came to the idea that:

  1. the fingerprint may not be correct.
  2. the public key is corrupted.

Therefore, I asked myself if it is possible to regenerate the public key from the private key?

That's what solved our problem as it turns out that the ssh-keygen command let us do that through the -y argument.

To quote the manual:

-y      This option will read a private OpenSSH format file and print an OpenSSH
public key to stdout.

Therefore, a simple:

$ ssh-keygen -y -f [ssh-private-key-file] > [ssh-private-key-file].pub

regenerate the public key from the private key.

Facit

We learned that:

  • we should first try to regenerate the public key from the private key instead of regenerating the whole pair.

  • the ssh-keygen command lets us do much more thing than just the generation of a key pair.

  • mastering or understanding some of the most useful arguments of the ssh-keygen command - mentioned in this post - may save us some time in the future.