I get a invalid digest error when connecting to a SFTP server but only with a particular key and server combination

[markbeazley]

Not sure if this is an actual issue with SSH.NET or not, but I'm a bit stumped at figuring out what exactly is going wrong. I suspect it might be some issue with the SSH/SFTP server itself as its SSH-2.0-OpenSSH_for_Windows_8.6 but I'm not getting the same issues using the command line sftp program on my linux development machine.

I am trying to connect to the above SFTP server with SSH.NET, using a private key file )4096-bit RSA with no password), I get this error/stack trace, when I call Connect() on the client

Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest
   at Interop.Crypto.RsaSignHash(SafeEvpPKeyHandle pkey, RSASignaturePaddingMode paddingMode, IntPtr digestAlgorithm, ReadOnlySpan`1 hash, Span`1 destination)
   at System.Security.Cryptography.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature)
   at System.Security.Cryptography.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
   at System.Security.Cryptography.RSA.SignData(Byte[] data, Int32 offset, Int32 count, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
   at Renci.SshNet.Security.Cryptography.RsaDigitalSignature.Sign(Byte[] input)
   at Renci.SshNet.Security.KeyHostAlgorithm.Sign(Byte[] data)
   at Renci.SshNet.PrivateKeyAuthenticationMethod.Authenticate(Session session)
   at Renci.SshNet.AuthenticationMethod.Renci.SshNet.IAuthenticationMethod.Authenticate(ISession session)
   at Renci.SshNet.ClientAuthentication.TryAuthenticate(ISession session, AuthenticationState authenticationState, String[] allowedAuthenticationMethods, SshAuthenticationException& authenticationException)
   at Renci.SshNet.ClientAuthentication.Authenticate(IConnectionInfoInternal connectionInfo, ISession session)
   at Renci.SshNet.ConnectionInfo.Authenticate(ISession session, IServiceFactory serviceFactory)
   at Renci.SshNet.Session.Connect()
   at Renci.SshNet.BaseClient.CreateAndConnectSession()
   at Renci.SshNet.BaseClient.Connect()

I can connect to the server fine if I use a username and password, or if I use a ssh key provided by GPG using a GPG key stored on a hardware token using the AGENT protocol with this library so I don't think its a problem with SSH.NET and the server itself,.

So this made me think it was an issue with the key, so I set up a linux machine locally, added the public key to it, and SSH.NET was able to connect fine using the same private key.

After this I tested connecting to the original server using the sftp command in linux with that private key and it works.

Which shows the error only happens when using this one key, this one server and SSH.NET. Presumably some subtle algorithm/config difference in SSH.NET/the .net Crypto libraries vs the linux command line tools on my computer.

Edit:

Just tested rolling back to 2024.0.0 from 2025.0.0 and it all works, so must be some change between the two.

Edit 2:

Narrowed down to a change in 2024.1.0 not sure what exactly though.

Edit 3:

Still happening as of 2025.0.1-prerelease.17

[markbeazley]

I think this is due to the switch to System.Security.Cryptography and I think some choice of algorithm ending up picking one that isn't supported I think rsa-sha1. I think the other server that works must be triggering a different algorithm which exists and works.

[markbeazley]

Not sure if its something that can be fixed or is an issue for SSH.NET, but I've swapped to an ed25519 key and that works fine.

Not sure what exactly was causing the RSA key and that servers config that was causing the broken choice of algorithm.

[Rob-Hague]

It looks like the machine that you are getting the error on has disabled SHA-1 signature support in the OS (at least, in OpenSSL). Does that sound right? RSA keys do indeed try to authenticate with a SHA-1 signature, and you are right about the cause being the switch to S.S.C which calls into an underlying crypto provider (OpenSSL). It is possible to hack around this by removing the "ssh-rsa" algorithm from PrivateKeyFile.HostAlgorithms.

ed25519 uses SHA-512 (the library also doesn't use S.S.C for it) so it that explains why that works.