Adding SetApplicationName seems to have broken token validation for some identity apps on the same host (cannot decrypt new tokens generated after change)

[jjts001]

IdentityServer version

Duende IdentityServer 7.0.8

.NET version

.NET 8.0

Description

Exception thrown from Microsoft.AspNetCore.Identity.DataProtectionTokenProvider.ValidateAsync when performing token decryption (either Email Confirmation token, or Password Reset token) for one of the ID Server web apps on a VM hosting multiple instances of ID Server (customer segregation of apps and databases). Another instance with the exact same configuration and release version does work however.

In app startup we have configured -

builder.Services.AddDataProtection()
       .PersistKeysToFileSystem(new DirectoryInfo(keyDirectory))
       .SetApplicationName(Assembly.GetEntryAssembly().GetName().Name);

where the Keys folder is local to each ID Server web application. But even though each app has its own Keys folder it does appear that app isolation may have been broken in some way by the change to add SetApplicationName? as it was working on the same host in the earlier version, but since the product upgrade which added that line this issue has been encountered.

We cannot reproduce this issue for only one instance of the ID Server web app per host and so we did not encounter it until the production hosting.

We may have to issue a patch release to take away the call to SetApplicationName? unless there is something we can do as a workaround? We've tried deleting the contents of the Keys folder.

Any info/suggestions much appreciated.

Reproduction steps

No response

Expected behavior

No response

Logs

No response

Additional context

No response

[maartenba]

Before setting the application name, ASP.NET defaulted to using the content root path of your site as the application name. Adding an explicit name now means this has changed, where ASP.NET uses a new data protection key.

It may be a good idea to remove the (explicit) application name again in order to restore access to your existing keys. You can then also request the application name ASP.NET uses for the application, e.g. using this code in your startup:

// ...

var app = builder.Build();

// ...
var discriminator = app.Services.GetRequiredService<IOptions<DataProtectionOptions>>() .Value.ApplicationDiscriminator;

// discriminator is now the (auto-generated?) application name

Regarding isolation: if all your applications are now using Assembly.GetEntryAssembly().GetName().Name as their application name, there is a chance they all share the same application name only if the assembly name for each separate application is the same.

[jjts001]

Many thanks for the response Maarten.

What doesn't make sense to me is that apparently the tokens are being encrypted using keys which the application now does not have access to (the "existing" keys) and therefore cannot decrypt them? Is that correct? I was under the impression that given each application instance has its own locally defined Keys folder, and we had deleted them and restarted the id server app, and it apparently generated new keys i.e. new files appeared in its local Keys folder, that it would be able to issue new tokens and decrypt them successfully, but that is not the case, as after doing so the application still cannot decrypt the tokens. So it seems that changing the application name means that instead of just defining new keys as we expected it is still using old keys under the old discriminator, even though it is configured now with a new application name?

Also the example on https://docs.duendesoftware.com/identityserver/v7/deployment/data_protection/ states "explicitly set an application name to prevent issues with key isolation", setting this seems to have done the opposite for us.

What i mean is, is it true to say that SetApplicationName is a method by which an application can invalidate itself, rather than re-configure itself?

[maartenba]

Going from the default (no discriminator specified) to having an explicit discriminator specified effectively means the keys have changed, and old data can no longer be decrypted. If you have the old data from your keys folder still around, I'd recommend checking the old discriminator name (see my first response) and setting that as the discriminator.

If you have lost the data from the keys folder, that would mean you've lost the keys (and thus won't be able to decrypt data that was encrypted before).

setting this seems to have done the opposite for us (re: isolation)

Is the assembly name the same for each deployed app? If yes, and they are all stored in the same folder, this effectively means all apps use the same application name. If the assembly name is different, that should be OK and isolated.

[jjts001]

Thanks Maarten, fully understood - each application has its own local separate Keys folder - and in this case the tokens that fail to decrypt are newly generated i.e. after the application has been deployed with the new discriminator configuration, and also after we deleted the contents of the Keys folder and restarted the application, so none of the tokens are "old" tokens encrypted using the old keys - these are new tokens which we tried to use immediately and encountered an exception in ValidateAsync.

So what I still don't understand is that the application is configured with an explicit keys path, local to itself, and it was requested to generate a new email confirmation token, which was used immediately afterwards but it failed to decrypt it.

[maartenba]

Got it, thanks! That's slightly different indeed. Do you happen to have the full exception message and stack trace?

[jjts001]

Thanks again. No unfortunately very little information, all we have is "ValidateAsync failed: unhandled exception was thrown."

I assume it is somewhere under Protect.Unprotect as there isn't a more explicit message -

[maartenba]

You mention keys are stored on disk. Is your application single-instance or are you running multiple instances?

If you have the ability to run this on your affected environment (and add logging around it), this could help see if data protection is configured correctly:

var protector = app.ApplicationServices.GetRequiredService<IDataProtector>();
try
{
    var protectedString = protector.Protect("this is a test");
    // todo: log
    try
    {
        var unprotectedString = protector.Unprotect(protectedString);
        // todo: log
    }
    catch (Exception e)
    {
        // todo: log
    }
}
catch (Exception e)
{
    // todo: log
}

[jjts001]

Thank you very much Maarten again. Our production hosting has multiple instances of the same application on the same VM, each with their own local Keys folder, in test we only have one instance on the same VM and it works fine, and so we only found out in production of this issue. Thanks for the suggestion we can create a new application to run on the same environment using the same configuration etc. to log more details of the error, will get back to you with more info when we have it.

[craigfowler]

Hi all, I work with @jjts001 . He's on leave this week but we have recently made a discovery which can shed a lot more light on this issue. I'm recording that here, mainly for the benefit of others who might find this conversation in the future and have suffered similar symptoms.

That discovery is that our ops team had made a setup/configuration mistake (human error). I think, intending to perform a server migration, they forgot to decommission the old instance. This meant that they had two instances of Identity Server running, on different hosts, with no knowledge of one another, sharing the same persistence database. Edit to clarify: We routinely have multiple ID Servers on the same host, but for different tenants/different databases. This problem was caused by two ID Servers on different hosts, for the same tenant/database. But, being on different hosts they would have been using different operating systems with different private keys and different filesystems where some other data protection material was stored.

Once we detected it (whilst working on a different issue), we resolved this configuration error by shutting one of the instances down. The symptom which led to this conversation (tokens in things like passwords reset emails failing to round-trip) then went away with no further change required.

I'm not saying that anything said here is explicitly incorrect; but in our case it turns out that the primary culprit was something else entirely!