Detecting IIS 503s caused by multiple host instances

[Tratcher]

https://techcommunity.microsoft.com/t5/iis-support-blog/asp-net-core-503-server-has-been-shutdown/ba-p/3830338

In IIS in-proc, if a developer creates and disposes an extra host for some reason before creating the real host they intend to start, the disposal of the first host puts IIS in a faulted state that the second one can't recover from. This results in the app returning 503s after the first few requests.

We should see if it's feasible to detect this state when starting the second host and at least produce a good error. e.g. "IIS can't be started because something in the process has already shut it down. Remove any extra Hosts."

public static void Main(string[] args)
{
    WebHost.CreateDefaultBuilder(args).UseStartup<EnvironmentStartup>().Build()
        .Dispose();
    CreateHostBuilder(args, EnvironmentStartup.Configuration).Build().Run();
}

public static IHostBuilder CreateHostBuilder(string[] args, IConfiguration config)
{
    return Host.CreateDefaultBuilder(args).ConfigureAppConfiguration(delegate(HostBuilderContext hostingContext, IConfigurationBuilder config)
    {
        [truncated]
    }).ConfigureWebHostDefaults(delegate(IWebHostBuilder webBuilder)
    {
        if (config.GetValue<bool>("UseIISHosting"))
        {
            webBuilder.UseIIS();
        }
        else
        {
            webBuilder.UseKestrel();
        }
        webBuilder.UseStartup<Startup>();
    });
}

[mahamr]

While I believe the host disposal may cause issues, the main problem above is going to stem from the calls to ConfigureWebHostDefaults and UseIIS. Both of those lead to a IISNativeApplication instance being added as a singleton to the DI container, thus ejecting one of them for finalization later. The finalization whenever it happens will trigger the 503s.

[amcasey]

thus ejecting one of them for finalization later

I'm probably missing something, but I thought all the instances ended up in an IEnumerable or something and then you could either retrieve a individual one (the last) or all of them. How did you determine that the older one was being finalized?

[davidfowl]

I'm probably missing something, but I thought all the instances ended up in an IEnumerable or something and then you could either retrieve a individual one (the last) or all of them. How did you determine that the older one was being finalized?

All are added to the container, and the last one is disposed explicitly here.

Which means the old one isn't disposed, which leads to it being finalized.

[Tratcher]

The original customer reported no more issues after removing the duplicate host. We can check on the finalizer, but it doesn't seem to be an issue for them.