Phantom app_offline.htm Detection Events and Application Shutdowns

[twinmind]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

We run ASP.NET Core web application on dotnet 6 in IIS 10 with in-process ANCM hosting model on Windows Server 2019 Datacenter instances in AWS Elastic Beanstalk. We started seeing 1-2 application shutdowns per machine within 48 hour period causing 503 errors on the application load balancer. Our servers live only 48 hours at most as we replace them with new VMs using autoscaling rules in our Beanstalk environment.

After investigation it was found that there are events with ID 1012 and event data looks like this:

Application 'C:\inetpub\wwwroot\' was recycled after detecting app_offline.htm.

We don't perform any application deployments during the time when we see these events and we don't see any other ways how app_offline.htm file could appear in C:\inetpub\wwwroot\.
We are not using Web Deploy for deployments and from the logs it looks like Beanstalk wasn't deploying anything at the time when we see those events.
The downtime usually lasts less than a minute and then application comes back online. On the screenshot you can see we got event ID 1032 Application 'C:\inetpub\wwwroot\' started successfully. in 20 seconds after app_offline.htm detection event.

We also don't see any system metrics anomalies during these events, CPU and memory look as usual.

Please advice what could be adding app_offline.htm to application directory and whether it can be a phantom message.

Symptom wise it seems very similar to this #24428

Expected Behavior

No app_offline.htm file is expected in site directory C:\inetpub\wwwroot\.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

6.0.29

Anything else?

.NET SDK:
Version: 8.0.204
Commit: c338c7548c
Workload version: 8.0.200-manifests.9f663350

Runtime Environment:
OS Name: Windows
OS Version: 10.0.17763
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\8.0.204\

.NET workloads installed:
There are no installed workloads to display.

Host:
Version: 8.0.4
Architecture: x64
Commit: 2d7eea2529

.NET SDKs installed:
6.0.421 [C:\Program Files\dotnet\sdk]
8.0.204 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

[BrennanConroy]

We found a few cases where app shutdown could be triggered incorrectly including false notifications of app_offline.htm. See #57735

Since this kind of issue is hard to reproduce and diagnose we can't say for sure it'll fix the problem you're having unless you test it out and don't see phantom app_offline notifications anymore.

[BrennanConroy]

A fix should be available in the 10.0 daily builds accessible via https://github.com/dotnet/sdk/blob/main/documentation/package-table.md
Technically you shouldn't need to use .NET 10 if you don't want to, just need to copy the aspnetcorev2_inprocess.dll into your published app (if self-contained) or into .dotnet/shared/Microsoft.AspNetCore.App/{version}/

@twinmind if possible would love if you could try it out!

[BrennanConroy]

I'm going to close. I hope this is fixed in 10.0 by the change I linked above.