Static files / compression / caching question

[DanielLoth]

Hi everyone,

I've been curious about how, if at all, it would be possible to use out-of-the-box tooling and middleware to handle immutable static files.

For those aspnetcore3.1 applications running in Dockerised environments, as immutable infrastructure, there is likely no expectation of ever changing the static content within the wwwroot folder after building and deploying the Docker image artifacts.

I've been experimenting with the following pipeline configuration:

            app.Map("/css/normalize.css", app => {
                app.Use(async (context, next) => {
                    // Rudimentary URL rewrite
                    context.Request.Path = context.Request.PathBase;
                    context.Request.PathBase = PathString.Empty;

                    await next.Invoke();
                });

                app.UseResponseCaching();

                app.Use(async (context, next) => {
                    await next.Invoke(); // Debug breakpoint to observe whether request terminates in response caching middleware
                });

                app.UseResponseCompression();

                app.Use(async (context, next) => {
                    await next.Invoke(); // Debug breakpoint
                });

                app.UseStaticFiles(new StaticFileOptions() {
                    FileProvider = new PhysicalFileProvider(env.WebRootPath)
                });

                app.Use(async (context, next) => {
                    await next.Invoke(); // Debug breakpoint - never hit, as a static file exists  - /css/normalize.css
                });

                app.Run(async context => {
                    await context.Response.WriteAsync("File not found."); // Never hit when file exists
                });
            });

This arrangement is not quite what I want. The caching middleware might be misunderstood on my part, or insufficiently configured.

What I would like is the following:

1. On the very first request, read the file from disk. Compress with gzip / brotli as appropriate. Cache the resulting file using the response caching middleware for the duration of the runtime.
2. On a subsequent request, for the same file and accepting the same encoding, return the cached content without exercising the ResponseCompression or StaticFiles middleware again.
3. On a subsequent request, for the same file, but accepting a different encoding -> Defer to the StaticFiles and ResponseCompression middleware again, and cache this response.

So after three requests:

1. One asking for gzip
2. One asking for br
3. One that doesn't accept encoding

I would want it to never reach the StaticFiles middleware again for the duration of the runtime.

Is this configuration possible?

[Tratcher]

What version is this? We've fixed a few issues with this flow, but there's certainly more we could do.

[DanielLoth]

3.1.8.

[Tratcher]

For starters, StaticFiles doesn't set cache headers be default, or at least not the ones ResponseCaching is looking for. It does have a callback on the options that let's you set your own headers to enable caching. Have you gotten that far?

Also, I recall an issue with caching and compression where it would only cache one representation at a time, though I can't find the issue for that.

[DanielLoth]

I've worked it to this point, which to some extent answers my original question:

            app.UseStaticFiles(new StaticFileOptions() {
                ContentTypeProvider = GetContentTypeProvider(),
                OnPrepareResponse = ctx => {
                    var cacheTimeSeconds = (int) TimeSpan.FromDays(365).TotalSeconds;

                    ctx.Context.Response.Headers[HeaderNames.CacheControl] =
                        $"public,max-age={cacheTimeSeconds},immutable";

                    // If, at this point, we knew we were serving a gzipped / brotli-encoded
                    // representation, pre-compressed and stored on the file system, we could
                    // do the following:
                    //var hasAcceptEncodingHeader =
                    //    ctx.Context.Request.GetTypedHeaders().AcceptEncoding?.Any() ?? false;

                    //if (hasAcceptEncodingHeader)
                    //{
                    //    ctx.Context.Response.Headers[HeaderNames.Vary] = new string[] { "Accept-Encoding" };
                    //}
                }
            });

Suppose I have main.css, main.css.gzip, and main.css.br on disk in wwwroot.

The OnPrepareResponse callback happens after a file has been located and made ready to serve.
At this point it's too late to say 'hey, check if there's a compressed representation of that file on disk and prefer to serve that instead'.

One solution might be to include some kind of Action<StaticFileLocatedContext> that is invoked when a candidate file exists, providing to adjust the file name from say main.css to main.css.gzip.

Another might be to add a convention behaviour-wise where, for a given file, you can configure the StaticFiles middleware to prefer a gzipped / brotli version if the appropriate Accept-Encoding header is present.

[Tratcher]

That's static caching which is a very different beast. At that point, you might as well compress them before publishing the site like blazor does. #19157 @javiercn might be able to explain how they serve the precompressed files.

[DanielLoth]

Yeah, a little different. Sorry, I was a little unclear in my original post.

I've got the BundlerMinifier tool in my application (https://github.com/madskristensen/BundlerMinifier/tree/master/src) which currently does gzip but not brotli.

So I've got a main.css and main.css.gz readily available.

I suspect I can modify my original code sample above so it selectively rewrites the requested file based on the Accept-Encoding header.
I'll give it a whirl in a few hours and, if working as intended, share the code for anyone else.