Unexpected high number of image optimizations

[canta2899]

Summary

Hi! I have a use case where I need to display a gallery of high-resolution images. To reduce bandwidth usage, I'm using image optimization to serve smaller versions with less quality when rendering the gallery. However, the images are private and must be fetched via pre-signed URLs.

To handle this, I implemented a middleware that intercepts requests to paths like /presigned/:path*. It extracts the key from the URL (e.g., /presigned/a/b/c becomes a/b/c), generates a pre-signed URL, and redirects to it. This allows me to keep the src attribute of the image fixed.

The same middleware also handles requests to /_next/image and ensures only authenticated users can access optimized images by validating the auth cookie.

This setup works: in the browser dev tools, I can see that image requests go to /_next/image?url=/presigned/key, and the optimized versions are correctly served on repeat visits (most of the times they are actually served from memory/disk cache). However, according to Vercel’s dashboard, the number of image optimizations is surprisingly high. For instance, with just 10 images in the gallery, I see over 150 transformations in 12 hours. I expected the number of optimizations to roughly match the number of images since they need to be optimized just once.

I've been wondering if that's considered "normal" behavior and if so why, or if the presigned url redirection may be causing unexpected cache misses. I'm leaving the code of my Image component and the code of my middleware in case it helps.

Thanks!

Additional information

// The image component


<Image
  src={`/presigned/${image.imageKey}`}
  alt={image.name}
  loading="lazy"
  placeholder="empty"
  quality={50}
  width={400}
  height={400}
  className="object-cover w-full h-full"
  draggable={false}
  onError={() => {
    statusRef.current = "error";
  }}
/>


// The middleware

export const middleware = auth(async (req) => {
  const auth = req.auth;
  const { pathname, searchParams } = req.nextUrl;

  if (pathname.startsWith("/presigned")) {
    if (!auth) return new NextResponse("Unauthorized", { status: 401 });

    const key = extractKey(pathname); // returns the key of the object

    const url = await getPresignedUrl(key);

    return NextResponse.rewrite(url);
  }

  if (
    pathname.startsWith("/_next/image") &&
    searchParams.get("url")?.startsWith("/presigned") &&
    !auth
  ) {
    return new NextResponse("Unauthorized");
  }

  return NextResponse.next();
});

export const config = {
  matcher: ["/_next/image", "/presigned/:path*"],
};

[Yasir-Rafique]

You’re seeing a high number of image optimizations because each time someone requests a private image, your backend creates a new pre-signed URL (which includes a unique signature and expiry). Even though it’s always the same image, the URL is slightly different every time due to the way pre-signed URLs work.

Vercel’s image optimizer uses the image URL as the “key” for its cache. So, if the URL changes, even a little bit it thinks it’s a brand new image, and will re-optimize it, instead of serving the cached optimized version. That’s why you’re seeing way more optimizations than the number of actual images.

Is this normal?
Yes, this is a common issue when using pre-signed URLs with image optimization services. The cache simply can’t work if the URL isn’t stable.

How do we fix it?
The best way is to make sure the optimizer always sees the same, stable URL for each image (like /presigned/my-image.jpg).
Instead of redirecting to a new pre-signed S3 URL every time, have your middleware fetch the image from S3 using the pre-signed URL in the backend, and then send the image data directly to the browser (this is called proxying). That way, from the optimizer’s and browser’s point of view, the URL never changes, so caching works as expected.

[canta2899]

As far as I know and according to the docs, NextResponse.rewrite() proxies the given URL while preserving the original URL. I was using rewrite exactly with the purpose of keeping the URL stable to allow for image optimization.

However, I tried your solution in the middleware but I started getting several errors due to middleware invocation failed, so I tried to do the same thing via an API route that fetches the image from S3 and streams it back to the client with appropriate headers. This works and images are being optimized and I see cache HITs on subsequent requests to the same image.

The main problem though is that I cannot ensure authentication for the api handler. If I do, the Image Optimization service fails with error 401 'cause I guess the optimization runs independently and doesn't have any session token despite it's being triggered by the authenticated user. I tried to check the auth token at handler and middleware level but in both cases it fails. So I end up either having a public endpoint that allows everyone to fetch images or having a restricted endpoint that also restricts access to the image optimization itself 🤣

This is the API route I implemented

import { getFromOrg } from "@repo/s3";
import { NextRequest, NextResponse } from "next/server";

export const GET = async (
  req: NextRequest,
  { params }: { params: Promise<{ p: string[] }> }
) => {
  const { p } = await params;
  const [orgId, ...keyPath] = p;
  
  // const session = await auth();  <-- this returns null when the user agent is 'vercel-image-optimization'

  if (!orgId || !keyPath)
    return new NextResponse("Bad request", { status: 400 });

  const key = keyPath.join("/");

  const response = await getFromOrg({ key, orgId });

  if (!response.Body) return new NextResponse("Not found", { status: 404 });

  const headers = new Headers();

  headers.set("Content-Type", response.ContentType || "image/png");

  if (response.ContentLength)
    headers.set("Content-Length", response.ContentLength.toString());

  if (response.ContentDisposition)
    headers.set("Content-Disposition", response.ContentDisposition);

  if (response.ContentEncoding)
    headers.set("Content-Encoding", response.ContentEncoding);

  if (response.ETag) headers.set("ETag", response.ETag);

  headers.set("Cache-Control", "public, max-age=2678400, immutable");

  const stream = response.Body.transformToWebStream();

  return new NextResponse(stream, { headers });
};

I don't know if there's anything I'm doing wrong but I think I've tried every possible solution and nothing has been working so far, the only option I have left is using cloudfront with public/private key pair to restrict access, create a custom loader and take vercel image optimization completely out of the equation.

[Yasir-Rafique]

Hey, you’ve done a fantastic job digging into this! This is one of those tricky corners of the Next.js + Vercel ecosystem where things can get a little frustrating, especially when mixing authentication, private images, and the built-in image optimizer.

The optimizer essentially acts like a new client. It fetches your image using only the info in the 'src' URL with no auth. This is by design for performance/scalability, but it does make protecting private images tough.

You would need to optimize images before upload, or use something like CloudFront (you mentioned) or a dedicated image proxy with signed URLs

There is no “perfect” way to have both secure, authenticated private images and use Vercel’s built-in image optimization. The way Vercel works, you generally have to pick one: either Private images or Optimized images

If strict privacy is a must, most teams use CloudFront or another CDN with signed URLs/cookies and take image optimization outside of Vercel’s pipeline (custom loader, pre-processing, etc.).

[canta2899]

Alright! I’ll spin up a CloudFront distribution with restricted access using private keys and hook up a few Lambdas for image optimization. I wish I could’ve just used Vercel to keep things simple, but hey, we do what we gotta do 🤣

Thanks a ton for your help! Really appreciate it.

[Yasir-Rafique]

No worries. I am glad I helped. 😎