Splitting content snapshot into chunks: enable deployment to platforms with file size limits (Cloudflare)

[dmgawel]

Body

Summary

Introduce chunked snapshot generation for content collections to prevent deployment failures when content collections snapshot files exceed platform file size limits (specifically the 25MB hard limit on Cloudflare Workers/Pages).

Background & Motivation

The current Content Layer API in Astro generates a single snapshot file containing all content collection data. While this approach works well for smaller sites, it creates deployment issues for content-based sites when deploying to platforms with file size restrictions:

• Cloudflare Workers/Pages enforces a hard 25MB per-file limit due to underlying KV infrastructure constraints
• Users importing medium datasets are hitting this limit
• The limit cannot be increased through paid plans - it's a technical constraint

This prevents successful deployment of otherwise valid Astro sites to a popular deployment platform.

Goals

Introduce configurable chunking for content collections snapshots with the following features:

• Build-Time Chunking: Generate multiple smaller snapshot files instead of one large file when content exceeds a configurable threshold
• Runtime Reassembly: Automatically detect and load chunked snapshots during collection queries - no changes required to existing getCollection() and getEntry() calls. Lazy loading of chunks only when needed to maintain performance and memory usage.
• Configuration: Allow configuring chunking and max chunk size.

Splitting strategy to be discussed (per collection? adaptive?).

An accompanying solution could be exploring snapshot file compression with a method like CBOR.

Example

// astro.config.js
export default defineConfig({
  adapter: cloudflare(),
  contentLayer: {
    chunking: {
      enabled: true,
      maxChunkSize: '20MB'
    }
  }
});

Build output:

Building content collections...
✓ blog collection: 200 entries → 2 chunks (19.8MB, 15.2MB)
✓ docs collection: 50 entries → 1 chunk (8.4MB)
✓ Generated 3 chunk files + manifest
⚠️  Large content detected - chunking enabled for Cloudflare deployment

[delucis]

Given most content bundles should be relatively small, it might even make sense to chunk by default at some reasonable size as a first step and then see if there’s still a need to make it configurable.

Do you have an example of the kinds and quantity of content that are passing 25 MB? As an example, the bundle for astro.build’s content is 3.2 MB, which includes ~150 Markdown/MDX blog posts and nearly 3,000 small data files.

[dmgawel]

Given most content bundles should be relatively small, it might even make sense to chunk by default at some reasonable size as a first step and then see if there’s still a need to make it configurable.

Agree 👍

Do you have an example of the kinds and quantity of content that are passing 25 MB? As an example, the bundle for astro.build’s content is 3.2 MB, which includes ~150 Markdown/MDX blog posts and nearly 3,000 small data files.

Real-world scenario we've hit: ~200 Zendesk Help Center articles (docs ref) translated to 19 languages.

[delucis]

Thanks for the example! If there’s no user-facing config, we could probably even add this behaviour without the full RFC process as it would just be an internal implementation detail. I’ll see what other maintainers think.

Would you be interested in implementing this potentially?