Mutable Content Store for On-Demand Routes

[dmgawel]

Body

Summary

Extend Astro's Content Collections with mutable storage capabilities for on-demand routes, allowing content to be synced at runtime without requiring full site rebuilds.

Background & Motivation

Astro's Content Collections provide an excellent developer experience for managing content, but currently have limitations around content syncing. The content store is immutable after build time, requiring full site rebuilds for any content changes.

Users building more dynamic sites face a dilemma: use Content Collections and sacrifice content freshness, or bypass collections entirely and implement custom data fetching logic. Live content loaders could be a solution for highly-dynamic pages, like news sites, but there is still a gap for content that changes occasionally but not on every request.

Many websites need a middle ground - content that updates periodically or in response to webhooks (post saved, etc.), without fetching from the source on every page load. For example, help centers, documentation sites, blogs, typically change throughout the day but don't require real-time data for every visitor. This gap becomes particularly problematic when integrating with external APIs that have rate limits.

A mutable content store would provide the best of both worlds: the type-safety and developer experience of Content Collections with the ability to update content without rebuilds, while protecting against API rate limits or slow APIs through efficient data reuse.

Goals

• Runtime content updates via webhooks or other mechanisms without requiring full site rebuilds
• Persistent content storage for on-demand rendered pages that survives across serverless invocations
• Adapter-specific optimization to use the most appropriate storage for each deployment platform

"Non-goals":

• Updating pre-rendered routes during runtime. This would still require a rebuild.

Example

Mutable store would be provided by adapters:

import { defineConfig } from 'astro/config';
import cloudflare from '@astrojs/cloudflare';

export default defineConfig({
  output: 'server',
  adapter: cloudflare(),
});

It could reuse the work already done for Sessions.

During initialization, data store snapshot would be fetched from a configured storage (e.g. Cloudflare KV) instead of a JSON file, with a fallback to a standard data store.

It would be possible to re-sync selected collections and write the resulting snapshot to the mutable store, so it's later used in on-demand routes.

When it comes to invoking re-sync, I'm thinking maybe via webhook:

import { refreshContent } from 'astro:content';

export async function POST() {
  // Trigger refresh
  await refreshContent('articles');
  
  return new Response('Content updated', { status: 200 });
}

or maybe periodically:

import { defineCollection } from 'astro:content';
import { zendeskLoader } from './loaders/zendesk';

const articles = defineCollection({
  loader: zendeskLoader({
    apiKey: process.env.ZENDESK_API_KEY
  }),
  // Optional sync configuration
  sync: {
    interval: 3600
  },
});

export const collections = { articles };

[delucis]

This feels related to the "Live Content Loaders" proposal which we recently moved to Stage 2 of our RFC process: #1151

It might make sense to consolidate proposals and leave any feedback you have in that issue.

[dmgawel]

I appreciate your suggestion, I can see the connection. I already left my feedback to "Live Content Loaders" few days ago when it was in Stage 1: #1137 (comment)

However, I believe these are actually distinct features addressing different problems, though they could certainly complement each other.

Live Content Loaders seem primarily focused on standardizing runtime data fetching with a familiar API, essentially providing a more elegant alternative to direct fetch() calls in components, which allows reusing loaders more easily. This is perfect for dynamic data that needs to be fresh on every request.

In contrast, the Mutable Content Store proposal addresses a different need: persisting content between invocations and providing update mechanisms for content that changes occasionally but is accessed frequently. It builds up on the powerful and resilient data snapshot feature Astro has. The focus is on reducing redundant API calls and preventing rate limit issues when many pages need overlapping data.

I think both features would be valuable for different use cases. A documentation site with hourly updates might benefit more from a mutable store, while a real-time dashboard would leverage live loaders.

Of course, I'm happy to continue the discussion and explore how these ideas might intersect or even merge if that makes sense for Astro's architecture. I'll leave a note about this proposal in the Stage 2 issue you've linked.

[florian-lefebvre]

Matt said he plans to make it integrate with route caching (#1140). I think that may address this proposal