@shortishly this is a fascinating project! Thank you for creating it.

I was curious how the schema registry worked related to schema evolution so I decided to ask copilot right here on GitHub. So if this is wrong feel free to disregard 😄

Summary

If I have this correct, currently, the schema registry caches schemas in memory the first time a topic receives a message. If the schema file in S3/local directory is updated (e.g., for schema evolution like adding a field), the broker continues to use the old cached schema and does not pick up changes until it is restarted.

Feature Request

• Add support for live schema reloading so that when a schema file is updated in the backing store (S3 or local directory), the broker picks up the new schema without requiring a restart.

• Possible approaches could include:

  • Periodic polling of schema files for changes.
  • Invalidation of the in-memory cache on a configurable interval.
  • An admin API or CLI trigger to reload schemas for a topic.

There might need to be some validation mechanism to ensure that schema evolution is handled safely (e.g., backward/forward compatibility checks, error reporting if a new schema is incompatible with in-flight messages).

What do you think?