Chunk Metadata

[mrchantey]

Currently chunks are split into an Iterator<str> without context for what part of the document they came from. For instance markdown subheadings dont have context for their parent heading.

Instead of returning raw strings we could instead provide some more context:

struct Chunk<'text>{
  /// information for which part of a document hierarchy the chunk belongs to.
  /// This will differ depending on the splitter.
  breadcrumbs: Vec<&'text str>,
  /// The raw content of the split text
  content: &'text str
}

Related to #116, it would also be valuable for code chunks. I haven't looked into tree-sitter but we could at least do this for rust files using syn.

As an alternative maybe Chunk could be a trait, and each splitter has its own metadata types.

trait Chunk<'text>{
  fn content(&self)-> &'text str;
}

[benbrandt]

@mrchantey thanks for the info. syn will likely be way too heavyweight for most folks, but tree-sitter queries should help here for sure.

There are also other variants already besides just str, with both byte and char offset variants, so this is some pattern for this. But yes, it would be nice to have this be a per-splitter decision since there are differing needs.

And this should likely be opt-in since there will be some performance cost, so we should only calculate these if requested to not slowdown use cases that don't need the extra information.