Rework flatbread config into a list of plugins

[odama626]

This will allow for multiple sources and allow for more advanced plugins when they all have access to a more streamlined interface api

Open Questions:

• when creating a new record, how do we determine which transformer should be used
• would it be better to have a separate "pipeline" for each content type and allow composition?

//flatbread.config.js

function defineConfig(t: any) {
}

// NOTES:
// - Plugin order matters!

export default defineConfig({

  plugins: [
    sourceFilesystem({
      content: {
        Author: {
          path: 'content/markdown/authors',
          refs: {
            friend: 'author'
          }
        },
        Post: {
          path: 'content/markdown/posts',
          refs: {
            authors: 'Author'
          }
        }
      }
      // default value
      // contentType determines what can be done with the output
      outputContentType: 'vfile' 
    }),
    sourceGitRepo({
      repo: 'git:asdf'
      key: ENV_GIT_KEY,
      content: {
        Author: {
          path: //
          ref:
        }
      }
      // default value
      outputContentType: 'vfile'
    }),
    sourceIPFS({

    }),
    sourcePostgres({
      connection: 'psql://asdfasdf',
      content: {
        Cat: {
          table: 'cat',
        }
      }
      contentType: 'json' // default types
    }),
    transformMarkdown({
      // maybe there should be a notion of content-type instead? 
      // where source-github / source-filesystem could create content-type: vfile?
      contentType: ['vfile'] // default types
      outputContentType: 'json'
    }),
    transformYaml({
      contentType: ['vfile'] // default types
      outputContentType: 'json'
    }),
    fieldOverrides({
      content: {
        Author: [
          { field: 'deeply.nested', type: 'String', resolve: () => {}}
        ]
      }
    }),
    svimgFields({
      defaultConfig: {
        inputDir: 'static/authorImages',
        outputDir: 'static/g',
        srcGenerator: path => '/g/' + path
      }
      content: {
        // can supply svimg config to override defaults per field
        Author: [ { field: 'deeply.nested'}],

      }
    })
  ]
})

// Plugin lifecycle

interface FlatbreadFetchActions {
  
}

interface  FlatbreadContext {
  addRecord(
    collection: LoadedCollectionEntry,
    record: EntryNode,
    context?: Context
  ): void;
  // all the data that is needed for each function 
}

interface FlatbreadPlugin<PluginConfig, PluginContext> {
  constructor(config: PluginConfig);
  initialize?();

  // provide content
  fetch?(flatbread: FlatbreadContext, ctx: PluginContext)

  // transform content
  parse?(flatbread: FlatbreadContext, ctx: PluginContext);

  // attach resolvers to ObjectComposer
  resolve?(flatbread: FlatbreadContext, ctx: PluginContext)

  // reverse transform -- time to mutate
  serialize?(flatbread: FlatbreadContext, ctx: PluginContext)

  // commit content to storage
  put?(flatbread: FlatbreadContext, ctx: PluginContext)
}

[tonyketcham]

Nice one setting up this discussion. I'll paste in my thoughts from our chat:

Transformers are logically tethered pretty close to sources, as the source output can be unusable to Flatbread until it’s parsed and transformed to JSON. I like the idea of being direct about how a source gets transformed. To that end, we can simplify the plugin API by making transformation the responsibility of the source plugin.

With multiple sources, a user could then easily reuse the same transformer config or break away from it per-call of the transformer on different sources.

// NOTES:
// - Plugin order matters!

export default defineConfig({
  plugins: [
    sourceFilesystem({
      transformers: [transformYaml(), transformMarkdown()],
      content: {
        Author: {
          path: 'content/yaml/authors',
          refs: {
            friend: 'Author',
          },
        },
        Post: {
          path: 'content/markdown/posts',
          refs: {
            authors: 'Author',
          },
        },
      },
    }),
    sourcePostgres({
      connection: 'psql://asdfasdf',
      content: {
        Cat: {
          table: 'cat',
        },
      },
    }),
    fieldOverrides({
      content: {
        Author: [
          { field: 'deeply.nested', type: 'String', resolve: () => {} },
        ],
      },
    }),
    svimgFields({
      defaultConfig: {
        inputDir: 'static/authorImages',
        outputDir: 'static/g',
        srcGenerator: (path) => '/g/' + path,
      },
      content: {
        // can supply svimg config to override defaults per field
        Author: [{ field: 'deeply.nested' }],
      },
    }),
  ],
});

[odama626]

I like this idea a lot, although after thinking about it for a couple days, I think I might prefer keeping the transformers out of the sources.

with this config setup we would be requiring all sources to implement a lot of functionality that they wouldn't otherwise need to handle