Configuration Format Overhaul

[miniscruff]

Currently there is a bit of inconsistency in whats supported as well as a few overloaded terms, like header and format, that I really want to clean up. Additionally with the addition of projects and monorepo support, the configuration file has gotten quite big.

It was always part of the design of changie to support a wide range of projects, formats, inputs, outputs and so on, so this is not unexpected really. But, I think we can do better on making the config format more clear.

Some notes so far:

• Use Markdown link references #602 would require changelog level footers, it would also make sense to provide the header equivalent.
• Leveraging changie for ADRs (architectural decision records) #600 has some alternative uses that may be possible to support as well
• there has been a lot of suggestions or issues related to dependabot/renovate bot and how changie may or may not be able to include these updates automatically, I am not quite sold on this but something to consider.

One thing to note is that, any existing functionality will remain as is, really what I want to do is clean up some wording and organization of the config file.

To give an example there are 3 config values that look like they are editing the same thing, but in practice are not really.

1. versionFooterPath
2. versionFormat
3. versionHeaderPath

Version footer path and header path are file paths for per version headers and footers, as in the generated release notes file. While the version format is the line in that file that relates to the version.

This discussion will be my note taking space until it feels more stable, for now know that existing configs will work and remain working until, if ever, a changie v2 is required. I will definitely find a way to make sure renamed/moved configs are properly documented as well.

Design

Changie works in multiple steps:

• Prompt users for an update, producing a changie fragment
• Batching unreleased changie fragments into a brand new release note
• Merging release notes into a changelog

Changie aims to allow you to configure every aspect of all steps.
Additionally, for configurations that are similar/shared between both steps
we try and be consistent.
For example: header and footer configs match between release note files and changelog files except for the data that is passed to the templating.

Shared types

newline_type:
    before: 0 # amount of newlines to add before writing text
    after: 1 # amount of newlines to add after writing text

line_prompt_type:
    key: key of our prompt output used in file/fragment templates
    label: display label when rendering prompts ( defaults to key if empty )
    line format: line format of our component ( defaults to line format of all components )
        ??? can projects have line formats? What would that do?

file_writer_type:
( custom type for what data to pass to template cache, we need to document )
    file path: file path + template cache to generate text
    line format: string + template cache to generate text
    newlines: see newline_type

Full Config

rootDir: .changes ( from ChangesDir )
env prefix: CHANGIE_ ( unchanged )
fragment:
    dir: directory for unrelease fragments relative to root dir ( from UnreleasedDir )
    file format: template format for generating the file name of fragments ( from FragmentFileFormat )
    posts: post processing ( from root posts )
    prompts: ( from Customs )
project:
    version separator: ( from project version separator )
    options: ( from Projects )
        - key: key of our component used in file/fragment outputs
          label: output label when selecting in prompts ( defaults to key if empty )
          ( maybe inline file_writer_type if projects have a use for line_prompt_type )
          replacements: ( unchanged )
          changelog: ( matches the root changelog config and will merge the root with each project )
component:
    line format: output format of our component line, can be empty
    newlines: see newline_type
    options:
    - line_prompt_type ( inline these values )
      post: post process options ( just like kind config, ran before kinds if both configured )
kind:
    line format: output format of our kind line, required
    newlines: see type
    options:
        - line_prompt_type ( inline these values )
          change format: per kind option of the change format ( takes precedence over default change line format )
          post: post process options (unchanged kind config)
          auto level: unchanged
          skip: ( extra level needed? )
            body: unchanged
            default prompts: unchanged
            default post: unchanged
          ??? prompts: ( from additional choices )
          ??? should this be wrapped in a prompts and have skip global under it?
change: ( renamed from body? )
    line_prompt_type ( inline these values ) ( default to body )
    min_length: min length validation
    max_length: max length validation
    skip: true/false
    newlines: see type
releaseNotes:
    extension: File extension for release notes ( from VersionExt )
    version: see file_writer_type
    header: see file_writer_type
    footer: see file_writer_type
    newlines: see newline_type
changelog: ( shared type between root and project.options[*].changelog )
    output: output path to changelog file ( from ChangelogPath )
    header: see file_writer_type
    footer: see file_writer_type
    newlines: see newline_type
    replacements: moved to better support projects

Release note file format:

<header file path>
<header line format>

<version line format>
<for each component if enabled>
    <for each kind if enabled>
        <for each change>
            <change line format>
        <end change>
    <end kind>
<end component>

<footer line format ( FooterFormat )>
<footer file path>

Changelog format:

<header file path>
<header line format>

<for each release note>
    <release note file>
<end release note>

<footer line format>
<footer file path>

Changie's changie config under new format

rootDir: .changes
fragment:
  dir: unreleased
  prompts:
    - key: Issue
      type: int
      minInt: 1
changelog:
  output: CHANGELOG.md
  newlines:
  header:
    filePath: header.tpl.md
    newlines:
      after: 1
  replacements:
    - path: 'docs/config.toml'
      find: "  version = '.*"
      replace: "  version = '{{.Version}}'"
    - path: 'docs/static/version.json'
      find: '    "latest": ".*"'
      replace: '    "latest": "{{.Version}}"'
    - path: package.json
      find: '  "version": ".*",'
      replace: '  "version": "{{.VersionNoPrefix}}",'
releaseNote:
  extension: md
  version:
    lineFormat: '## {{.Version}} on {{.Time.Format "2006-01-02"}}'
kind:
  lineFormat: '### {{.Kind}}'
  newlines:
    before: 1
    after: 1
  options:
    - label: Added
      auto: minor
    - label: Changed
      auto: major
    - label: Deprecated
      auto: minor
    - label: Removed
      auto: major
    - label: Fixed
      auto: patch
    - label: Security
      auto: patch
change:
  lineFormat: '* [#{{.Custom.Issue}}](https://github.com/miniscruff/changie/issues/{{.Custom.Issue}}) {{.Body}}'
  newlines:
    after: 1

Current Notes:

1. The use of skip vs enabled. Skip is like a negative not a positive value of what we are doing
2. Migration plan: Do we mark all existing config options as optional pointers without defaults?

[sheldonhull]

I like the idea so far. Seems like you are trying to unify certain things to have the same flexibility as right now you have some assumed defaults and this tries to make it more configurable.

This goes beyond what I'd thought about but seems promising.

[miniscruff]

Yup that is the goal, if anyone wants me to show that your config might look like in the new format that might be an interesting exercise.

[miniscruff]

Ok after a long time sitting here I think its finally time to open this up to final design and starting to work on it. So I am going to recheck the configs and build out the expected config format but with:

1. JSON Schema support as that was added since this discussion was started
2. Version keys for v1,v2 and so on
3. Keeping the old v1 documented and schema hosted
4. What the migration script could look like

[Blacksmoke16]

Another use case I'd like to pitch is being able to have different formats for the intermediary changelog file that you get from changie batch and the resulting merged changelog from changie merge. The reason being if you're using the intermediary changelog files for GH release notes then it makes sense to use contributor's GH username @firstlast as it provides some nice integration with GH's UI. However for the CHANGELOG.md file itself you may want to use their plain name First Last so that it's not coupled to GH.

[Blacksmoke16]

I have been thinking about this more and did some testing. Unfortunately doing something like [George Dietrich](@blacksmoke16) or using a link to their GH profile doesn't result in the same level of integration as compared to using @blacksmoke16 directly.

I'm also unable to change the intermediary file because that would alter the downstream changelogs after running merge. But more importantly I lose the context changie had after running batch since it deletes the change files of course.

I'll have to see if there'd be a good way to just include both in some way as you say as that'd likely be the easiest solution. Having some way to persist the metadata related to a batch could be a neat extension point tho. E.g. something like if there was a way to dump (optionally?) a vx.x.x.json file along with the intermediary .md file that consists of a JSON representation of the batch. I don't really like the idea of having double the amount of files hanging around tho. A single file key'ed by version might be sufficient too to keep things a bit more tidy.

Tho after typing this out I could prob get away with doing like <!-- {{.Custom.Username}} --> at the end of the change format. Would make the raw markdown code a bit messier but ultimately be hidden when rendered. Then from here could have some extra processing steps to sed it out. 🤷 I'll have to dwell on it more.

[miniscruff]

hm, interesting, its possible to keep the unreleased changelogs and even move them via --keep or --move PATH options in batch https://changie.dev/cli/changie_batch/#options . But this was mostly created to assist with prerelease notes being scrubbed when the final version came out. So you would move all prerelease notes to a "stable pending" type directory for later batching.

Maybe if you share a final looking changelog we could reverse backwards to find out how to get there. My initial thoughts of just having an optional Author input for your github account is all I would do haha.

[Blacksmoke16]

My initial thoughts of just having an optional Author input for your github account is all I would do haha.

This is what I do yea. I have two custom properties. One for FullName and another for Username. E.g. George Dietrich and blacksmoke16. Changie handles changelog generation just fine using FullName, the problem stems from my desire to use the intermediary version file as release notes on a GH release, but use the user's username instead of their fullname.

E.g. the changelog file itself should be like - The change body ([#123]) (George Dietrich) but the GH release should be - The change body ([#123]) (@blacksmoke16). This is where the trouble is because I can't actually change the intermediary file without changing the end CHANGELOG.md file, but still need a way to know what username maps to which change even after the batch.

Honestly so far the comment at the end approach seems to be a decent enough solution. But happy to chat more about the use case if you want.

EDIT: Had ChatGPT help come up with a one-liner:

perl -0777 -pe 's{\([^)]+\)\s*<!--\s*([^>]*\S)\s*-->}{ "(" . join(", ", map { s/^\s+|\s+$//g; s/^@?/@/; $_ } grep { length } split(/\s*,\s*/, $1)) . ")" }ge' -i ./path/to/changelog

This would turn:

- Change number 1 ([#123]) (George Dietrich) <!-- blacksmoke16 -->
- Change number 2 ([#456], [#493]) (Jon Snow, George Dietrich) <!-- @jsnow, @blacksmoke16 -->

Into:

- Change number 1 ([#123]) (@blacksmoke16)
- Change number 2 ([#456], [#493]) (@jsnow, @blacksmoke16)

Adds @ if missing, trims spaces, and enforces a single space after commas. Seems like it would work pretty well as part of my CD pipeline.

[miniscruff]

ooh, I like the idea. yq is an option for processing the unreleased files before batching too.

[Blacksmoke16]

To close the loop on this, it turned out quite well!

https://github.com/athena-framework/athena/blob/afb13c8fe5c1c92d78d36defda0ce109e9c06cbe/.github/workflows/sync.yml#L117-L131

The comment stores the metadata (hidden when viewing the rendered version of either merged or batched changelog file). Then does some light transformation, replace authors with username in comment, then pipe it into gh release create. Good stuff!