Should we update first octoherd-scripts with latest features?

[oscard0m]

I was checking already existing octoherd-scripts and most of them don't have test.yml. I guess other "features" are missing which new ones will have if we use create-octoherd-script.

Should we plan a task to migrate already existing ones with this script?

A second question comes to my mind, should we think about potential new features we would like to propagate to already existing octoherd-scripts? Maybe the GitHub App we are installing or that would be too intrusive?

[gr2m]

Yes to both questions.

The problem of keeping repositories up-to-date based on a template repository they were created with, or a setup script such as npm init octoherd-script is quite common. And I think that @octoherd could be the perfect project to make that possible, and we could implement it ourselves as we have that exact need for https://github.com/octokit/create-octokit-project.js/

There are several challenges we face

1. How do we know the set of repositories that needs to be updated?
   • In the case of @octoherd we could utilize the https://github.com/apps/octoherd/ app.
   • In fact, we could in general rely on GitHub app installations.
   • Another common case is all repositories in a GitHub organization, but we could consider this out of scope for now. Also it would be easily covered by creating a new GitHub app and use that for keeping track of repositories
2. Say we do have the set of repositories that needs to be updated, what does the process look like of creating and running a script? I think I already mentioned this somewhere else, but I like the model of migrations from Ruby on Rails. The problem it solves is to keep a database schema and its data up-to-date with the latest version of the Rails app. When a database change is required, a migration needs to be implemented. When the change gets merged, then all deployment targets need to run the migration script before restarting the app. The same is true for local development environments of all developers.
   • I could imagine we could create a folder "migrations". And each time we add a new feature, change repository settings, or change common repository files, an @octoherd script needs to be added
   • In its simplest form, a migration script could create an issue explaining what changed and prompting the respective repository owners to upgrade their script repository. It's up to them to act on it
   • In its ideal form, we would create pull request that the script owner can directly merge
   • In terms of process / development environment, I think we need
     • testing: We don't yet have a good testing story for @octoherd scripts. But if we create migration scripts that might potentially be run against thousands of repositories, than we better are confident in what we are doing.
     • previews: before creating any mutations such as creating an issue/pull request, or updating repository settings, we should be able to have a "dry" run, which could be a summary report of all the repositories the script was run on, how many repositories where changed, which repositories were not changed, and what the changes would be
     • staging: once we are confident, we should just run the script on all repositories, but instead should run it in a safe staging environment first. We could use a separate GitHub App for that https://github.com/apps/octoherd-staging, which would be installed on a few test scripts. That way we can run the migration script against real repositories and see the real outcome. We could also create a staging GitHub organization and copy/clone selected scripts into that organization. That way we can selectively run the script on some scripts. We could even randomize the script repositories that we want to copy/clone as part of the staging process
3. How do we keep state of which repositories we need to run the migration script on. We probably wouldn't want to run a migration on a repository that is younger than the migration script. A migration script might have failed on some but not all repositories, we might want to fix the migration script and then run it again, but only on the repositories where the migration script failed.
4. Lastly, migration scripts might fail in the middle of a repository migration. Ideally, each script could revert all its changes if one of them failed, but that's hard and not always possible. If things fail, repositories might end up in an inconsistent state, which might need to be resolved manually. This is just something we have to keep in mind. Good logs will help with that.

Anyway, these are my thoughts on the subject :D Does that all make sense?