DOC: updating the notes on how to release

doc/source/developing/releasing.rst

@@ -10,38 +10,76 @@ from the version number used. Version numbers should follow the scheme
 
   These releases should contain only fixes for bugs discovered in
   earlier releases and should not contain new features or API changes. Bugfix
-  releases only increment the ``PATCH`` version number. Bugfix releases should
-  *not* be generated by merging from the ``main`` branch, instead bugfix pull
-  requests should be backported to a dedicated branch.
-  See :ref:`doing-a-bugfix-release`. Version ``3.2.2`` is a bugfix release.
+  releases only increment the ``PATCH`` version number. Bugfix releases are
+  generated from a dedicated backport branch that contains cherry-picked
+  commits for bug fixes (handled semi-automatically as pull requests are
+  merged), see :ref:`doing-a-bugfix-release` for more details. Version
+  ``3.2.2`` is a bugfix release.
 
 * Minor releases
 
-  These releases happen when new features are deemed ready to be merged into the
-  ``stable`` branch and should not happen on a regular schedule. Minor releases
-  can also include fixes for bugs if the fix is determined to be too invasive
-  for a bugfix release. Minor releases should *not* include
-  backwards-incompatible changes and should not change APIs.  If an API change
+  Minor releases include new features and fixes for bugs if the fix is
+  determined to be too invasive for a bugfix release. Minor releases
+  should *not* include backwards-incompatible changes and should not change APIs.  If an API change
   is deemed to be necessary, the old API should continue to function but might
-  trigger deprecation warnings. Minor releases should happen by merging the
-  ``main`` branch into the ``stable`` branch. Minor releases should increment the
+  trigger deprecation warnings. Minor releases should happen by creating a new series
+  branch off of ``main`` branch. Minor releases should increment the
   ``MINOR`` version number and reset the ``PATCH`` version number to zero.
-  Version ``3.3.0`` is a minor release.
+  Version ``3.3.0`` is a minor release and has a corresponding series branch ``yt-3.3.x``.
+  After release, the new series branch becomes the backport branch for any bug fix releases
+  in the series.
 
 * Major releases
 
   These releases happen when the development community decides to make major
   backwards-incompatible changes intentionally. In principle a major version release could
   include arbitrary changes to the library. Major version releases should only
   happen after extensive discussion and vetting among the developer and user
-  community. Like minor releases, a major release should happen by merging the
-  ``main`` branch into the ``stable`` branch. Major releases should increment the
-  ``MAJOR`` version number and reset the ``MINOR`` and ``PATCH`` version numbers
-  to zero. If it ever happens, version ``4.0.0`` will be a major release.
+  community. Like minor releases, a major release should happen by creating
+  a new series branch off of the ``main`` branch from which to release. Major releases should
+  increment the ``MAJOR`` version number and reset the ``MINOR`` and ``PATCH``
+  version numbers to zero. Version ``4.0.0`` is a major release.
 
 The job of doing a release differs depending on the kind of release. Below, we
-describe the necessary steps for each kind of release in detail.
+describe the necessary steps for each kind of release in detail. Several of the
+following steps require that you have write privileges for the main yt GitHub
+repository (if you're reading this, you probably already do).
 
+.. note::
+
+  This documentation assumes that your local copy of the yt repository has the
+  main yt repository as the ``upstream`` remote (the default configuration when
+  forking with ``gh fork yt-project/yt``). You can double check with:
+  ``git remote -v``, which will display all the remote sources you have setup
+  as well as their addresses. If you are missing an ``upstream`` remote (for
+  example, after a standard ``git clone`` of your yt fork), you can add it
+  with ``git remote add upstream git@github.com:yt-project/yt.git``.
+
+.. _prepping-release-notes:
+
+Prepping Release Notes
+~~~~~~~~~~~~~~~~~~~~~~
+
+Before starting the release process, it's useful to create a draft of a GitHub release. Doing this
+at the start lets you share the upcoming release notes with other developers before the release
+actually happens. To create a new draft release, go to
+https://github.com/yt-project/yt/releases and click "Draft a new release". Use the version tag
+as the release title (e.g., ``yt-4.4.1``). For the target tag, enter the tag that you will use
+(see  :ref:`tagging-a-release-release`) and select the branch from which the release will be cut.
+
+Now it's time to generate some release notes. To get a nice starting point, try using the ``uv`` script
+`draft_yt_release_notes.py <https://gist.github.com/chrishavlin/248adea4296abb7bcdbaac952f304cf0>`_. You can
+run the script directly from the gist using ``uv`` and the url to the raw script (check the readme at the gist).
+This script will pull all the issues and pull requests that have been tagged to a specified GitHub milestone
+and do some initial categorizing to produce a decent draft for release notes. You can then create a draft
+release manually via the GitHub interface and copy in your draft notes, or if using ``gh``, you can do so
+from the command line with ``gh release create --draft --notes-file <file> --target <branch>``. The initial
+notes created by the ``draft_yt_release_notes.py`` will still need some manual attention: you should update
+the release summary text, add any highlights if desired, add PRs that are missing. At present, you'll also
+need to manually sort the frontend-specific changes by frontend to match previous release notes.
+
+When updating the draft via the GitHub interface, click "save" at any point to save a draft.
+Do NOT publish the draft yet.
 
 .. _doing-a-bugfix-release:
 
@@ -52,22 +90,30 @@ As described above, bugfix releases are regularly scheduled updates for minor
 releases to ensure fixes for bugs make their way out to users in a timely
 manner. Since bugfix releases should not include new features, we do not issue
 bugfix releases by simply merging from the development ``main`` branch into
-the ``stable`` branch. Instead, commits are cherry-picked from the ``main``
-branch to a backport branch, which is itself merged into ``stable`` when a release
-happens.
+the designated backport branch for the series. Instead, commits are cherry-picked
+from the ``main`` branch to a backport branch, and the backport branch is tagged
+with the release tags.
 
-Backport branches are named after the minor version then descend from, followed by
+Backport branches are named after the minor version they descend from, followed by
 an ``x``. For instance, ``yt-4.0.x`` is the backport branch for all releases in the 4.0 series.
+The backport branches are initially created during the release of a new minor or major
+version (see :ref:`doing-a-minor-or-major-release`).
 
 Backporting bugfixes can be done automatically using the `MeeseeksBox bot
-<https://meeseeksbox.github.io>`_.
+<https://meeseeksbox.github.io>`_ with a GitHub milestone linked to a backport branch.
+To set up a new milestone linked to a backport branch, click the New Milestone button
+on the `GitHub interface <https://github.com/yt-project/yt/milestones>`_` then name it
+with the version for the future release and a note within the description field:
+``on-merge: backport to <name of backport branch>``, for example:
+``on-merge: backport to yt-4.0.x``.
 
-This necessitates having a Github milestone dedicated to the release, configured
-with a comment in its description such as ``on-merge: backport to yt-4.0.x``.
 Then, every PR that was triaged into the milestone will be replicated as a
 backport PR by the bot when it's merged into main. Some backports are non-trivial and
 require human attention; if conflicts occur, the bot will provide detailed
-instructions to perfom the task manually.
+instructions to perfom the task manually. If you forget to assign a backport branch label
+before merging a PR, you can tag the bot in a comment on the merged PR to have it
+create a new backport PR for the already merged PR (see `here <https://github.com/scientific-python/MeeseeksDev>`_
+for a list of commands understood by the Meeseeks bot).
 
 In short, a manual backport consist of 4 steps
 
@@ -76,47 +122,65 @@ In short, a manual backport consist of 4 steps
 - cherry-picking the merge commit from the original PR with ``git cherry-pick -m1 <commit sha>``
 - opening a PR to the backport branch
 
+.. _doing-a-minor-or-major-release:
 
 Doing a Minor or Major Release
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-This is much simpler than a bugfix release. First, make sure that every
+This is simpler than a bugfix release. First, make sure that all
 deprecated features targeted for removal in the new release are removed from the
 ``main`` branch, ideally in a single PR. Such a PR can be issued at any point
-between the previous minor or major release and the new one. Then, all that
-needs to happen is the ``main`` branch must get merged into the ``stable``
-branch, and any conflicts that happen must be resolved, almost always in favor
-of the state of the code on the ``main`` branch.
+between the previous minor or major release and the new one. Then, create a new
+series branch off of the ``main`` branch (for example ``git switch -c yt-4.5.x``)
+and push the new branch up to the yt repository.
 
+.. code-block:: bash
+  git fetch upstream
+  git switch upstream/main
+  git switch -c yt-4.5.x
+  git push --set-upstream upstream yt-4.5.x
+
+After the series branch is up, you will bump the version number and generate a git tag
+as described below.
+
+After the completion of the release, the new series branch becomes the
+backport branch for subsequent bugfix releases.
 
-Incrementing Version Numbers and Tagging a Release
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Incrementing Version Numbers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Before creating the tag for the release, you must increment the version numbers
 that are hard-coded in a few files in the yt source so that version metadata
-for the code is generated correctly. This includes things like ``yt.__version__``
-and the version that gets read by the Python Package Index (PyPI) infrastructure.
-
-The paths relative to the root of the repository for the three files that need
-to be edited are:
+for the code is generated correctly. The paths relative to the root of the
+repository for the three files that need to be edited are:
 
 * ``doc/source/conf.py``
 
   The ``version`` and ``release`` variables need to be updated.
 
-* ``setup.py``
+* ``pyproject.toml``
 
-  The ``VERSION`` variable needs to be updated
+  The ``version`` variable needs to be updated
 
-* ``yt/__init__.py``
+* ``yt/_version.py``
 
   The ``__version__`` variable must be updated.
 
-Once these files have been updated, commit these updates. This is the commit we
+To update these files, check out and update the branch that will be released (``main``
+if doing a major or minor release, or the backport branch if doing a bugfix release).
+
+Once these files have been updated, commit these updates and submit a pull request
+targeting the branch that will be released. This is the commit we
 will tag for the release.
 
-To actually create the tag, issue the following command from the ``stable``
-branch:
+
+.. _tagging-a-release:
+
+Tagging a Release
+~~~~~~~~~~~~~~~~~
+
+After incrementing version numbers, checkout and update the branch that will be released
+and actually create the tag by issuing the following command:
 
 .. code-block:: bash
 
@@ -133,35 +197,6 @@ yt git repository. If you are doing a minor or major version number release, you
 will also need to update back to the development branch and update the
 development version numbers in the same files.
 
-Uploading to yt-project.org
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Before uploading the release to the Python Package Index (pypi.org) we will
-first upload the package to yt-project.org. This facilitates building binary
-wheels for pypi and binary conda packages on conda-forge before doing the
-"official" release. This also ensures that there isn't a period of time when
-users do ``pip install yt`` and end up downloading the source distribution
-instead of one of the binary wheels.
-
-To create the source distribution, issue the following command in the root of
-the yt repository::
-
-  $ python setup.py sdist
-
-This will generate a tarball in a ``dist/`` directory located in the root of the
-repository.
-
-Access to yt-project.org mediated via SSH login. Please contact one of the
-current yt developers for access to the webserver running yt-project.org if you
-do not already have it. You will need a copy of your SSH public key so that your
-key can be added to the list of authorized keys. Once you login, use
-e.g. ``scp`` to upload a copy of the source distribution tarball to
-https://yt-project.org/sdist, like so::
-
-  $ scp dist/yt-3.5.1.tar.gz yt_analysis@dickenson.dreamhost.com:yt-project.org/sdist
-
-You may find it helpful to set up an ssh config for dickenson to make this
-command a bit easier to execute.
 
 Publishing
 ~~~~~~~~~~
@@ -232,12 +267,9 @@ Announcing
 ~~~~~~~~~~
 
 After the release is uploaded to `PyPI <https://pypi.org/project/yt/#files>`_ and
-`conda-forge <https://anaconda.org/conda-forge/yt>`_,
-you should send out an announcement
-e-mail to the yt mailing lists as well as other possibly interested mailing
-lists for all but bugfix releases.
-
-Creating a Github release attached to the tag also offers a couple advantages.
-Auto-generated release notes can be a good starting point, though it's best to
-edit out PRs that not directly affecting users, and these notes can be edited
-before (draft mode) and after the release, so errors can be corrected after the fact.
+`conda-forge <https://anaconda.org/conda-forge/yt>`_, you should publish the
+GitHub draft release (see :ref:`prepping-release-notes`) and then you should
+send out an announcement e-mail to the yt mailing lists and the yt slack. Optionally,
+for major and minor releases, you can send the announcement to other interested
+mailing lists. Include a brief overview of the changes in the release and link to
+the GitHub release.