############### Release Process ############### The page provides the steps needed to make and publish a release of OpenDDS. **************************** Notes About OpenDDS Releases **************************** Temporary Notes =============== List of current problems with the release process and their workarounds. If there's nothing to note then the list should just consist of ``- N/A``. .. attention:: - N/A File Updates ============ This guide will document what should be updated manually. Almost all manual updates should be done before the release and are documented in :ref:`release-update-files`. All authoritative references to OpenDDS versions in the source code (like :ghfile:`VERSION.txt` or :ghfile:`dds/Version.h`) will be automatically updated by the release script, so normally there’s no need to do it manually. Other files such as :ghfile:`NEWS.md` and :ghfile:`README.md` shouldn't be updated manually specifically for a release. References to ACE/TAO versions are handled using :ghfile:`acetao.ini`, which is automatically updated by :ghfile:`.github/workflows/update-ace-tao.yml`. Versioning ========== .. _release-micro-release: Micro release (a.k.a. patch release) Releases with ``X.Y.Z`` versions. They should only contain fixes. They should avoid additional functionality and breaking changes if possible. Some parts of the release process are different for micro releases. See :ref:`release-make-micro-release` for more information. Minor release Releases with ``X.Y.0`` versions. They should only contain fixes and additional functionality. They should avoid breaking changes if possible. Major release Releases with ``X.0.0`` versions. They are generally the only time breaking changes can be made. Version metadata Text that follows the version in most circumstances. Currently there's only two uses for this: - The default state of the repo is for the metadata to be ``dev`` (e.g. ``4.0.0-dev``). The release script will remove the metadata before a release and restore it afterwards. - A preview release/prerelease/release candidate can be indicated by ``pre.N`` where ``N`` starts at 1 (e.g. ``4.0.0-pre.2``). Currently this isn't supported by the release script, so it would have to be manually created. .. _release-latest-release: Latest release The release that has the highest version number. Some parts of the release process are only done for the latest release. ******************** Prior to the Release ******************** Address Automated Code Analysis Results ======================================= - Address any serious compiler warnings seen on the CI - Address any serious diagnostic messages generated by Coverity at https://scan.coverity.com/projects/opendds?tab=overview .. _release-update-files: Update Files in the Repo as Needed ================================== These are files or the parts of the files that the release script won't be able to automatically update or should be checked beforehand to make sure they are correct. - Check :doc:`/news` to make sure the release notes for the next release look correct. If they need changes, then the news fragments should be changed before release. .. seealso:: :ref:`docs-news` documents how news fragments works. - The :ghfile:`AUTHORS` file should also be checked. This file is automatically generated from git history by the release script during the release. It can also be generated ahead of time to get a preview: - Fetch the latest ``master`` branch and run ``tools/scripts/gitrelease.pl --update-authors`` to see what the file will look like after release. This doesn't require any other arguments like the release script normally does. - If the file changed, but something is wrong, then it might be good to correct it before the release. For example a contributor might have used a Git client that is inserting an name/email that might not be what they want in the file or including a name/email might be in addition to their existing name/email combo. Corrections will have to be inserted into :ghfile:`.mailmap` using the format described in the git documentation listed in the file. In addition to that there are general rules in the release script such as ignoring bots, preferring ``objectcomputing.com`` email addresses to ``ociweb.com`` ones, and dealing with GitHub-specific issues. - Run ``tools/scripts/gitrelease.pl --update-authors`` again to make sure the changes worked. - Update :ghfile:`README.md` and :doc:`/devguide/building/dependencies` for any platform or dependency changes. - Document changes to building OpenDDS, at least in :doc:`/devguide/building/index`, but possibly also in :ghfile:`java/README` and :ghfile:`java/INSTALL`. Update the Modeling SDK version numbers and release notes ========================================================= Our convention recently has been to only update these if changes have been made to the Modeling SDK plugins in the current release cycle. Notes are in :ghfile:`tools/modeling/plugins/org.opendds.modeling.help/html/gettingstarted/maintopic.html`. Version numbers are updated by running :ghfile:`tools/modeling/update_version.pl`. Generate the Modeling SDK Eclipse update site ============================================= Our convention recently has been to only update these if changes have been made to the Modeling SDK plugins in the current release cycle. Follow all the steps in :ghfile:`tools/modeling/features/org.opendds.modeling.site/README.txt` The step dealing with version numbers is already taken care of by the above section "Update the Modeling SDK version numbers and release notes". The result of this process is adding the update site contents to the repository for http://www.opendds.org (which will be synced to the live site in the steps below). Update the opendds.org Website ============================== The www.opendds.org website is hosted by GitHub as special branch named ``gh-pages`` in the OpenDDS repository. The website is updated when changes are pushed to that branch. To do this follow these steps: - Clone the OpenDDS repository and checkout the branch named ``website-next-release`` to make changes to website source files This branch is used instead of ``gh-pages`` to hold changes that shouldn't be public until the release is made. When the release script runs it has a step for merging ``website-next-release`` into ``gh-pages``. .. code-block:: bash git clone -b gh-pages git@github.com:OpenDDS/OpenDDS.git website cd website git checkout -B website-next-release - To have Jekyll generate and serve ``gh-pages`` website locally in order to validate website changes: - You will need to have `Ruby `__ and `bundler `__ installed. - Run the commands: .. code-block:: bash bundle install bundle exec jekyll serve - This will generate the website from the Jekyll source files and serve the generated website on localhost port TCP/4000 .. seealso:: `The gh-pages README `__ Detailed instructions `Testing your GitHub Pages site locally with Jekyll `__ Official GitHub tutorial - Update the website source files and commit local changes to the ``website-next-release`` branch. - Push local ``website-next-release`` branch changes to the central repository or your forked repository - If using a forked repo, generate a pull request for the ``website-next-release`` branch to get the changes in to the central OpenDDS/OpenDDS repository. The release script will merge ``website-next-release`` into ``gh-pages`` on the ``OpenDDS/OpenDDS`` repository during the release process. .. _release-workflows: Check if GHA Workflows need Updating ==================================== .. note:: This should only be done for the :ref:`latest release `. For non-micro releases, if the release workflows haven't been updated in a while, manually trigger them to make sure they work. If any don't, then make changes as necessary. For micro releases check to see if the workflows have been updated on master and backport those changes to the release series branch. Then manually trigger them to make sure they work. The workflows are: - :ref:`shapes-demo`: :ghfile:`(file) <.github/workflows/ishapes.yml>`, `(runs) `__, - `OMG RTPS interoperability test `__: :ghfile:`(file) <.github/workflows/dds-rtps.yml>`, `(runs) `__, These will be triggered by the release script, then must be :ref:`uploaded after release `. **************** Making a Release **************** The release script (:ghfile:`tools/scripts/gitrelease.pl`) performs or validates the release steps. All the steps can be listed using the ``--list-all`` option. The steps that would be ran with the full set arguments can be listed with the ``--list`` options. By default it will try to run all the steps it can or you can run an arbitrary subset of the steps using the ``--steps`` option. Some manual steps are required. It will make modifications to the repository of the current working directory while using a directory of your choosing for intermediate and release files. Before Running the Release Script ================================= - Release Script Prerequisites: - Commands available: - ``md5sum`` and ``sha256sum`` - ``zip``, ``unzip``, and ``zipinfo`` - Git version 2.5 or later - :ref:`Python 3 for News Generation ` - Your GitHub account meets the following requirements: - It has been added as a member of the `OpenDDS organization `__ with the appropriate permissions. - It has permissions to update the release artifacts for the `OMG RTPS Interop repo `__ for :ref:`release-upload-artifacts`. - `You have uploaded your SSH public key to your GitHub account `__ - `You have created a Personal Access Token for your GitHub account `__ - You are a maintainer on the `OpenDDS Read the Docs project `__. - Maintainers can add new maintainers `here `__. - You will need a `API token `__. - The following `Perl CPAN modules `__ are required (`Perl core modules `__ should not be listed here): - ``Pithub`` - ``Net::SFTP::Foreign`` - ``Time::Piece`` - ``LWP::UserAgent`` - ``LWP::Protocol::https`` To install them run: .. code-block:: bash cpan -T -i Pithub Net::SFTP::Foreign Time::Piece LWP::UserAgent LWP::Protocol::https If Pithub gives errors about https url not be supported, then it might be necessary to install `Net::SSLeay `__ which also requires openssl. - Choose a directory for the ``WORKSPACE`` argument. It doesn't have to exist but the release script must be able to create it if it doesn't. It should not contain files created by previous release (mocked or otherwise). - You should start a new clone of the OpenDDS repository for just for this release. That clone shouldn't be inside the directory being passed as ``WORKSPACE``. .. code-block:: bash git clone git@github.com:OpenDDS/OpenDDS.git For micro releases, check out the relevant branch that the release will come from and pass ``--branch=BRANCH`` along with the ``--micro`` argument. - Set tokens the release script needs to interact with web services by exporting them as environment variables. These are ``GITHUB_TOKEN`` for your `GitHub Personal Access Token `__ and ``READ_THE_DOCS_TOKEN`` for your `Read the Docs API token `__ as shown below: .. code-block:: bash export GITHUB_TOKEN=ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00 export READ_THE_DOCS_TOKEN=ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00 Releasing from a Container ========================== Performing the release from a container is recommended as it provides a standardized and repeatable environment. The following script demonstrates how to prepare a container to perform a release: .. code-block:: bash podman run --rm -ti ubuntu /bin/bash apt update apt install zip git g++ make libssl-dev zlib1g-dev rustc cargo python3 python3-venv vim cpan -T -i Pithub Net::SFTP::Foreign Time::Piece LWP::UserAgent LWP::Protocol::https Net::SSLeay git config --global user.email "YOUR EMAIL" git config --global user.name "YOUR NAME" export GITHUB_TOKEN="YOUR GITHUB TOKEN" export READ_THE_DOCS_TOKEN="YOUR READ THE DOCS TOKEN" mkdir ~/.ssh # Copy in your SSH private key for GitHub vim ~/.ssh/id_rsa # Copy in your SSH public key for GitHub vim ~/.ssh/id_rsa.pub chmod 400 ~/.ssh/id_rsa # Create a workspace mkdir workspace # Clone OpenDDS git clone git@github.com:OpenDDS/OpenDDS.git cd OpenDDS Then, you can execute the release script or, for a micro release, check out the appropriate branch and then execute the release script. Running the Release Script ========================== The release script is located at :ghfile:`tools/scripts/gitrelease.pl` and should be ran from the root of the repo. (See above note in mock releases for the exception) There are two required arguments, the ``WORKSPACE`` and ``VERSION`` arguments: - ``WORKSPACE`` is the directory where the script will place all intermediate files. If it doesn't exist the script will try to create it for you. This should be different for different releases of OpenDDS. - ``VERSION`` is the version to release. Run the script with just the required arguments to validate each step of the process. It will stop at the first error and give you instructions of what to do. In most cases ``--remedy`` should be used to continue. When the script wants to commit something, it will show you the ``git diff``. Press ``q`` and it will ask you for confirmation that it's okay to commit it. The most important options are: - ``--list``, which lists the steps with their number and description - ``--remedy``, which tells the release script to attempt to resolve issues with the release - ``--steps``, which will specify the steps to run If one of the steps isn't verifying correctly, but you already manually fixed it, you can skip the step by passing ``--step ^STEP`` where ``STEP`` is the step you want to skip. You can also skip whole ranges of the steps. See ``--help`` for the notation it accepts. - ``--micro``, which excludes the steps that probably are not desired when doing a micro release and requires ``--branch``. Run ``perl tools/scripts/gitrelease.pl --help`` to see the full help. Here is an example of what to run for a version 1.0.0 release command assuming that the release script can take care of everything for us: .. code-block:: bash tools/scripts/gitrelease.pl ../1.0.0-release-workspace 1.0.0 --remedy .. _release-make-micro-release: Micro Releases -------------- As stated earlier, a micro release only contains fixes. If a bug is found in a release version, the fix should first be applied to the development branch for the corresponding major version and the master branch if they are not the same. To start a micro release, cherry pick one or more fixes from the development branch to the release branch. See the ``--cherry-pick-prs`` option of the release script for help with this task. Release branches have the form ``branch-DDS-X.Y``. The cherry-picked commits should include the relevant news files. The release script has a ``--micro`` option which skips steps that probably are not relevant to :ref:`micro releases `. You must pass the ``--branch`` argument as you should be on the release branch for the minor release. As of writing these steps skipped are: - Merging ``website-next-release`` with ``gh-pages`` Some other notes about using ``--micro``: - The notation of the version argument has no effect on if the script is doing a micro release. - Steps are skipped if they are one of the ones listed above, even if that step number is the only one explicitly passed in. Otherwise the script should behave the same way. Here is an example of what to run for a version 1.0.1 release assuming that the release script can take care of everything for us: .. code-block:: bash git checkout branch-DDS-1.0 tools/scripts/gitrelease.pl ../1.0.1-release-workspace 1.0.1 --micro --branch=branch-DDS-1.0 --remedy Doing Mock Releases with the Release Script ------------------------------------------- It is possible to do a mock release where basically everything is tested, but the script will make sure it's not making any real changes to the real thing. To set this up, you must do the following: - Fork OpenDDS on GitHub. - To avoid conflicts with regular work on a fork you might already have, it's recommended to create a new organization for this purpose and create a token for the repository just like for an actual release. Pass the organization name using ``--github-user``. - This can be skipped if code involving GitHub doesn't need to be tested and ``--skip-github`` is passed. - Pass ``--mock``. This actually isn't absolutely necessary, but it is useful as it does some basic checks to make sure the mock release won't interfere with the actual releases. It's possible to use and edit ``gitrelease.pl`` without having to commit changes to it for a mock release if you use two repos. One repo, lets call it ``$MOCK_ROOT``, is the one cloned from the mock organization mentioned in the previous instructions and is where the release is going to happen. The other, ``$WORKING_ROOT``, is a normal repo where you can edit ``gitrelease.pl`` and other files and push changes to your normal GitHub fork. Running ``$WORKING_ROOT/tools/scripts/gitrelease.pl`` from ``$MOCK_ROOT`` will work because ``gitrelease.pl`` does everything relative to the current working directory. This also might be possible with ``git worktree`` instead of fully separate repos but this hasn't been tested. After Running the Release Script ================================ Test the release package ------------------------ A simple test of Messenger will do. The git tag is already cloned for you as part of the release process. Update News on Master --------------------- .. note:: This should only be done for the :ref:`latest release ` that's also a :ref:`micro release `. The news on the master branch has to be updated to account for the micro release. Updating the news consists of: - Copy the micro release entry in :ghfile:`NEWS.md`. - Copy the micro release file in :ghfile:`docs/news.d/_releases`. - Remove the :ref:`news fragments ` in :ghfile:`docs/news.d` for the PRs that were backported. .. _release-upload-artifacts: Upload Artifacts from Release Workflows --------------------------------------- .. note:: This should only be done for the :ref:`latest release `. The release script has steps that trigger :ref:`release workflows ` on GitHub Actions and print out links to the runs. After they have finished successfully, run the release script with the version and workspace arguments and the ``--upload-artifacts`` option. If the workflows are still in progress it will say so and give the links again. If the workflows were successful, it will download the artifacts, package them, and upload them to GitHub. Remove Files Used for Release ----------------------------- Once everything is been finished, the repo and workspace directory used for release can usually be safely deleted. Erring on the side of caution though, they could be kept around for at least a few days after the release to help rerun steps if necessary or inspecting contents of the workspace directory for debugging purposes.