Development workflows

    A key aspect of all workflows is that none of them blocks another. Forinstance, a bug fix can be backported and merged to a stable branchwhile the next point release is being published. For that specificexample to work, a branch should be created to avoid anyinterference. In practice it is not necessary for Ceph because:

    • there are few people involved

    • the frequency of backports is not too high

    • the reviewers, who know a release is being published, are unlikelyto merge anything that may cause issues

    This ad-hoc approach implies the workflows are changed on a regularbasis to adapt. For instance, quality engineers were not involvedin the workflow to publish dumpling point releases. The number ofcommits being backported to firefly made it impractical for developerstasked to write code or fix bugs to also run and verify the full suiteof integration tests. Inserting quality engineers makes itpossible for someone to participate in the workflow by analyzing testresults.

    The workflows are not enforced when they impose an overhead that doesnot make sense. For instance, if the release notes for a point releasewere not written prior to checking all integration tests, they can becommitted to the stable branch and the result sent for publicationwithout going through another run of integration tests.

    Four times a year, the development roadmap is discussed online duringthe Ceph Developer Summit. Anew stable release (hammer, infernalis, jewel …) is published at the samefrequency. Every other release (firefly, hammer, jewel…) is a . See Understanding the release cycle for more information.

    Merging bug fixes or features

    The development branch is master and the workflow followed by alldevelopers can be summarized as follows:

    • The developer prepares a series of commits

    • The developer submits the series of commits via a pull request

    • A reviewer is assigned the pull request

    • When the pull request looks good to the reviewer, it is merged intoan integration branch by the tester

    • After a successful run of integration tests, the pull request ismerged by the tester

    The developer is the author of a series of commits. Thereviewer is responsible for providing feedback to the developer ona regular basis and the developer is invited to ping the reviewer ifnothing happened after a week. After the reviewer is satisfiedwith the pull request, (s)he passes it to the tester. Thetester is responsible for running teuthology integration tests onthe pull request. If nothing happens within a month the reviewer isinvited to ping the tester.

    All bug reports and feature requests are in the issue tracker and the workflow can be summarized asfollows:

    • The reporter creates the issue with priority Normal

    • A developer may pick the issue right away

    • During a bi-weekly bug scrub, the team goes over all new issue andassign them a priority

    Each team is responsible for a project, managed by .

    The developer assigned to an issue is responsible for it. Thestatus of an open issue can be:

    • New: it is unclear if the issue needs work.

    • Verified: the bug can be reproduced or showed up multiple times

    • In Progress: the developer is working on it this week

    For each issue, there exists at least one issuein the Backport tracker to record the work done to cherry pick thenecessary commits from the master branch to the target stable branch.See the backporter manual for moreinformation.

    Running and interpreting teuthology integration tests

    The Sepia community test lab runs integration tests on a regular basis and theresults are posted on and theceph-qa mailing list.

    • The job failures are

    • If the cause is environmental (e.g. network connectivity), an issueis created in the sepia lab project

    • If the bug is known, a pulpito URL to the failed job is added to the issue

    • If the bug is new, an issue is created

    The quality engineer is either a developer or a member of the QEteam. There is at least one integration test suite per project:

    and many others such as

    A release is prepared in a dedicated branch, different from themaster branch.

    • For a stable releases it is the branch matching the release codename (dumpling, firefly, etc.)

    • For a development release it is the next branch

    The workflow expected of all developers to stabilize the releasecandidate is the same as the normal development workflow with thefollowing differences:

    • The pull requests must target the stable branch or next instead ofmaster

    • The reviewer rejects pull requests that are not bug fixes

    Cutting a new stable release

    A new stable release can be cut when:

    • all Backport issues with priority Urgent are fixed

    Publishing a new stable release implies a risk of regression ordiscovering new bugs during the upgrade, no matter how carefully it istested. The decision to cut a release must take this into account: itmay not be wise to publish a stable release that only fixes a fewminor bugs. For instance if only one commit has been backported to astable release that is not a LTS, it is better to wait until there aremore.

    When a stable release is to be retired, it may be safer torecommend an upgrade to the next LTS release instead ofproposing a new point release to fix a problem. For instance, thedumpling v0.67.11 release has bugs related to backfilling which havebeen fixed in firefly v0.80.x. A backport fixing these backfillingbugs has been tested in the draft point release dumpling v0.67.12 butthey are large enough to introduce a risk of regression. As dumplingis to be retired, users suffering from this bug canupgrade to firefly to fix it. Unless users manifest themselves and askfor dumpling v0.67.12, this draft release may never be published.

    • The Ceph lead decides a new stable release must be published

    • The release master gets approval from all leads

    • The release master writes and commits the release notes

    • The release master informs the quality engineer that thebranch is ready for testing

    • The quality engineer runs additional integration tests

    • If the discovers new bugs that require anUrgent Backport, the release goes back to being prepared, itwas not ready after all

    • The quality engineer informs the publisher that the branchis ready for release

    • The publisher creates the packages and sets the release tag

    The person responsible for each role is:

    • Sage Weil is the Ceph lead

    • Sage Weil is the release master for major stable releases(firefly 0.80, hammer 0.94 etc.)

    • Loic Dachary is the release master for stable point releases(firefly 0.80.10, hammer 0.94.1 etc.)

    • Yuri Weinstein is the quality engineer

    • Alfredo Deza is the publisher

    The publication workflow of a development release is the same aspreparing a new release and cutting it, with the followingdifferences:

    • The next branch is reset to the tip of master afterpublication

    • The quality engineer is not required to run additional tests,the release master directly informs the publisher that therelease is ready to be published.

    Publishing point releases and backporting

    The publication workflow of the point releases is the same aspreparing a new release and cutting it, with the followingdifferences:

    • The backport field of each issue contains the code name of thestable release

    • There is exactly one issue in the Backport tracker for eachstable release to which the issue is backported

    See the backporter manual for moreinformation.