Publish of Apache Doris

    The general process of publication is as follows:

    1. Preparing your setup
    2. Preparing for release candidates
      1. launching DISCUSS in the community
      2. cutting a release branch
      3. clean up issues
      4. merging necessary patch to release branch
    3. Verify branch
      1. QA stability test
      2. Verify the correctness of the compiled image
      3. Prepare Release Nodes
    4. Running the voting process for a release
      1. singing a tag and upload it to Apache dev svn repo (opens new window)
      2. calling votes from
      3. send result email to Doris community
      4. calling votes from
      5. send result email to general@incubator.apache.org
    5. Finalizing and posting a release
      1. Upload the signature package to Apache release repoPublish of Apache Doris - 图2 (opens new window) and generate relevant links
      2. Publish download links on Doris website and GitHub, clean up old version packages on svn
      3. Send Announce mail to general@incubator.apache.org

    If you are a new Release Manager, you can read up on the process from the followings:

    1. release signing
    2. gpg for signing https://www.apache.org/dev/openpgp.html
    3. svn

    Release manager needs Mr. A to sign his own public key before publishing and upload it to the public key server. Then he can use this public key to sign the package ready for publication. If your key already exists in [key] (https://dist.apache.org/repos/dist/dev/initiator/doris/keys), you can skip this step.

    Installation and configuration of signature software GnuPG

    GnuPG

    In 1991, programmer Phil Zimmermann developed the encryption software PGP to avoid government surveillance. This software is very useful, spread quickly, has become a necessary tool for many programmers. However, it is commercial software and cannot be used freely. So the Free Software Foundation decided to develop a replacement for PGP, called GnuPG. This is the origin of GPG.

    Installation Configuration

    CentOS installation command:

    After installation, the default configuration file gpg.conf will be placed in the home directory.

    If this directory or file does not exist, you can create an empty file directly. Edit gpg.conf, modify or add KeyServer configuration:

    1. keyserver hkp http://keys.gnupg.net

    Apache signature recommends SHA512, which can be done by configuring gpg. Edit gpg.conf and add the following three lines:

    1. personal-digest-preferences SHA512
    2. cert -digest -something SHA512
    3. default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

    Generating new signatures

    Prepare to Sign

    Recommended settings for generating new signatures:

    We must log in to user account directly through SecureCRT and other terminals. We can’t transfer it through Su - user or ssh. Otherwise, the password input box will not show up and make an error.

    Let’s first look at the version of GPG and whether it supports SHA512.

    1. $ gpg --version
    2. gpg (GnuPG) 2.0.22
    3. libgcrypt 1.5.3
    4. Copyright (C) 2013 Free Software Foundation, Inc.
    5. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    6. This is free software: you are free to change and redistribute it.
    7. There is NO WARRANTY, to the extent permitted by law.
    9. Home: ~/.gnupg
    10. Supported algorithms:
    11. Pubkey: RSA, ?, ?, ELG, DSA
    12. Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
    13.         CAMELLIA128, CAMELLIA192, CAMELLIA256
    14. Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
    15. Compression: Uncompressed, ZIP, ZLIB, BZIP2
    Generating new signatures
    1. $ gpg --gen-key
    2. gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
    3. This is free software: you are free to change and redistribute it.
    4. There is NO WARRANTY, to the extent permitted by law.
    6. Please select what kind of key you want:
    7.    (1) RSA and RSA (default)
    8.    (2) DSA and Elgamal
    9.    (3) DSA (sign only)
    10.    (4) RSA (sign only)
    11. Your selection? 1
    12. RSA keys may be between 1024 and 4096 bits long.
    13. What keysize do you want? (2048) 4096
    14. Requested keysize is 4096 bits
    15. Please specify how long the key should be valid.
    16.          0 = key does not expire
    17.       <n>  = key expires in n days
    18.       <n>w = key expires in n weeks
    19.       <n>m = key expires in n months
    20.       <n>y = key expires in n years
    21. Key is valid for? (0)
    22. Key does not expire at all
    24. GnuPG needs to construct a user ID to identify your key.
    26. Real name: xxx
    27. Name must be at least 5 characters long
    28. Real name: xxx-yyy
    29. Email address: xxx@apache.org
    30. Comment: xxx's key
    31. You selected this USER-ID:
    32.     "xxx-yyy (xxx's key) <xxx@apache.org>"
    34. Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o

    Real name needs to be consistent with the ID shown in ID. apache. org. Email address is apache’s mailbox.

    Enter passphrase, you need to enter it twice. The passphrase must more than 8 characters.

    The passphrase here must be remembered, it will be used when signing later

    View and Output

    The first line displays the public key file name (pubring.gpg), the second line displays the characteristics of the public key (4096 bits, Hash string and the generation time), the third line displays “User ID”, comments, email, and the fourth line displays Private key characteristics.

    1. $ gpg --list-keys
    2. /home/lide/.gnupg/pubring.gpg
    3. -----------------------------
    4. pub   4096R/33DBF2E0 2018-12-06
    5. uid                  xxx-yyy  (xxx's key) <xxx@apache.org>
    6. sub   4096R/0E8182E6 2018-12-06

    xxx-yy is the user ID.

    1. gpg --armor --output public-key.txt --export [UserID]
    1. $ gpg --armor --output public-key.txt --export xxx-yyy
    2. 文件‘public-key.txt’已存在。 是否覆盖？(y/N)y
    3. $ cat public-key.txt
    4. -----BEGIN PGP PUBLIC KEY BLOCK-----
    5. Version: GnuPG V2.0.22 (GNU /Linux)
    7. mQINBFwJEQ0BEACwqLluHfjBqD/RWZ4uoYxNYHlIzZvbvxAlwS2mn53BirLIU/G3
    8. 9opMWNplvmK+3+gNlRlFpiZ7EvHsF/YJOAP59HmI2Z...

    Upload signature public key

    Public key servers are servers that store users’public keys exclusively on the network. The send-keys parameter uploads the public key to the server.

    1. gpg --send-keys xxxx

    Where XXX is the last step — the string after pub in the list-keys result, as shown above: 33DBF2E0

    You can also upload the contents of the above public-key.txt through the following website:

    1. http://keys.gnupg.net

    After successful upload, you can query the website and enter 0x33DBF2E0:

    Queries on the site are delayed and may take an hour.

    Generate fingerprint and upload it to Apache user information

    Fingerprint parameter generates public key fingerprints:

    1. $ gpg --fingerprint xxx-yyy
    2. pub   4096R/33DBF2E0 2018-12-06
    3.       Key fingerprint = 07AA E690 B01D 1A4B 469B  0BEF 5E29 CE39 33DB F2E0
    4. uid                  xxx-yyy (xxx's key) <xxx@apache.org>
    5. sub   4096R/0E8182E6 2018-12-06

    Paste the fingerprint above (i.e. 07AA E690 B01D 1A4B 469B 0BEF 5E29 CE39 33DB F2E0) into your user information:

    OpenPGP Public Key Primary Fingerprint:

    Generating keys

    1. svn co https://dist.apache.org/repos/dist/dev/incubator/doris/
    2. # edit doris/KEY file
    3. gpg --list-sigs [用户 ID] >> doris/KEYS
    4. gpg --armor --export [用户 ID] >> doris/KEYS

    Launching DISCUSS in the Community

    If you think you’ve fixed a lot of bugs and developed more important features, any IPMC member can initiate DISCUSS discussions to release a new version. An e-mail entitled [DISCUSS] x.y.z release can be launched to discuss within the community what bugs have been fixed and what features have been developed. If DISCUSS mail is supported, we can proceed to the next step.

    Preparatory Branch

    Before publishing, we need to build a new branch, For example:

    1. $ git checkout -b branch-0.9

    This branch needs to be fully tested to make functions available, bug convergence, and important bugs fixed.

    This process needs to wait for the community to see if a necessary patch needs to be merged in this version, and if so, it needs to be cherry picked to the release branch.

    clean up issue

    Go through all the issues belonging to this version, close those that have been completed, and if they cannot be completed, postpone them to a later version.

    Merge necessary patches

    During the release waiting process, there may be more important patch merging. If someone in the community says that there is an important bug to merge, then release manager needs to evaluate and merge the important patches into the release branch.

    Give the prepared branch to QA students for stability testing. If there is a problem that needs to be fixed during the testing process, if there is a problem that needs to be fixed during the testing process, after the problem is fixed, the PR that fixes the problem needs to be merged into the branch of the release version.

    After the entire branch is stable, the release can be prepared.

    Verify the correctness of the compiled image

    1. Download the compiled image

      1.

      docker pull apachedoris/doris-dev:build-env-1.2

    2. Use official documents to compile the new branch, see Docker Development Mirror Compilation (opens new window)

      1.   After entering the mirror, the compilation may take about 3~4 hours, please be patient.
      3.   If the compilation fails due to the lack of some third-party libraries, the compilation image needs to be updated.

    3. Rebuild the image

    dozen Tags

    When the above branches are stable, tags can be made on them. Remember to modify the build_version variable in gensrc/script/gen_build_version.sh when creating tags. For example, build_version='0.10.0-release'.

    For example:

    1. $ git checkout branch-0.9
    2. $ git tag -a 0.9.0-rc01 -m "0.9.0 release candidate 01"
    3. $ git push origin 0.9.0-rc01
    4. Counting objects: 1, done.
    5. Writing objects: 100% (1/1), 165 bytes | 0 bytes/s, done.
    6. Total 1 (delta 0), reused 0 (delta 0)
    7. To git@github.com:apache/incubator-doris.git
    8.  * [new tag]         0.9.0-rc01 -> 0.9.0-rc01
    10. $ git tag

    Packing Signature

    The following steps also need to log into user accounts directly through terminals such as SecureCRT, and can not be transferred through Su - user or ssh, otherwise the password input box will not show and error will be reported.

    1. $ git checkout 0.9.0-rc01
    3. $ git archive --format=tar 0.9.0-rc01 --prefix=apache-doris-0.9.0-incubating-src/ | gzip > apache-doris-0.9.0-incubating-src.tar.gz
    5. $ gpg -u xxx@apache.org --armor --output apache-doris-0.9.0-incubating-src.tar.gz.asc --detach-sign apache-doris-0.9.0-incubating-src.tar.gz
    7. $ gpg --verify apache-doris-0.9.0-incubating-src.tar.gz.asc apache-doris-0.9.0-incubating-src.tar.gz
    9. $ sha512sum apache-doris-0.9.0-incubating-src.tar.gz > apache-doris-0.9.0-incubating-src.tar.gz.sha512
    11. $ sha512sum --check apache-doris-0.9.0-incubating-src.tar.gz.sha512

    Upload signature packages and KEYS files to DEV SVN

    First, download the SVN library:

    1. svn co https://dist.apache.org/repos/dist/dev/incubator/doris/

    Organize all previous files into the following SVN paths

    1. ./doris/
    2. |-- 0.11.0-rc1
    3. |   |-- apache-doris-0.11.0-incubating-src.tar.gz
    4. |   |-- apache-doris-0.11.0-incubating-src.tar.gz.asc
    5. |   `-- apache-doris-0.11.0-incubating-src.tar.gz.sha512
    6. `-- KEYS

    Upload these files

    1. svn add 0.9.0-rc1
    2. svn commit -m "Release Apache Doris (incubating) 0.9.0 rc1"

    [VOTE] Release Apache Doris 0.9.0-incubating-rc01

    Email Result after the vote is passed

    1. Thanks to everyone, and this vote is now closed.
    3. It has passed with 4 +1 (binding) votes and no 0 or -1 votes.
    5. Binding:
    6. Zhao Chun
    7. +1 xxx
    8. + 1 Li Chaoyong
    9. +1 Mingyu Chen
    11. Best Regards,
    12. xxx

    Send an e-mail to general@incubator.apache.org for a vote.

    [VOTE] Release Apache Doris 0.9.0-incubating-rc01

    1. Hi all,
    3. Please review and vote on Apache Doris 0.9.0-incubating-rc01 release.
    5. Apache Doris is an MPP-based interactive SQL data warehousing for reporting and analysis.
    7. The Apache Doris community has voted on and approved this release:
    8. https://lists.apache.org/thread.html/d70f7c8a8ae448bf6680a15914646005c6483564464cfa15f4ddc2fc@%3Cdev.doris.apache.org%3E
    10. The vote result email thread:
    11. https://lists.apache.org/thread.html/64d229f0ba15d66adc83306bc8d7b7ccd5910ecb7e842718ce6a61da@%3Cdev.doris.apache.org%3E
    12. The release candidate has been tagged in GitHub as 0.9.0-rc01, available here:
    13. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
    15. There is no CHANGE LOG file because this is the first release of Apache Doris.
    16. Thanks to everyone who has contributed to this release, and there is a simple release notes can be found here:
    17. https://github.com/apache/incubator-doris/issues/406
    19. The artifacts (source, signature and checksum) corresponding to this release candidate can be found here:
    20. https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc01/
    22. This has been signed with PGP key 33DBF2E0, corresponding to lide@apache.org.
    23. KEYS file is available here:
    24. https://dist.apache.org/repos/dist/dev/incubator/doris/KEYS
    25. It is also listed here:
    26. https://people.apache.org/keys/committer/lide.asc
    28. The vote will be open for at least 72 hours.
    30. [ ] +0 No opinion
    31. [ ] -1 Do not release this package because ...
    33. To verify and build, you can refer to following instruction:
    35. Firstly, you must be install and start docker service, and then you could build Doris as following steps:
    37. Step1: Pull the docker image with Doris building environment
    38. $ docker pull apachedoris/doris-dev:build-env
    39. You can check it by listing images, its size is about 3.28GB.
    41. Step2: Run the Docker image
    42. You can run image directly:
    43. $ docker run -it apachedoris/doris-dev:build-env
    45. Step3: Download Doris source
    46. Now you should in docker environment, and you can download Doris source package.
    47. (If you have downloaded source and it is not in image, you can map its path to image in Step2.)
    48. $ wget https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc01/apache-doris-0.9.0.rc01-incubating-src.tar.gz
    50. Step4: Build Doris
    51. Now you can decompress and enter Doris source path and build Doris.
    52. $ tar zxvf apache-doris-0.9.0.rc01-incubating-src.tar.gz
    53. $ cd apache-doris-0.9.0.rc01-incubating-src
    54. $ sh build.sh
    56. Best Regards,
    57. xxx
    59. ----
    60. DISCLAIMER-WIP: 
    61. Apache Doris is an effort undergoing incubation at The Apache Software Foundation (ASF), 
    62. sponsored by the Apache Incubator. Incubation is required of all newly accepted projects 
    63. until a further review indicates that the infrastructure, communications, and decision 
    64. making process have stabilized in a manner consistent with other successful ASF projects. 
    65. While incubation status is not necessarily a reflection of the completeness or stability 
    66. of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
    68. Some of the incubating project’s releases may not be fully compliant with ASF policy. For 
    69. example, releases may have incomplete or un-reviewed licensing conditions. What follows is 
    70. a list of known issues the project is currently aware of (note that this list, by definition, 
    71. is likely to be incomplete): 
    73.  * Releases may have incomplete licensing conditions
    75. If you are planning to incorporate this work into your product/project, please be aware that
    76. you will need to conduct a thorough licensing review to determine the overall implications of 
    77. including this work. For the current status of this project through the Apache Incubator 
    78. visit: https://incubator.apache.org/projects/doris.html

    The threaded connection for mail can be found here:

    https://lists.apache.org/list.html?dev@doris.apache.org

    Email Result to general@incubator.apache.org

    [RESULT][VOTE] Release Apache Doris 0.9.0-incubating-rc01

    1. Hi,
    3. Thanks to everyone, and the vote for releasing Apache Doris 0.9.0-incubating-rc01 is now closed.
    5. It has passed with 4 +1 (binding) votes and no 0 or -1 votes.
    7. Binding:
    8. +1 Willem Jiang
    9. +1 Justin Mclean
    10. +1 ShaoFeng Shi
    11. +1 Makoto Yui
    13. The vote thread:
    14. https://lists.apache.org/thread.html/da05fdd8d84e35de527f27200b5690d7811a1e97d419d1ea66562130@%3Cgeneral.incubator.apache.org%3E
    16. Best Regards,
    17. xxx

    Upload package to release

    When the formal voting is successful, email [Result] first, and then prepare the release package. Copy the source package, signature file and hash file from the corresponding RC folder published under dev to another directory 0.9.0-incubating. Note that the file name does not need rcxx (rename, but do not recalculate signatures, hash can recalculate, the results will not change)

    KEYS files also need to be copied if they are first released. Then add to SVN release.

    1. After the add is successful, you can see the file you posted on the following URL
    2. https://dist.apache.org/repos/dist/release/incubator/doris/0.9.0-incubating/
    4. After waiting for a while, you can see on the official apache website:
    5. http://www.apache.org/dist/incubator/doris/0.9.0-incubating/

    Download link:

    wget —trust-server-names “https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz“

    Original location:

    http://www.apache.org/dyn/closer.cgi/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz

    Source package:

    ASC: http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.asc

    sha512:

    KEYS: http://archive.apache.org /dist /incubator /doris /KEYS

    refer to:

    Prepare release note

    The following two areas need to be modified:

    1. Github’s release page
    1. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
    1. Doris Official Website Download Page
    1. http://doris.apache.org /downloads.html

    clean up the old release package from svn

    1. clean up old release package

    Since svn only needs to save the latest version of the package, when a new version is released, the old version of the package should be cleaned up from svn.

    1. https://dist.apache.org/repos/dist/release/incubator/doris/
    2. https://dist.apache.org/repos/dist/dev/incubator/doris/

    Keep only the latest version of the package in these two addresses.

    1. Change the download link of the old version package on the download page to the address of the archive page
    1. Download page: http://doris.apache.org/downloads.html
    2. Archive page: http://archive.apache.org/dist/incubator/doris

    Apache will have a synchronization mechanism to archive historical release versions. For details, see: (how to archive)[https://www.apache.org/legal/release-policy.html#how-to-archive\] So even if the old package is removed from svn, it can still be found on the archive page.

    Send Announce e-mail to general@incubator.apache.org

    Title:

    1. [ANNOUNCE] Apache Doris (incubating) 0.9.0 Release

    Send mail group:

    Mail text: