Backward Incompatible Change

• Function (operator %) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added alias mod. This closes #7323. (alexey-milovidov).
• Deprecate special printing of zero Date/DateTime values as 0000-00-00 and 0000-00-00 00:00:00. (alexey-milovidov).
• The function groupArrayMoving* was not working for distributed queries. It’s result was calculated within incorrect data type (without promotion to the largest type). The function groupArrayMovingAvg was returning integer number that was inconsistent with the avg function. This fixes . #12622 ().
• Add sanity check for MergeTree settings. If the settings are incorrect, the server will refuse to start or to create a table, printing detailed explanation to the user. #13153 ().
• Protect from the cases when user may set background_pool_size to value lower than number_of_free_entries_in_pool_to_execute_mutation or number_of_free_entries_in_pool_to_lower_max_size_of_merge. In these cases ALTERs won’t work or the maximum size of merge will be too limited. It will throw exception explaining what to do. This closes #10897. (alexey-milovidov).

New Feature

• Polygon dictionary type that provides efficient “reverse geocoding” lookups - to find the region by coordinates in a dictionary of many polygons (world map). It is using carefully optimized algorithm with recursive grids to maintain low CPU and memory usage. #9278 ().
• Added support of LDAP authentication for preconfigured users (“Simple Bind” method). #11234 ().
• Introduce setting alter_partition_verbose_result which outputs information about touched parts for some types of ALTER TABLE ... PARTITION ... queries (currently ATTACH and FREEZE). Closes #8076. (alesapin).
• Add bayesAB function for bayesian-ab-testing. (achimbab).
• Added system.crash_log table into which stack traces for fatal errors are collected. This table should be empty. (alexey-milovidov).
• Added http headers X-ClickHouse-Database and X-ClickHouse-Format which may be used to set default database and output format. (hcz).
• Add minMap and maxMap functions support to SimpleAggregateFunction. (Ildus Kurbangaliev).
• Add setting allow_non_metadata_alters which restricts to execute ALTER queries which modify data on disk. Disabled be default. Closes . #12635 ().
• A function formatRow is added to support turning arbitrary expressions into a string via given format. It’s useful for manipulating SQL outputs and is quite versatile combined with the columns function. #12574 ().
• Add FROM_UNIXTIME function for compatibility with MySQL, related to 12149. (flynn).
• Allow Nullable types as keys in MergeTree tables if allow_nullable_key table setting is enabled. Closes . #12433 ().
• Integration with COS. (fastio).
• Add mapAdd and mapSubtract functions for adding/subtracting key-mapped values. (Ildus Kurbangaliev).

Bug Fix

• Fix premature ON CLUSTER timeouts for queries that must be executed on a single replica. Fixes #6704, , #13361, . #13450 ().
• Fix crash in mark inclusion search introduced in #12277. (Amos Bird).
• Fix race condition in external dictionaries with cache layout which can lead server crash. (alesapin).
• Fix visible data clobbering by progress bar in client in interactive mode. This fixes and #13369 and and fixes #12964. (alexey-milovidov).
• Fixed incorrect sorting order for LowCardinality columns when ORDER BY multiple columns is used. This fixes . #14223 ().
• Removed hardcoded timeout, which wrongly overruled query_wait_timeout_milliseconds setting for cache-dictionary. #14105 ().
• Fixed wrong mount point in extra info for Poco::Exception: no space left on device. #14050 ().
• Fix wrong query optimization of select queries with DISTINCT keyword when subqueries also have DISTINCT in case optimize_duplicate_order_by_and_distinct setting is enabled. #13925 ().
• Fixed potential deadlock when renaming Distributed table. #13922 ().
• Fix incorrect sorting for FixedString columns when ORDER BY multiple columns is used. Fixes #13182. (Nikolai Kochetov).
• Fix potentially lower precision of topK/topKWeighted aggregations (with non-default parameters). (Azat Khuzhin).
• Fix reading from MergeTree table with INDEX of type SET fails when compared against NULL. This fixes . #13793 ().
• Fix step overflow in function range(). #13790 ().
• Fixed Directory not empty error when concurrently executing DROP DATABASE and CREATE TABLE. #13756 ().
• Add range check for h3KRing function. This fixes #13633. (alexey-milovidov).
• Fix race condition between DETACH and background merges. Parts may revive after detach. This is continuation of that did not fix the issue but introduced a test that started to fail in very rare cases, demonstrating the issue. #13746 ().
• Fix logging Settings.Names/Values when log_queries_min_type greater than QUERY_START. #13737 ().
• Fix incorrect message in clickhouse-server.init while checking user and group. #13711 ().
• Do not optimize any(arrayJoin()) to arrayJoin() under optimize_move_functions_out_of_any. #13681 ().
• Fixed possible deadlock in concurrent ALTER ... REPLACE/MOVE PARTITION ... queries. #13626 ().
• Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. #13624 ().
• Fix secondary indices corruption in compact parts (compact parts is an experimental feature). #13538 ().
• Fix wrong code in function netloc. This fixes #13335. (alexey-milovidov).
• Fix error in parseDateTimeBestEffort function when unix timestamp was passed as an argument. This fixes . #13441 ().
• Fix invalid return type for comparison of tuples with NULL elements. Fixes #12461. (Nikolai Kochetov).
• Fix wrong optimization caused aggregate function any(x) is found inside another aggregate function in query error with SET optimize_move_functions_out_of_any = 1 and aliases inside any(). (Artem Zuikov).
• Fix possible race in StorageMemory. (Nikolai Kochetov).
• Fix empty output for Arrow and Parquet formats in case if query return zero rows. It was done because empty output is not valid for this formats. (hcz).
• Fix select queries with constant columns and prefix of primary key in ORDER BY clause. (Anton Popov).
• Fix PrettyCompactMonoBlock for clickhouse-local. Fix extremes/totals with PrettyCompactMonoBlock. Fixes . #13394 ().
• Fixed deadlock in system.text_log. #12452 (). It is a part of #12339. This fixes . #13386 ().
• Fixed File(TSVWithNames*) (header was written multiple times), fixed clickhouse-local --format CSVWithNames* (lacks header, broken after #12197), fixed clickhouse-local --format CSVWithNames* with zero rows (lacks header). (Azat Khuzhin).
• Fix segfault when function groupArrayMovingSum deserializes empty state. Fixes . #13341 ().
• Throw error on arrayJoin() function in JOIN ON section. #13330 ().
• Fix crash in LEFT ASOF JOIN with join_use_nulls=1. #13291 ().
• Fix possible error Totals having transform was already added to pipeline in case of a query from delayed replica. #13290 ().
• The server may crash if user passed specifically crafted arguments to the function h3ToChildren. This fixes #13275. (alexey-milovidov).
• Fix potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes . #13254 ().
• Fix assertion in KeyCondition when primary key contains expression with monotonic function and query contains comparison with constant whose type is different. This fixes #12465. (alexey-milovidov).
• Return passed number for numbers with MSB set in function roundUpToPowerOfTwoOrZero(). It prevents potential errors in case of overflow of array sizes. (Azat Khuzhin).
• Fix function if with nullable constexpr as cond that is not literal NULL. Fixes . #13226 ().
• Fix assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. (alexey-milovidov).
• Fix DateTime64 conversion functions with constant argument. (Azat Khuzhin).
• Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes , #12527. (Vitaly Baranov).
• Fix access to redis dictionary after connection was dropped once. It may happen with cache and direct dictionary layouts. (Anton Popov).
• Fix wrong index analysis with functions. It could lead to some data parts being skipped when reading from MergeTree tables. Fixes . Fixes #12406. (Anton Popov).
• Fix error Cannot convert column because it is constant but values of constants are different in source and result for remote queries which use deterministic functions in scope of query, but not deterministic between queries, like now(), now64(), randConstant(). Fixes . #13075 ().
• Fix crash which was possible for queries with ORDER BY tuple and small LIMIT. Fixes #12623. (Nikolai Kochetov).
• Fix Block structure mismatch error for queries with UNION and JOIN. Fixes . #12989 ().
• Corrected merge_with_ttl_timeout logic which did not work well when expiration affected more than one partition over one time interval. (Authored by @excitoon). #12982 ().
• Fix columns duplication for range hashed dictionary created from DDL query. This fixes #10605. (alesapin).
• Fix unnecessary limiting for the number of threads for selects from local replica. (Nikolai Kochetov).
• Fix rare bug when ALTER DELETE and ALTER MODIFY COLUMN queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows in count.txt and as a consequence incorrect data in part. Also, fix a small bug with simultaneous ALTER RENAME COLUMN and ALTER ADD COLUMN. (alesapin).
• Wrong credentials being used when using clickhouse dictionary source to query remote tables. (sundyli).
• Fix CAST(Nullable(String), Enum()). (Azat Khuzhin).
• Fix performance with large tuples, which are interpreted as functions in IN section. The case when user writes WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. (Anton Popov).
• Fix memory tracking for input_format_parallel_parsing (by attaching thread to group). (Azat Khuzhin).
• Fix wrong optimization optimize_move_functions_out_of_any=1 in case of any(func(<lambda>)). (Artem Zuikov).
• Fixed fix bloom filter index with const expression. #12659 ().
• Fix SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 ().
• Add support for function if with Array(UUID) arguments. This fixes #11066. (alexey-milovidov).
• CREATE USER IF NOT EXISTS now doesn’t throw exception if the user exists. This fixes . #12646 ().
• Exception There is no supertype... can be thrown during ALTER ... UPDATE in unexpected cases (e.g. when subtracting from UInt64 column). This fixes #7306. This fixes . #12633 ().
• Fix possible Pipeline stuck error for queries with external sorting. Fixes #12617. (Nikolai Kochetov).
• Fix error Output of TreeExecutor is not sorted for OPTIMIZE DEDUPLICATE. Fixes . #12613 ().
• Fix the issue when alias on result of function any can be lost during query optimization. #12593 ().
• Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 ().
• Now ClickHouse will recalculate checksums for parts when file checksums.txt is absent. Broken since #9827. (alesapin).
• Fix bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes . #12543 ().
• Fixing race condition in live view tables which could cause data duplication. LIVE VIEW is an experimental feature. #12519 ().
• Fix backwards compatibility in binary format of AggregateFunction(avg, ...) values. This fixes #12342. (alexey-milovidov).
• Fix crash in JOIN with dictionary when we are joining over expression of dictionary key: t JOIN dict ON expr(dict.id) = t.id. Disable dictionary join optimisation for this case. (Artem Zuikov).
• Fix overflow when very large LIMIT or OFFSET is specified. This fixes . This fixes #11372. (alexey-milovidov).
• kafka: fix SIGSEGV if there is a message with error in the middle of the batch. (Azat Khuzhin).

Improvement

• Keep smaller amount of logs in ZooKeeper. Avoid excessive growing of ZooKeeper nodes in case of offline replicas when having many servers/tables/inserts. #13100 ().
• Now exceptions forwarded to the client if an error happened during ALTER or mutation. Closes #11329. (alesapin).
• Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds to system.events, along with system.metrics, processes, query_log, etc. (ianton-ru).
• Added SelectedRows and SelectedBytes to system.events, along with system.metrics, processes, query_log, etc. (ianton-ru).
• Added current_database information to system.query_log. (Amos Bird).
• Allow TabSeparatedRaw as input format. (hcz).
• Now joinGet supports multi-key lookup. (Amos Bird).
• Allow *Map aggregate functions to work on Arrays with NULLs. Fixes . #13225 ().
• Avoid overflow in parsing of DateTime values that will lead to negative unix timestamp in their timezone (for example, 1970-01-01 00:00:00 in Moscow). Saturate to zero instead. This fixes #3470. This fixes . #12443 ().
• AvroConfluent: Skip Kafka tombstone records - Support skipping broken records #13203 ().
• Fix wrong error for long queries. It was possible to get syntax error other than Max query size exceeded for correct query. #13928 ().
• Fix data race in lgamma function. This race was caught only in tsan, no side effects really happened. #13842 ().
• Fix a ‘Week’-interval formatting for ATTACH/ALTER/CREATE QUOTA-statements. #13417 ().
• Now broken parts are also reported when encountered in compact part processing. Compact parts is an experimental feature. #13282 ().
• Fix assert in geohashesInBox. This fixes #12554. (alexey-milovidov).
• Fix assert in parseDateTimeBestEffort. This fixes . #13227 ().
• Minor optimization in Processors/PipelineExecutor: breaking out of a loop because it makes sense to do so. #13058 ().
• Support TRUNCATE table without TABLE keyword. #12653 ().
• Fix explain query format overwrite by default. This fixes #12541. (BohuTANG).
• Allow to set JOIN kind and type in more standad way: LEFT SEMI JOIN instead of SEMI LEFT JOIN. For now both are correct. (Artem Zuikov).
• Changes default value for multiple_joins_rewriter_version to 2. It enables new multiple joins rewriter that knows about column names. (Artem Zuikov).
• Add several metrics for requests to S3 storages. (ianton-ru).
• Use correct default secure port for clickhouse-benchmark with --secure argument. This fixes . #12440 ().
• Rollback insertion errors in Log, TinyLog, StripeLog engines. In previous versions insertion error lead to inconsisent table state (this works as documented and it is normal for these table engines). This fixes #12402. (alexey-milovidov).
• Implement RENAME DATABASE and RENAME DICTIONARY for Atomic database engine - Add implicit {uuid} macro, which can be used in ZooKeeper path for ReplicatedMergeTree. It works with CREATE ... ON CLUSTER ... queries. Set show_table_uuid_in_table_create_query_if_not_nil to true to use it. - Make ReplicatedMergeTree engine arguments optional, /clickhouse/tables/{uuid}/{shard}/ and {replica} are used by default. Closes . - Minor fixes. - These changes break backward compatibility of Atomic database engine. Previously created Atomic databases must be manually converted to new format. Atomic database is an experimental feature. #12343 ().
• Separated AWSAuthV4Signer into different logger, removed excessive AWSClient: AWSClient from log messages. #12320 ().
• Better exception message in disk access storage. #12625 ().
• Better exception for function in with invalid number of arguments. #12529 ().
• Fix error message about adaptive granularity. #12624 ().
• Fix SETTINGS parse after FORMAT. #12480 ().
• If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed #7941. (alexey-milovidov).
• Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). (Azat Khuzhin).

Performance Improvement

• Lower memory usage for some operations up to 2 times. #12424 ().
• Optimize PK lookup for queries that match exact PK range. #12277 ().
• Slightly optimize very short queries with LowCardinality. #14129 ().
• Slightly improve performance of aggregation by UInt8/UInt16 keys. #13091 and (alexey-milovidov).
• Push down LIMIT step for query plan (inside subqueries). (Nikolai Kochetov).
• Parallel primary key lookup and skipping index stages on parts, as described in . #12589 ().
• Converting String-type arguments of function “if” and “transform” into enum if set optimize_if_transform_strings_to_enum = 1. #12515 ().
• Replaces monotonic functions with its argument in ORDER BY if set optimize_monotonous_functions_in_order_by=1. #12467 ().
• Add order by optimization that rewrites ORDER BY x, f(x) with ORDER by x if set optimize_redundant_functions_in_order_by = 1. #12404 ().
• Allow pushdown predicate when subquery contains WITH clause. This fixes #12293 (Winter Zhang).
• Improve performance of reading from compact parts. Compact parts is an experimental feature. (Anton Popov).
• Attempt to implement streaming optimization in DiskS3. DiskS3 is an experimental feature. (Vladimir Chebotarev).

Build/Testing/Packaging Improvement

• Use shellcheck for sh tests linting. #13200 (alexey-milovidov).
• Add script which set labels for pull requests in GitHub hook. (alesapin).
• Remove some of recursive submodules. See . #13379 ().
• Ensure that all the submodules are from proper URLs. Continuation of #13379. This fixes . #13397 ().
• Added support for user-declared settings, which can be accessed from inside queries. This is needed when ClickHouse engine is used as a component of another system. #13013 ().
• Added testing for RBAC functionality of INSERT privilege in TestFlows. Expanded tables on which SELECT is being tested. Added Requirements to match new table engine tests. #13340 ().
• Fix timeout error during server restart in the stress test. #13321 ().
• Now fast test will wait server with retries. #13284 ().
• Function materialize() (the function for ClickHouse testing) will work for NULL as expected - by transforming it to non-constant column. #13212 ().
• Fix libunwind build in AArch64. This fixes #13204. (alexey-milovidov).
• Even more retries in zkutil gtest to prevent test flakiness. (alexey-milovidov).
• Small fixes to the RBAC TestFlows. (vzakaznikov).
• Fixing 00960_live_view_watch_events_live.py test. (vzakaznikov).
• Improve cache purge in documentation deploy script. (alesapin).
• Rewrote some orphan tests to gtest. Removed useless includes from tests. (Nikita Mikhaylov).
• Added tests for RBAC functionality of SELECT privilege in TestFlows. (Ritaank Tiwari).
• Rerun some tests in fast test check. (alesapin).
• Fix MSan error in “rdkafka” library. This closes . Updated rdkafka to version 1.5 (master). #12991 ().
• Fix UBSan report in base64 if tests were run on server with AVX-512. This fixes #12318. Author: @qoega. (alexey-milovidov).
• Fix UBSan report in HDFS library. This closes . #12453 ().
• Check an ability that we able to restore the backup from an old version to the new version. This closes #8979. (alesapin).
• Do not build helper_container image inside integrational tests. Build docker container in CI and use pre-built helper_container in integration tests. (Ilya Yatsishin).
• Add a test for ALTER TABLE CLEAR COLUMN query for primary key columns. (alesapin).
• Increased timeouts in testflows tests. (vzakaznikov).
• Fix build of test under Mac OS X. This closes . #12772 ().
• Connector-ODBC updated to mysql-connector-odbc-8.0.21. #12739 ().
• Adding RBAC syntax tests in TestFlows. #12642 ().
• Improve performance of TestKeeper. This will speedup tests with heavy usage of Replicated tables. #12505 ().
• Now we check that server is able to start after stress tests run. This fixes #12473. (alesapin).
• Update fmtlib to master (7.0.1). (alexey-milovidov).
• Add docker image for fast tests. (alesapin).
• Rework configuration paths for integration tests. (Ilya Yatsishin).
• Add compiler option to control that stack frames are not too large. This will help to run the code in fibers with small stack size. (alexey-milovidov).
• Update gitignore-files. (vladimir-golovchenko).

ClickHouse release 20.6

ClickHouse release v20.6.3.28-stable

New Feature

• Added an initial implementation of EXPLAIN query. Syntax: EXPLAIN SELECT .... This fixes #1118. (Nikolai Kochetov).
• Added storage RabbitMQ. (Kseniia Sumarokova).
• Implemented PostgreSQL-like ILIKE operator for . #12125 ().
• Supported RIGHT and FULL JOIN with SET join_algorithm = 'partial_merge'. Only ALL strictness is allowed (ANY, SEMI, ANTI, ASOF are not). #12118 ().
• Added a function initializeAggregation to initialize an aggregation based on a single value. #12109 ().
• Supported ALTER TABLE ... [ADD|MODIFY] COLUMN ... FIRST #4006. (Winter Zhang).
• Added function parseDateTimeBestEffortUS. (flynn).
• Support format ORC for output (was supported only for input). (Kruglov Pavel).

Bug Fix

• Fixed aggregate function any(x) is found inside another aggregate function in query error with SET optimize_move_functions_out_of_any = 1 and aliases inside any(). #13419 ().
• Fixed PrettyCompactMonoBlock for clickhouse-local. Fixed extremes/totals with PrettyCompactMonoBlock. This fixes #7746. (Azat Khuzhin).
• Fixed possible error Totals having transform was already added to pipeline in case of a query from delayed replica. (Nikolai Kochetov).
• The server may crash if user passed specifically crafted arguments to the function h3ToChildren. This fixes . #13277 ().
• Fixed potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. (alexey-milovidov).
• Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes . #13226 ().
• Fixed assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. (alexey-milovidov).
• Fixed DateTime64 conversion functions with constant argument. (Azat Khuzhin).
• Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from MergeTree tables. Fixes . Fixes #12406. (Anton Popov).
• Fixed error Cannot convert column because it is constant but values of constants are different in source and result for remote queries which use deterministic functions in scope of query, but not deterministic between queries, like now(), now64(), randConstant(). Fixes . #13075 ().
• Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 ().
• Fixed rare bug when ALTER DELETE and ALTER MODIFY COLUMN queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows in count.txt and as a consequence incorrect data in part. Also, fix a small bug with simultaneous ALTER RENAME COLUMN and ALTER ADD COLUMN. #12760 ().
• Fixed CAST(Nullable(String), Enum()). #12745 ().
• Fixed a performance with large tuples, which are interpreted as functions in IN section. The case when user write WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. #12700 ().
• Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 ().
• Fixed bloom filter index with const expression. This fixes #10572. (Winter Zhang).
• Fixed SIGSEGV in StorageKafka when broker is unavailable (and not only). (Azat Khuzhin).
• Added support for function if with Array(UUID) arguments. This fixes . #12648 ().
• CREATE USER IF NOT EXISTS now doesn’t throw exception if the user exists. This fixes #12507. (Vitaly Baranov).
• Better exception message in disk access storage. (alesapin).
• The function groupArrayMoving* was not working for distributed queries. It’s result was calculated within incorrect data type (without promotion to the largest type). The function groupArrayMovingAvg was returning integer number that was inconsistent with the avg function. This fixes . #12622 ().
• Fixed lack of aliases with function any. #12593 ().
• Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 ().
• Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 ().
• Fixed bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes #12536. (alesapin).
• Better exception for function in with invalid number of arguments. (Anton Popov).
• Fixing race condition in live view tables which could cause data duplication. (vzakaznikov).
• Fixed performance issue, while reading from compact parts. (Anton Popov).
• Fixed backwards compatibility in binary format of AggregateFunction(avg, ...) values. This fixes . #12486 ().
• Fixed SETTINGS parse after FORMAT. #12480 ().
• Fixed the deadlock if text_log is enabled. #12452 ().
• Fixed overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes . #12427 ().
• Fixed possible segfault if StorageMerge. This fixes #12054. (tavplubix).
• Reverted change introduced in to resolve #12098. (Mike).
• Additional check for arguments of bloom filter index. This fixes . #12388 ().
• Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. (alexey-milovidov).
• Allowed to CLEAR column even if there are depending DEFAULT expressions. This fixes . #12378 ().
• Fix TOTALS/ROLLUP/CUBE for aggregate functions with -State and Nullable arguments. This fixes #12163. (alexey-milovidov).
• Fixed error message and exit codes for ALTER RENAME COLUMN queries, when RENAME is not allowed. Fixes and #12303. (alesapin).
• Fixed very rare race condition in ReplicatedMergeTreeQueue. (alexey-milovidov).
• When using codec Delta or DoubleDelta with non fixed width types, exception with code LOGICAL_ERROR was returned instead of exception with code BAD_ARGUMENTS (we ensure that exceptions with code logical error never happen). This fixes . #12308 ().
• Fixed order of columns in WITH FILL modifier. Previously order of columns of ORDER BY statement wasn’t respected. #12306 ().
• Avoid “bad cast” exception when there is an expression that filters data by virtual columns (like _table in Merge tables) or by “index” columns in system tables such as filtering by database name when querying from system.tables, and this expression returns Nullable type. This fixes #12166. (alexey-milovidov).
• Fixed TTL after renaming column, on which depends TTL expression. (Anton Popov).
• Fixed SIGSEGV if there is an message with error in the middle of the batch in Kafka Engine. (Azat Khuzhin).
• Fixed the situation when some threads might randomly hang for a few seconds during DNS cache updating. (tavplubix).
• Fixed typo in setting name. (alexey-milovidov).
• Show error after TrieDictionary failed to load. (Vitaly Baranov).
• The function arrayFill worked incorrectly for empty arrays that may lead to crash. This fixes . #12279 ().
• Implement conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes . #12275 ().
• Fixed the behaviour on reaching redirect limit in request to S3 storage. #12256 ().
• Fixed the behaviour when during multiple sequential inserts in StorageFile header for some special types was written more than once. This fixed #6155. (Nikita Mikhaylov).
• Fixed logical functions for UInt8 values when they are not equal to 0 or 1. (Alexander Kazakov).
• Cap max_memory_usage* limits to the process resident memory. (Azat Khuzhin).
• Fix dictGet arguments check during GROUP BY injective functions elimination. (Azat Khuzhin).
• Fixed the behaviour when SummingMergeTree engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes . #12173 ().
• Don’t split the dictionary source’s table name into schema and table name itself if ODBC connection doesn’t support schema. #12165 ().
• Fixed wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes . #12153 ().
• Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. (alexey-milovidov).
• Fixed bad code in redundant ORDER BY optimization. The bug was introduced in . #12148 ().
• Fixed potential overflow in integer division. This fixes #12119. (alexey-milovidov).
• Fixed potential infinite loop in greatCircleDistance, geoDistance. This fixes . #12137 ().
• Normalize “pid” file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. (alexey-milovidov).
• Fixed bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes . #12121 ().
• Avoid “There is no query” exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 ().
• Fixed handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes . #12116 ().
• Format Parquet now properly works with LowCardinality and LowCardinality(Nullable) types. Fixes #12086, . #12108 ().
• Fixed performance for selects with UNION caused by wrong limit for the total number of threads. Fixes #12030. (Nikolai Kochetov).
• Fixed segfault with -StateResample combinators. (Anton Popov).
• Fixed empty result_rows and result_bytes metrics in system.quey_log for selects. Fixes . #12089 ().
• Fixed unnecessary limiting the number of threads for selects from VIEW. Fixes #11937. (Nikolai Kochetov).
• Fixed SIGSEGV in StorageKafka on DROP TABLE. (Azat Khuzhin).
• Fixed possible crash while using wrong type for PREWHERE. Fixes , #12060. (Nikolai Kochetov).
• Fixed error Cannot capture column for higher-order functions with Tuple(LowCardinality) argument. Fixes . #12055 ().
• Fixed constraints check if constraint is a constant expression. This fixes #11360. (alexey-milovidov).
• Fixed wrong result and potential crash when invoking function if with arguments of type FixedString with different sizes. This fixes . #12021 ().

Improvement

• Allowed to set JOIN kind and type in more standard way: LEFT SEMI JOIN instead of SEMI LEFT JOIN. For now both are correct. (Artem Zuikov).
• lifetime_rows/lifetime_bytes for Buffer engine. (Azat Khuzhin).
• Write the detail exception message to the client instead of ‘MySQL server has gone away’. (BohuTANG).
• Allows to change a charset which is used for printing grids borders. Available charsets are following: UTF-8, ASCII. Setting output_format_pretty_grid_charset enables this feature. (Sabyanin Maxim).
• Supported MySQL ‘SELECT DATABASE()’ 2. Add MySQL replacement query integration test. #12314 ().
• Added KILL QUERY [connection_id] for the MySQL client/driver to cancel the long query, issue #12038. (BohuTANG).
• Added support for %g (two digit ISO year) and %G (four digit ISO year) substitutions in formatDateTime function. (vivarum).
• Added ‘type’ column in system.disks. (ianton-ru).
• Improved REVOKE command: now it requires grant/admin option for only access which will be revoked. For example, to execute REVOKE ALL ON *.* FROM user1 now it doesn’t require to have full access rights granted with grant option. Added command REVOKE ALL FROM user1 - it revokes all granted roles from user1. (Vitaly Baranov).
• Added replica priority for load_balancing (for manual prioritization of the load balancing). (Azat Khuzhin).
• Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. (Vladimir Chebotarev).

Performance Improvement

• Improved performace of ‘ORDER BY’ and ‘GROUP BY’ by prefix of sorting key (enabled with optimize_aggregation_in_order setting, disabled by default). #11696 ().
• Removed injective functions inside uniq*() if set optimize_injective_functions_inside_uniq=1. #12337 ().
• Index not used for IN operator with literals”, performance regression introduced around v19.3. This fixes “#10574. (nvartolomei).
• Implemented single part uploads for DiskS3 (experimental feature). (Vladimir Chebotarev).

Experimental Feature

• Added new in-memory format of parts in MergeTree-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholds min_rows_for_compact_part and min_bytes_for_compact_part. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by setting in_memory_parts_enable_wal. #10697 ().

Build/Testing/Packaging Improvement

• Implement AST-based query fuzzing mode for clickhouse-client. See for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and 00746_sql_fuzzy.pl. #12111 ().
• Add new type of tests based on Testflows framework. #12090 ().
• Added S3 HTTPS integration test. #12412 ().
• Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. #12313 ().
• Now functional and stress tests will be able to run with old version of clickhouse-test script. #12287 ().
• Remove strange file creation during build in orc. #12258 ().
• Place common docker compose files to integration docker container. #12168 ().
• Fix warnings from CodeQL. CodeQL is another static analyzer that we will use along with clang-tidy and PVS-Studio that we use already. #12138 ().
• Minor CMake fixes for UNBUNDLED build. #12131 ().
• Added a showcase of the minimal Docker image without using any Linux distribution. #12126 ().
• Perform an upgrade of system packages in the clickhouse-server docker image. #12124 ().
• Add UNBUNDLED flag to system.build_options table. Move skip lists for clickhouse-test to clickhouse repo. #12107 ().
• Regular check by Anchore Container Analysis security analysis tool that looks for in clickhouse-server Docker image. Also confirms that Dockerfile is buildable. Runs daily on master and on pull-requests to Dockerfile. #12102 ().
• Daily check by GitHub CodeQL security analysis tool that looks for . #12101 ().
• Install ca-certificates before the first apt-get update in Dockerfile. #12095 ().

ClickHouse release v20.5.4.40-stable 2020-08-10

Bug Fix

• Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from MergeTree tables. Fixes #13060. Fixes . #13081 ().
• Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 ().
• Fixed performance with large tuples, which are interpreted as functions in IN section. The case when user write WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. #12700 ().
• Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 ().
• Fixed bloom filter index with const expression. This fixes #10572. (Winter Zhang).
• Fixed SIGSEGV in StorageKafka when broker is unavailable (and not only). (Azat Khuzhin).
• Added support for function if with Array(UUID) arguments. This fixes . #12648 ().
• Fixed lack of aliases with function any. #12593 ().
• Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 ().
• Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 ().
• Fixed bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes #12536. (alesapin).
• Better exception for function in with invalid number of arguments. (Anton Popov).
• Fixed race condition in live view tables which could cause data duplication. (vzakaznikov).
• Fixed performance issue, while reading from compact parts. (Anton Popov).
• Fixed backwards compatibility in binary format of AggregateFunction(avg, ...) values. This fixes . #12486 ().
• Fixed the deadlock if text_log is enabled. #12452 ().
• Fixed overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes . #12427 ().
• Fixed possible segfault if StorageMerge. Closes #12054. (tavplubix).
• Reverts change introduced in to resolve #12098. (Mike).
• Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes . #12384 ().
• Allow to CLEAR column even if there are depending DEFAULT expressions. This fixes #12333. (alexey-milovidov).
• Fixed TOTALS/ROLLUP/CUBE for aggregate functions with -State and Nullable arguments. This fixes . #12376 ().
• Fixed SIGSEGV if there is an message with error in the middle of the batch in Kafka Engine. #12302 ().
• Fixed the behaviour when SummingMergeTree engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. (Nikita Mikhaylov).
• Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes . #12151 ().
• Fixed bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. (alesapin).
• Fixed unnecessary limiting the number of threads for selects from VIEW. Fixes . #12085 ().
• Fixed crash in JOIN with LowCardinality type with join_algorithm=partial_merge. #12035 ().
• Fixed wrong result for if() with NULLs in condition. #11807 ().

Performance Improvement

• Index not used for IN operator with literals”, performance regression introduced around v19.3. This fixes “. #12062 ().

Build/Testing/Packaging Improvement

• Install ca-certificates before the first apt-get update in Dockerfile. (Ivan Blinkov).

ClickHouse release v20.5.2.7-stable 2020-07-02

Backward Incompatible Change

• Return non-Nullable result from COUNT(DISTINCT), and uniq aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. (alexey-milovidov).
• Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in users.xml inside <profile> section for specific user profile (or in <default> for default settings). The server won’t start with exception message in log. This fixes . If you want to skip the check, you can either move settings to the appropriate place or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> to config.xml. #11449 ().
• The setting input_format_with_names_use_header is enabled by default. It will affect parsing of input formats -WithNames and -WithNamesAndTypes. #10937 ().
• Remove experimental_use_processors setting. It is enabled by default. #10924 ().
• Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages Data after merge is not byte-identical to data on another replicas. with explanation. These messages are Ok and you should not worry. This change is backward compatible but we list it here in changelog in case you will wonder about these messages. #10663 ().
• Added a check for meaningless codecs and a setting allow_suspicious_codecs to control this check. This closes #4966. (alexey-milovidov).
• Several Kafka setting changes their defaults. See .

New Feature

• TTL DELETE WHERE and TTL GROUP BY for automatic data coarsening and rollup in tables. (expl0si0nn).
• Implementation of PostgreSQL wire protocol. (Movses).
• Added system tables for users, roles, grants, settings profiles, quotas, row policies; added commands SHOW USER, SHOW [CURRENT|ENABLED] ROLES, SHOW SETTINGS PROFILES. (Vitaly Baranov).
• Support writes in ODBC Table function (ageraab). (tavplubix).
• Add query performance metrics based on Linux perf_events (these metrics are calculated with hardware CPU counters and OS counters). It is optional and requires CAP_SYS_ADMIN to be set on clickhouse binary. Andrey Skobtsov. (Alexander Kuzmenkov).
• Now support NULL and NOT NULL modifiers for data types in CREATE query. (Павел Потемкин).
• Add ArrowStream input and output format. (hcz).
• Support Cassandra as external dictionary source. (favstovol).
• Added a new layout direct which loads all the data directly from the source for each query, without storing or caching data. (Artem Streltsov).
• Added new complex_key_direct layout to dictionaries, that does not store anything locally during query execution. (Artem Streltsov).
• Added support for MySQL style global variables syntax (stub). This is needed for compatibility of MySQL protocol. (alexey-milovidov).
• Added syntax highligting to clickhouse-client using replxx. (Tagir Kuskarov).
• minMap and maxMap functions were added. (Ildus Kurbangaliev).
• Add the system.asynchronous_metric_log table that logs historical metrics from system.asynchronous_metrics. (Alexander Kuzmenkov).
• Add functions extractAllGroupsHorizontal(haystack, re) and extractAllGroupsVertical(haystack, re). (Vasily Nemkov).
• Add SHOW CLUSTER(S) queries. (hexiaoting).
• Add netloc function for extracting network location, similar to urlparse(url), netloc in python. (Guillaume Tassery).
• Add 2 more virtual columns for engine=Kafka to access message headers. (filimonov).
• Add _timestamp_ms virtual column for Kafka engine (type is Nullable(DateTime64(3))). (filimonov).
• Add function randomFixedString. (Andrei Nekrashevich).
• Add function fuzzBits that randomly flips bits in a string with given probability. (Andrei Nekrashevich).
• Allow comparison of numbers with constant string in comparison operators, IN and VALUES sections. (alexey-milovidov).
• Add round_robin load_balancing mode. (Azat Khuzhin).
• Add cast_keep_nullable setting. If set CAST(something_nullable AS Type) return Nullable(Type). (Artem Zuikov).
• Added column position to system.columns table and column_position to system.parts_columns table. It contains ordinal position of a column in a table starting with 1. This closes . #11655 ().
• ON CLUSTER support for SYSTEM {FLUSH DISTRIBUTED,STOP/START DISTRIBUTED SEND}. #11415 ().
• Add system.distribution_queue table. #11394 ().
• Support for all format settings in Kafka, expose some setting on table level, adjust the defaults for better performance. #11388 ().
• Add port function (to extract port from URL). #11120 ().
• Now dictGet* functions accept table names. #11050 ().
• The clickhouse-format tool is now able to format multiple queries when the -n argument is used. #10852 ().
• Possibility to configure proxy-resolver for DiskS3. #10744 ().
• Make pointInPolygon work with non-constant polygon. PointInPolygon now can take Array(Array(Tuple(…, …))) as second argument, array of polygon and holes. #10623 () #11421 ().
• Added move_ttl_info to system.parts in order to provide introspection of move TTL functionality. #10591 ().
• Possibility to work with S3 through proxies. #10576 ().
• Add NCHAR and NVARCHAR synonims for data types. #11025 ().
• Resolved #7224: added FailedQuery, FailedSelectQuery and FailedInsertQuery metrics to system.events table. (Nikita Orlov).
• Add more jemalloc statistics to system.asynchronous_metrics, and ensure that we see up-to-date values for them. (Alexander Kuzmenkov).
• Allow to specify default S3 credentials and custom auth headers. (Grigory Pervakov).
• Added new functions to import/export DateTime64 as Int64 with various precision: to-/fromUnixTimestamp64Milli/-Micro/-Nano. (Vasily Nemkov).
• Allow specifying mongodb:// URI for MongoDB dictionaries. (Alexander Kuzmenkov).
• OFFSET keyword can now be used without an affiliated LIMIT clause. (Guillaume Tassery).
• Added system.licenses table. This table contains licenses of third-party libraries that are located in contrib directory. This closes . #10795 ().
• New function function toStartOfSecond(DateTime64) -> DateTime64 that nullifies sub-second part of DateTime64 value. #10722 ().
• Add new input format JSONAsString that accepts a sequence of JSON objects separated by newlines, spaces and/or commas. #10607 ().
• Allowed to profile memory with finer granularity steps than 4 MiB. Added sampling memory profiler to capture random allocations/deallocations. #10598 ().
• SimpleAggregateFunction now also supports sumMap. #10000 ().
• Support ALTER RENAME COLUMN for the distributed table engine. Continuation of #10727. Fixes . #10887 ().

Bug Fix

• Fix UBSan report in Decimal parse. This fixes . #10512 ().
• Fix potential floating point exception when parsing DateTime64. This fixes #11374. (alexey-milovidov).
• Fix rare crash caused by using Nullable column in prewhere condition. #11608 (Nikolai Kochetov).
• Don’t allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes . #11846 ().
• Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. (alexey-milovidov).
• Fix wrong result for if with NULLs in condition. (Artem Zuikov).
• Fix using too many threads for queries. (Nikolai Kochetov).
• Fixed Scalar doesn't exist exception when using WITH <scalar subquery> ... in SELECT ... FROM merge_tree_table ... . #11767 ().
• Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 ().
• Now replicated fetches will be cancelled during metadata alter. #11744 ().
• Parse metadata stored in zookeeper before checking for equality. #11739 ().
• Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 ().
• Fix ORDER BY ... WITH FILL over const columns. #11697 ().
• Fix very rare race condition in SYSTEM SYNC REPLICA. If the replicated table is created and at the same time from the separate connection another client is issuing SYSTEM SYNC REPLICA command on that table (this is unlikely, because another client should be aware that the table is created), it’s possible to get nullptr dereference. #11691 ().
• Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 ().
• Fix LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. #11689 ().
• Fix possible Pipeline stuck for selects with parallel FINAL. Fixes #11636. (Nikolai Kochetov).
• Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes . #11681 ().
• Fix syntax hilite in CREATE USER query. #11664 ().
• Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes . #11649 ().
• Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. (alexey-milovidov).
• Fix bloom filters for String (data skipping indices). (Azat Khuzhin).
• Without -q option the database does not get created at startup. (giordyb).
• Fix error Block structure mismatch for queries with sampling reading from Buffer table. (Nikolai Kochetov).
• Fix wrong exit code of the clickhouse-client, when exception.code() % 256 == 0. (filimonov).
• Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree. Continue to work if the table was not removed completely from ZooKeeper or not created successfully. This fixes . #11592 ().
• Fix trivial error in log message about “Mark cache size was lowered” at server startup. This closes #11399. (alexey-milovidov).
• Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. (Nikolai Kochetov).
• Fixed rare segfault in SHOW CREATE TABLE Fixes . #11579 ().
• All queries in HTTP session have had the same query_id. It is fixed. #11578 ().
• Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 ().
• Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. (alexey-milovidov).
• Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). (Azat Khuzhin).
• Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication. (Azat Khuzhin).
• Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes . #11496 ().
• Fix Pipeline stuck exception for INSERT SELECT FINAL where SELECT (max_threads>1) has multiple streams but INSERT has only one (max_insert_threads\==0). #11455 ().
• Fix wrong result in queries like select count() from t, u. #11454 ().
• Fix return compressed size for codecs. #11448 ().
• Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. (alesapin).
• Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. (alexey-milovidov).
• Fix crash in JOIN over LowCarinality(T) and Nullable(T). . #11414 ().
• Fix error code for wrong USING key. #11373. (Artem Zuikov).
• Fixed geohashesInBox with arguments outside of latitude/longitude range. (Vasily Nemkov).
• Better errors for joinGet() functions. (Artem Zuikov).
• Fix possible Pipeline stuck error for queries with external sort and limit. Fixes . #11366 ().
• Remove redundant lock during parts send in ReplicatedMergeTree. #11354 ().
• Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. (alexey-milovidov).
• Fix potential segfault when using Lazy database. (alexey-milovidov).
• Fix crash in direct selects from Join table engine (without JOIN) and wrong nullability. (Artem Zuikov).
• Fix crash in quantilesExactWeightedArray. (Nikolai Kochetov).
• Now merges stopped before change metadata in ALTER queries. (alesapin).
• Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes . #11330 ().
• Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 ().
• Fix very rare race condition in ThreadPool. #11314 ().
• Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 ().
• Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 ().
• Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. (alexey-milovidov).
• Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes . This fixes #5732. This fixes . Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 ().
• Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. (Vitaly Baranov).
• Fix crash while reading malformed data in Protobuf format. This fixes , fixes https://github.com/ClickHouse/ClickHouse/issues/11203. (Vitaly Baranov).
• Fixed a bug when cache dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. (Nikita Mikhaylov).
• Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes . #11205 ().
• Fix possible exception Invalid status for associated output. #11200 ().
• Now primary.idx will be checked if it’s defined in CREATE query. #11199 ().
• Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 ().
• Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 ().
• If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 ().
• Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 ().
• Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 ().
• Remove logging from mutation finalization task if nothing was finalized. #11109 ().
• Fixed deadlock during server startup after update with changes in structure of system log tables. #11106 ().
• Fixed memory leak in registerDiskS3. #11074 ().
• Fix error No such name in Block::erase() when JOIN appears with PREWHERE or optimize_move_to_prewhere makes PREWHERE from WHERE. #11051 ().
• Fixes the potential missed data during termination of Kafka engine table. #11048 ().
• Fixed parseDateTime64BestEffort argument resolution bugs. #10925. (Vasily Nemkov).
• Now it’s possible to ADD/DROP and RENAME the same one column in a single ALTER query. Exception message for simultaneous MODIFY and RENAME became more clear. Partially fixes . #11037 ().
• Fixed parsing of S3 URLs. #11036 ().
• Fix memory tracking for two-level GROUP BY when there is a LIMIT. #11022 ().
• Fix very rare potential use-after-free error in MergeTree if table was not created successfully. #10986 ().
• Fix metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 ().
• Fix server crash on concurrent ALTER and DROP DATABASE queries with Atomic database engine. #10968 ().
• Fix incorrect raw data size in method getRawData(). #10964 ().
• Fix incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 ().
• Avoid sending partially written files by the DistributedBlockOutputStream. #10940 ().
• Fix crash in SELECT count(notNullIn(NULL, [])). #10920 ().
• Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #10910 ().
• Now it’s possible to execute multiple ALTER RENAME like a TO b, c TO a. #10895 ().
• Fix possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way (which I found) it can happen is when you use finalizeAggregation function while reading from table with Memory engine which stores AggregateFunction state for quanite* function. #10890 ().
• Fix backward compatibility with tuples in Distributed tables. #10889 ().
• Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 ().
• Fixed WATCH hangs after LiveView table was dropped from database with Atomic engine. #10859 ().
• Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 ().
• Now constraints are updated if the column participating in CONSTRAINT expression was renamed. Fixes #10844. (alesapin).
• Fix potential read of uninitialized memory in cache dictionary. (alexey-milovidov).
• Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). (Azat Khuzhin).
• Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes . #10821 ().
• Fix UBSan and MSan report in DateLUT. #10798 ().
• Make use of src_type for correct type conversion in key conditions. Fixes #6287. (Andrew Onyshchuk).
• Get rid of old libunwind patches. This allows to disable -fno-omit-frame-pointer in clang builds that improves performance at least by 1% in average. #10761 ().
• Fix avgWeighted when using floating-point weight over multiple shards. #10758 ().
• Fix parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. (Nikolai Kochetov).
• Fix combinator -OrNull and -OrDefault when combined with -State. (hcz).
• Fix crash in generateRandom with nested types. Fixes . #10734 ().
• Fix data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. (Nikolai Kochetov).
• Fix usage of primary key wrapped into a function with ‘FINAL’ modifier and ‘ORDER BY’ optimization. (Anton Popov).
• Fix possible buffer overflow in function h3EdgeAngle. (alexey-milovidov).
• Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes . #10698 ().
• Fix atomicity of HTTP insert. This fixes #9666. (Andrew Onyshchuk).
• Fix multiple usages of IN operator with the identical set in one query. (Anton Popov).
• Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes , #7019, , #7091. (tavplubix).
• Fix order of parameters in AggregateTransform constructor. (palasonic1).
• Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes . #10664 ().
• Fix possible incorrect number of rows for queries with LIMIT. Fixes #10566, . #10660 ().
• Fix bug which locks concurrent alters when table has a lot of parts. #10659 ().
• Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 ().
• Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, . #10621 ().
• Fix optimize_skip_unused_shards with LowCardinality. #10611 ().
• Fix segfault in StorageBuffer when exception on server startup. Fixes #10550. (tavplubix).
• On SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. (tavplubix).
• Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. (Nikolai Kochetov).
• Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. (Vladimir Chebotarev).
• Fixes possible crash createDictionary() is called before loadStoredObject() has finished. (Vitaly Baranov).
• Fix error the BloomFilter false positive must be a double number between 0 and 1 . #10569 ().
• Fix SELECT of column ALIAS which default expression type different from column type. #10563 ().
• Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 ().
• Fix index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 ().
• Disable GROUP BY sharding_key optimization by default (optimize_distributed_group_by_sharding_key had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is if in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 ().
• Fixes: #10263 (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess): internal_replication == false, multiple local shards (activates the hardlinking code) and distributed_storage_policy (makes link(2) fails on EXDEV)). #10486 ().
• Fixed error with “max_rows_to_sort” limit. #10268 ().
• Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 ().

Improvement

• Apply TTL for old data, after ALTER MODIFY TTL query. This behaviour is controlled by setting materialize_ttl_after_modify, which is enabled by default. (Anton Popov).
• When parsing C-style backslash escapes in string literals, VALUES and various text formats (this is an extension to SQL standard that is endemic for ClickHouse and MySQL), keep backslash if unknown escape sequence is found (e.g. \% or \w) that will make usage of LIKE and match regular expressions more convenient (it’s enough to write name LIKE 'used\_cars' instead of name LIKE 'used\\_cars') and more compatible at the same time. This fixes . #11208 ().
• When reading Decimal value, cut extra digits after point. This behaviour is more compatible with MySQL and PostgreSQL. This fixes #10202. (alexey-milovidov).
• Allow to DROP replicated table if the metadata in ZooKeeper was already removed and does not exist (this is also the case when using TestKeeper for testing and the server was restarted). Allow to RENAME replicated table even if there is an error communicating with ZooKeeper. This fixes . #11652 ().
• Slightly improve diagnostic of reading decimal from string. This closes #10202. (alexey-milovidov).
• Fix sleep invocation in signal handler. It was sleeping for less amount of time than expected. (alexey-milovidov).
• (Only Linux) OS related performance metrics (for CPU and I/O) will work even without CAP_NET_ADMIN capability. (Alexander Kazakov).
• Added hostname as an alias to function hostName. This feature was suggested by Victor Tarnavskiy from Yandex.Metrica. (alexey-milovidov).
• Added support for distributed DDL (update/delete/drop partition) on cross replication clusters. (Nikita Mikhaylov).
• Emit warning instead of error in server log at startup if we cannot listen one of the listen addresses (e.g. IPv6 is unavailable inside Docker). Note that if server fails to listen all listed addresses, it will refuse to startup as before. This fixes . #11687 ().
• Default user and database creation on docker image starting. #10637 ().
• When multiline query is printed to server log, the lines are joined. Make it to work correct in case of multiline string literals, identifiers and single-line comments. This fixes #3853. (alexey-milovidov).
• Multiple names are now allowed in commands: CREATE USER, CREATE ROLE, ALTER USER, SHOW CREATE USER, SHOW GRANTS and so on. (Vitaly Baranov).
• Add support for distributed DDL (UPDATE/DELETE/DROP PARTITION) on cross replication clusters. (frank lee).
• Clear password from command line in clickhouse-client and clickhouse-benchmark if the user has specified it with explicit value. This prevents password exposure by ps and similar tools. (alexey-milovidov).
• Don’t use debug info from ELF file if it doesn’t correspond to the running binary. It is needed to avoid printing wrong function names and source locations in stack traces. This fixes . #11657 ().
• Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. (alexey-milovidov).
• Skip empty parameters in requested URL. They may appear when you write http://localhost:8123/?&a=b or http://localhost:8123/?a=b&&c=d. This closes . #11651 ().
• Allow using groupArrayArray and groupUniqArrayArray as SimpleAggregateFunction. #11650 ().
• Allow comparison with constant strings by implicit conversions when analysing index conditions on other types. This may close #11630. (alexey-milovidov).
• Support config default HTTPHandlers. #11628 ().
• Make more input formats to work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when kafka_num_consumers is greater than number of partitions in topic. #11599 ().
• Improve multiple_joins_rewriter_version=2 logic. Fix unknown columns error for lambda aliases. #11587 ().
• Better exception message when cannot parse columns declaration list. This closes #10403. (alexey-milovidov).
• Improve enable_optimize_predicate_expression=1 logic for VIEW. (Artem Zuikov).
• Adding support for PREWHERE in live view tables. (vzakaznikov).
• Automatically update DNS cache, which is used to check if user is allowed to connect from an address. (tavplubix).
• OPTIMIZE FINAL will force merge even if concurrent merges are performed. This closes and closes #11322. (alexey-milovidov).
• Suppress output of cancelled queries in clickhouse-client. In previous versions result may continue to print in terminal even after you press Ctrl+C to cancel query. This closes . #11342 ().
• Now history file is updated after each query and there is no race condition if multiple clients use one history file. This fixes #9897. (Tagir Kuskarov).
• Better log messages in while reloading configuration. (alexey-milovidov).
• Remove trailing whitespaces from formatted queries in clickhouse-client or clickhouse-format in some cases. (alexey-milovidov).
• Add setting “output_format_pretty_max_value_width”. If value is longer, it will be cut to avoid output of too large values in terminal. This closes . #11324 ().
• Better exception message in case when there is shortage of memory mappings. This closes #11027. (alexey-milovidov).
• Support (U)Int8, (U)Int16, Date in ASOF JOIN. (Artem Zuikov).
• Support kafka_client_id parameter for Kafka tables. It also changes the default client.id used by ClickHouse when communicating with Kafka to be more verbose and usable. (filimonov).
• Keep the value of DistributedFilesToInsert metric on exceptions. In previous versions, the value was set when we are going to send some files, but it is zero, if there was an exception and some files are still pending. Now it corresponds to the number of pending files in filesystem. (alexey-milovidov).
• Add support for multi-word data type names (such as DOUBLE PRECISION and CHAR VARYING) for better SQL compatibility. (Павел Потемкин).
• Provide synonyms for some data types. (Павел Потемкин).
• The query log is now enabled by default. (Ivan Blinkov).
• Show authentication type in table system.users and while executing SHOW CREATE USER query. (Vitaly Baranov).
• Remove data on explicit DROP DATABASE for Memory database engine. Fixes . #11021 ().
• Set thread names for internal threads of rdkafka library. Make logs from rdkafka available in server logs. #10983 ().
• Support for unicode whitespaces in queries. This helps when queries are copy-pasted from Word or from web page. This fixes #10896. (alexey-milovidov).
• Allow large UInt types as the index in function tupleElement. (hcz).
• Respect prefer_localhost_replica/load_balancing on INSERT into Distributed. (Azat Khuzhin).
• Introduce min_insert_block_size_rows_for_materialized_views, min_insert_block_size_bytes_for_materialized_views settings. This settings are similar to min_insert_block_size_rows and min_insert_block_size_bytes, but applied only for blocks inserted into MATERIALIZED VIEW. It helps to control blocks squashing while pushing to MVs and avoid excessive memory usage. (Azat Khuzhin).
• Get rid of exception from replicated queue during server shutdown. Fixes . #10841 ().
• Ensure that varSamp, varPop cannot return negative results due to numerical errors and that stddevSamp, stddevPop cannot be calculated from negative variance. This fixes #10532. (alexey-milovidov).
• Better DNS exception message. This fixes . #10828 ().
• Change HTTP response code in case of some parse errors to 400 Bad Request. This fix #10636. (alexey-milovidov).
• Print a message if clickhouse-client is newer than clickhouse-server. (alexey-milovidov).
• Adding support for INSERT INTO [db.]table WATCH query. (vzakaznikov).
• Allow to pass quota_key in clickhouse-client. This closes . #10270 ().

Performance Improvement

• Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes . #11639 () #11795 ().
• Optimization of GROUP BY with respect to table sorting key, enabled with optimize_aggregation_in_order setting. #9113 ().
• Selects with final are executed in parallel. Added setting max_final_threads to limit the number of threads used. #10463 ().
• Improve performance for INSERT queries via INSERT SELECT or INSERT with clickhouse-client when small blocks are generated (typical case with parallel parsing). This fixes #11275. Fix the issue that CONSTRAINTs were not working for DEFAULT fields. This fixes . Fix the issue that CONSTRAINTS were ignored for TEMPORARY tables. This fixes #11274. (alexey-milovidov).
• New optimization that takes all operations out of any function, enabled with optimize_move_functions_out_of_any (Ruslan).
• Improve performance of clickhouse-client in interactive mode when Pretty formats are used. In previous versions, significant amount of time can be spent calculating visible width of UTF-8 string. This closes . #11323 ().
• Improved performance for queries with ORDER BY and small LIMIT (less, then max_block_size). #11171 ().
• Add runtime CPU detection to select and dispatch the best function implementation. Add support for codegeneration for multiple targets. This closes #1017. (DimasKovas).
• Enable mlock of clickhouse binary by default. It will prevent clickhouse executable from being paged out under high IO load. (alexey-milovidov).
• Make queries with sum aggregate function and without GROUP BY keys to run multiple times faster. (alexey-milovidov).
• Improving radix sort (used in ORDER BY with simple keys) by removing some redundant data moves. (Arslan Gumerov).
• Sort bigger parts of the left table in MergeJoin. Buffer left blocks in memory. Add partial_merge_join_left_table_buffer_bytes setting to manage the left blocks buffers sizes. (Artem Zuikov).
• Remove duplicate ORDER BY and DISTINCT from subqueries, this optimization is enabled with optimize_duplicate_order_by_and_distinct (Mikhail Malafeev).
• This feature eliminates functions of other keys in GROUP BY section, enabled with optimize_group_by_function_keys (xPoSx).
• New optimization that takes arithmetic operations out of aggregate functions, enabled with optimize_arithmetic_operations_in_aggregate_functions (Ruslan).
• Use HTTP client for S3 based on Poco instead of curl. This will improve performance and lower memory usage of s3 storage and table functions. (Pavel Kovalenko).
• Fix Kafka performance issue related to reschedules based on limits, which were always applied. (filimonov).
• Enable percpu_arena:percpu for jemalloc (This will reduce memory fragmentation due to thread pool). (Azat Khuzhin).
• Optimize memory usage when reading a response from an S3 HTTP client. (Pavel Kovalenko).
• Adjust the default Kafka settings for better performance. (filimonov).

Experimental Feature

• Add data type Point (Tuple(Float64, Float64)) and Polygon (Array(Array(Tuple(Float64, Float64))). #10678 ().
• Add’s a hasSubstr function that allows for look for subsequences in arrays. Note: this function is likely to be renamed without further notice. #11071 ().
• Added OpenCL support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag -DENABLE_OPENCL=1. For using bitonic sort algorithm instead of others you need to set bitonic_sort for Setting’s option special_sort and make sure that OpenCL is available. This feature does not improve performance or anything else, it is only provided as an example and for demonstration purposes. It is likely to be removed in near future if there will be no further development in this direction. #10232 ().

Build/Testing/Packaging Improvement

• Enable clang-tidy for programs and utils. (alexey-milovidov).
• Remove dependency on tzdata: do not fail if /usr/share/zoneinfo directory does not exist. Note that all timezones work in ClickHouse even without tzdata installed in system. (alexey-milovidov).
• Added MSan and UBSan stress tests. Note that we already have MSan, UBSan for functional tests and “stress” test is another kind of tests. (alexey-milovidov).
• Print compiler build id in crash messages. It will make us slightly more certain about what binary has crashed. Added new function buildId. (alexey-milovidov).
• Added a test to ensure that mutations continue to work after FREEZE query. (alexey-milovidov).
• Don’t allow tests with “fail” substring in their names because it makes looking at the tests results in browser less convenient when you type Ctrl+F and search for “fail”. (alexey-milovidov).
• Removes unused imports from HTTPHandlerFactory. (Bharat Nallan).
• Added a random sampling of instances where copier is executed. It is needed to avoid Too many simultaneous queries error. Also increased timeout and decreased fault probability. (Nikita Mikhaylov).
• Fix missed include. (Matwey V. Kornilov).
• Speed up build by removing old example programs. Also found some orphan functional test. (alexey-milovidov).
• Increase ccache size for builds in CI. (alesapin).
• Leave only unit_tests_dbms in deb build. (Ilya Yatsishin).
• Update librdkafka to version . #11256 ().
• Refactor CMake build files. #11390 ().
• Fix several flaky integration tests. #11355 ().
• Add support for unit tests run with UBSan. #11345 ().
• Remove redundant timeout from integration test test_insertion_sync_fails_with_timeout. #11343 ().
• Better check for hung queries in clickhouse-test. #11321 ().
• Emit a warning if server was build in debug or with sanitizers. #11304 ().
• Now clickhouse-test check the server aliveness before tests run. #11285 ().
• Fix potentially flacky test 00731_long_merge_tree_select_opened_files.sh. It does not fail frequently but we have discovered potential race condition in this test while experimenting with ThreadFuzzer: #9814 See .html#fail1) for the example. #11270 ().
• Repeat test in CI if curl invocation was timed out. It is possible due to system hangups for 10+ seconds that are typical in our CI infrastructure. This fixes #11267. (alexey-milovidov).
• Add a test for Join table engine from @donmikel. This closes . #11265 ().
• Fix several non significant errors in unit tests. #11262 ().
• Now parts of linker command for cctz library will not be shuffled with other libraries. #11213 ().
• Split /programs/server into actual program and library. #11186 ().
• Improve build scripts for protobuf & gRPC. #11172 ().
• Enable performance test that was not working. #11158 ().
• Create root S3 bucket for tests before any CH instance is started. #11142 ().
• Add performance test for non-constant polygons. #11141 ().
• Fixing 00979_live_view_watch_continuous_aggregates test. #11024 ().
• Add ability to run zookeeper in integration tests over tmpfs. #11002 ().
• Wait for odbc-bridge with exponential backoff. Previous wait time of 200 ms was not enough in our CI environment. #10990 ().
• Fix non-deterministic test. #10989 ().
• Added a test for empty external data. #10926 ().
• Database is recreated for every test. This improves separation of tests. #10902 ().
• Added more asserts in columns code. #10833 ().
• Better cooperation with sanitizers. Print information about query_id in the message of sanitizer failure. #10832 ().
• Fix obvious race condition in “Split build smoke test” check. #10820 ().
• Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. (alexey-milovidov).
• Add MSan suppression for MariaDB Client library. (alexey-milovidov).
• GRPC make couldn’t find protobuf files, changed make file by adding the right link. (mnkonkova).
• Enable extra warnings (-Weverything) for base, utils, programs. Note that we already have it for the most of the code. (alexey-milovidov).
• Suppressions of warnings from libraries was mistakenly declared as public in . #10776 ().
• Restore a patch that was accidentially deleted in #10396. (alexey-milovidov).
• Fix performance tests errors, part 2. (alexey-milovidov).
• Fix performance test errors. (alexey-milovidov).
• Update cross-builds to use clang-10 compiler. (Ivan).
• Update instruction to install RPM packages. This was suggested by Denis (TG login @ldviolet) and implemented by Arkady Shejn. (alexey-milovidov).
• Trying to fix tests/queries/0_stateless/01246_insert_into_watch_live_view.py test. (vzakaznikov).
• Fixing and re-enabling 00979_live_view_watch_continuous_aggregates.py test. (vzakaznikov).
• Fix OOM in ASan stress test. (alexey-milovidov).
• Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. (alexey-milovidov).
• Remove external call to ld (bfd) linker during tzdata processing in compile time. (alesapin).
• Allow to use lld to link blobs (resources). (alexey-milovidov).
• Fix UBSan report in LZ4 library. (alexey-milovidov). See also
• Update LZ4 to the latest dev branch. #10630 ().
• Added auto-generated machine-readable file with the list of stable versions. #10628 ().
• Fix capnproto version check for capnp::UnalignedFlatArrayMessageReader. #10618 ().
• Lower memory usage in tests. #10617 ().
• Fixing hard coded timeouts in new live view tests. #10604 ().
• Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. #10599 ().
• Enable ThinLTO for clang builds, continuation of https://github.com/ClickHouse/ClickHouse/pull/10435. (Amos Bird).
• Adding fuzzers and preparing for oss-fuzz integration. (kyprizel).
• Fix FreeBSD build. (Ivan).
• Add new build for query tests using pytest framework. (Ivan).

ClickHouse release v20.4

ClickHouse release v20.4.8.99-stable 2020-08-10

Bug Fix

• Fixed error in parseDateTimeBestEffort function when unix timestamp was passed as an argument. This fixes #13362. (alexey-milovidov).
• Fixed potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes . #13254 ().
• Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. (alexey-milovidov).
• Fixed assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes . #13224 ().
• Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from MergeTree tables. Fixes #13060. Fixes . #13081 ().
• Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 ().
• Fixed possible extra overflow row in data which could appear for queries WITH TOTALS. #12747 ().
• Fixed performance with large tuples, which are interpreted as functions in IN section. The case when user write WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. #12700 ().
• Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 ().
• Fixed #12293 allow push predicate when subquery contains with clause. (Winter Zhang).
• Fixed fix bloom filter index with const expression. #12659 ().
• Fixed SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 ().
• Added support for function if with Array(UUID) arguments. This fixes #11066. (alexey-milovidov).
• Fixed race condition in external dictionaries with cache layout which can lead server crash. (alesapin).
• Removed data for Distributed tables (blocks from async INSERTs) on DROP TABLE. (Azat Khuzhin).
• Fixed bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes . #12543 ().
• Better exception for function in with invalid number of arguments. #12529 ().
• Fixed performance issue, while reading from compact parts. #12492 ().
• Fixed crash in JOIN with dictionary when we are joining over expression of dictionary key: t JOIN dict ON expr(dict.id) = t.id. Disable dictionary join optimisation for this case. #12458 ().
• Fixed possible segfault if StorageMerge. Closes #12054. (tavplubix).
• Fixed order of columns in WITH FILL modifier. Previously order of columns of ORDER BY statement wasn’t respected. (Anton Popov).
• Avoid “bad cast” exception when there is an expression that filters data by virtual columns (like _table in Merge tables) or by “index” columns in system tables such as filtering by database name when querying from system.tables, and this expression returns Nullable type. This fixes . #12305 ().
• Show error after TrieDictionary failed to load. #12290 ().
• The function arrayFill worked incorrectly for empty arrays that may lead to crash. This fixes #12263. (alexey-milovidov).
• Implemented conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes . This fixes #4342. (alexey-milovidov).
• Fixed the behaviour when during multiple sequential inserts in StorageFile header for some special types was written more than once. This fixed . #12197 ().
• Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 ().
• Cap max_memory_usage* limits to the process resident memory. #12182 ().
• Fixed dictGet arguments check during GROUP BY injective functions elimination. #12179 ().
• Don’t split the dictionary source’s table name into schema and table name itself if ODBC connection doesn’t support schema. #12165 ().
• Fixed wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes . #12153 ().
• Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. (alexey-milovidov).
• Fixed potential overflow in integer division. This fixes . #12140 ().
• Fixed potential infinite loop in greatCircleDistance, geoDistance. This fixes #12117. (alexey-milovidov).
• Normalize “pid” file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes . #12133 ().
• Fixed handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes . #12116 ().
• Fixed performance for selects with UNION caused by wrong limit for the total number of threads. Fixes #12030. (Nikolai Kochetov).
• Fixed segfault with combinators. (Anton Popov).
• Fixed empty result_rows and result_bytes metrics in system.quey_log for selects. Fixes . #12089 ().
• Fixed unnecessary limiting the number of threads for selects from VIEW. Fixes #11937. (Nikolai Kochetov).
• Fixed possible crash while using wrong type for PREWHERE. Fixes , #12060. (Nikolai Kochetov).
• Fixed error Expected single dictionary argument for function for function defaultValueOfArgumentType with LowCardinality type. Fixes . #12056 ().
• Fixed error Cannot capture column for higher-order functions with Tuple(LowCardinality) argument. Fixes #9766. (Nikolai Kochetov).
• Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. (tavplubix).
• Make topK aggregate function return Enum for Enum types. This fixes . #12043 ().
• Fixed constraints check if constraint is a constant expression. This fixes #11360. (alexey-milovidov).
• Fixed incorrect comparison of tuples with Nullable columns. Fixes . #12039 ().
• Fixed calculation of access rights when allow_introspection_functions=0. #12031 ().
• Fixed wrong result and potential crash when invoking function if with arguments of type FixedString with different sizes. This fixes #11362. (alexey-milovidov).
• A query with function neighbor as the only returned expression may return empty result if the function is called with offset -9223372036854775808. This fixes . #12019 ().
• Fixed calculation of access rights when allow_ddl=0. #12015 ().
• Fixed potential array size overflow in generateRandom that may lead to crash. This fixes #11371. (alexey-milovidov).
• Fixed potential floating point exception. This closes . #12005 ().
• Fixed wrong setting name in log message at server startup. #11997 ().
• Fixed Query parameter was not set in Values format. Fixes #11918. (tavplubix).
• Keep aliases for substitutions in query (parametrized queries). This fixes . #11916 ().
• Fixed bug with no moves when changing storage policy from default one. #11893 ().
• Fixed potential floating point exception when parsing DateTime64. This fixes #11374. (alexey-milovidov).
• Fixed memory accounting via HTTP interface (can be significant with wait_end_of_query=1). (Azat Khuzhin).
• Parse metadata stored in zookeeper before checking for equality. (Azat Khuzhin).

Performance Improvement

• Index not used for IN operator with literals”, performance regression introduced around v19.3. This fixes “#10574. (nvartolomei).

Build/Testing/Packaging Improvement

• Install ca-certificates before the first apt-get update in Dockerfile. #12095 ().

ClickHouse release v20.4.6.53-stable 2020-06-25

Bug Fix

• Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11608. (Nikolai Kochetov).
• Don’t allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes . #11846 ().
• Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. (alexey-milovidov).
• Fix wrong result for if() with NULLs in condition. (Artem Zuikov).
• Fix using too many threads for queries. (Nikolai Kochetov).
• Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. (hexiaoting).
• Now replicated fetches will be cancelled during metadata alter. (alesapin).
• Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. (tavplubix).
• Fix ORDER BY ... WITH FILL over const columns. (Anton Popov).
• Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. (alexey-milovidov).
• Fix LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. (Anton Popov).
• Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes . #11681 ().
• Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes . #11649 ().
• Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. (alexey-milovidov).
• Fix bloom filters for String (data skipping indices). (Azat Khuzhin).
• Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with somehow). #11608 ().
• Fix error Block structure mismatch for queries with sampling reading from Buffer table. #11602 ().
• Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 ().
• Fix trivial error in log message about “Mark cache size was lowered” at server startup. This closes #11399. (alexey-milovidov).
• Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. (Nikolai Kochetov).
• Fixed rare segfault in SHOW CREATE TABLE Fixes . #11579 ().
• All queries in HTTP session have had the same query_id. It is fixed. #11578 ().
• Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 ().
• Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 ().
• Fix race condition which may lead to an exception during table drop. It’s a bit tricky and not dangerous at all. If you want an explanation, just notice me in telegram. #11523 ().
• Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. (alexey-milovidov).
• If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. (Azat Khuzhin).
• Get rid of old libunwind patches. This allows to disable -fno-omit-frame-pointer in clang builds that improves performance at least by 1% in average. #10761 ().
• Fix usage of primary key wrapped into a function with ‘FINAL’ modifier and ‘ORDER BY’ optimization. #10715 ().

Build/Testing/Packaging Improvement

• Fix several non significant errors in unit tests. (alesapin).
• Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in . #10801 ().

ClickHouse release v20.4.5.36-stable 2020-06-10

Bug Fix

• Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. (alexey-milovidov).
• Fix return compressed size for codecs. (Nikolai Kochetov).
• Fix server crash when a column has compression codec with non-literal arguments. Fixes . #11431 ().
• Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. (Alexey Ilyukhov).
• Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. (alexey-milovidov).
• Fixed geohashesInBox with arguments outside of latitude/longitude range. (Vasily Nemkov).
• Fix possible Pipeline stuck error for queries with external sort and limit. Fixes . #11366 ().
• Remove redundant lock during parts send in ReplicatedMergeTree. #11354 ().
• Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. (alexey-milovidov).
• Fix potential segfault when using Lazy database. (alexey-milovidov).
• Fix crash in quantilesExactWeightedArray. (Nikolai Kochetov).
• Now merges stopped before change metadata in ALTER queries. (alesapin).
• Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes . #11330 ().
• Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 ().
• Fix very rare race condition in ThreadPool. #11314 ().
• Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 ().
• Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 ().
• Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. (alexey-milovidov).
• Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes . This fixes #5732. This fixes . Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 ().
• Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. (Vitaly Baranov).
• Fix crash while reading malformed data in Protobuf format. This fixes , fixes https://github.com/ClickHouse/ClickHouse/issues/11203. (Vitaly Baranov).
• Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. (Nikita Mikhaylov).
• Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes . #11205 ().
• Fix possible exception Invalid status for associated output. #11200 ().
• Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 ().
• Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 ().
• If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 ().
• Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 ().
• Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 ().
• Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 ().
• Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, . #10621 ().

Build/Testing/Packaging Improvement

• Fix several flaky integration tests. (alesapin).

ClickHouse release v20.4.4.18-stable 2020-05-26

ClickHouse release v20.4.3.16-stable 2020-05-23

Bug Fix

• Removed logging from mutation finalization task if nothing was finalized. #11109 ().
• Fixed memory leak in registerDiskS3. #11074 ().
• Fixed the potential missed data during termination of Kafka engine table. #11048 ().
• Fixed parseDateTime64BestEffort argument resolution bugs. #11038 ().
• Fixed very rare potential use-after-free error in MergeTree if table was not created successfully. #10986, (alexey-milovidov).
• Fixed metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. (Azat Khuzhin).
• Fixed server crash on concurrent ALTER and DROP DATABASE queries with Atomic database engine. (tavplubix).
• Fixed incorrect raw data size in getRawData() method. (Igr).
• Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. (alexey-milovidov).
• Fixed sending partially written files by the DistributedBlockOutputStream. (Azat Khuzhin).
• Fixed crash in SELECT count(notNullIn(NULL, [])). (Nikolai Kochetov).
• Fixed the hang which was happening sometimes during DROP of Kafka table engine. (or during server restarts). (filimonov).
• Fixed the impossibility of executing multiple ALTER RENAME like a TO b, c TO a. (alesapin).
• Fixed possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way it can happen is when you use finalizeAggregation function while reading from table with Memory engine which stores AggregateFunction state for quantile* function. (Nikolai Kochetov).
• Fixed backward compatibility with tuples in Distributed tables. (Anton Popov).
• Fixed SIGSEGV in StringHashTable if such a key does not exist. (Azat Khuzhin).
• Fixed WATCH hangs after LiveView table was dropped from database with Atomic engine. (tavplubix).
• Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. (tavplubix).
• Now constraints are updated if the column participating in CONSTRAINT expression was renamed. Fixes . #10847 ().
• Fixed potential read of uninitialized memory in cache-dictionary. #10834 ().
• Fixed columns order after Block::sortColumns(). #10826 ().
• Fixed the issue with ODBC bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). (alexey-milovidov).
• Fixed UBSan and MSan report in DateLUT. (alexey-milovidov).
• Fixed incorrect type conversion in key conditions. Fixes . #10791 ().
• Fixed parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. (Nikolai Kochetov).
• Fixed combinator -OrNull and -OrDefault when combined with -State. (hcz).
• Fixed possible buffer overflow in function h3EdgeAngle. (alexey-milovidov).
• Fixed bug which locks concurrent alters when table has a lot of parts. (alesapin).
• Fixed nullptr dereference in StorageBuffer if server was shutdown before table startup. (alexey-milovidov).
• Fixed optimize_skip_unused_shards with LowCardinality. (Azat Khuzhin).
• Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. (Vladimir Chebotarev).
• Fixed possible crash when createDictionary() is called before loadStoredObject() has finished. (Vitaly Baranov).
• Fixed SELECT of column ALIAS which default expression type different from column type. (Azat Khuzhin).
• Implemented comparison between DateTime64 and String values. (Vasily Nemkov).
• Disable GROUP BY sharding_key optimization by default (optimize_distributed_group_by_sharding_key had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is if in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. (Azat Khuzhin).
• Fixed . #10486 ().
• Added tests about max_rows_to_sort setting. #10268 ().
• Added backward compatibility for create bloom filter index. #10551. (Winter Zhang).

ClickHouse release v20.4.2.9, 2020-05-12

Backward Incompatible Change

• System tables (e.g. system.query_log, system.trace_log, system.metric_log) are using compact data part format for parts smaller than 10 MiB in size. Compact data part format is supported since version 20.3. If you are going to downgrade to version less than 20.3, you should manually delete table data for system logs in /var/lib/clickhouse/data/system/.
• When string comparison involves FixedString and compared arguments are of different sizes, do comparison as if smaller string is padded to the length of the larger. This is intented for SQL compatibility if we imagine that FixedString data type corresponds to SQL CHAR. This closes . #10363 ()
• Make SHOW CREATE TABLE multiline. Now it is more readable and more like MySQL. #10049 ()
• Added a setting validate_polygons that is used in pointInPolygon function and enabled by default. #9857 ()

New Feature

• Add support for secured connection from ClickHouse to Zookeeper (Konstantin Lebedev)
• Support custom HTTP handlers. See ISSUES-5436 for description. (Winter Zhang)
• Add MessagePack Input/Output format. (Kruglov Pavel)
• Add Regexp input format. (Kruglov Pavel)
• Added output format Markdown for embedding tables in markdown documents. (Kruglov Pavel)
• Added support for custom settings section in dictionaries. Also fixes issue . #10137 ()
• Added custom settings support in DDL-queries for CREATE DICTIONARY #10465 ()
• Add simple server-wide memory profiler that will collect allocation contexts when server memory usage becomes higher than the next allocation threshold. #10444 ()
• Add setting always_fetch_merged_part which restrict replica to merge parts by itself and always prefer dowloading from other replicas. #10379 ()
• Add function JSONExtractKeysAndValuesRaw which extracts raw data from JSON objects #10378 ()
• Add memory usage from OS to system.asynchronous_metrics. #10361 ()
• Added generic variants for functions least and greatest. Now they work with arbitrary number of arguments of arbitrary types. This fixes #4767 (alexey-milovidov)
• Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: strict_max_lifetime_seconds, which is max_lifetime by default, and query_wait_timeout_milliseconds, which is one minute by default. The first settings is also useful with allow_read_expired_keys settings (to forbid reading very expired keys). (Nikita Mikhaylov)
• Add log_queries_min_type to filter which entries will be written to query_log (Azat Khuzhin)
• Added function isConstant. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. (alexey-milovidov)
• add joinGetOrNull to return NULL when key is missing instead of returning the default value. (Amos Bird)
• Consider NULL to be equal to NULL in IN operator, if the option transform_null_in is set. (achimbab)
• Add ALTER TABLE ... RENAME COLUMN for MergeTree table engines family. (alesapin)
• Support parallel distributed INSERT SELECT. (vxider)
• Add ability to query Distributed over Distributed (w/o distributed_group_by_no_merge) … (Azat Khuzhin)
• Add function arrayReduceInRanges which aggregates array elements in given ranges. (hcz)
• Add Dictionary Status on prometheus exporter. (Guillaume Tassery)
• Add function arrayAUC (taiyang-li)
• Support DROP VIEW statement for better TPC-H compatibility. (Amos Bird)
• Add ‘strict_order’ option to windowFunnel() (achimbab)
• Support DATE and TIMESTAMP SQL operators, e.g. SELECT date '2001-01-01' (Artem Zuikov)

Experimental Feature

• Added experimental database engine Atomic. It supports non-blocking DROP and RENAME TABLE queries and atomic EXCHANGE TABLES t1 AND t2 query #7512 ()
• Initial support for ReplicatedMergeTree over S3 (it works in suboptimal way) #10126 ()

Bug Fix

• Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table (Nikolai Kochetov)
• Fixed bug, which caused HTTP requests to get stuck on client closing connection when readonly=2 and cancel_http_readonly_queries_on_client_close=1. (tavplubix)
• Fix segfault in StorageBuffer when exception is thrown on server startup. Fixes #10609 ()
• The querySYSTEM DROP DNS CACHE now also drops caches used to check if user is allowed to connect from some IP addresses #10608 ()
• Fix usage of multiple IN operators with an identical set in one query. Fixes #10539 (Anton Popov)
• Fix crash in generateRandom with nested types. Fixes . #10734 ()
• Fix data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. (Nikolai Kochetov)
• Fix logic for aggregation_memory_efficient_merge_threads setting. (palasonic1)
• Fix disappearing totals. Totals could have being filtered if query had JOIN or subquery with external WHERE condition. Fixes #10698 ()
• Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655 (Nikolai Kochetov)
• Fix possible incorrect number of rows for queries with LIMIT. Fixes , #10709 (Nikolai Kochetov)
• Fix index corruption, which may occur in some cases after merging compact parts into another compact part. (Anton Popov)
• Fix the situation, when mutation finished all parts, but hung up in is_done=0. (alesapin)
• Fix overflow at beginning of unix epoch for timezones with fractional offset from UTC. Fixes . #10513 ()
• Better diagnostics for input formats. Fixes #10204 (tavplubix)
• Fix numeric overflow in simpleLinearRegression() over large integers (hcz)
• Fix use-after-free in Distributed shutdown, avoid waiting for sending all batches (Azat Khuzhin)
• Add CA certificates to clickhouse-server docker image (filimonov)
• Fix a rare endless loop that might have occurred when using the addressToLine function or AggregateFunctionState columns. (Alexander Kuzmenkov)
• Handle zookeeper “no node error” during distributed query (Daniel Chen)
• Fix bug when server cannot attach table after column’s default was altered. (alesapin)
• Implicitly cast the default expression type to the column type for the ALIAS columns (Azat Khuzhin)
• Don’t remove metadata directory if ATTACH DATABASE fails (Winter Zhang)
• Avoid dependency on system tzdata. Fixes loading of Africa/Casablanca timezone on CentOS 8. Fixes #10425 ()
• Fix some issues if data is inserted with quorum and then gets deleted (DROP PARTITION, TTL, etc.). It led to stuck of INSERTs or false-positive exceptions in SELECTs. Fixes #9946 (Nikita Mikhaylov)
• Check the number and type of arguments when creating BloomFilter index #10431 ()
• Prefer fallback_to_stale_replicas over skip_unavailable_shards, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev ) #10422 ()
• Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. Fixes #10226. (Vadim Plakhtinskiy)
• Add database name to dictionary name after DETACH/ATTACH. Fixes system.dictionaries table and SYSTEM RELOAD query (Azat Khuzhin)
• Fix possible incorrect result for extremes in processors pipeline. (Nikolai Kochetov)
• Fix possible segfault when the setting distributed_group_by_no_merge is enabled (introduced in 20.3.7.46 by ). #10399 ()
• Fix wrong flattening of Array(Tuple(...)) data types. Fixes #10259 (alexey-milovidov)
• Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN (Alexander Kuzmenkov)
• Fix order of columns after Block::sortColumns() (Azat Khuzhin)
• Fix possible Pipeline stuck error in ConcatProcessor which may happen in remote query. (Nikolai Kochetov)
• Don’t make disk reservations for aggregations. Fixes #10375 ()
• Fix wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). Fixes #7202 (alexey-milovidov)
• Avoid infinite loop in dictIsIn function. Fixes #515 (alexey-milovidov)
• Disable GROUP BY sharding_key optimization by default and fix it for WITH ROLLUP/CUBE/TOTALS (Azat Khuzhin)
• Check for error code when checking parts and don’t mark part as broken if the error is like “not enough memory”. Fixes #10364 ()
• Show information about not loaded dictionaries in system tables. #10234 ()
• Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 ()
• Fixed DROP vs OPTIMIZE race in ReplicatedMergeTree. DROP could left some garbage in replica path in ZooKeeper if there was concurrent OPTIMIZE query. #10312 ()
• Fix ‘Logical error: CROSS JOIN has expressions’ error for queries with comma and names joins mix. Fixes #9910 (Artem Zuikov)
• Fix queries with max_bytes_before_external_group_by. (Artem Zuikov)
• Fix the issue with limiting maximum recursion depth in parser in certain cases. This fixes This fix may introduce minor incompatibility: long and deep queries via clickhouse-client may refuse to work, and you should adjust settings max_query_size and max_parser_depth accordingly. #10295 ()
• Allow to use count(*) with multiple JOINs. Fixes #9853 (Artem Zuikov)
• Fix error Pipeline stuck with max_rows_to_group_by and group_by_overflow_mode = 'break'. (Nikolai Kochetov)
• Fix ‘Cannot add column’ error while creating range_hashed dictionary using DDL query. Fixes . #10235 ()
• Fix rare possible exception Cannot drain connections: cancel first. #10239 ()
• Fixed bug where ClickHouse would throw “Unknown function lambda.” error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 ()
• Fixed reasonably rare segfault in StorageSystemTables that happens when SELECT … FROM system.tables is run on a database with Lazy engine. #10209 ()
• Fix possible infinite query execution when the query actually should stop on LIMIT, while reading from infinite source like system.numbers or system.zeros. #10206 ()
• Fixed “generateRandom” function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. (alexey-milovidov)
• Convert types if the table definition of a View does not correspond to the SELECT query. This fixes and #10022 (alexey-milovidov)
• Fix parseDateTimeBestEffort for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10214 ()
• Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 ()
• Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092 (alexey-milovidov)
• Fix issue with separator appearing in SCRAMBLE for native mysql-connector-java (JDBC) (BohuTANG)
• Fix using the current database for an access checking when the database isn’t specified. (Vitaly Baranov)
• Fix ALTER of tables with compact parts. (Anton Popov)
• Add the ability to relax the restriction on non-deterministic functions usage in mutations with allow_nondeterministic_mutations setting. (filimonov)
• Fix DROP TABLE invoked for dictionary (Azat Khuzhin)
• Convert blocks if structure does not match when doing INSERT into Distributed table (Azat Khuzhin)
• The number of rows was logged incorrectly (as sum across all parts) when inserted block is split by parts with partition key. (alexey-milovidov)
• Add some arguments check and support identifier arguments for MySQL Database Engine (Winter Zhang)
• Fix incorrect index_granularity_bytes check while creating new replica. Fixes . #10121 ()
• Fix bug in CHECK TABLE query when table contain skip indices. #10068 ()
• Fix Distributed-over-Distributed with the only one shard in a nested table #9997 ()
• Fix possible rows loss for queries with JOIN and UNION ALL. Fixes #9826, . … #10099 ()
• Fix bug in dictionary when local clickhouse server is used as source. It may caused memory corruption if types in dictionary and source are not compatible. #10071 ()
• Fixed replicated tables startup when updating from an old ClickHouse version where /table/replicas/replica_name/metadata node doesn’t exist. Fixes #10037. (alesapin)
• Fix error Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform. It happened when setting distributed_aggregation_memory_efficient was enabled, and distributed query read aggregating data with mixed single and two-level aggregation from different shards. (Nikolai Kochetov)
• Fix deadlock when database with materialized view failed attach at start (Azat Khuzhin)
• Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (, #8925). … (Alexander Kuzmenkov)
• Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9714 (Artem Zuikov)
• Fix possible deadlock in SYSTEM RESTART REPLICAS (tavplubix)
• Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from Distributed table was executed simultaneously on local and remote shards. Fixes #9971 ()
• Fixed DeleteOnDestroy logic in ATTACH PART which could lead to automatic removal of attached part and added few tests #9410 ()
• Fix a bug with ON CLUSTER DDL queries freezing on server startup. #9927 ()
• Fix bug in which the necessary tables weren’t retrieved at one of the processing stages of queries to some databases. Fixes #9699. (achulkov2)
• Fix ‘Not found column in block’ error when JOIN appears with TOTALS. Fixes #9939 ()
• Fix parsing multiple hosts set in the CREATE USER command #9924 ()
• Fix TRUNCATE for Join table engine (#9917). (Amos Bird)
• Fix race condition between drop and optimize in ReplicatedMergeTree. (alesapin)
• Fix DISTINCT for Distributed when optimize_skip_unused_shards is set. (Azat Khuzhin)
• Fix “scalar doesn’t exist” error in ALTERs (). … #9904 ()
• Fix error with qualified names in distributed_product_mode=\'local\'. Fixes #4756 (Artem Zuikov)
• For INSERT queries shards now do clamp the settings from the initiator to their constraints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix . #9852 ()
• Add some retries when commiting offsets to Kafka broker, since it can reject commit if during offsets.commit.timeout.ms there were no enough replicas available for the __consumer_offsets topic #9884 ()
• Fix Distributed engine behavior when virtual columns of the underlying table used in WHERE #9847 ()
• Fixed some cases when timezone of the function argument wasn’t used properly. #9574 ()
• Fix ‘Different expressions with the same alias’ error when query has PREWHERE and WHERE on distributed table and SET distributed_product_mode = 'local'. #9871 ()
• Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. (alesapin)
• Fix calculating grants for introspection functions from the setting allow_introspection_functions. (Vitaly Baranov)
• Fix max_distributed_connections (w/ and w/o Processors) (Azat Khuzhin)
• Fix possible exception Got 0 in totals chunk, expected 1 on client. It happened for queries with JOIN in case if right joined table had zero rows. Example: select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;. Fixes . … #9823 ()
• Fix ‘COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query’ error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782 (Artem Zuikov)
• Fix server crashing when optimize_skip_unused_shards is set and expression for key can’t be converted to its field type (Azat Khuzhin)
• Fix empty string handling in splitByString. (hcz)
• Fix broken ALTER TABLE DELETE COLUMN query for compact parts. (alesapin)
• Fixed missing rows_before_limit_at_least for queries over http (with processors pipeline). Fixes #9757 ()
• Fix excessive memory consumption in ALTER queries (mutations). This fixes #9533 and . #9754 ()
• Fix possible permanent “Cannot schedule a task” error. #9154 ()
• Fix bug in backquoting in external dictionaries DDL. Fixes #9619. (alesapin)
• Fixed data race in text_log. It does not correspond to any real bug. (alexey-milovidov)
• Fix bug in a replication that doesn’t allow replication to work if the user has executed mutations on the previous version. This fixes . #9652 ()
• Fixed incorrect internal function names for sumKahan and sumWithOverflow. It led to exception while using this functions in remote queries. #9636 ()
• Add setting use_compact_format_in_distributed_parts_names which allows to write files for INSERT queries into Distributed table with more compact format. This fixes #9647. (alesapin)
• Fix RIGHT and FULL JOIN with LowCardinality in JOIN keys. (Artem Zuikov)
• Fix possible exceptions Size of filter doesn't match size of column and Invalid number of rows in Chunk in MergeTreeRangeReader. They could appear while executing PREWHERE in some cases. (Anton Popov)
• Allow ALTER ON CLUSTER of Distributed tables with internal replication. This fixes #9617 ()
• Fix issue when timezone was not preserved if you write a simple arithmetic expression like time + 1 (in contrast to an expression like time + INTERVAL 1 SECOND). This fixes #5743 (alexey-milovidov)

Improvement

• Use time zone when comparing DateTime with string literal. This fixes #5206. (alexey-milovidov)
• Print verbose diagnostic info if Decimal value cannot be parsed from text input format. (alexey-milovidov)
• Add tasks/memory metrics for distributed/buffer schedule pools (Azat Khuzhin)
• Display result as soon as it’s ready for SELECT DISTINCT queries in clickhouse-local and HTTP interface. This fixes #9559 ()
• Allow to use SAMPLE OFFSET query instead of cityHash64(PRIMARY KEY) % N == n for splitting in clickhouse-copier. To use this feature, pass --experimental-use-sample-offset 1 as a command line argument. #10414 ()
• Allow to parse BOM in TSV if the first column cannot contain BOM in its value. This fixes #10301 (alexey-milovidov)
• Add Avro nested fields insert support (Andrew Onyshchuk)
• Allowed to alter column in non-modifying data mode when the same type is specified. (Vladimir Chebotarev)
• Auto distributed_group_by_no_merge on GROUP BY sharding key (if optimize_skip_unused_shards is set) (Azat Khuzhin)
• Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (Azat Khuzhin)
• Added a setting max_server_memory_usage to limit total memory usage of the server. The metric MemoryTracking is now calculated without a drift. The setting max_memory_usage_for_all_queries is now obsolete and does nothing. This closes . #10362 ()
• Add config option system_tables_lazy_load. If it’s set to false, then system tables with logs are loaded at the server startup. Alexander Burmak, , #9642 (alexey-milovidov)
• Use background thread pool (background_schedule_pool_size) for distributed sends (Azat Khuzhin)
• Use background thread pool for background buffer flushes. (Azat Khuzhin)
• Support for one special case of removing incompletely written parts. This fixes . #10221 ()
• Use isInjective() over manual list of such functions for GROUP BY optimization. #10342 ()
• Avoid printing error message in log if client sends RST packet immediately on connect. It is typical behaviour of IPVS balancer with keepalived and VRRP. This fixes #1851 (alexey-milovidov)
• Allow to parse +inf for floating point types. This closes #10272 ()
• Implemented generateRandom table function for Nested types. This closes #9903 (alexey-milovidov)
• Provide max_allowed_packed in MySQL compatibility interface that will help some clients to communicate with ClickHouse via MySQL protocol. (BohuTANG)
• Allow literals for GLOBAL IN (i.e. SELECT * FROM remote('localhost', system.one) WHERE dummy global in (0)) (Azat Khuzhin)
• Fix various small issues in interactive mode of clickhouse-client (alexey-milovidov)
• Avoid superfluous dictionaries load (system.tables, DROP/SHOW CREATE TABLE) (Azat Khuzhin)
• Update to RWLock: timeout parameter for getLock() + implementation reworked to be phase fair (Alexander Kazakov)
• Enhanced compatibility with native mysql-connector-java(JDBC) (BohuTANG)
• The function toString is considered monotonic and can be used for index analysis even when applied in tautological cases with String or LowCardinality(String) argument. (Amos Bird)
• Add ON CLUSTER clause support to commands {CREATE|DROP} USER/ROLE/ROW POLICY/SETTINGS PROFILE/QUOTA, GRANT. (Vitaly Baranov)
• Virtual hosted-style support for S3 URI (Pavel Kovalenko)
• Now layout type for dictionaries with no arguments can be specified without round brackets in dictionaries DDL-queries. Fixes . #10064 ()
• Add ability to use number ranges with leading zeros in filepath #9989 ()
• Better memory usage in CROSS JOIN. #10029 ()
• Try to connect to all shards in cluster when getting structure of remote table and skip_unavailable_shards is set. #7278 ()
• Add total_rows/total_bytes into the system.tables table. #9919 ()
• System log tables now use polymorpic parts by default. #9905 ()
• Add type column into system.settings/merge_tree_settings #9909 ()
• Check for available CPU instructions at server startup as early as possible. #9888 ()
• Remove ORDER BY stage from mutations because we read from a single ordered part in a single thread. Also add check that the rows in mutation are ordered by sorting key and this order is not violated. #9886 ()
• Implement operator LIKE for FixedString at left hand side. This is needed to better support TPC-DS queries. #9890 ()
• Add force_optimize_skip_unused_shards_no_nested that will disable force_optimize_skip_unused_shards for nested Distributed table #9812 ()
• Now columns size is calculated only once for MergeTree data parts. #9827 ()
• Evaluate constant expressions for optimize_skip_unused_shards (i.e. SELECT * FROM foo_dist WHERE key=xxHash32(0)) #8846 ()
• Check for using Date or DateTime column from TTL expressions was removed. #9967 ()
• DiskS3 hard links optimal implementation. #9760 ()
• If set multiple_joins_rewriter_version = 2 enables second version of multiple JOIN rewrites that keeps not clashed column names as is. It supports multiple JOINs with USING and allow select * for JOINs with subqueries. #9739 ()
• Implementation of “non-blocking” alter for StorageMergeTree #9606 ()
• Add MergeTree full support for DiskS3 #9646 ()
• Extend splitByString to support empty strings as separators. #9742 ()
• Add a timestamp_ns column to system.trace_log. It contains a high-definition timestamp of the trace event, and allows to build timelines of thread profiles (“flame charts”). #9696 ()
• When the setting send_logs_level is enabled, avoid intermixing of log messages and query progress. #9634 ()
• Added support of MATERIALIZE TTL IN PARTITION. #9581 ()
• Support complex types inside Avro nested fields #10502 ()

Performance Improvement

• Better insert logic for right table for Partial MergeJoin. (Artem Zuikov)
• Improved performance of row-oriented formats (more than 10% for CSV and more than 35% for Avro in case of narrow tables). (Andrew Onyshchuk)
• Improved performance of queries with explicitly defined sets at right side of IN operator and tuples on the left side. (Anton Popov)
• Use less memory for hash table in HashJoin. (Artem Zuikov)
• Special HashJoin over StorageDictionary. Allow rewrite dictGet() functions with JOINs. It’s not backward incompatible itself but could uncover on some installations. #10133 ()
• Enable parallel insert of materialized view when its target table supports. #10052 ()
• Improved performance of index analysis with monotonic functions. #9607 (Anton Popov)
• Using SSE2 or SSE4.2 SIMD intrinsics to speed up tokenization in bloom filters. (Vasily Nemkov)
• Improved performance of queries with explicitly defined sets at right side of IN operator. This fixes performance regression in version 20.3. (Anton Popov)
• Now clickhouse-copier splits each partition in number of pieces and copies them independently. (Nikita Mikhaylov)
• Adding more aggregation methods. For example TPC-H query 1 will now pick FixedHashMap<UInt16, AggregateDataPtr> and gets 25% performance gain (Amos Bird)
• Use single row counter for multiple streams in pre-limit transform. This helps to avoid uniting pipeline streams in queries with limit but without order by (like select f(x) from (select x from t limit 1000000000)) and use multiple threads for further processing. (Nikolai Kochetov)

Build/Testing/Packaging Improvement

• Use a fork of AWS SDK libraries from ClickHouse-Extras #10527 ()
• Add integration tests for new ALTER RENAME COLUMN query. #10654 ()
• Fix possible signed integer overflow in invocation of function now64 with wrong arguments. This fixes #8973 (alexey-milovidov)
• Split fuzzer and sanitizer configurations to make build config compatible with Oss-fuzz. (kyprizel)
• Fixes for clang-tidy on clang-10. (alexey-milovidov)
• Display absolute paths in error messages. Otherwise KDevelop fails to navigate to correct file and opens a new file instead. (alexey-milovidov)
• Added ASAN_OPTIONS environment variable to investigate errors in CI stress tests with Address sanitizer. (Nikita Mikhaylov)
• Enable ThinLTO for clang builds (experimental). (alexey-milovidov)
• Remove accidential dependency on Z3 that may be introduced if the system has Z3 solver installed. (alexey-milovidov)
• Move integration tests docker files to docker/ directory. (Ilya Yatsishin)
• Allow to use clang-10 in CI. It ensures that is fixed. #10384 ()
• Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error and SSL Exception: error:2400006E:random number generator::error retrieving entropy. The issue was present in version 20.1. #8956 ()
• Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238 (Amos Bird)
• Add performance test for . #10345 ()
• Fix flaky test test_settings_constraints_distributed.test_insert_clamps_settings. #10346 ()
• Add util to test results upload in CI ClickHouse #10330 ()
• Convert test results to JSONEachRow format in junit_to_html tool #10323 ()
• Update cctz. #10215 ()
• Allow to create HTML report from the purest JUnit XML report. #10247 ()
• Update the check for minimal compiler version. Fix the root cause of the issue #10250 (alexey-milovidov)
• Initial support for live view tables over distributed (vzakaznikov)
• Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in . #10801 ()
• clickhouse-docker-util #10151 ()
• Update pdqsort to recent version #10171 ()
• Update libdivide to v3.0 #10169 ()
• Add check with enabled polymorphic parts. #10086 ()
• Add cross-compile build for FreeBSD. This fixes #9465 (Ivan)
• Add performance test for #6980 ()
• Add support of /dev/null in the File engine for better performance testing #8455 ()
• Move all folders inside /dbms one level up #9974 ()
• Add a test that checks that read from MergeTree with single thread is performed in order. Addition to #9670 (alexey-milovidov)
• Fix the 00964_live_view_watch_events_heartbeat.py test to avoid race condition. (vzakaznikov)
• Fix integration test test_settings_constraints (Vitaly Baranov)
• Every function in its own file, part 12. (alexey-milovidov)
• Added performance test for the case of extremely slow analysis of array of tuples. (alexey-milovidov)
• Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages Data after merge is not byte-identical to data on another replicas. with explanation. These messages are Ok and you should not worry. (alexey-milovidov)
• Fix TSan report in system.stack_trace. (alexey-milovidov)
• Removed dependency on clock_getres. (alexey-milovidov)
• Added identifier names check with clang-tidy. (alexey-milovidov)
• Update “builder” docker image. This image is not used in CI but is useful for developers. (alexey-milovidov)
• Remove old performance-test tool that is no longer used in CI. clickhouse-performance-test is great but now we are using way superior tool that is doing comparison testing with sophisticated statistical formulas to achieve confident results regardless to various changes in environment. (alexey-milovidov)
• Added most of clang-static-analyzer checks. (alexey-milovidov)
• Update Poco to 1.9.3 in preparation for MongoDB URI support. (Alexander Kuzmenkov)
• Fix build with -DUSE_STATIC_LIBRARIES=0 -DENABLE_JEMALLOC=0 (Artem Zuikov)
• For change log script, if merge commit was cherry-picked to release branch, take PR name from commit description. (Nikolai Kochetov)
• Support vX.X-conflicts tag in backport script. (Nikolai Kochetov)
• Fix auto-label for backporting script. (Nikolai Kochetov)
• Use libc++ in Darwin cross-build to make it consistent with native build. (Hui Wang)
• Fix flacky test 01017_uniqCombined_memory_usage. Continuation of . #9667 ()
• Fix build for native MacOS Clang compiler #9649 ()
• Allow to add various glitches around pthread_mutex_lock, pthread_mutex_unlock functions. #9635 ()
• Add support for clang-tidy in packager script. #9625 ()
• Add ability to use unbundled msgpack. #10168 ()

Bug Fix

• Fixed error in parseDateTimeBestEffort function when unix timestamp was passed as an argument. This fixes . #13441 ().
• Fixed potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. (alexey-milovidov).
• Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes . #13226 ().
• Fixed assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. (alexey-milovidov).
• Fixed unnecessary limiting for the number of threads for selects from local replica. (Nikolai Kochetov).
• Fixed possible extra overflow row in data which could appear for queries WITH TOTALS. (Nikolai Kochetov).
• Fixed performance with large tuples, which are interpreted as functions in IN section. The case when user write WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. (Anton Popov).
• Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). (Azat Khuzhin).
• Fixed allow push predicate when subquery contains with clause. #12663 ().
• Fixed #10572 fix bloom filter index with const expression. (Winter Zhang).
• Fixed SIGSEGV in StorageKafka when broker is unavailable (and not only). (Azat Khuzhin).
• Fixed race condition in external dictionaries with cache layout which can lead server crash. (alesapin).
• Fixed bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes . #12543 ().
• Better exception for function in with invalid number of arguments. #12529 ().
• Fixed performance issue, while reading from compact parts. #12492 ().
• Fixed the deadlock if text_log is enabled. #12452 ().
• Fixed possible segfault if StorageMerge. Closes #12054. (tavplubix).
• Fixed TOTALS/ROLLUP/CUBE for aggregate functions with -State and Nullable arguments. This fixes . #12376 ().
• Fixed order of columns in WITH FILL modifier. Previously order of columns of ORDER BY statement wasn’t respected. #12306 ().
• Avoid “bad cast” exception when there is an expression that filters data by virtual columns (like _table in Merge tables) or by “index” columns in system tables such as filtering by database name when querying from system.tables, and this expression returns Nullable type. This fixes #12166. (alexey-milovidov).
• Show error after TrieDictionary failed to load. (Vitaly Baranov).
• The function arrayFill worked incorrectly for empty arrays that may lead to crash. This fixes . #12279 ().
• Implement conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes . #12275 ().
• Fixed the behaviour when during multiple sequential inserts in StorageFile header for some special types was written more than once. This fixed #6155. (Nikita Mikhaylov).
• Fixed logical functions for UInt8 values when they are not equal to 0 or 1. (Alexander Kazakov).
• Fixed dictGet arguments check during GROUP BY injective functions elimination. (Azat Khuzhin).
• Fixed wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes . This closes #12106. (alexey-milovidov).
• Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes . #12151 ().
• Fixed potential overflow in integer division. This fixes #12119. (alexey-milovidov).
• Fixed potential infinite loop in greatCircleDistance, geoDistance. This fixes . #12137 ().
• Avoid There is no query exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 ().
• Fixed performance for selects with UNION caused by wrong limit for the total number of threads. Fixes #12030. (Nikolai Kochetov).
• Fixed segfault with -StateResample combinators. (Anton Popov).
• Fixed unnecessary limiting the number of threads for selects from VIEW. Fixes . #12085 ().
• Fixed possible crash while using wrong type for PREWHERE. Fixes #12053, . #12060 ().
• Fixed error Expected single dictionary argument for function for function defaultValueOfArgumentType with LowCardinality type. Fixes #11808. (Nikolai Kochetov).
• Fixed error Cannot capture column for higher-order functions with Tuple(LowCardinality) argument. Fixes . #12055 ().
• Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. #12045 ().
• Make topK aggregate function return Enum for Enum types. This fixes #3740. (alexey-milovidov).
• Fixed constraints check if constraint is a constant expression. This fixes . #12042 ().
• Fixed incorrect comparison of tuples with Nullable columns. Fixes #11985. (Nikolai Kochetov).
• Fixed wrong result and potential crash when invoking function if with arguments of type FixedString with different sizes. This fixes . #12021 ().
• A query with function neighbor as the only returned expression may return empty result if the function is called with offset -9223372036854775808. This fixes #11367. (alexey-milovidov).
• Fixed potential array size overflow in generateRandom that may lead to crash. This fixes . #12013 ().
• Fixed potential floating point exception. This closes #11378. (alexey-milovidov).
• Fixed wrong setting name in log message at server startup. (alexey-milovidov).
• Fixed Query parameter was not set in Values format. Fixes . #11936 ().
• Keep aliases for substitutions in query (parametrized queries). This fixes #11914. (alexey-milovidov).
• Fixed potential floating point exception when parsing DateTime64. This fixes . #11875 ().
• Fixed memory accounting via HTTP interface (can be significant with wait_end_of_query=1). #11840 ().
• Fixed wrong result for if() with NULLs in condition. #11807 ().
• Parse metadata stored in zookeeper before checking for equality. #11739 ().
• Fixed LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. #11689 ().
• Fix potential read of uninitialized memory in cache dictionary. #10834 ().

Performance Improvement

• Index not used for IN operator with literals”, performance regression introduced around v19.3. This fixes “. #12062 ().

ClickHouse release v20.3.12.112-lts 2020-06-25

Bug Fix

• Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11608. (Nikolai Kochetov).
• Don’t allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes . #11846 ().
• Fix using too many threads for queries. #11788 ().
• Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 ().
• Now replicated fetches will be cancelled during metadata alter. #11744 ().
• Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 ().
• Fix ORDER BY ... WITH FILL over const columns. #11697 ().
• Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 ().
• Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes #11611. (alesapin).
• Add support for regular expressions with case-insensitive flags. This fixes and fixes #11506. (alexey-milovidov).
• Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes . #11644 ().
• Fix bloom filters for String (data skipping indices). #11638 ().
• Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with #11572 somehow). (Nikolai Kochetov).
• Fix error Block structure mismatch for queries with sampling reading from Buffer table. (Nikolai Kochetov).
• Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. (filimonov).
• Fix trivial error in log message about “Mark cache size was lowered” at server startup. This closes . #11589 ().
• Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 ().
• All queries in HTTP session have had the same query_id. It is fixed. #11578 ().
• Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 ().
• Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 ().
• Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. (alexey-milovidov).
• Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9714. (Artem Zuikov).

ClickHouse release v20.3.11.97-lts 2020-06-10

New Feature

• Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: strict_max_lifetime_seconds, which is max_lifetime by default and query_wait_timeout_milliseconds, which is one minute by default. The first settings is also useful with allow_read_expired_keys settings (to forbid reading very expired keys). (Nikita Mikhaylov).

Bug Fix

• Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. (alexey-milovidov).
• Fix return compressed size for codecs. (Nikolai Kochetov).
• Fix server crash when a column has compression codec with non-literal arguments. Fixes . #11431 ().
• Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. (Alexey Ilyukhov).
• Fix crash in JOIN over LowCarinality(T) and Nullable(T). . #11414 ().
• Fix error code for wrong USING key. #11373. (Artem Zuikov).
• Fixed geohashesInBox with arguments outside of latitude/longitude range. (Vasily Nemkov).
• Better errors for joinGet() functions. (Artem Zuikov).
• Fix possible Pipeline stuck error for queries with external sort and limit. Fixes . #11366 ().
• Remove redundant lock during parts send in ReplicatedMergeTree. #11354 ().
• Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. (alexey-milovidov).
• Fix crash in direct selects from StorageJoin (without JOIN) and wrong nullability. (Artem Zuikov).
• Fix crash in quantilesExactWeightedArray. (Nikolai Kochetov).
• Now merges stopped before change metadata in ALTER queries. (alesapin).
• Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes . #11330 ().
• Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 ().
• Fix very rare race condition in ThreadPool. #11314 ().
• Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 ().
• Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. (alexey-milovidov).
• Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes . This fixes #5732. This fixes . Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 ().
• Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes . #11258 ().
• Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 ().
• Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes #11181. (Nikolai Kochetov).
• Fix possible exception Invalid status for associated output. (Nikolai Kochetov).
• Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. (Nikolai Kochetov).
• Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. (Vladimir Chebotarev).
• If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. (Azat Khuzhin).
• Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). (Azat Khuzhin).
• Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: , #11413. (Azat Khuzhin).
• Introduce commit retry logic to decrease the possibility of getting duplicates from Kafka in rare cases when offset commit was failed. (filimonov).

Performance Improvement

• Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 ().

Build/Testing/Packaging Improvement

• Fix several flaky integration tests. (alesapin).

ClickHouse release v20.3.10.75-lts 2020-05-23

Bug Fix

• Removed logging from mutation finalization task if nothing was finalized. (alesapin).
• Fixed parseDateTime64BestEffort argument resolution bugs. (Vasily Nemkov).
• Fixed incorrect raw data size in method getRawData(). (Igr).
• Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. (alexey-milovidov).
• Fixed backward compatibility with tuples in Distributed tables. (Anton Popov).
• Fixed SIGSEGV in StringHashTable if such a key does not exist. (Azat Khuzhin).
• Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. (tavplubix).
• Fixed columns order after Block::sortColumns(). (Azat Khuzhin).
• Fixed the issue with ODBC bridge when no quoting of identifiers is requested. Fixes [#7984] (). #10821 ().
• Fixed UBSan and MSan report in DateLUT. #10798 ().
• Fixed incorrect type conversion in key conditions. Fixes #6287. (Andrew Onyshchuk)
• Fixed parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes . #10757 ().
• Fixed combinator -OrNull and -OrDefault when combined with -State. #10741 ().
• Fixed crash in generateRandom with nested types. Fixes #10583. (Nikolai Kochetov).
• Fixed data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes . #10721 ().
• Fixed possible buffer overflow in function h3EdgeAngle. #10711 ().
• Fixed disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. (Nikolai Kochetov).
• Fixed multiple usages of IN operator with the identical set in one query. (Anton Popov).
• Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes , #7019, , #7091. (tavplubix).
• Fixed order of parameters in AggregateTransform constructor. (palasonic1).
• Fixed the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes . #10664 ().
• Fixed possible incorrect number of rows for queries with LIMIT. Fixes #10566, . #10660 ().
• Fixed a bug which locks concurrent alters when table has a lot of parts. #10659 ().
• Fixed a bug when on SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 ().
• Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. #10603 ().
• Fixed SELECT of column ALIAS which default expression type different from column type. #10563 ().
• Implemented comparison between DateTime64 and String values. #10560 ().
• Fixed index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 ().
• Fixed the situation, when mutation finished all parts, but hung up in is_done=0. #10526 ().
• Fixed overflow at beginning of unix epoch for timezones with fractional offset from UTC. This fixes #9335. (alexey-milovidov).
• Fixed improper shutdown of Distributed storage. (Azat Khuzhin).
• Fixed numeric overflow in simpleLinearRegression over large integers. (hcz).

Build/Testing/Packaging Improvement

• Fix UBSan report in LZ4 library. #10631 ().
• Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238. (Amos Bird).
• Added failing tests about max_rows_to_sort setting. (alexey-milovidov).
• Added some improvements in printing diagnostic info in input formats. Fixes . #10418 ().
• Added CA certificates to clickhouse-server docker image. #10476 ().

Bug fix

• 10551. #10569 ().

ClickHouse release v20.3.8.53, 2020-04-23

Bug Fix

• Fixed wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). This fixes #7202 (alexey-milovidov)
• Fix possible segfault with distributed_group_by_no_merge enabled (introduced in 20.3.7.46 by ). #10399 ()
• Fix wrong flattening of Array(Tuple(...)) data types. This fixes #10259 (alexey-milovidov)
• Drop disks reservation in Aggregator. This fixes bug in disk space reservation, which may cause big external aggregation to fail even if it could be completed successfully (Azat Khuzhin)
• Fixed DROP vs OPTIMIZE race in ReplicatedMergeTree. DROP could left some garbage in replica path in ZooKeeper if there was concurrent OPTIMIZE query. (tavplubix)
• Fix bug when server cannot attach table after column default was altered. (alesapin)
• Do not remove metadata directory when attach database fails before loading tables. (Winter Zhang)
• Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes #10188 ()
• Fix possible Pipeline stuck error in ConcatProcessor which could have happened in remote query. #10381 ()
• Fixed wrong behavior in HashTable that caused compilation error when trying to read HashMap from buffer. #10386 ()
• Allow to use count(*) with multiple JOINs. Fixes #9853 (Artem Zuikov)
• Prefer fallback_to_stale_replicas over skip_unavailable_shards, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev). Fixes: . #10422 ()
• Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes #10226. Author: . #10427 ()
• Check the number and type of arguments when creating BloomFilter index #9623 (Winter Zhang)

Performance Improvement

• Improved performance of queries with explicitly defined sets at right side of IN operator and tuples in the left side. This fixes performance regression in version 20.3. #9740, (Anton Popov)

ClickHouse release v20.3.7.46, 2020-04-17

Bug Fix

• Fix Logical error: CROSS JOIN has expressions error for queries with comma and names joins mix. (Artem Zuikov).
• Fix queries with max_bytes_before_external_group_by. (Artem Zuikov).
• Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes . #10195 ().
• Add the ability to relax the restriction on non-deterministic functions usage in mutations with allow_nondeterministic_mutations setting. #10186 ().

ClickHouse release v20.3.6.40, 2020-04-16

New Feature

• Added function isConstant. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 ().

Bug Fix

• Fix error Pipeline stuck with max_rows_to_group_by and group_by_overflow_mode = 'break'. (Nikolai Kochetov).
• Fix rare possible exception Cannot drain connections: cancel first. (Nikolai Kochetov).
• Fixed bug where ClickHouse would throw “Unknown function lambda.” error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. (Alexander Kazakov).
• Fixed “generateRandom” function for Date type. This fixes . Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 ().
• Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and . #10217 ().
• Fix parseDateTimeBestEffort for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082. (alexey-milovidov).
• Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. (alexey-milovidov).
• Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like system.numbers or system.zeros. (Nikolai Kochetov).
• Fix using the current database for access checking when the database isn’t specified. (Vitaly Baranov).
• Fix possible incorrect result for extremes in processors pipeline. (Nikolai Kochetov).
• Fix some kinds of alters with compact parts. (Anton Popov).
• Fix incorrect index_granularity_bytes check while creating new replica. Fixes . #10121 ().
• Fix SIGSEGV on INSERT into Distributed table when its structure differs from the underlying tables. #10105 ().
• Fix possible rows loss for queries with JOIN and UNION ALL. Fixes #9826, . #10099 ().
• Fixed replicated tables startup when updating from an old ClickHouse version where /table/replicas/replica_name/metadata node doesn’t exist. Fixes #10037. (alesapin).
• Add some arguments check and support identifier arguments for MySQL Database Engine. (Winter Zhang).
• Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. (alesapin).
• Fix bug in CHECK TABLE query when table contain skip indices. (alesapin).
• Fix error Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform. It happened when setting distributed_aggregation_memory_efficient was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). (Nikolai Kochetov).
• Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (, #8925). (Alexander Kuzmenkov).
• Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from Distributed table was executed simultaneously on local and remote shards. Fixes . #9971 ().
• Fix bug in which the necessary tables weren’t retrieved at one of the processing stages of queries to some databases. Fixes #9699. (achulkov2).
• Fix ‘Not found column in block’ error when JOIN appears with TOTALS. Fixes . #9939 ().
• Fix a bug with ON CLUSTER DDL queries freezing on server startup. #9927 ().
• Fix parsing multiple hosts set in the CREATE USER command, e.g. CREATE USER user6 HOST NAME REGEXP 'lo.?*host', NAME REGEXP 'lo*host'. #9924 ().
• Fix TRUNCATE for Join table engine (#9917). (Amos Bird).
• Fix “scalar doesn’t exist” error in ALTERs (). #9904 ().
• Fix race condition between drop and optimize in ReplicatedMergeTree. #9901 ().
• Fix error with qualified names in distributed_product_mode='local'. Fixes #4756. (Artem Zuikov).
• Fix calculating grants for introspection functions from the setting ‘allow_introspection_functions’. (Vitaly Baranov).

Build/Testing/Packaging Improvement

• Fix integration test test_settings_constraints. #9962 ().
• Removed dependency on clock_getres. #9833 ().

ClickHouse release v20.3.5.21, 2020-03-27

Bug Fix

• Fix ‘Different expressions with the same alias’ error when query has PREWHERE and WHERE on distributed table and SET distributed_product_mode = 'local'. #9871 ().
• Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. (alesapin).
• For INSERT queries shard now clamps the settings got from the initiator to the shard’s constaints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix . #9852 ().
• Fix ‘COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query’ error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782. (Artem Zuikov).
• Fix possible exception Got 0 in totals chunk, expected 1 on client. It happened for queries with JOIN in case if right joined table had zero rows. Example: select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;. Fixes . #9823 ().
• Fix SIGSEGV with optimize_skip_unused_shards when type cannot be converted. #9804 ().
• Fix broken ALTER TABLE DELETE COLUMN query for compact parts. #9779 ().
• Fix max_distributed_connections (w/ and w/o Processors). #9673 ().
• Fixed a few cases when timezone of the function argument wasn’t used properly. #9574 ().

Improvement

• Remove order by stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. (alesapin).

ClickHouse release v20.3.4.10, 2020-03-20

Bug Fix

• This release also contains all bug fixes from 20.1.8.41
• Fix missing rows_before_limit_at_least for queries over http (with processors pipeline). This fixes . #9757 ()

ClickHouse release v20.3.3.6, 2020-03-17

Bug Fix

• This release also contains all bug fixes from 20.1.7.38
• Fix bug in a replication that doesn’t allow replication to work if the user has executed mutations on the previous version. This fixes #9645. (alesapin). It makes version 20.3 backward compatible again.
• Add setting use_compact_format_in_distributed_parts_names which allows to write files for INSERT queries into Distributed table with more compact format. This fixes . #9653 (). It makes version 20.3 backward compatible again.

Backward Incompatible Change

• Fixed the issue file name too long when sending data for Distributed tables for a large number of replicas. Fixed the issue that replica credentials were exposed in the server log. The format of directory name on disk was changed to [shard{shard_index}[_replica{replica_index}]]. (Mikhail Korotov) After you upgrade to the new version, you will not be able to downgrade without manual intervention, because old server version does not recognize the new directory format. If you want to downgrade, you have to manually rename the corresponding directories to the old format. This change is relevant only if you have used asynchronous INSERTs to Distributed tables. In the version 20.3.3 we will introduce a setting that will allow you to enable the new format gradually.
• Changed the format of replication log entries for mutation commands. You have to wait for old mutations to process before installing the new version.
• Implement simple memory profiler that dumps stacktraces to system.trace_log every N bytes over soft allocation limit (Ivan) (alexey-milovidov) The column of system.trace_log was renamed from timer_type to trace_type. This will require changes in third-party performance analysis and flamegraph processing tools.
• Use OS thread id everywhere instead of internal thread number. This fixes Old clickhouse-client cannot receive logs that are send from the server when the setting send_logs_level is enabled, because the names and types of the structured log messages were changed. On the other hand, different server versions can send logs with different types to each other. When you don’t use the send_logs_level setting, you should not care. #8954 ()
• Remove indexHint function #9542 ()
• Remove findClusterIndex, findClusterValue functions. This fixes #8641. If you were using these functions, send an email to [[email protected]](https://clickhouse.tech/cdn-cgi/l/email-protection) (alexey-milovidov)
• Now it’s not allowed to create columns or add columns with SELECT subquery as default expression. (alesapin)
• Require aliases for subqueries in JOIN. (Artem Zuikov)
• Improved ALTER MODIFY/ADD queries logic. Now you cannot ADD column without type, MODIFY default expression doesn’t change type of column and MODIFY type doesn’t loose default expression value. Fixes . #9227 ()
• Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see #8696). (Alexander Kuzmenkov)
• The setting experimental_use_processors is enabled by default. This setting enables usage of the new query pipeline. This is internal refactoring and we expect no visible changes. If you will see any issues, set it to back zero. (alexey-milovidov)

New Feature

• Add Avro and AvroConfluent input/output formats #8571 () #8957 () #8717 ()
• Multi-threaded and non-blocking updates of expired keys in cache dictionaries (with optional permission to read old ones). #8303 ()
• Add query ALTER ... MATERIALIZE TTL. It runs mutation that forces to remove expired data by TTL and recalculates meta-information about TTL in all parts. #8775 ()
• Switch from HashJoin to MergeJoin (on disk) if needed #9082 ()
• Added MOVE PARTITION command for ALTER TABLE #4729 (Guillaume Tassery)
• Reloading storage configuration from configuration file on the fly. (Vladimir Chebotarev)
• Allowed to change storage_policy to not less rich one. (Vladimir Chebotarev)
• Added support for globs/wildcards for S3 storage and table function. (Vladimir Chebotarev)
• Implement bitAnd, bitOr, bitXor, bitNot for FixedString(N) datatype. (Guillaume Tassery)
• Added function bitCount. This fixes . #8708 () #8749 ()
• Add generateRandom table function to generate random rows with given schema. Allows to populate arbitrary test table with data. #8994 ()
• JSONEachRowFormat: support special case when objects enclosed in top-level array. #8860 ()
• Now it’s possible to create a column with DEFAULT expression which depends on a column with default ALIAS expression. #9489 ()
• Allow to specify --limit more than the source data size in clickhouse-obfuscator. The data will repeat itself with different random seed. #9155 ()
• Added groupArraySample function (similar to groupArray) with reservior sampling algorithm. #8286 ()
• Now you can monitor the size of update queue in cache/complex_key_cache dictionaries via system metrics. #9413 ()
• Allow to use CRLF as a line separator in CSV output format with setting output_format_csv_crlf_end_of_line is set to 1 #8934 #8963 ()
• Implement more functions of the H3 API: h3GetBaseCell, h3HexAreaM2, h3IndexesAreNeighbors, h3ToChildren, h3ToString and stringToH3 (Nico Mandery)
• New setting introduced: max_parser_depth to control maximum stack size and allow large complex queries. This fixes and #7668. (Maxim Smirnov)
• Add a setting force_optimize_skip_unused_shards setting to throw if skipping of unused shards is not possible (Azat Khuzhin)
• Allow to configure multiple disks/volumes for storing data for send in Distributed engine (Azat Khuzhin)
• Support storage policy (<tmp_policy>) for storing temporary data. (Azat Khuzhin)
• Added X-ClickHouse-Exception-Code HTTP header that is set if exception was thrown before sending data. This implements . #8786 ()
• Added function ifNotFinite. It is just a syntactic sugar: ifNotFinite(x, y) = isFinite(x) ? x : y. #8710 ()
• Added last_successful_update_time column in system.dictionaries table #9394 ()
• Add blockSerializedSize function (size on disk without compression) #8952 ()
• Add function moduloOrZero #9358 ()
• Added system tables system.zeros and system.zeros_mt as well as tale functions zeros() and zeros_mt(). Tables (and table functions) contain single column with name zero and type UInt8. This column contains zeros. It is needed for test purposes as the fastest method to generate many rows. This fixes #6604 (Nikolai Kochetov)

Experimental Feature

• Add new compact format of parts in MergeTree-family tables in which all columns are stored in one file. It helps to increase performance of small and frequent inserts. The old format (one file per column) is now called wide. Data storing format is controlled by settings min_bytes_for_wide_part and min_rows_for_wide_part. #8290 ()
• Support for S3 storage for Log, TinyLog and StripeLog tables. #8862 ()

Bug Fix

• Fixed inconsistent whitespaces in log messages. (alexey-milovidov)
• Fix bug in which arrays of unnamed tuples were flattened as Nested structures on table creation. (achulkov2)
• Fixed the issue when “Too many open files” error may happen if there are too many files matching glob pattern in File table or file table function. Now files are opened lazily. This fixes #8861 ()
• DROP TEMPORARY TABLE now drops only temporary table. #8907 ()
• Remove outdated partition when we shutdown the server or DETACH/ATTACH a table. #8602 ()
• For how the default disk calculates the free space from data subdirectory. Fixed the issue when the amount of free space is not calculated correctly if the data directory is mounted to a separate device (rare case). This fixes #7441 (Mikhail Korotov)
• Allow comma (cross) join with IN () inside. (Artem Zuikov)
• Allow to rewrite CROSS to INNER JOIN if there’s [NOT] LIKE operator in WHERE section. (Artem Zuikov)
• Fix possible incorrect result after GROUP BY with enabled setting distributed_aggregation_memory_efficient. Fixes . #9289 ()
• Found keys were counted as missed in metrics of cache dictionaries. #9411 ()
• Fix replication protocol incompatibility introduced in #8598. (alesapin)
• Fixed race condition on queue_task_handle at the startup of ReplicatedMergeTree tables. (alexey-milovidov)
• The token NOT didn’t work in SHOW TABLES NOT LIKE query #8940 ()
• Added range check to function h3EdgeLengthM. Without this check, buffer overflow is possible. #8945 ()
• Fixed up a bug in batched calculations of ternary logical OPs on multiple arguments (more than 10). #8718 ()
• Fix error of PREWHERE optimization, which could lead to segfaults or Inconsistent number of columns got from MergeTreeRangeReader exception. #9024 ()
• Fix unexpected Timeout exceeded while reading from socket exception, which randomly happens on secure connection before timeout actually exceeded and when query profiler is enabled. Also add connect_timeout_with_failover_secure_ms settings (default 100ms), which is similar to connect_timeout_with_failover_ms, but is used for secure connections (because SSL handshake is slower, than ordinary TCP connection) #9026 ()
• Fix bug with mutations finalization, when mutation may hang in state with parts_to_do=0 and is_done=0. #9022 ()
• Use new ANY JOIN logic with partial_merge_join setting. It’s possible to make ANY|ALL|SEMI LEFT and ALL INNER joins with partial_merge_join=1 now. #8932 ()
• Shard now clamps the settings got from the initiator to the shard’s constaints instead of throwing an exception. This fix allows to send queries to a shard with another constraints. #9447 ()
• Fixed memory management problem in MergeTreeReadPool. #8791 ()
• Fix toDecimal*OrNull() functions family when called with string e. Fixes #8312 (Artem Zuikov)
• Make sure that FORMAT Null sends no data to the client. (Alexander Kuzmenkov)
• Fix bug that timestamp in LiveViewBlockInputStream will not updated. LIVE VIEW is an experimental feature. (vxider) (vxider)
• Fixed ALTER MODIFY TTL wrong behavior which did not allow to delete old TTL expressions. (Vladimir Chebotarev)
• Fixed UBSan report in MergeTreeIndexSet. This fixes #9365 ()
• Fixed the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9160 (alexey-milovidov) (alexey-milovidov)
• Avoid throwing from destructor in Apache Avro 3rd-party library. (Andrew Onyshchuk)
• Don’t commit a batch polled from Kafka partially as it can lead to holes in data. (filimonov)
• Fix joinGet with nullable return types. #9014 ()
• Fix data incompatibility when compressed with T64 codec. #9016 () Fix data type ids in T64 compression codec that leads to wrong (de)compression in affected versions. #9033 ()
• Add setting enable_early_constant_folding and disable it in some cases that leads to errors. #9010 ()
• Fix pushdown predicate optimizer with VIEW and enable the test #9011 ()
• Fix segfault in Merge tables, that can happen when reading from File storages #9387 ()
• Added a check for storage policy in ATTACH PARTITION FROM, REPLACE PARTITION, MOVE TO TABLE. Otherwise it could make data of part inaccessible after restart and prevent ClickHouse to start. #9383 ()
• Fix alters if there is TTL set for table. #8800 ()
• Fix race condition that can happen when SYSTEM RELOAD ALL DICTIONARIES is executed while some dictionary is being modified/added/removed. #8801 ()
• In previous versions Memory database engine use empty data path, so tables are created in path directory (e.g. /var/lib/clickhouse/), not in data directory of database (e.g. /var/lib/clickhouse/db_name). #8753 ()
• Fixed wrong log messages about missing default disk or policy. #9530 ()
• Fix not(has()) for the bloom_filter index of array types. #9407 ()
• Allow first column(s) in a table with Log engine be an alias #9231 ()
• Fix order of ranges while reading from MergeTree table in one thread. It could lead to exceptions from MergeTreeRangeReader or wrong query results. #9050 ()
• Make reinterpretAsFixedString to return FixedString instead of String. #9052 ()
• Avoid extremely rare cases when the user can get wrong error message (Success instead of detailed error description). #9457 ()
• Do not crash when using Template format with empty row template. #8785 ()
• Metadata files for system tables could be created in wrong place #8653 () Fixes #8581.
• Fix data race on exception_ptr in cache dictionary . #9379 ()
• Do not throw an exception for query ATTACH TABLE IF NOT EXISTS. Previously it was thrown if table already exists, despite the IF NOT EXISTS clause. #8967 ()
• Fixed missing closing paren in exception message. #8811 ()
• Avoid message Possible deadlock avoided at the startup of clickhouse-client in interactive mode. #9455 ()
• Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9500 ()
• Prevent losing data in Kafka in rare cases when exception happens after reading suffix but before commit. Fixes #9378 (filimonov)
• Fixed exception in DROP TABLE IF EXISTS (Nikita Vasilev)
• Fix crash when a user tries to ALTER MODIFY SETTING for old-formated MergeTree table engines family. (alesapin)
• Support for UInt64 numbers that don’t fit in Int64 in JSON-related functions. Update SIMDJSON to master. This fixes #9344 ()
• Fixed execution of inversed predicates when non-strictly monotinic functional index is used. #9223 ()
• Don’t try to fold IN constant in GROUP BY #8868 ()
• Fix bug in ALTER DELETE mutations which leads to index corruption. This fixes #9019 and . Additionally fix extremely rare race conditions in ReplicatedMergeTree ALTER queries. #9048 ()
• When the setting compile_expressions is enabled, you can get unexpected column in LLVMExecutableFunction when we use Nullable type #8910 ()
• Multiple fixes for Kafka engine: 1) fix duplicates that were appearing during consumer group rebalance. 2) Fix rare ‘holes’ appeared when data were polled from several partitions with one poll and committed partially (now we always process / commit the whole polled block of messages). 3) Fix flushes by block size (before that only flushing by timeout was working properly). 4) better subscription procedure (with assignment feedback). 5) Make tests work faster (with default intervals and timeouts). Due to the fact that data was not flushed by block size before (as it should according to documentation), that PR may lead to some performance degradation with default settings (due to more often & tinier flushes which are less optimal). If you encounter the performance issue after that change - please increase kafka_max_block_size in the table to the bigger value ( for example CREATE TABLE ...Engine=Kafka ... SETTINGS ... kafka_max_block_size=524288). Fixes #7259 (filimonov)
• Fix Parameter out of bound exception in some queries after PREWHERE optimizations. (Baudouin Giard)
• Fixed the case of mixed-constness of arguments of function arrayZip. (alexey-milovidov)
• When executing CREATE query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes , #3492 (tavplubix)
• Now it’s not possible to create or add columns with simple cyclic aliases like a DEFAULT b, b DEFAULT a. (alesapin)
• Fixed a bug with double move which may corrupt original part. This is relevant if you use (Vladimir Chebotarev)
• Allow interval identifier to correctly parse without backticks. Fixed issue when a query cannot be executed even if the interval identifier is enclosed in backticks or double quotes. This fixes . #9142 ()
• Fixed fuzz test and incorrect behaviour of bitTestAll/bitTestAny functions. #9143 ()
• Fix possible crash/wrong number of rows in LIMIT n WITH TIES when there are a lot of rows equal to n’th row. #9464 ()
• Fix mutations with parts written with enabled insert_quorum. #9463 ()
• Fix data race at destruction of Poco::HTTPServer. It could happen when server is started and immediately shut down. #9468 ()
• Fix bug in which a misleading error message was shown when running SHOW CREATE TABLE a_table_that_does_not_exist. #8899 ()
• Fixed Parameters are out of bound exception in some rare cases when we have a constant in the SELECT clause when we have an ORDER BY and a LIMIT clause. #8892 ()
• Fix mutations finalization, when already done mutation can have status is_done=0. #9217 ()
• Prevent from executing ALTER ADD INDEX for MergeTree tables with old syntax, because it doesn’t work. #8822 ()
• During server startup do not access table, which LIVE VIEW depends on, so server will be able to start. Also remove LIVE VIEW dependencies when detaching LIVE VIEW. LIVE VIEW is an experimental feature. #8824 ()
• Fix possible segfault in MergeTreeRangeReader, while executing PREWHERE. #9106 ()
• Fix possible mismatched checksums with column TTLs. #9451 ()
• Fixed a bug when parts were not being moved in background by TTL rules in case when there is only one volume. #8672 ()
• Fixed the issue Method createColumn() is not implemented for data type Set. This fixes #7799. (alexey-milovidov)
• Now we will try finalize mutations more frequently. (alesapin)
• Fix intDiv by minus one constant (hcz)
• Fix possible race condition in BlockIO. (Nikolai Kochetov)
• Fix bug leading to server termination when trying to use / drop Kafka table created with wrong parameters. (filimonov)
• Added workaround if OS returns wrong result for timer_create function. (alexey-milovidov)
• Fixed error in usage of min_marks_for_seek parameter. Fixed the error message when there is no sharding key in Distributed table and we try to skip unused shards. (Azat Khuzhin)

Improvement

• Implement ALTER MODIFY/DROP queries on top of mutations for ReplicatedMergeTree* engines family. Now ALTERS blocks only at the metadata update stage, and don’t block after that. #8701 ()
• Add ability to rewrite CROSS to INNER JOINs with WHERE section containing unqialified names. #9512 ()
• Make SHOW TABLES and SHOW DATABASES queries support the WHERE expressions and FROM/IN #9076 ()
• Added a setting deduplicate_blocks_in_dependent_materialized_views. #9070 ()
• After recent changes MySQL client started to print binary strings in hex thereby making them not readable (#9032). The workaround in ClickHouse is to mark string columns as UTF-8, which is not always, but usually the case. (Yuriy Baranov)
• Add support of String and FixedString keys for sumMap (Baudouin Giard)
• Support string keys in SummingMergeTree maps (Baudouin Giard)
• Signal termination of thread to the thread pool even if the thread has thrown exception (Ding Xiang Fei)
• Allow to set query_id in clickhouse-benchmark (Anton Popov)
• Don’t allow strange expressions in ALTER TABLE ... PARTITION partition query. This addresses #8835 ()
• The table system.table_engines now provides information about feature support (like supports_ttl or supports_sort_order). #8830 ()
• Enable system.metric_log by default. It will contain rows with values of ProfileEvents, CurrentMetrics collected with “collect_interval_milliseconds” interval (one second by default). The table is very small (usually in order of megabytes) and collecting this data by default is reasonable. #9225 ()
• Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries. Fixes #6964 (Ivan)
• Now temporary LIVE VIEW is created by CREATE LIVE VIEW name WITH TIMEOUT [42] ... instead of CREATE TEMPORARY LIVE VIEW ..., because the previous syntax was not consistent with CREATE TEMPORARY TABLE ... (tavplubix)
• Add text_log.level configuration parameter to limit entries that goes to system.text_log table (Azat Khuzhin)
• Allow to put downloaded part to a disks/volumes according to TTL rules (Vladimir Chebotarev)
• For external MySQL dictionaries, allow to mutualize MySQL connection pool to “share” them among dictionaries. This option significantly reduces the number of connections to MySQL servers. (Clément Rodriguez)
• Show nearest query execution time for quantiles in clickhouse-benchmark output instead of interpolated values. It’s better to show values that correspond to the execution time of some queries. (alexey-milovidov)
• Possibility to add key & timestamp for the message when inserting data to Kafka. Fixes #8969 ()
• If server is run from terminal, highlight thread number, query id and log priority by colors. This is for improved readability of correlated log messages for developers. #8961 ()
• Better exception message while loading tables for Ordinary database. #9527 ()
• Implement arraySlice for arrays with aggregate function states. This fixes #9388 (alexey-milovidov)
• Allow constant functions and constant arrays to be used on the right side of IN operator. (Anton Popov)
• If zookeeper exception has happened while fetching data for system.replicas, display it in a separate column. This implements #9138 ()
• Atomically remove MergeTree data parts on destroy. #8402 ()
• Support row-level security for Distributed tables. #8926 ()
• Now we recognize suffix (like KB, KiB…) in settings values. #8072 ()
• Prevent out of memory while constructing result of a large JOIN. #8637 ()
• Added names of clusters to suggestions in interactive mode in clickhouse-client. #8709 ()
• Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries #8820 ()
• Added column exception_code in system.query_log table. #8770 ()
• Enabled MySQL compatibility server on port 9004 in the default server configuration file. Fixed password generation command in the example in configuration. #8771 ()
• Prevent abort on shutdown if the filesystem is readonly. This fixes #9094 (alexey-milovidov)
• Better exception message when length is required in HTTP POST query. (alexey-milovidov)
• Add _path and _file virtual columns to HDFS and File engines and hdfs and file table functions (Olga Khvostikova)
• Fix error Cannot find column while inserting into MATERIALIZED VIEW in case if new column was added to view’s internal table. #8788 () #8788 (Nikolai Kochetov) (Nikolai Kochetov)
• Fix progress over native client-server protocol, by send progress after final update (like logs). This may be relevant only to some third-party tools that are using native protocol. (Azat Khuzhin)
• Add a system metric tracking the number of client connections using MySQL protocol (). #9015 ()
• From now on, HTTP responses will have X-ClickHouse-Timezone header set to the same timezone value that SELECT timezone() would report. #9493 ()

Performance Improvement

• Improve performance of analysing index with IN (Anton Popov)
• Simpler and more efficient code in Logical Functions + code cleanups. A followup to #8728 ()
• Overall performance improvement (in range of 5%..200% for affected queries) by ensuring even more strict aliasing with C++20 features. #9304 ()
• More strict aliasing for inner loops of comparison functions. #9327 ()
• More strict aliasing for inner loops of arithmetic functions. #9325 ()
• A ~3 times faster implementation for ColumnVector::replicate(), via which ColumnConst::convertToFullColumn() is implemented. Also will be useful in tests when materializing constants. #9293 ()
• Another minor performance improvement to ColumnVector::replicate() (this speeds up the materialize function and higher order functions) an even further improvement to #9293 (Alexander Kazakov)
• Improved performance of stochasticLinearRegression aggregate function. This patch is contributed by Intel. (alexey-milovidov)
• Improve performance of reinterpretAsFixedString function. (alexey-milovidov)
• Do not send blocks to client for Null format in processors pipeline. (Nikolai Kochetov) (Alexander Kuzmenkov)

Build/Testing/Packaging Improvement

• Exception handling now works correctly on Windows Subsystem for Linux. See https://github.com/ClickHouse-Extras/libunwind/pull/3 This fixes #9564 ()
• Replace readline with replxx for interactive line editing in clickhouse-client #8416 ()
• Better build time and less template instantiations in FunctionsComparison. #9324 ()
• Added integration with clang-tidy in CI. See also #6044 (alexey-milovidov)
• Now we link ClickHouse in CI using lld even for gcc. (alesapin)
• Allow to randomize thread scheduling and insert glitches when THREAD_FUZZER_* environment variables are set. This helps testing. (alexey-milovidov)
• Enable secure sockets in stateless tests (tavplubix)
• Make SPLIT_SHARED_LIBRARIES=OFF more robust (Azat Khuzhin)
• Make “performance_introspection_and_logging” test reliable to random server stuck. This may happen in CI environment. See also #9528 ()
• Validate XML in style check. #9550 ()
• Fixed race condition in test 00738_lock_for_inner_table. This test relied on sleep. #9555 ()
• Remove performance tests of type once. This is needed to run all performance tests in statistical comparison mode (more reliable). #9557 ()
• Added performance test for arithmetic functions. #9326 ()
• Added performance test for sumMap and sumMapWithOverflow aggregate functions. Follow-up for #8933 (alexey-milovidov)
• Ensure style of ErrorCodes by style check. (alexey-milovidov)
• Add script for tests history. (alesapin)
• Add GCC warning -Wsuggest-override to locate and fix all places where override keyword must be used. (kreuzerkrieg)
• Ignore weak symbol under Mac OS X because it must be defined (Deleted user)
• Normalize running time of some queries in performance tests. This is done in preparation to run all the performance tests in comparison mode. (alexey-milovidov)
• Fix some tests to support pytest with query tests (Ivan)
• Enable SSL in build with MSan, so server will not fail at startup when running stateless tests (tavplubix)
• Fix database substitution in test results (Ilya Yatsishin)
• Build fixes for miscellaneous platforms (proller) (proller) (proller)
• Added disks section to stateless-with-coverage test docker image (Pavel Kovalenko)
• Get rid of in-source-tree files when building with GRPC (Amos Bird)
• Slightly faster build time by removing SessionCleaner from Context. Make the code of SessionCleaner more simple. (alexey-milovidov)
• Updated checking for hung queries in clickhouse-test script (Alexander Kazakov)
• Removed some useless files from repository. (alexey-milovidov)
• Changed type of math perftests from once to loop. (Nikolai Kochetov)
• Add docker image which allows to build interactive code browser HTML report for our codebase. (alesapin) See
• Suppress some test failures under MSan. #8780 ()
• Speedup “exception while insert” test. This test often time out in debug-with-coverage build. #8711 ()
• Updated libcxx and libcxxabi to master. In preparation to #9304 (alexey-milovidov)
• Fix flacky test 00910_zookeeper_test_alter_compression_codecs. (alexey-milovidov)
• Clean up duplicated linker flags. Make sure the linker won’t look up an unexpected symbol. (Amos Bird)
• Add clickhouse-odbc driver into test images. This allows to test interaction of ClickHouse with ClickHouse via its own ODBC driver. (filimonov)
• Fix several bugs in unit tests. (alesapin)
• Enable -Wmissing-include-dirs GCC warning to eliminate all non-existing includes - mostly as a result of CMake scripting errors (kreuzerkrieg)
• Describe reasons if query profiler cannot work. This is intended for #9144 ()
• Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error and SSL Exception: error:2400006E:random number generator::error retrieving entropy. The issue was present in version 20.1. #8956 ()
• Update Dockerfile for server #8893 ()
• Minor fixes in build-gcc-from-sources script #8774 ()
• Replace numbers to zeros in perftests where number column is not used. This will lead to more clean test results. #9600 ()
• Fix stack overflow issue when using initializer_list in Column constructors. #9367 ()
• Upgrade librdkafka to v1.3.0. Enable bundled rdkafka and gsasl libraries on Mac OS X. #9000 ()
• build fix on GCC 9.2.0 #9306 ()

ClickHouse release v20.1

ClickHouse release v20.1.16.120-stable 2020-60-26

Bug Fix

• Fix rare crash caused by using Nullable column in prewhere condition. Continuation of . #11869 ().
• Don’t allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. (alexey-milovidov).
• Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. (hexiaoting).
• Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. (tavplubix).
• Fix ORDER BY ... WITH FILL over const columns. (Anton Popov).
• Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. (alexey-milovidov).
• Add support for regular expressions with case-insensitive flags. This fixes and fixes #11506. (alexey-milovidov).
• Fix bloom filters for String (data skipping indices). (Azat Khuzhin).
• Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with somehow). #11608 ().
• Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 ().
• Fix trivial error in log message about “Mark cache size was lowered” at server startup. This closes #11399. (alexey-milovidov).
• Now clickhouse-server docker container will prefer IPv6 checking server aliveness. (Ivan Starkov).
• Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes . #11496 ().
• Fix usage of primary key wrapped into a function with ‘FINAL’ modifier and ‘ORDER BY’ optimization. #10715 ().

ClickHouse release v20.1.15.109-stable 2020-06-19

Bug Fix

• Fix excess lock for structure during alter. #11790 ().

ClickHouse release v20.1.14.107-stable 2020-06-11

Bug Fix

• Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 ().

ClickHouse release v20.1.13.105-stable 2020-06-10

Bug Fix

• Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. (alexey-milovidov).
• Fix return compressed size for codecs. (Nikolai Kochetov).
• Fix server crash when a column has compression codec with non-literal arguments. Fixes . #11431 ().
• Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. (Alexey Ilyukhov).
• Fixed geohashesInBox with arguments outside of latitude/longitude range. (Vasily Nemkov).
• Fix possible Pipeline stuck error for queries with external sort and limit. Fixes . #11366 ().
• Fix crash in quantilesExactWeightedArray. #11337 ().
• Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. (Nikolai Kochetov).
• Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. (Ewout).
• Fix very rare race condition in ThreadPool. (alexey-milovidov).
• Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). (alexey-milovidov).
• Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes . #11303 ().
• Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes . This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. (alexey-milovidov).
• Fix crash while reading malformed data in Protobuf format. This fixes , fixes https://github.com/ClickHouse/ClickHouse/issues/11203. (Vitaly Baranov).
• Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. (Nikolai Kochetov).
• If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. (Azat Khuzhin).
• Remove logging from mutation finalization task if nothing was finalized. (alesapin).
• Fixed parseDateTime64BestEffort argument resolution bugs. . #11038 ().
• Fix incorrect raw data size in method getRawData(). #10964 ().
• Fix backward compatibility with tuples in Distributed tables. #10889 ().
• Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 ().
• Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 ().
• Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 ().
• Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. (alexey-milovidov).
• Fix UBSan and MSan report in DateLUT. (alexey-milovidov).
• • Make use of src_type for correct type conversion in key conditions. Fixes . #10791 ().
• Fix parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. (Nikolai Kochetov).
• Fix combinator -OrNull and -OrDefault when combined with -State. (hcz).
• Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes . #10698 ().
• Fix multiple usages of IN operator with the identical set in one query. #10686 ().
• Fix order of parameters in AggregateTransform constructor. #10667 ().
• Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655. (Nikolai Kochetov).
• Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: , #11413. (Azat Khuzhin).
• Fix error the BloomFilter false positive must be a double number between 0 and 1 . #10569 ().
• Fix SELECT of column ALIAS which default expression type different from column type. #10563 ().
• • Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 ().

ClickHouse release v20.1.12.86, 2020-05-26

Bug Fix

• Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 ().
• Fixed data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. (Nikolai Kochetov).
• Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes , #7019, , #7091. (tavplubix).
• Fixed a bug when on SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. (tavplubix).
• Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. (Nikolai Kochetov).
• Fixed the situation when mutation finished all parts, but hung up in is_done=0. (alesapin).
• Fixed overflow at beginning of unix epoch for timezones with fractional offset from UTC. This fixes . #10513 ().
• Fixed improper shutdown of Distributed storage. #10491 ().
• Fixed numeric overflow in simpleLinearRegression over large integers. #10474 ().
• Fixed removing metadata directory when attach database fails. #10442 ().
• Added a check of number and type of arguments when creating BloomFilter index #9623. (Winter Zhang).
• Fixed the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes . #10427 ().
• Prefer fallback_to_stale_replicas over skip_unavailable_shards. #10422 ().
• Fixed wrong flattening of Array(Tuple(...)) data types. This fixes #10259. (alexey-milovidov).
• Fixed wrong behavior in HashTable that caused compilation error when trying to read HashMap from buffer. (palasonic1).
• Fixed possible Pipeline stuck error in ConcatProcessor which could have happened in remote query. (Nikolai Kochetov).
• Fixed error Pipeline stuck with max_rows_to_group_by and group_by_overflow_mode = 'break'. (Nikolai Kochetov).
• Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes . #10188 ().
• Fixed incompatibility when versions prior to 18.12.17 are used on remote servers and newer is used on initiating server, and GROUP BY both fixed and non-fixed keys, and when two-level group by method is activated. #3254 ().

Build/Testing/Packaging Improvement

• Added CA certificates to clickhouse-server docker image. (filimonov).

ClickHouse release v20.1.10.70, 2020-04-17

Bug Fix

• Fix rare possible exception Cannot drain connections: cancel first. (Nikolai Kochetov).
• Fixed bug where ClickHouse would throw 'Unknown function lambda.' error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. (Alexander Kazakov).
• Fix parseDateTimeBestEffort for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes . #10214 ().
• Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 ().
• Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like system.numbers or system.zeros. #10206 ().
• Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092. (alexey-milovidov).
• Add the ability to relax the restriction on non-deterministic functions usage in mutations with allow_nondeterministic_mutations setting. (filimonov).
• Convert blocks if structure does not match on INSERT into table with Distributed engine. (Azat Khuzhin).
• Fix SIGSEGV on INSERT into Distributed table when its structure differs from the underlying tables. (Azat Khuzhin).
• Fix possible rows loss for queries with JOIN and UNION ALL. Fixes , #10113. (Nikolai Kochetov).
• Add arguments check and support identifier arguments for MySQL Database Engine. (Winter Zhang).
• Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. (alesapin).
• Fix error Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform. It happened when setting distributed_aggregation_memory_efficient was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). (Nikolai Kochetov).
• Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (, #8925). (Alexander Kuzmenkov).
• Fix bug in which the necessary tables weren’t retrieved at one of the processing stages of queries to some databases. Fixes . #9949 ().
• Fix 'Not found column in block' error when JOIN appears with TOTALS. Fixes #9839. (Artem Zuikov).
• Fix a bug with ON CLUSTER DDL queries freezing on server startup. (Gagan Arneja).
• Fix TRUNCATE for Join table engine (). #9920 ().
• Fix 'scalar doesn't exist' error in ALTER queries (#9878). (Amos Bird).
• Fix race condition between drop and optimize in ReplicatedMergeTree. (alesapin).
• Fixed DeleteOnDestroy logic in ATTACH PART which could lead to automatic removal of attached part and added few tests. (Vladimir Chebotarev).

Build/Testing/Packaging Improvement

• Fix unit test collapsing_sorted_stream. #9367 ().

ClickHouse release v20.1.9.54, 2020-03-28

Bug Fix

• Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and SET distributed_product_mode = 'local'. #9871 ().
• Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. (alesapin).
• For INSERT queries shard now clamps the settings got from the initiator to the shard’s constaints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix . #9852 ().
• Fix possible exception Got 0 in totals chunk, expected 1 on client. It happened for queries with JOIN in case if right joined table had zero rows. Example: select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;. Fixes #9777. (Nikolai Kochetov).
• Fix SIGSEGV with optimize_skip_unused_shards when type cannot be converted. (Azat Khuzhin).
• Fixed a few cases when timezone of the function argument wasn’t used properly. (Vasily Nemkov).

Improvement

• Remove ORDER BY stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. #9886 ().

Build/Testing/Packaging Improvement

• Clean up duplicated linker flags. Make sure the linker won’t look up an unexpected symbol. (Amos Bird).

ClickHouse release v20.1.8.41, 2020-03-20

Bug Fix

• Fix possible permanent Cannot schedule a task error (due to unhandled exception in ParallelAggregatingBlockInputStream::Handler::onFinish/onFinishThread). This fixes . #9154 ()
• Fix excessive memory consumption in ALTER queries (mutations). This fixes #9533 and . #9754 ()
• Fix bug in backquoting in external dictionaries DDL. This fixes #9619. (alesapin)

ClickHouse release v20.1.7.38, 2020-03-18

Bug Fix

• Fixed incorrect internal function names for sumKahan and sumWithOverflow. I lead to exception while using this functions in remote queries. (Azat Khuzhin). This issue was in all ClickHouse releases.
• Allow ALTER ON CLUSTER of Distributed tables with internal replication. This fixes . #9617 (). This issue was in all ClickHouse releases.
• Fix possible exceptions Size of filter doesn't match size of column and Invalid number of rows in Chunk in MergeTreeRangeReader. They could appear while executing PREWHERE in some cases. Fixes #9132. (Anton Popov)
• Fixed the issue: timezone was not preserved if you write a simple arithmetic expression like time + 1 (in contrast to an expression like time + INTERVAL 1 SECOND). This fixes . #9323 (). This issue was in all ClickHouse releases.
• Now it’s not possible to create or add columns with simple cyclic aliases like a DEFAULT b, b DEFAULT a. #9603 ()
• Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9500 ()
• Fix data race at destruction of Poco::HTTPServer. It could happen when server is started and immediately shut down. #9468 ()
• Fix possible crash/wrong number of rows in LIMIT n WITH TIES when there are a lot of rows equal to n’th row. #9464 ()
• Fix possible mismatched checksums with column TTLs. #9451 ()
• Fix crash when a user tries to ALTER MODIFY SETTING for old-formated MergeTree table engines family. #9435 ()
• Now we will try finalize mutations more frequently. #9427 ()
• Fix replication protocol incompatibility introduced in #8598. (alesapin)
• Fix not(has()) for the bloom_filter index of array types. (achimbab)
• Fixed the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9163 () #9345 ()

Build/Testing/Packaging Improvement

• Exception handling now works correctly on Windows Subsystem for Linux. See This fixes #6480 (sobolevsv)

Bug Fix

• Fix data incompatibility when compressed with T64 codec.
  #9039
• Fix order of ranges while reading from MergeTree table in one thread. Fixes #8964.
  (CurtizJ)
• Fix possible segfault in MergeTreeRangeReader, while executing PREWHERE. Fixes .
  #9106
• Fix reinterpretAsFixedString to return FixedString instead of String.
  #9052
• Fix joinGet with nullable return types. Fixes #8919
  (amosbird)
• Fix fuzz test and incorrect behaviour of bitTestAll/bitTestAny functions.
  (alexey-milovidov)
• Fix the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. Fixes
  #9163
• Fixed execution of inversed predicates when non-strictly monotinic functional index is used. Fixes #9034
  (Akazz)
• Allow to rewrite CROSS to INNER JOIN if there’s [NOT] LIKE operator in WHERE section. Fixes
  #9229
• Allow first column(s) in a table with Log engine be an alias.
  #9231
• Allow comma join with IN() inside. Fixes #7314.
  (4ertus2)
• Improve ALTER MODIFY/ADD queries logic. Now you cannot ADD column without type, MODIFY default expression doesn’t change type of column and MODIFY type doesn’t loose default expression value. Fixes .
  #9227
• Fix mutations finalization, when already done mutation can have status is_done=0.
  #9217
• Support “Processors” pipeline for system.numbers and system.numbers_mt. This also fixes the bug when max_execution_time is not respected.
  #7796
• Fix wrong counting of DictCacheKeysRequestedFound metric.
  #9411
• Added a check for storage policy in ATTACH PARTITION FROM, REPLACE PARTITION, MOVE TO TABLE which otherwise could make data of part inaccessible after restart and prevent ClickHouse to start.
  #9383
• Fixed UBSan report in MergeTreeIndexSet. This fixes #9250
  (alexey-milovidov)
• Fix possible datarace in BlockIO.
  (KochetovNicolai)
• Support for UInt64 numbers that don’t fit in Int64 in JSON-related functions. Update SIMDJSON to master. This fixes
  #9344
• Fix the issue when the amount of free space is not calculated correctly if the data directory is mounted to a separate device. For default disk calculate the free space from data subdirectory. This fixes #7441
  (millb)
• Fix the issue when TLS connections may fail with the message OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error and SSL Exception: error:2400006E:random number generator::error retrieving entropy. Update OpenSSL to upstream master.
  (alexey-milovidov)
• When executing CREATE query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes , #3492. Also fix check for local address in ClickHouseDictionarySource.
  (tabplubix)
• Fix segfault in StorageMerge, which can happen when reading from StorageFile.
  (tabplubix)
• Prevent losing data in Kafka in rare cases when exception happens after reading suffix but before commit. Fixes . Related: #7175
  (filimonov)
• Fix bug leading to server termination when trying to use / drop Kafka table created with wrong parameters. Fixes . Incorporates #9507.
  (filimonov)

New Feature

• Add deduplicate_blocks_in_dependent_materialized_views option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity.
  #9070

ClickHouse release v20.1.2.4, 2020-01-22

Backward Incompatible Change

• Make the setting merge_tree_uniform_read_distribution obsolete. The server still recognizes this setting but it has no effect. #8308 ()
• Changed return type of the function greatCircleDistance to Float32 because now the result of calculation is Float32. #7993 ()
• Now it’s expected that query parameters are represented in “escaped” format. For example, to pass string a<tab>b you have to write a\tb or a\<tab>b and respectively, a%5Ctb or a%5C%09b in URL. This is needed to add the possibility to pass NULL as \N. This fixes #7488. (alexey-milovidov)
• Enable use_minimalistic_part_header_in_zookeeper setting for ReplicatedMergeTree by default. This will significantly reduce amount of data stored in ZooKeeper. This setting is supported since version 19.1 and we already use it in production in multiple services without any issues for more than half a year. Disable this setting if you have a chance to downgrade to versions older than 19.1. (alexey-milovidov)
• Data skipping indices are production ready and enabled by default. The settings allow_experimental_data_skipping_indices, allow_experimental_cross_to_join_conversion and allow_experimental_multiple_joins_emulation are now obsolete and do nothing. (alexey-milovidov)
• Add new ANY JOIN logic for StorageJoin consistent with JOIN operation. To upgrade without changes in behaviour you need add SETTINGS any_join_distinct_right_table_keys = 1 to Engine Join tables metadata or recreate these tables after upgrade. (Artem Zuikov)
• Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see ). #8707 ()

New Feature

• Added information about part paths to system.merges. (Vladimir Chebotarev)
• Add ability to execute SYSTEM RELOAD DICTIONARY query in ON CLUSTER mode. (Guillaume Tassery)
• Add ability to execute CREATE DICTIONARY queries in ON CLUSTER mode. (alesapin)
• Now user’s profile in users.xml can inherit multiple profiles. (Mikhail f. Shiryaev)
• Added system.stack_trace table that allows to look at stack traces of all server threads. This is useful for developers to introspect server state. This fixes . #8344 ()
• Add DateTime64 datatype with configurable sub-second precision. #7170 ()
• Add table function clusterAllReplicas which allows to query all the nodes in the cluster. #8493 ()
• Add aggregate function categoricalInformationValue which calculates the information value of a discrete feature. #8117 ()
• Speed up parsing of data files in CSV, TSV and JSONEachRow format by doing it in parallel. #7780 ()
• Add function bankerRound which performs banker’s rounding. #8112 ()
• Support more languages in embedded dictionary for region names: ‘ru’, ‘en’, ‘ua’, ‘uk’, ‘by’, ‘kz’, ‘tr’, ‘de’, ‘uz’, ‘lv’, ‘lt’, ‘et’, ‘pt’, ‘he’, ‘vi’. #8189 ()
• Improvements in consistency of ANY JOIN logic. Now t1 ANY LEFT JOIN t2 equals t2 ANY RIGHT JOIN t1. #7665 ()
• Add setting any_join_distinct_right_table_keys which enables old behaviour for ANY INNER JOIN. #7665 ()
• Add new SEMI and ANTI JOIN. Old ANY INNER JOIN behaviour now available as SEMI LEFT JOIN. #7665 ()
• Added Distributed format for File engine and file table function which allows to read from .bin files generated by asynchronous inserts into Distributed table. #8535 ()
• Add optional reset column argument for runningAccumulate which allows to reset aggregation results for each new key value. #8326 ()
• Add ability to use ClickHouse as Prometheus endpoint. #7900 ()
• Add section <remote_url_allow_hosts> in config.xml which restricts allowed hosts for remote table engines and table functions URL, S3, HDFS. #7154 ()
• Added function greatCircleAngle which calculates the distance on a sphere in degrees. #8105 ()
• Changed Earth radius to be consistent with H3 library. #8105 ()
• Added JSONCompactEachRow and JSONCompactEachRowWithNamesAndTypes formats for input and output. #7841 ()
• Added feature for file-related table engines and table functions (File, S3, URL, HDFS) which allows to read and write gzip files based on additional engine parameter or file extension. #7840 ()
• Added the randomASCII(length) function, generating a string with a random set of ASCII printable characters. (BayoNet)
• Added function JSONExtractArrayRaw which returns an array on unparsed json array elements from JSON string. (Oleg Matrokhin)
• Add arrayZip function which allows to combine multiple arrays of equal lengths into one array of tuples. (Winter Zhang)
• Add ability to move data between disks according to configured TTL-expressions for *MergeTree table engines family. (Vladimir Chebotarev)
• Added new aggregate function avgWeighted which allows to calculate weighted average. (Andrey Bodrov)
• Now parallel parsing is enabled by default for TSV, TSKV, CSV and JSONEachRow formats. (Nikita Mikhaylov)
• Add several geo functions from H3 library: h3GetResolution, h3EdgeAngle, h3EdgeLength, h3IsValid and h3kRing. (Konstantin Malanchev)
• Added support for brotli (br) compression in file-related storages and table functions. This fixes . #8526 ()
• Add groupBit* functions for the SimpleAggregationFunction type. #8485 ()

Bug Fix

• Fix rename of tables with Distributed engine. Fixes issue . #8306 ()
• Now dictionaries support EXPRESSION for attributes in arbitrary string in non-ClickHouse SQL dialect. #8098 ()
• Fix broken INSERT SELECT FROM mysql(...) query. This fixes #8070 and . #8234 ()
• Fix error “Mismatch column sizes” when inserting default Tuple from JSONEachRow. This fixes #5653. (tavplubix)
• Now an exception will be thrown in case of using WITH TIES alongside LIMIT BY. Also add ability to use TOP with LIMIT BY. This fixes . #7637 ()
• Fix unintendent dependency from fresh glibc version in clickhouse-odbc-bridge binary. #8046 ()
• Fix bug in check function of *MergeTree engines family. Now it doesn’t fail in case when we have equal amount of rows in last granule and last mark (non-final). #8047 ()
• Fix insert into Enum* columns after ALTER query, when underlying numeric type is equal to table specified type. This fixes #7836. (Anton Popov)
• Allowed non-constant negative “size” argument for function substring. It was not allowed by mistake. This fixes . #7703 ()
• Fix parsing bug when wrong number of arguments passed to (O|J)DBC table engine. #7709 ()
• Using command name of the running clickhouse process when sending logs to syslog. In previous versions, empty string was used instead of command name. #8460 ()
• Fix check of allowed hosts for localhost. This PR fixes the solution provided in #8241. (Vitaly Baranov)
• Fix rare crash in argMin and argMax functions for long string arguments, when result is used in runningAccumulate function. This fixes #8341 ()
• Fix memory overcommit for tables with Buffer engine. #8345 ()
• Fixed potential bug in functions that can take NULL as one of the arguments and return non-NULL. #8196 ()
• Better metrics calculations in thread pool for background processes for MergeTree table engines. #8194 ()
• Fix function IN inside WHERE statement when row-level table filter is present. Fixes #6687 (Ivan)
• Now an exception is thrown if the integral value is not parsed completely for settings values. (Mikhail Korotov)
• Fix exception when aggregate function is used in query to distributed table with more than two local shards. (小路)
• Now bloom filter can handle zero length arrays and doesn’t perform redundant calculations. (achimbab)
• Fixed checking if a client host is allowed by matching the client host to host_regexp specified in users.xml. (Vitaly Baranov)
• Relax ambiguous column check that leads to false positives in multiple JOIN ON section. (Artem Zuikov)
• Fixed possible server crash (std::terminate) when the server cannot send or write data in JSON or XML format with values of String data type (that require UTF-8 validation) or when compressing result data with Brotli algorithm or in some other rare cases. This fixes #8384 ()
• Fix race condition in StorageDistributedDirectoryMonitor found by CI. This fixes #8364. (Nikolai Kochetov)
• Now background merges in *MergeTree table engines family preserve storage policy volume order more accurately. (Vladimir Chebotarev)
• Now table engine Kafka works properly with Native format. This fixes #7337 . #8016 ()
• Fixed formats with headers (like CSVWithNames) which were throwing exception about EOF for table engine Kafka. #8016 ()
• Fixed a bug with making set from subquery in right part of IN section. This fixes #5767 and . #7755 ()
• Fix possible crash while reading from storage File. #7756 ()
• Fixed reading of the files in Parquet format containing columns of type list. #8334 ()
• Fix error Not found column for distributed queries with PREWHERE condition dependent on sampling key if max_parallel_replicas > 1. #7913 ()
• Fix error Not found column if query used PREWHERE dependent on table’s alias and the result set was empty because of primary key condition. #7911 ()
• Fixed return type for functions rand and randConstant in case of Nullable argument. Now functions always return UInt32 and never Nullable(UInt32). #8204 ()
• Disabled predicate push-down for WITH FILL expression. This fixes #7784. (Winter Zhang)
• Fixed incorrect count() result for SummingMergeTree when FINAL section is used. #7786 ()
• Fix possible incorrect result for constant functions from remote servers. It happened for queries with functions like version(), uptime(), etc. which returns different constant values for different servers. This fixes #7666. (Nikolai Kochetov)
• Fix complicated bug in push-down predicate optimization which leads to wrong results. This fixes a lot of issues on push-down predicate optimization. (Winter Zhang)
• Fix crash in CREATE TABLE .. AS dictionary query. (Azat Khuzhin)
• Several improvements ClickHouse grammar in .g4 file. (taiyang-li)
• Fix bug that leads to crashes in JOINs with tables with engine Join. This fixes #8254 #8100. (Artem Zuikov)
• Fix redundant dictionaries reload on CREATE DATABASE. (Azat Khuzhin)
• Limit maximum number of streams for read from StorageFile and StorageHDFS. Fixes . #7981 ()
• Fix bug in ALTER ... MODIFY ... CODEC query, when user specify both default expression and codec. Fixes 8593. (alesapin)
• Fix error in background merge of columns with SimpleAggregateFunction(LowCardinality) type. (Nikolai Kochetov)
• Fixed type check in function toDateTime64. (Vasily Nemkov)
• Now server do not crash on LEFT or FULL JOIN with and Join engine and unsupported join_use_nulls settings. (Artem Zuikov)
• Now DROP DICTIONARY IF EXISTS db.dict query doesn’t throw exception if db doesn’t exist. (Vitaly Baranov)
• Fix possible crashes in table functions (file, mysql, remote) caused by usage of reference to removed IStorage object. Fix incorrect parsing of columns specified at insertion into table function. (tavplubix)
• Ensure network be up before starting clickhouse-server. This fixes . #8570 ()
• Fix timeouts handling for secure connections, so queries doesn’t hang indefenitely. This fixes #8126. (alexey-milovidov)
• Fix clickhouse-copier‘s redundant contention between concurrent workers. (Ding Xiang Fei)
• Now mutations doesn’t skip attached parts, even if their mutation version were larger than current mutation version. (Zhichang Yu) (alesapin)
• Ignore redundant copies of *MergeTree data parts after move to another disk and server restart. (Vladimir Chebotarev)
• Fix crash in FULL JOIN with LowCardinality in JOIN key. (Artem Zuikov)
• Forbidden to use column name more than once in insert query like INSERT INTO tbl (x, y, x). This fixes , #7681. (alesapin)
• Added fallback for detection the number of physical CPU cores for unknown CPUs (using the number of logical CPU cores). This fixes . #7726 ()
• Fix There's no column error for materialized and alias columns. #8210 ()
• Fixed sever crash when EXISTS query was used without TABLE or DICTIONARY qualifier. Just like EXISTS t. This fixes #8172. This bug was introduced in version 19.17. (alexey-milovidov)
• Fix rare bug with error "Sizes of columns doesn't match" that might appear when using SimpleAggregateFunction column. (Boris Granveaud)
• Fix bug where user with empty allow_databases got access to all databases (and same for allow_dictionaries). (DeifyTheGod)
• Fix client crash when server already disconnected from client. (Azat Khuzhin)
• Fix ORDER BY behaviour in case of sorting by primary key prefix and non primary key suffix. (Anton Popov)
• Check if qualified column present in the table. This fixes . #7758 ()
• Fixed behavior with ALTER MOVE ran immediately after merge finish moves superpart of specified. Fixes #8103. (Vladimir Chebotarev)
• Fix possible server crash while using UNION with different number of columns. Fixes . #7929 ()
• Fix size of result substring for function substr with negative size. #8589 ()
• Now server does not execute part mutation in MergeTree if there are not enough free threads in background pool. #8588 ()
• Fix a minor typo on formatting UNION ALL AST. #7999 ()
• Fixed incorrect bloom filter results for negative numbers. This fixes #8317. (Winter Zhang)
• Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that will cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. (alexey-milovidov)
• Fix incorrect result because of integers overflow in arrayIntersect. (Nikolai Kochetov)
• Now OPTIMIZE TABLE query will not wait for offline replicas to perform the operation. (javi santana)
• Fixed ALTER TTL parser for Replicated*MergeTree tables. (Vladimir Chebotarev)
• Fix communication between server and client, so server read temporary tables info after query failure. (Azat Khuzhin)
• Fix bitmapAnd function error when intersecting an aggregated bitmap and a scalar bitmap. (Yue Huang)
• Refine the definition of ZXid according to the ZooKeeper Programmer’s Guide which fixes bug in clickhouse-cluster-copier. (Ding Xiang Fei)
• odbc table function now respects external_table_functions_use_nulls setting. (Vasily Nemkov)
• Fixed bug that lead to a rare data race. (Alexander Kazakov)
• Now SYSTEM RELOAD DICTIONARY reloads a dictionary completely, ignoring update_field. This fixes . #8037 ()
• Add ability to check if dictionary exists in create query. #8032 ()
• Fix Float* parsing in Values format. This fixes #7817. (tavplubix)
• Fix crash when we cannot reserve space in some background operations of *MergeTree table engines family. (Vladimir Chebotarev)
• Fix crash of merge operation when table contains SimpleAggregateFunction(LowCardinality) column. This fixes . #8522 ()
• Restore support of all ICU locales and add the ability to apply collations for constant expressions. Also add language name to system.collations table. #8051 ()
• Fix bug when external dictionaries with zero minimal lifetime (LIFETIME(MIN 0 MAX N), LIFETIME(N)) don’t update in background. #7983 ()
• Fix crash when external dictionary with ClickHouse source has subquery in query. #8351 ()
• Fix incorrect parsing of file extension in table with engine URL. This fixes #8157. (Andrey Bodrov)
• Fix CHECK TABLE query for *MergeTree tables without key. Fixes . #7979 ()
• Fixed conversion of Float64 to MySQL type. #8079 ()
• Now if table was not completely dropped because of server crash, server will try to restore and load it. #8176 ()
• Fixed crash in table function file while inserting into file that doesn’t exist. Now in this case file would be created and then insert would be processed. #8177 ()
• Fix rare deadlock which can happen when trace_log is in enabled. #7838 ()
• Add ability to work with different types besides Date in RangeHashed external dictionary created from DDL query. Fixes 7899. (alesapin)
• Fixes crash when now64() is called with result of another function. (Vasily Nemkov)
• Fixed bug with detecting client IP for connections through mysql wire protocol. (Dmitry Muzyka)
• Fix empty array handling in arraySplit function. This fixes . #7747 ()
• Fixed the issue when pid-file of another running clickhouse-server may be deleted. #8487 ()
• Fix dictionary reload if it has invalidate_query, which stopped updates and some exception on previous update tries. #8029 ()
• Fixed error in function arrayReduce that may lead to “double free” and error in aggregate function combinator Resample that may lead to memory leak. Added aggregate function aggThrow. This function can be used for testing purposes. #8446 ()

Improvement

• Improved logging when working with S3 table engine. (Grigory Pervakov)
• Printed help message when no arguments are passed when calling clickhouse-local. This fixes . #8230 ()
• Add setting mutations_sync which allows to wait ALTER UPDATE/DELETE queries synchronously. #8237 ()
• Allow to set up relative user_files_path in config.xml (in the way similar to format_schema_path). #7632 ()
• Add exception for illegal types for conversion functions with -OrZero postfix. #7880 ()
• Simplify format of the header of data sending to a shard in a distributed query. #8044 ()
• Live View table engine refactoring. #8519 ()
• Add additional checks for external dictionaries created from DDL-queries. #8127 ()
• Fix error Column ... already exists while using FINAL and SAMPLE together, e.g. select count() from table final sample 1/2. Fixes #5186. (Nikolai Kochetov)
• Now table the first argument of joinGet function can be table indentifier. (Amos Bird)
• Allow using MaterializedView with subqueries above Kafka tables. (filimonov)
• Now background moves between disks run it the seprate thread pool. (Vladimir Chebotarev)
• SYSTEM RELOAD DICTIONARY now executes synchronously. (Vitaly Baranov)
• Stack traces now display physical addresses (offsets in object file) instead of virtual memory addresses (where the object file was loaded). That allows the use of addr2line when binary is position independent and ASLR is active. This fixes . #8387 ()
• Support new syntax for row-level security filters: <table name='table_name'>…</table>. Fixes #5779. (Ivan)
• Now cityHash function can work with Decimal and UUID types. Fixes . #7693 ()
• Removed fixed index granularity (it was 1024) from system logs because it’s obsolete after implementation of adaptive granularity. #7698 ()
• Enabled MySQL compatibility server when ClickHouse is compiled without SSL. #7852 ()
• Now server checksums distributed batches, which gives more verbose errors in case of corrupted data in batch. #7914 ()
• Support DROP DATABASE, DETACH TABLE, DROP TABLE and ATTACH TABLE for MySQL database engine. #8202 ()
• Add authentication in S3 table function and table engine. #7623 ()
• Added check for extra parts of MergeTree at different disks, in order to not allow to miss data parts at undefined disks. #8118 ()
• Enable SSL support for Mac client and server. #8297 ()
• Now ClickHouse can work as MySQL federated server (see https://dev.mysql.com/doc/refman/5.7/en/federated-create-server.html). (Maxim Fedotov)
• clickhouse-client now only enable bracketed-paste when multiquery is on and multiline is off. This fixes (#7757)[]. #7761 ()
• Support Array(Decimal) in if function. #7721 ()
• Support Decimals in arrayDifference, arrayCumSum and arrayCumSumNegative functions. #7724 ()
• Added lifetime column to system.dictionaries table. #6820 (kekekekule)
• Improved check for existing parts on different disks for *MergeTree table engines. Addresses . #8440 ()
• Integration with AWS SDK for S3 interactions which allows to use all S3 features out of the box. #8011 ()
• Added support for subqueries in Live View tables. #7792 ()
• Check for using Date or DateTime column from TTL expressions was removed. #7920 ()
• Information about disk was added to system.detached_parts table. #7833 ()
• Now settings max_(table|partition)_size_to_drop can be changed without a restart. #7779 ()
• Slightly better usability of error messages. Ask user not to remove the lines below Stack trace:. #7897 ()
• Better reading messages from Kafka engine in various formats after #7935. (Ivan)
• Better compatibility with MySQL clients which don’t support sha2_password auth plugin. (Yuriy Baranov)
• Support more column types in MySQL compatibility server. (Yuriy Baranov)
• Implement ORDER BY optimization for Merge, Buffer and Materilized View storages with underlying MergeTree tables. (Anton Popov)
• Now we always use POSIX implementation of getrandom to have better compatibility with old kernels (< 3.17). (Amos Bird)
• Better check for valid destination in a move TTL rule. (Vladimir Chebotarev)
• Better checks for broken insert batches for Distributed table engine. (Azat Khuzhin)
• Add column with array of parts name which mutations must process in future to system.mutations table. (alesapin)
• Parallel merge sort optimization for processors. (Nikolai Kochetov)
• The settings mark_cache_min_lifetime is now obsolete and does nothing. In previous versions, mark cache can grow in memory larger than mark_cache_size to accomodate data within mark_cache_min_lifetime seconds. That was leading to confusion and higher memory usage than expected, that is especially bad on memory constrained systems. If you will see performance degradation after installing this release, you should increase the mark_cache_size. (alexey-milovidov)
• Preparation to use tid everywhere. This is needed for . #8276 ()

Performance Improvement

• Performance optimizations in processors pipeline. (Nikolai Kochetov)
• Non-blocking updates of expired keys in cache dictionaries (with permission to read old ones). (Nikita Mikhaylov)
• Compile ClickHouse without -fno-omit-frame-pointer globally to spare one more register. (Amos Bird)
• Speedup greatCircleDistance function and add performance tests for it. (Olga Khvostikova)
• Improved performance of function roundDown. (alexey-milovidov)
• Improved performance of max, min, argMin, argMax for DateTime64 data type. (Vasily Nemkov)
• Improved performance of sorting without a limit or with big limit and external sorting. (alexey-milovidov)
• Improved performance of formatting floating point numbers up to 6 times. (alexey-milovidov)
• Improved performance of modulo function. (Amos Bird)
• Optimized ORDER BY and merging with single column key. (alexey-milovidov)
• Better implementation for arrayReduce, -Array and -State combinators. (Amos Bird)
• Now PREWHERE should be optimized to be at least as efficient as WHERE. (Amos Bird)
• Improve the way round and roundBankers handling negative numbers. (hcz)
• Improved decoding performance of DoubleDelta and Gorilla codecs by roughly 30-40%. This fixes . #8019 ()
• Improved performance of base64 related functions. #8444 ()
• Added a function geoDistance. It is similar to greatCircleDistance but uses approximation to WGS-84 ellipsoid model. The performance of both functions are near the same. #8086 ()
• Faster min and max aggregation functions for Decimal data type. #8144 ()
• Vectorize processing arrayReduce. #7608 ()
• if chains are now optimized as multiIf. #8355 ()
• Fix performance regression of Kafka table engine introduced in 19.15. This fixes #7261. (filimonov)
• Removed “pie” code generation that gcc from Debian packages occasionally brings by default. (alexey-milovidov)
• Parallel parsing data formats (Nikita Mikhaylov)
• Enable optimized parser of Values with expressions by default (input_format_values_deduce_templates_of_expressions=1). (tavplubix)

Build/Testing/Packaging Improvement

• Build fixes for ARM and in minimal mode. #8304 ()
• Add coverage file flush for clickhouse-server when std::atexit is not called. Also slightly improved logging in stateless tests with coverage. #8267 ()
• Update LLVM library in contrib. Avoid using LLVM from OS packages. #8258 ()
• Make bundled curl build fully quiet. #8232 (Pavel Kovalenko)
• Fix some MemorySanitizer warnings. (Alexander Kuzmenkov)
• Use add_warning and no_warning macros in CMakeLists.txt. (Ivan)
• Add support of Minio S3 Compatible object () for better integration tests. #7863 (Pavel Kovalenko)
• Imported libc headers to contrib. It allows to make builds more consistent across various systems (only for x86_64-linux-gnu). (alexey-milovidov)
• Remove -fPIC from some libraries. (alexey-milovidov)
• Clean CMakeLists.txt for curl. See #8459 ()
• Silent warnings in CapNProto library. #8220 ()
• Add performance tests for short string optimized hash tables. #7679 ()
• Now ClickHouse will build on AArch64 even if MADV_FREE is not available. This fixes #8027. (Amos Bird)
• Update zlib-ng to fix memory sanitizer problems. #8206 ()
• Enable internal MySQL library on non-Linux system, because usage of OS packages is very fragile and usually doesn’t work at all. This fixes #5765. (alexey-milovidov)
• Fixed build on some systems after enabling libc++. This supersedes . #8380 ()
• Make Field methods more type-safe to find more errors. #7386 (Alexander Kuzmenkov)
• Added missing files to the libc-headers submodule. (alexey-milovidov)
• Fix wrong JSON quoting in performance test output. (Nikolai Kochetov)
• Now stack trace is displayed for std::exception and Poco::Exception. In previous versions it was available only for DB::Exception. This improves diagnostics. (alexey-milovidov)
• Porting clock_gettime and clock_nanosleep for fresh glibc versions. (Amos Bird)
• Enable part_log in example config for developers. (alexey-milovidov)
• Fix async nature of reload in 01036_no_superfluous_dict_reload_on_create_database*. (Azat Khuzhin)
• Fixed codec performance tests. (Vasily Nemkov)
• Add install scripts for .tgz build and documentation for them. #8591 ()
• Removed old ZSTD test (it was created in year 2016 to reproduce the bug that pre 1.0 version of ZSTD has had). This fixes #8618. (alexey-milovidov)
• Fixed build on Mac OS Catalina. (meo)
• Increased number of rows in codec performance tests to make results noticeable. (Vasily Nemkov)
• In debug builds, treat LOGICAL_ERROR exceptions as assertion failures, so that they are easier to notice. (Alexander Kuzmenkov)
• Make formats-related performance test more deterministic. (alexey-milovidov)
• Update lz4 to fix a MemorySanitizer failure. (Alexander Kuzmenkov)
• Suppress a known MemorySanitizer false positive in exception handling. (Alexander Kuzmenkov)
• Update gcc and g++ to version 9 in build/docker/build.sh (TLightSky)
• Add performance test case to test that PREWHERE is worse than WHERE. (Amos Bird)
• Progress towards fixing one flacky test. (alexey-milovidov)
• Avoid MemorySanitizer report for data from libunwind. (alexey-milovidov)
• Updated libc++ to the latest version. (alexey-milovidov)
• Build ICU library from sources. This fixes . #8219 ()
• Switched from libressl to openssl. ClickHouse should support TLS 1.3 and SNI after this change. This fixes #8171. (alexey-milovidov)
• Fixed UBSan report when using chacha20_poly1305 from SSL (happens on connect to ). #8214 ()
• Fix mode of default password file for .deb linux distros. #8075 ()
• Improved expression for getting clickhouse-server PID in clickhouse-test. #8063 ()
• Updated contrib/googletest to v1.10.0. #8587 ()
• Fixed ThreadSaninitizer report in base64 library. Also updated this library to the latest version, but it doesn’t matter. This fixes #8397. (alexey-milovidov)
• Fix 00600_replace_running_query for processors. (Nikolai Kochetov)
• Remove support for tcmalloc to make CMakeLists.txt simpler. (alexey-milovidov)
• Release gcc builds now use libc++ instead of libstdc++. Recently libc++ was used only with clang. This will improve consistency of build configurations and portability. (alexey-milovidov)
• Enable ICU library for build with MemorySanitizer. (alexey-milovidov)
• Suppress warnings from CapNProto library. (alexey-milovidov)
• Removed special cases of code for tcmalloc, because it’s no longer supported. (alexey-milovidov)
• In CI coverage task, kill the server gracefully to allow it to save the coverage report. This fixes incomplete coverage reports we’ve been seeing lately. (alesapin)
• Performance tests for all codecs against Float64 and UInt64 values. (Vasily Nemkov)
• termcap is very much deprecated and lead to various problems (f.g. missing “up” cap and echoing ^J instead of multi line) . Favor terminfo or bundled ncurses. (Amos Bird)
• Fix test_storage_s3 integration test. (Nikolai Kochetov)
• Support StorageFile(<format>, null) to insert block into given format file without actually write to disk. This is required for performance tests. (Amos Bird)
• Added argument --print-time to functional tests which prints execution time per test. (Nikolai Kochetov)
• Added asserts to KeyCondition while evaluating RPN. This will fix warning from gcc-9. (alexey-milovidov)
• Dump cmake options in CI builds. (Alexander Kuzmenkov)
• Don’t generate debug info for some fat libraries. (alexey-milovidov)
• Make log_to_console.xml always log to stderr, regardless of is it interactive or not. (Alexander Kuzmenkov)
• Removed some unused features from clickhouse-performance-test tool. (alexey-milovidov)
• Now we will also search for lld-X with corresponding clang-X version. (alesapin)
• Parquet build improvement. (maxulan)
• More GCC warnings (kreuzerkrieg)
• Package for Arch Linux now allows to run ClickHouse server, and not only client. (Vladimir Chebotarev)
• Fix test with processors. Tiny performance fixes. (Nikolai Kochetov)
• Update contrib/protobuf. (Matwey V. Kornilov)
• In preparation of switching to c++20 as a new year celebration. “May the C++ force be with ClickHouse.” (Amos Bird)

Experimental Feature

• Added experimental setting min_bytes_to_use_mmap_io. It allows to read big files without copying data from kernel to userspace. The setting is disabled by default. Recommended threshold is about 64 MB, because mmap/munmap is slow. #8520 ()
• Reworked quotas as a part of access control system. Added new table system.quotas, new functions currentQuota, currentQuotaKey, new SQL syntax CREATE QUOTA, ALTER QUOTA, DROP QUOTA, SHOW QUOTA. #7257 ()
• Allow skipping unknown settings with warnings instead of throwing exceptions. #7653 ()
• Reworked row policies as a part of access control system. Added new table system.row_policies, new function currentRowPolicies(), new SQL syntax CREATE POLICY, ALTER POLICY, DROP POLICY, SHOW CREATE POLICY, . #7808 ()

Security Fix

• Fixed the possibility of reading directories structure in tables with File table engine. This fixes . #8537 ()