Memory Limit Exceeded Analysis

    When the query or import error is reported, the possible reasons are: the process memory exceeds the limit, the remaining available memory of the system is insufficient, and the memory limit for a single query execution is exceeded.

    When the following error is returned, it means that the process memory exceeds the limit, or the remaining available memory of the system is insufficient. The specific reason depends on the memory statistics.

    1. Memory limit exceeded:<consuming tracker:<Query#Id=3c88608cf35c461d-95fe88969aa6fc30>: It is found that the memory limit is exceeded during the memory application process of query 3c88608cf35c461d-95fe88969aa6fc30.
    2. process memory used 2.68 GB exceed limit 2.47 GB or sys mem available 50.95 GB less than low water mark 3.20 GB, failed alloc size 2.00 MB: The reason for exceeding the limit is that the 2.68GB of memory used by the BE process exceeds the limit of 2.47GB limit, the value of limit comes from mem_limit * system MemTotal in be.conf, which is equal to 80% of the total memory of the operating system by default. The remaining available memory of the current operating system is 50.95 GB, which is still higher than the minimum water level of 3.2GB. This time, we are trying to apply for 2MB of memory.
    3. executing msg:<execute:<ExecNode:VAGGREGATION_NODE (id=7)>>, backend 172.24.47.117 process memory used 2.68 GB, limit 2.47 GB: The location of this memory application is ExecNode:VAGGREGATION_NODE (id= 7)>, the current IP of the BE node is 172.1.1.1, and print the memory statistics of the BE node again.

    At the same time, you can find the following log in log/be.INFO to confirm whether the memory usage of the current process meets expectations. The log is also divided into three parts:

    1. Process Memory Summary: process memory statistics.
    2. Alloc Stacktrace: The stack that triggers the memory overrun detection, which is not necessarily the location of the large memory application.
    3. Memory Tracker Summary: Process memory tracker statistics, refer to to analyze the location of memory usage. Notice:
    4. When the process memory exceeds the limit, BE will trigger memory GC.

    When the available memory of the system in the error message is less than the low water mark, it is also treated as a process memory limit. The value of the available memory of the system comes from MemAvailable in /proc/meminfo. When is insufficient, continue to use the memory The application may return std::bad_alloc or cause OOM of the BE process. Because both refreshing process memory statistics and BE memory GC have a certain lag, a small part of the memory buffer is reserved as a low water mark to avoid OOM as much as possible.

    The low water mark defaults to a maximum of 1.6G, calculated based on MemTotal, vm/min_free_kbytes, confg::mem_limit, config::max_sys_mem_available_low_water_mark_bytes, and avoid wasting too much memory. Among them, MemTotal is the total memory of the system, and the value also comes from /proc/meminfo; vm/min_free_kbytes is the buffer reserved by the operating system for the memory GC process, and the value is usually between 0.4% and 5%. vm/min_free_kbytes may be 5% on some cloud servers, which will lead to visually that the available memory of the system is less than the real value; increasing config::max_sys_mem_available_low_water_mark_bytes will reserve more for Full GC on machines with more than 16G memory If there are more memory buffers, otherwise, the memory will be fully used as much as possible.

    Query or import a single execution memory limit

    When the following error is returned, it means that the memory limit of a single execution has been exceeded.

    1. : It is found that the memory limit is exceeded during the memory application process of query f78208b15e064527-a84c5c0b04c04fcf.
    2. failed alloc size 1.03 MB, exceeded tracker:<Query#Id=f78208b15e064527-a84c5c0b04c04fcf>, limit 100.00 MB, peak used 99.29 MB, current used 99.25 MB: The memory requested this time is 1.03 MB The current consumption of f78208b15e064527-a84c5c0b04c04fcf memory tracker is 99.28MB plus 1.03MB, which exceeds the limit of 100MB. The value of limit comes from exec_mem_limit in session veriables, and the default is 4G.
    3. executing msg:<execute:<ExecNode:VHASH_JOIN_NODE (id=4)>>. backend 172.24.47.117 process memory used 1.13 GB, limit 98.92 GB. If query tracker exceeds, set exec_mem_limit=8G to change limit, details mem usage see be.INFO.: The location of this memory application is VHASH_JOIN_NODE (id=4), and it prompts that set exec_mem_limit can be used to increase the memory limit of a single query.

    After set global enable_profile=true, you can print a log in log/be.INFO when a single query memory exceeds the limit, to confirm whether the current query memory usage meets expectations. At the same time, you can find the following logs in log/be.INFO to confirm whether the current query memory usage meets expectations. The logs are also divided into three parts:

    1. Process Memory Summary: process memory statistics.
    2. Alloc Stacktrace: The stack that triggers the memory overrun detection, which is not necessarily the location of the large memory application.