Docker & Kernel Configuration

    When running M3DB inside Docker, it is recommended to add the SYS_RESOURCE capability to the container (using the --cap-add argument to docker run) so that it can raise its file limits:

    If M3DB is being run as a non-root user, M3’s setcap images are required:

      More information on Docker’s capability settings can be found here.

      M3DB uses a lot of mmap-ed files for performance, as a result, you might need to bump vm.max_map_count. We suggest setting this value to 3000000, so you don’t have to come back and debug issues later.

      On Linux, you can increase the limits by running the following command as root:

      vm.swappiness controls how much the virtual memory subsystem will try to swap to disk. By default, the kernel configures this value to 60, and will try to swap out items in memory even when there is plenty of RAM available to the system.

      We recommend sizing clusters such that M3DB is running on a substrate (hosts/containers) such that no-swapping is necessary, i.e. the process is only using 30-50% of the maximum available memory. And therefore recommend setting the value of vm.swappiness to 1. This tells the kernel to swap as little as possible, without altogether disabling swapping.

      On Linux, you can configure this by running the following as root:

      1. sysctl -w vm.swappiness=1

      To set this value permanently, update the vm.swappiness setting in /etc/sysctl.conf.

      M3DB also can use a high number of files and we suggest setting a high max open number of files due to per partition fileset volumes.

      and

      1. sysctl -n fs.nr_open

      to see the kernel and process limits respectively. If either of the values are less than three million (our minimum recommended value), then you can update them with the following commands:

        To set these values permanently, update the fs.file-max and fs.nr_open settings in /etc/sysctl.conf.

        Alternatively, if you wish to have M3DB run under systemd you can use our which will set sane defaults. Keep in mind that you’ll still need to configure the kernel and process limits because systemd will not allow a process to exceed them and will silently fallback to a default value which could cause M3DB to crash due to hitting the file descriptor limit. Also note that systemd has a system.conf file and a user.conf file which may contain limits that the service-specific configuration files cannot override. Be sure to check that those files aren’t configured with values lower than the value you configure at the service level.

        Before running the process make sure the limits are set, if running manually you can raise the limit for the current user with ulimit -n 3000000.