RocksDB’s data files are usually generated in an appending way. File system may choose buffer the write until the dirty pages hit a threshold and write out all of those pages all together. This can create a burst of write I/O and cause the online I/Os to wait too long and cause long query latency. Rather, you can ask RocksDB to periodically hint OS to write out outstanding dirty pages by setting for SST files and options.wal_bytes_per_sync for WAL files. Underlying it calls sync_file_range() on Linux every time a file is appended for such size. The most recent pages are not included in the range sync.

Rate Limiter

You can control the total rate RocksDB writes to data files through options.rate_limiter, in order to reserve enough I/O bandwidth to online queries. See Rate Limiter for details.

When appending a file, RocksDB has internal buffering of files before writing to the file system, unless an explicit fsync is needed. The max size of this buffer can be controlled by options.writable_file_max_buffer_size. Tuning this parameter is more critical in [[Direct IO] mode or to a file system without page cache. With non-direct I/O mode, enlarging this buffer only reduces number of write() system calls and is unlikely to change the I/O behavior, so unless this is what you want, it may be desirable to keep the default value 0 to save the memory.

File Deletion

Deletion of obsolete DB files can be rate limited by configuring the delete scheduler. This is especially useful on flash devices to limit read latency spikes due to a burst of deletions. See for details.

Unfortunately, there isn’t a good setting for mixed workload. There is an ongoing project to address this, by doing read-ahead for iterators inside RocksDB.

Compaction inputs

Compaction inputs are special. They are long sequential reads, so applying the same fadvise hint as user reads is usually not optimal. Also, usually, compaction input files are often going to be deleted soon, though there is no guarantee. RocksDB provides multiple ways to deal with that:

fadvise hint

RocksDB will call fadvise to any compaction input file according to . This can override the fadvise random setting since a file is picked as a compaction input.

Use a different file descriptor for compaction inputs

If options.new_table_reader_for_compaction_inputs = true, RocksDB will use different file descriptors for compaction inputs. This can avoid the mixture of fadvise setting for normal data files and compaction inputs. The limitation of the setting is that, RocksDB does not just create a new file descriptor, but read index, filter and other meta blocks again and store them in memory, so that it takes extra I/O and use more memory.

readahead for compaction inputs

It is critical to set the option if Direct IO is on or the file system doesn’t support readahead.

Rather than control the I/O through file system hints shown above, you can enable direct I/O in RocksDB to allow RocksDB to directly control I/O, using option use_direct_reads and/or . If direct I/O is enabled, some or all of the options introduced above will not be applicable. See more details in .

options.allow_mmap_reads and options.allow_mmap_writes make RocksDB mmap the whole data file while doing read or write, respectively. The benefit of the approach is to reduce the file system calls doing pread() and write(), and in many cases, reduce the memory copying too. options.allow_mmap_reads can usually significantly improve performance if the DB is run on ramfs. They can be used on file systems backed by block device too. However, based on our previous experience, file systems aren’t usually doing a perfect job maintaining this kind of memory mapping, and some times cause slow queries. In this case, we advise you try out this option only when necessary and with caution.

Cleanup on Iterator destruction and cleanup on column family destruction by default will try to delete obsolete files in the context of the thread calling the destructor, subject to deletion rate limits. This can result in an unexpected long latency in completing the operation. To avoid the long latency and defer the deletion of obsolete files to background threads, the following options are provided -

• ReadOptions::background_purge_on_iterator_cleanup - This option, when set in the call to DB::NewIterator(), will schedule the deletion of obsolete files in a background thread on iterator destruction.