Large datasets

    • Load 30 billion key-value records
    • Each write operation inserts a single record
    • Perform a read-heavy workload that does random reads in the presence of some writes
    • Perform a read-heavy workload that does reads of a subset of data in the presence of some writes

    Each record is a key-value record of size almost 300 bytes.

    • Value size: 256 Bytes (chosen to be not very compressible)

    Note that the load tester was run from a separate machine in the same availability zone (AZ).

    A machine in the Amazon Web Services (AWS) cloud with the following specifications was chosen: 32-vcpus, 240 GB RAM, 4 x 1.9TB nvme SSD.

    • Cloud: AWS
    • Node type: i3.8xlarge

    Cluster creation

    Create a standard four-node cluster, with replication factor (RF) of . Pass the following option to the YugabyteDB processes.

    The yb_num_shards_per_tserver was set to 20 (default value is 8). This is done because the i3.8xlarge nodes have four disks. In future, YugabyteDB will automatically pick better defaults for nodes with multiple disks.

    The data was loaded at a steady rate over about 4 days using the CassandraKeyValue workload. To load the data, run the following command:

    You should see a steady 85,000 inserts per second with write latencies of around 2.5 milliseconds. This is shown graphically below.

    Data set size growth rate

    The graph below shows the steady growth in SSTables size at a node from Sep 4 to beyond which it stabilizes at 6.5 TB.

    Load Phase Results

    Note

    The uncompressed dataset size per node is 8 TB, while the compressed size is 6.5 TB. This is because the load generator generates random bytes, which are not very compressible.

    Real world workloads generally have much more compressible data.

    The results you see should be similar to the observations shown below.

    Read-heavy workload results

    NameObservation
    Random-data read heavy workload185K reads/sec and 1K writes/sec
    Recent-data read heavy Workload385K reads/sec and 6.5K writes/sec
    • Expanded from four to five nodes in about eight hours
      • Deliberately rate limited at 200 MB/sec
    • New node takes traffic as soon the first tablet arrives
      • Pressure relieved from old nodes very quickly
    • Induced one node failure in five-node cluster