Docker

    This tutorial uses Docker Machine to create multiple nodes on your desktop. These nodes can even be on multiple machines on the cloud platform of your choice.

    macOS

    • Docker Engine 1.12 or later installed using Docker for Mac. Docker Machine is already included with Docker for Mac.

    • VirtualBox 5.2 or later for creating the swarm nodes.

    • Docker Engine 1.12 or later installed using . Docker Machine is already included with Docker for Windows.

    • Microsoft Hyper-V driver for creating the swarm nodes.

    1. Create swarm nodes

    Following bash script is a simpler form of Docker’s own swarm beginner tutorial bash script. You can use this for Linux and macOS. If you are using Windows, then download and change the of the same script.

    • The script first instantiates three nodes using Docker Machine and VirtualBox. Thereafter, it initializes the swarm cluster by creating a swarm on the first node. Finally, it adds the remaining nodes as workers to the cluster. It also pulls the container image into each of the nodes to expedite the next steps.

    NoteIn more fault-tolerant setups, there will be multiple manager nodes and they will be independent of the worker nodes. A 3-node master and 3-node worker setup is used in the Docker tutorial script referenced above.

    • Review all the nodes created.
    1. $ docker-machine ls
    1. NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
    2. worker1   -        virtualbox   Running   tcp://192.168.99.100:2376           v18.05.0-ce
    3. worker2   -        virtualbox   Running   tcp://192.168.99.101:2376           v18.05.0-ce
    4. worker3   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.05.0-ce

    2. Create overlay network

    • SSH into the worker1 node where the swarm manager is running.
    1. $ docker-machine ssh worker1
    • Create an overlay network that the swarm services can use to communicate with each other. The attachable option allows standalone containers to connect to swarm services on the network.
    1. $ docker network create --driver overlay --attachable yugabytedb
    • Create 3 YB-Master services each with replicas set to 1. This is the only way in Docker Swarm today to get stable network identities for each of the YB-Master containers that we will need to provide as input for creating the YB-TServer service in the next step.

    Note for Kubernetes usersDocker Swarm lacks an equivalent of . The concept of replicated services is similar to Kubernetes deployments.

    1. $ docker service create \
    2. --replicas 1 \
    3. --name yb-master1 \
    4. --network yugabytedb \
    5. --mount type=volume,source=yb-master1,target=/mnt/data0 \
    6. --publish 7000:7000 \
    7. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-master \
    8. --master_addresses=yb-master1:7100,yb-master2:7100,yb-master3:7100 \
    9. --replication_factor=3
    1. $ docker service create \
    2. --replicas 1 \
    3. --name yb-master2 \
    4. --network yugabytedb \
    5. --mount type=volume,source=yb-master2,target=/mnt/data0 \
    6. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-master \
    7. --fs_data_dirs=/mnt/data0 \
    8. --master_addresses=yb-master1:7100,yb-master2:7100,yb-master3:7100 \
    9. --replication_factor=3
    1. $ docker service create \
    2. --replicas 1 \
    4. --network yugabytedb \
    5. --mount type=volume,source=yb-master3,target=/mnt/data0 \
    6. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-master \
    7. --fs_data_dirs=/mnt/data0 \
    8. --master_addresses=yb-master1:7100,yb-master2:7100,yb-master3:7100 \
    9. --replication_factor=3
    • Run the command below to see the services that are now live.
    1. $ docker service ls
    • View the yb-master Admin UI by going to the port 7000 of any node, courtesy of the publish option used when yb-master1 was created. For example, we can see from Step 1 that worker2’s IP address is 192.168.99.101. So, takes us to the yb-master Admin UI.

    4. Create yb-tserver service

    • Create a single yb-tserver service so that swarm can then automatically spawn 1 container/task on each worker node. Each time we add a node to the swarm, the swarm orchestrator creates a task and the scheduler assigns the task to the new node.

    Note for Kubernetes UsersThe global services concept in Docker Swarm is similar to Kubernetes DaemonSets.

    1. $ docker service create \
    2. --mode global \
    3. --name yb-tserver \
    4. --network yugabytedb \
    5. --mount type=volume,source=yb-tserver,target=/mnt/data0 \
    6. --publish 9000:9000 \
    7. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-tserver \
    8. --fs_data_dirs=/mnt/data0 \
    9. --tserver_master_addrs=yb-master1:7100,yb-master2:7100,yb-master3:7100

    TipUse remote volumes instead of local volumes (used above) when you want to scale-out or scale-in your swarm cluster.

    • Run the command below to see the services that are now live.
    1. $ docker service ls
    1. ID                  NAME                MODE                REPLICAS            IMAGE                        PORTS
    2. jfnrqfvnrc5b        yb-master1          replicated          1/1                 yugabytedb/yugabyte:latest   *:7000->7000/tcp
    3. kqp6eju3kq88        yb-master2          replicated          1/1                 yugabytedb/yugabyte:latest
    4. ah6wfodd4noh        yb-master3          replicated          1/1                 yugabytedb/yugabyte:latest
    5. n6padh2oqjk7        yb-tserver          global              3/3                 yugabytedb/yugabyte:latest   *:9000->9000/tcp
    • Now we can go to to see the yb-tserver admin UI.

    5. Test the APIs

    YSQL API

    • Connect to the ysqlsh client in yb-tserver.
      1. ...
      2. ysqlsh (11.2-YB-2.0.1.0-b0)
      3. Type "help" for help.
      5. yugabyte=#

      • Connect to that container using that container ID.

      1. $ docker exec -it <ybtserver_container_id> /home/yugabyte/bin/cqlsh
      1. Connected to local cluster at 127.0.0.1:9042.
      2. [cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
      3. Use HELP for help.
      4. cqlsh>
      • Follow the test instructions as noted in .

      YEDIS API

      • Find the container ID of the yb-master running on worker1. Use the first param of docker ps output.

      • Initialize the YEDIS API.

      1. $ docker exec -it <ybmaster_container_id> /home/yugabyte/bin/yb-admin -- --master_addresses yb-master1:7100,yb-master2:7100,yb-master3:7100 setup_redis_table
      • Follow the test instructions as noted in .

      Docker Swarm ensures that the yb-tserver global service will always have 1 yb-tserver container running on every node. If the yb-tserver container on any node dies, then Docker Swarm will bring it back on.

      1. $ docker kill <ybtserver_container_id>

      Observe the output of docker ps every few seconds till you see that the yb-tserver container is re-spawned by Docker Swarm.

      7. Test auto-scaling with node addition

      • On the host machine, get worker token for new worker nodes to use to join the existing swarm.
      1. $ docker-machine ssh worker1 "docker swarm join-token worker -q"
      1. SWMTKN-1-aadasdsadas-2ja2q2esqsivlfx2ygi8u62yq
      • Create a new node worker4.
      1. $ docker-machine create -d virtualbox worker4
      • Pull the YugabyteDB container.
      1. $ docker-machine ssh worker4 "docker pull yugabytedb/yugabyte"
      • Join worker4 with existing swarm.
      1. $ docker-machine ssh worker4 \
      2. "docker swarm join \
      3. --token SWMTKN-1-aadasdsadas-2ja2q2esqsivlfx2ygi8u62yq \
      4. --listen-addr $(docker-machine ip worker4) \
      5. --advertise-addr $(docker-machine ip worker4) \
      6. $(docker-machine ip worker1)"
      • Observe that Docker Swarm adds a new yb-tserver instance to the newly added worker4 node and changes its replica status from 3 / 3 to 4 / 4.
      1. $ docker service ls
      1. ID                  NAME                MODE                REPLICAS            IMAGE                        PORTS
      2. jfnrqfvnrc5b        yb-master1          replicated          1/1                 yugabytedb/yugabyte:latest   *:7000->7000/tcp
      3. kqp6eju3kq88        yb-master2          replicated          1/1                 yugabytedb/yugabyte:latest
      4. ah6wfodd4noh        yb-master3          replicated          1/1                 yugabytedb/yugabyte:latest
      5. n6padh2oqjk7        yb-tserver          global              4/4                 yugabytedb/yugabyte:latest   *:9000->9000/tcp

      8. Remove services and destroy nodes

      • Stop the machines.
      • Remove the machines.