Administration

    Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system.

    When adding a stream using the nats CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored on multiple nodes in your cluster using the RAFT protocol as above.

    Example output extract:

    1. ....
    2. Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
    3. ....
    4. Configuration:
    5. ....
    6.              Replicas: 3
    8. Cluster Information:
    10.                  Name: C1
    11.                Leader: n1-c1
    12.               Replica: n4-c1, current, seen 0.07s ago
    13.               Replica: n3-c1, current, seen 0.07s ago

    Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using nats stream info.

    Here we have a stream in the NATS cluster C1, its current leader is a node n1-c1 and it has 2 followers - n4-c1 and n3-c1.

    The current indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently.

    The replica count cannot be edited once configured.

    Users can get overall statistics about their streams and also where these streams are placed:

    1. nats stream report

    Example output

    1. Obtaining Stream stats
    2. +----------+-----------+----------+--------+---------+------+---------+----------------------+
    3. | Stream   | Consumers | Messages | Bytes  | Storage | Lost | Deleted | Cluster              |
    4. +----------+-----------+----------+--------+---------+------+---------+----------------------+
    5. | ORDERS   | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
    6. | ORDERS_3 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
    7. | ORDERS_4 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
    8. | ORDERS_5 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1, n2-c1, n3-c1* |
    9. | ORDERS_2 | 4         | 1,385    | 13 MiB | File    | 0    | 1       | n1-c1, n2-c1, n3-c1* |
    10. | ORDERS_0 | 4         | 1,561    | 14 MiB | File    | 0    | 0       | n1-c1, n2-c1*, n3-c1 |
    11. +----------+-----------+----------+--------+---------+------+---------+----------------------+

    Forcing Stream and Consumer leader election

    Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election.

    1. nats stream cluster step-down ORDERS

    Example output

    The same is true for consumers, nats consumer cluster step-down ORDERS NEW.

    Systems users can view state of the Meta Group - but not individual Stream or Consumers.

    We have a high level report of cluster state:

    1. nats server report jetstream --user system

    Example output

    1. +--------------------------------------------------------------------------------------------------+
    2. |                                        JetStream Summary                                         |
    3. | Server | Cluster | Streams | Consumers | Messages | Bytes  | Memory | File   | API Req | API Err |
    5. | n3-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 1       | 0       |
    6. | n3-c1  | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 3       | 0       |
    7. | n2-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 3       | 0       |
    8. | n1-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 14      | 2       |
    9. | n2-c1  | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 15      | 0       |
    10. | n1-c1* | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 31      | 0       |
    11. +--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
    12. |        |         | 18      | 72        | 8,838    | 80 MiB | 0 B    | 80 MiB | 67      | 2       |
    13. +--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
    14. +---------------------------------------------------+
    15. |            RAFT Meta Group Information            |
    16. +-------+--------+---------+---------+--------+-----+
    17. | Name  | Leader | Current | Offline | Active | Lag |
    18. +-------+--------+---------+---------+--------+-----+
    19. | n1-c1 | yes    | true    | false   | 0.00s  | 0   |
    20. | n1-c2 |        | true    | false   | 0.05s  | 0   |
    21. | n2-c1 |        | true    | false   | 0.05s  | 0   |
    22. | n2-c2 |        | true    | false   | 0.05s  | 0   |
    23. | n3-c1 |        | true    | false   | 0.05s  | 0   |
    24. | n3-c2 |        | true    | false   | 0.05s  | 0   |
    25. +-------+--------+---------+---------+--------+-----+

    This is a full cluster wide report, the report can be limited to a specific account using --account.

    Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group.

    In the Meta Group report the server n2-c1 is not current and has not been seen for 9 seconds, it’s also behind by 2 raft operations.

    1. nats server req jetstream --help

    Output

    1. usage: nats server request jetstream [<flags>] [<wait>]
    3. Show JetStream details
    5. Flags:
    6.   -h, --help                    Show context-sensitive help (also try --help-long and --help-man).
    7.       --version                 Show application version.
    8.   -s, --server=NATS_URL         NATS server urls
    9.       --user=NATS_USER          Username or Token
    10.       --password=NATS_PASSWORD  Password
    11.       --creds=NATS_CREDS        User credentials
    12.       --nkey=NATS_NKEY          User NKEY
    13.       --tlscert=NATS_CERT       TLS public certificate
    15.       --timeout=NATS_TIMEOUT    Time to wait on responses from NATS
    16.       --js-api-prefix=PREFIX    Subject prefix for access to JetStream API
    17.       --js-event-prefix=PREFIX  Subject prefix for access to JetStream Advisories
    18.       --js-domain=DOMAIN        JetStream domain to access
    19.       --context=CONTEXT         Configuration context
    20.       --trace                   Trace API interactions
    21.       --limit=2048              Limit the responses to a certain amount of records
    22.       --offset=0                Start at a certain record
    23.       --name=NAME               Limit to servers matching a server name
    24.       --host=HOST               Limit to servers matching a server host name
    25.       --cluster=CLUSTER         Limit to servers matching a cluster name
    26.       --tags=TAGS ...           Limit to servers with these configured tags
    27.       --account=ACCOUNT         Show statistics scoped to a specific account
    28.       --accounts                Include details about accounts
    29.       --streams                 Include details about Streams
    30.       --consumer                Include details about Consumers
    31.       --config                  Include details about configuration
    32.       --leader                  Request a response from the Meta-group leader only
    33.       --all                     Include accounts, streams, consumers and configuration
    35. Args:
    36.   [<wait>]  Wait for a certain number of responses

    This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only.

    Forcing Meta Group leader election

    Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a SYSTEM user.

    1. nats server raft step-down --user system

    Example output

    1. 17:44:24 Current leader: n2-c2
    2. 17:44:24 New leader: n1-c2

    Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down.

    There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it’s Consumers.

    After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream.

    1. nats stream cluster peer-remove ORDERS

    Example output

    1. ? Select a Peer n4-c1
    2. 14:38:50 Removing peer "n4-c1"
    3. 14:38:50 Requested removal of peer "n4-c1"

    At this point the stream and all consumers will have removed n4-c1 from the group, they will all start new peer selection and data replication.

    1. ....
    2. Cluster Information:
    4.                  Name: c1
    5.                Leader: n3-c1
    6.               Replica: n1-c1, current, seen 0.02s ago
    7.               Replica: n2-c1, outdated, seen 0.42s ago

    We can see a new replica was picked, the stream is back to replication level of 3 and n4-c1 is not active any more in this Stream or any of its Consumers.