Forward

    This plugin offers two different transports and modes:

    • Forward (TCP): It uses a plain TCP connection.
    • Secure Forward (TLS): when TLS is enabled, the plugin switch to Secure Forward mode.

    The following parameters are mandatory for either Forward for Secure Forward modes:

    When using Secure Forward mode, the TLS mode requires to be enabled. The following additional configuration parameters are available:

    Key Description Default
    Shared_Key A key string known by the remote Fluentd used for authorization.
    Self_Hostname Default value of the auto-generated certificate common name (CN).
    tls Enable or disable TLS support Off
    tls.verify Force certificate validation On
    tls.debug Set TLS debug verbosity level. It accept the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational) and 4 Verbose 1
    tls.ca_file Absolute path to CA certificate file
    tls.crt_file Absolute path to Certificate file.
    tls.key_file Absolute path to private Key file.
    tls.key_passwd Optional password for tls.key_file file.

    Before proceeding, make sure that is installed in your system, if it’s not the case please refer to the following Fluentd Installation document and go ahead with that.

    Once is installed, create the following configuration file example that will allow us to stream data into it:

    That configuration file specifies that it will listen for TCP connections on the port 24224 through the forward input type. Then for every message with a fluent_bit TAG, will print the message to the standard output.

    2. 2017-03-23 11:50:43 -0600 [info]: reading config file path="test.conf"
    3. 2017-03-23 11:50:43 -0600 [info]: starting fluentd-0.12.33
    4. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
    5. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-docker' version '0.1.0'
    6. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-elasticsearch' version '1.4.0'
    7. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flatten-hash' version '0.2.0'
    8. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.0.4'
    9. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-influxdb' version '0.2.8'
    10. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
    11. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
    12. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-out-http' version '0.1.3'
    13. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-parser' version '0.6.0'
    14. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-record-reformer' version '0.7.0'
    15. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.1'
    16. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-stdin' version '0.1.1'
    17. 2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-td' version '0.10.27'
    18. 2017-03-23 11:50:43 -0600 [info]: adding match pattern="fluent_bit" type="stdout"
    19. 2017-03-23 11:50:43 -0600 [info]: adding source type="forward"
    20. 2017-03-23 11:50:43 -0600 [info]: using configuration file: <ROOT>
    21.   <source>
    22.     bind 0.0.0.0
    23.     port 24224
    24.   </source>
    25.   <match fluent_bit>
    26.     type stdout
    27.   </match>
    29. 2017-03-23 11:50:43 -0600 [info]: listening fluent socket on 0.0.0.0:24224

    Now that Fluentd is ready to receive messages, we need to specify where the forward output plugin will flush the information using the following format:

    1. bin/fluent-bit -i INPUT -o forward://HOST:PORT

    If the TAG parameter is not set, the plugin will set the tag as fluent_bit. Keep in mind that TAG is important for routing rules inside .

    Using the CPU input plugin as an example we will flush CPU metrics to :

    Now on the Fluentd side, you will see the CPU metrics gathered in the last seconds:

    1. 2017-03-23 11:53:06 -0600 fluent_bit: {"cpu_p":0.0,"user_p":0.0,"system_p":0.0,"cpu0.p_cpu":0.0,"cpu0.p_user":0.0,"cpu0.p_system":0.0,"cpu1.p_cpu":0.0,"cpu1.p_user":0.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0}
    2. 2017-03-23 11:53:07 -0600 fluent_bit: {"cpu_p":2.25,"user_p":2.0,"system_p":0.25,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":1.0,"cpu1.p_user":1.0,"cpu1.p_system":0.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":3.0,"cpu3.p_user":2.0,"cpu3.p_system":1.0}
    3. 2017-03-23 11:53:08 -0600 fluent_bit: {"cpu_p":1.75,"user_p":1.0,"system_p":0.75,"cpu0.p_cpu":2.0,"cpu0.p_user":1.0,"cpu0.p_system":1.0,"cpu1.p_cpu":3.0,"cpu1.p_user":1.0,"cpu1.p_system":2.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0}
    4. 2017-03-23 11:53:09 -0600 fluent_bit: {"cpu_p":4.75,"user_p":3.5,"system_p":1.25,"cpu0.p_cpu":4.0,"cpu0.p_user":3.0,"cpu0.p_system":1.0,"cpu1.p_cpu":5.0,"cpu1.p_user":4.0,"cpu1.p_system":1.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":5.0,"cpu3.p_user":4.0,"cpu3.p_system":1.0}

    So we gathered metrics and flushed them out to Fluentd properly.

    Paste this content in a file called flb.conf:

    1. [SERVICE]
    2.     Flush      5
    3.     Daemon     off
    4.     Log_Level  info
    6. [INPUT]
    7.     Name       cpu
    8.     Tag        cpu_usage
    10. [OUTPUT]
    11.     Name          forward
    12.     Match         *
    13.     Host          127.0.0.1
    14.     Port          24284
    15.     Shared_Key    secret
    16.     Self_Hostname flb.local
    17.     tls           on
    18.     tls.verify    off

    Paste this content in a file called fld.conf:

    If you’re using Fluentd v1, set up it as below:

    1.   @type forward
    2.   <transport tls>
    4.     private_key_path /etc/td-agent/certs/fluentd.key
    5.     private_key_passphrase password
    6.   </transport>
    7.   <security>
    8.     self_hostname myserver.local
    9.     shared_key secret
    10.   </security>
    11. </source>
    13. <match **>
    14.  @type stdout
    15. </match>

    Start Fluentd:

    1. $ fluentd -c fld.conf

    Start Fluent Bit:

    After five seconds, Fluent Bit will write the records to Fluentd. In Fluentd output you will see a message like this:

    1. 2017-03-23 13:34:40 -0600 [info]: using configuration file: <ROOT>
    2.   <source>
    3.     @type secure_forward
    4.     self_hostname myserver.local
    5.     shared_key xxxxxx
    6.     secure no
    7.   </source>
    8.   <match **>
    9.     @type stdout
    10.   </match>
    11. </ROOT>
    12. 2017-03-23 13:34:41 -0600 cpu_usage: {"cpu_p":1.0,"user_p":0.75,"system_p":0.25,"cpu0.p_cpu":1.0,"cpu0.p_user":1.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":1.0,"cpu1.p_system":1.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0}
    13. 2017-03-23 13:34:42 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.75,"system_p":0.0,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0}
    14. 2017-03-23 13:34:43 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.25,"system_p":0.5,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0}