MCollective架构篇7-多MQ下MCollective高可用部署

一、配置Rabbitmq

安装（略），可参考http://kisspuppet.com/2013/11/10/mcollective-middleware/或

2. 添加tcp监听端口和范围

  3. [
  4.   {rabbitmq_stomp, [{tcp_listeners, [61613]}]}
  5. ].

备注：可参考

如果你以前配置过，建议将配置清空

  1. [root@linuxmaster1poc ~]# rabbitmqctl stop_app
  2. Stopping node rabbit@linuxmaster1poc ...
  3. ...done.
  4. [root@linuxmaster1poc ~]# rabbitmqctl reset
  5. Resetting node rabbit@linuxmaster1poc ...
  6. ...done.
  7. [root@linuxmaster1poc ~]# rabbitmqctl start_app
  8. Starting node rabbit@linuxmaster1poc ...
  9. ...done.

删除默认用户guest，添加三个用户（web_admin-http访问用，admin—管理员，mc_rabbitmq—mcollective链接用）

  1. [root@linuxmaster1poc ~]# rabbitmqctl list_users
  2. Listing users ...
  3. guest    [administrator]
  4. ...done.
  5. [root@linuxmaster1poc ~]# rabbitmqctl delete_user guest
  6. Deleting user "guest" ...
  7. ...done.
  8. [root@linuxmaster1poc ~]# rabbitmqctl add_user mc_rabbitmq 123.com
  9. Creating user "mc_rabbitmq" ...
  10. ...done.
  11. [root@linuxmaster1poc ~]# rabbitmqctl add_user admin password=123.com 
  12. Creating user "admin" ...
  13. ...done.
  14. [root@linuxmaster1poc ~]# rabbitmqctl add_user web_admin 123.com
  15. Creating user "web_admin" ...
  16. ...done.

设置用户的角色

  1. [root@linuxmaster1poc ~]# rabbitmqctl set_user_tags admin administrator
  2. Setting tags for user "admin" to [administrator] ...
  3. ...done.
  5. [root@linuxmaster1poc ~]# rabbitmqctl set_user_tags web_admin monitoring
  6. Setting tags for user "web_admin" to [monitoring] ...
  7. ...done.

创建虚拟主机组

  1. [root@linuxmaster1poc ~]# rabbitmqctl add_vhost /mcollective
  2. Creating vhost "/mcollective" ...
  3. ...done.

重启rabbitmq-server服务

  1. [root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server restart
  2. Restarting rabbitmq-server: SUCCESS
  3. rabbitmq-server.

查看用户以及角色是否创建成功

  1. [root@linuxmaster1poc ~]# rabbitmqctl list_users
  2. Listing users ...
  3. admin    [administrator]
  4. mc_rabbitmq    []
  5. web_admin    [monitoring]
  6. ...done.

查看虚拟主机组“/mcollective”中所有用户的权限

  1. [root@linuxmaster1poc ~]# rabbitmqctl list_permissions -p "/mcollective"
  2. Listing permissions in vhost "/mcollective" ...
  3. admin    .*    .*    .*
  4. mc_rabbitmq    .*    .*    .*
  5. web_admin    .*    .*    .*
  6. ...done.
  7. [root@linuxmaster1poc ~]#

4、登录

默认配置

  1. [root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p "/mcollective"
  2. Listing exchanges ...
  3.     direct
  4. amq.direct    direct
  5. amq.fanout    fanout
  6. amq.headers    headers
  7. amq.match    headers
  8. amq.rabbitmq.trace    topic
  9. amq.topic    topic
  10. ...done.

设置后更新配置

  1. [root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p "/mcollective"
  2. Listing exchanges ...
  3. amq.direct    direct
  4. amq.fanout    fanout
  5. amq.headers    headers
  6. amq.match    headers
  7. amq.rabbitmq.trace    topic
  8. amq.topic    topic
  9. mcollective_broadcast    topic
  11. ...done.

备注：可参考官网设置 https://www.rabbitmq.com/man/rabbitmqctl.1.man.html

二、 配置MCollective:

2. 配置mcollective server端

  1. [root@linux57poc tmp]# cat /etc/mcollective/server.cfg
  2. # --Global--
  3. topicprefix = /topic/
  4. main_collective = mcollective
  5. collectives = mcollective
  6. libdir = /usr/libexec/mcollective
  7. logfile = /var/log/puppet/mcollective.log
  8. loglevel = info
  9. daemonize = 1
  11. # --rabbitmq Plugins--
  12. securityprovider = psk
  13. plugin.psk = a36cd839414370e10fd281b8a38a4f48
  15. direct_addressing = 1
  16. connector = rabbitmq
  17. plugin.rabbitmq.vhost = /mcollective 
  18. plugin.rabbitmq.pool.size = 2
  19. plugin.rabbitmq.initial_reconnect_delay = 0.01
  20. plugin.rabbitmq.max_reconnect_delay = 30.0
  21. plugin.rabbitmq.use_exponential_back_off = true
  22. plugin.rabbitmq.back_off_multiplier = 2
  23. plugin.rabbitmq.max_reconnect_attempts = 0
  24. plugin.rabbitmq.randomize = false
  25. plugin.rabbitmq.timeout = -1
  27. plugin.rabbitmq.pool.1.host = 192.168.100.120
  28. plugin.rabbitmq.pool.1.port = 61613
  29. plugin.rabbitmq.pool.1.user = mc_rabbitmq
  30. plugin.rabbitmq.pool.1.password = 123.com
  31. plugin.rabbitmq.pool.1.ssl = false 
  33. plugin.rabbitmq.pool.2.host = 192.168.100.121
  34. plugin.rabbitmq.pool.2.port = 61613
  35. plugin.rabbitmq.pool.2.user = mc_rabbitmq
  36. plugin.rabbitmq.pool.2.password = 123.com
  37. plugin.rabbitmq.pool.2.ssl = false 
  39. # --Puppet provider specific options--
  40. plugin.service.provider = puppet
  41. plugin.service.puppet.hasstatus = true
  42. plugin.service.puppet.hasrestart = true
  44. plugin.puppet.command = puppet agent
  45. plugin.puppet.splay = true
  46. plugin.puppet.splaylimit = 30
  47. plugin.puppet.config = /etc/puppet/puppet.conf
  49. # --Facts--
  50. factsource = yaml
  51. ##factsource = facter
  52. plugin.yaml = /etc/mcollective/facts.yaml

三、高可用测试

特别注意： 节点mcollective的server.cfg中pool是有优先级的，默认数字小的生效，这点需要注意，也就是说当所有节点都连接在MQ2上的时候，启动MQ1，mco命令是无法使用的，因为它在运行的时候连接的是MQ1，而所有节点都连接在MQ2上。

1.1 先看当前的节点连接状态

  1. [root@linuxmaster1poc ~]# mco ping   #查看连接的节点
  2. linux57poc                               time=69.46 ms
  3. linux58poc                               time=70.05 ms
  4. linux64poc                               time=70.59 ms
  7. ---- ping statistics ----
  8. 3 replies max: 70.59 min: 69.46 avg: 70.03 
  9. [root@linuxmaster1poc ~]# mco shell "lsof -i:61613" #查看所有节点监听的端口情况，可以看到目前都连接在linuxmaster1poc上。
  10. Do you really want to send this command unfiltered? (y/n): y
  11. Discovering hosts using the mc method for 2 second(s) .... 3
  12. Host: linux64poc
  13. Statuscode: 0
  14. Output:
  15. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  16. ruby    36625 root    6u  IPv4  27771      0t0  TCP linux64poc:40493->linuxmaster1poc:61613 (ESTABLISHED)
  17. Host: linux58poc
  18. Statuscode: 0
  19. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  20. ruby    11060 root    6u  IPv4  34046      0t0  TCP linux58poc:36295->linuxmaster1poc:61613 (ESTABLISHED)
  21. Host: linux57poc
  23. Output:
  24. COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
  25. ruby    18076 root    6u  IPv4 1351365       TCP linux57poc:24698->linuxmaster1poc:61613 (ESTABLISHED)
  28. [root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server stop
  29. Stopping rabbitmq-server: rabbitmq-server.

1.2 再次运行mco查看切换状态

  1. [root@linuxmaster1poc ~]# mco ping
  2. linux58poc                               time=73.54 ms
  3. linux64poc                               time=74.61 ms
  4. linux57poc                               time=75.39 ms
  7. ---- ping statistics ----
  8. 3 replies max: 75.39 min: 73.54 avg: 74.51 
  9. [root@linuxmaster1poc ~]# mco shell "lsof -i:61613" 
  10. Do you really want to send this command unfiltered? (y/n): y
  11. Discovering hosts using the mc method for 2 second(s) .... 3
  12. Host: linux58poc
  13. Statuscode: 0
  14. Output:
  15. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  16. ruby    11060 root    6u  IPv4  34046      0t0  TCP linux58poc:36295->linuxmaster1poc:61613 (CLOSE_WAIT)
  17. ruby    11060 root    9u  IPv4  34137      0t0  TCP linux58poc:47200->linuxmaster2poc:61613 (ESTABLISHED)
  18. Host: linux64poc
  19. Statuscode: 0
  20. Output:
  21. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  22. ruby    36625 root    6u  IPv4  27771      0t0  TCP linux64poc:40493->linuxmaster1poc:61613 (CLOSE_WAIT)
  23. ruby    36625 root    8u  IPv4  27877      0t0  TCP linux64poc:37472->linuxmaster2poc:61613 (ESTABLISHED)
  24. Host: linux57poc
  25. Statuscode: 0
  26. Output:
  27. COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
  28. ruby    18076 root    9u  IPv4 1351484       TCP linux57poc:9309->linuxmaster2poc:61613 (ESTABLISHED)

通过日志查看

  1. [root@linuxmaster1poc ~]# mco shell "lsof -i:61613"
  2. Do you really want to send this command unfiltered? (y/n): y
  3. Discovering hosts using the mc method for 2 second(s) .... 3
  4. Host: linux58poc
  5. Statuscode: 0
  6. Output:
  7. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  8. ruby    11428 root    6u  IPv4  34283      0t0  TCP linux58poc:36300->linuxmaster1poc:61613 (CLOSE_WAIT)
  9. ruby    11428 root    8u  IPv4  34338      0t0  TCP linux58poc:47205->linuxmaster2poc:61613 (ESTABLISHED)
  10. Host: linux57poc
  11. Statuscode: 0
  12. Output:
  13. COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
  14. ruby    18447 root    6u  IPv4 1351559       TCP linux57poc:59343->linuxmaster1poc:61613 (CLOSE_WAIT)
  15. ruby    18447 root    8u  IPv4 1351622       TCP linux57poc:29757->linuxmaster2poc:61613 (ESTABLISHED)
  16. Host: linux64poc
  17. Statuscode: 0
  18. Output:
  19. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  20. ruby    37054 root    4u  IPv4  28036      0t0  TCP linux64poc:37476->linuxmaster2poc:61613 (ESTABLISHED)
  21. ruby    37054 root    6u  IPv4  27990      0t0  TCP linux64poc:40497->linuxmaster1poc:61613 (CLOSE_WAIT)

总结：可以看到之前的连接已经变成CLOSE_WAIT，新的连接被建立

2. 停止MQ2，启动MQ1 查看切换状态

  1. [root@linuxmaster2poc rabbitmq]# /etc/init.d/rabbitmq-server stop
  2. Stopping rabbitmq-server: rabbitmq-server.
  4. [root@linux57poc service]# lsof -i:61613
  5. COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
  6. ruby    18447 root    6u  IPv4 1351559       TCP linux57poc:59343->linuxmaster1poc:61613 (CLOSE_WAIT)
  7. ruby    18447 root    8u  IPv4 1351622       TCP linux57poc:29757->linuxmaster2poc:61613 (CLOSE_WAIT)
  8. [root@linux58poc ~]# lsof -i:61613
  9. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  10. ruby    11428 root    6u  IPv4  34283      0t0  TCP linux58poc:36300->linuxmaster1poc:61613 (CLOSE_WAIT)
  11. ruby    11428 root    8u  IPv4  34338      0t0  TCP linux58poc:47205->linuxmaster2poc:61613 (CLOSE_WAIT)
  12. [root@linux64poc ~]# lsof -i:61613
  13. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  14. ruby    37054 root    4u  IPv4  28036      0t0  TCP linux64poc:37476->linuxmaster2poc:61613 (CLOSE_WAIT)
  15. ruby    37054 root    6u  IPv4  27990      0t0  TCP linux64poc:40497->linuxmaster1poc:61613 (CLOSE_WAIT)
  18. [root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server start
  19. Starting rabbitmq-server: SUCCESS