Manage multi-tenant notifications with Notification Manager

    Supported senders includes:

    • Prometheus Alertmanager
    • Custom sender (Coming soon)

    Supported receivers includes:

    Notification Manager uses port and API path /api/v2/alerts to receive alerts sent from Prometheus Alertmanager of Kubesphere.

    To receive Alertmanager alerts, KubeSphere already added Alertmanager webhook and route configurations like below ( by editing the Secret alertmanager-main in the namespace kubesphere-monitoring-system ):

    Send Prometheus alerts to Notification Manager:

    Send event alerts to Notification Manager:

    1.     "receivers":
    2.     - "name": "event"
    3.       "webhook_configs":
    4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
    5.         "send_resolved": false
    6.     "route":
    7.       "routes":
    8.       - "match":
    9.           "alerttype": "event"
    10.         "receiver": "event"
    11.         "group_interval": "30s"

    Send auditing alerts to Notification Manager:

    1.     "receivers":
    2.     - "name": "auditing"
    3.       "webhook_configs":
    4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
    5.         "send_resolved": false
    6.     "route":
    7.       "routes":
    8.       - "match":
    9.           "alerttype": "auditing"
    10.         "receiver": "auditing"
    11.         "group_interval": "30s"

    The above is the default configuration. If you do not want to receive a certain type of alert, you can delete the corresponding configuration.

    Configure receivers

    Notification Manager now supports three types of receivers: Email, WeChat Work and Slack. Only the administrator can configure receivers.

    Email

    If a tenant named test-user who wants to receive notifications from email, just create an email receiver like this.

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   password: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: test-user-email-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: EmailConfig
    16. metadata:
    17.   labels:
    18.     app: notification-manager
    19.     type: tenant
    20.     user: test-user
    21.   name: test-user-config
    22.   namespace: kubesphere-monitoring-system
    23. spec:
    24.   authPassword:
    25.     key: password
    26.     name: test-user-email-secret
    27.   authUsername: abc1
    28.   from: [email protected]
    29.   requireTLS: true
    30.   smartHost:
    31.     host: imap.xyz.com
    32.     port: "25"
    34. ---
    35. apiVersion: notification.kubesphere.io/v1alpha1
    36. kind: EmailReceiver
    37. metadata:
    38.   labels:
    39.     app: notification-manager
    40.     type: tenant
    41.     user: test-user
    42.   name: test-user-receiver
    43.   namespace: kubesphere-monitoring-system
    44. spec:
    45.   emailConfigSelector:
    46.     matchLabels:
    47.       type: tenant
    48.       user: test-user
    49.   to:
    50.   - [email protected]
    51.   - [email protected]
    52. EOF

    The emailConfigSelector is a selector to select EmailConfig for email receiver, if the emailConfigSelector is not set, receiver will use the default email config. You can create a default email config like this.

    Email receivers with label type: tenant only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global email receiver with label type: global as below::

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: notification.kubesphere.io/v1alpha1
    3. metadata:
    4.   labels:
    5.     app: notification-manager
    6.     type: global
    8.   namespace: kubesphere-monitoring-system
    9. spec:
    10.   to:
    11.   - [email protected]
    12. EOF

    Note

    Global email receiver will use the default email config.

    Wechat Work

    Notification Manager supports sending notification to Wechat Work. If a tenant named test-user who wants to receive notifications from Wechat Work, just create a wechat receiver like this.

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   wechat: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: test-user-wechat-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: WechatConfig
    16. metadata:
    17.   name: test-user-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: tenant
    22.     user: test-user
    23. spec:
    24.   wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
    25.   wechatApiSecret:
    26.     key: wechat
    27.     name: test-user-wehat-secret
    28.   wechatApiCorpId: wwfd76b24f06513578
    29.   wechatApiAgentId: "1000002"
    31. ---
    32. apiVersion: notification.kubesphere.io/v1alpha1
    33. kind: WechatReceiver
    34. metadata:
    35.   name: test-user-wechat
    36.   namespace: kubesphere-monitoring-system
    37.   labels:
    38.     app: notification-manager
    39.     type: tenant
    40.     user: test-user
    41. spec:
    42.   wechatConfigSelector:
    43.     matchLabels:
    44.       type: tenant
    45.       user: test-user
    46.   # optional
    47.   # One of toUser, toParty, toParty should be specified.
    48.   toUser: user1 | user2
    49.   toParty: party1 | party2
    50.   toTag: tag1 | tag2
    51. EOF
    • wechatApiCorpId is the id of your Wechat Work.
    • wechatApiAgentId is the id of app sending message to user in your Wechat Work
    • wechatApiSecret is the secret of this app, you can get these two parameters in App Management of your Wechat Work.
    • Any user, party or tag who wants to receive notifications must be in the allowed users list of this app.

    The wechatConfigSelector is a selector to select WechatConfig for wechat receiver, if the wechatConfigSelector is not set, wechat receiver will use the default wechat config. You can create a default wechat config like this.

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   wechat: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: default-wechat-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: WechatConfig
    16. metadata:
    17.   name: default-wechat-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: default
    22. spec:
    23.   wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
    24.     key: wechat
    25.     name: default-wechat-secret
    26.   wechatApiCorpId: wwfd76b24f06513578
    27.   wechatApiAgentId: "1000002"
    28. EOF

    Wechat receivers with label can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global wechat receiver with label type: global as below:

    Note

    Global wechat receiver will use the default wechat config.

    Slack

    Notification Manager supports sending notification to slack channels. If a tenant named test-user who wants to receive notifications from slack, just create a slack receiver like this.

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   token: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: test-user-slack-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: SlackConfig
    16. metadata:
    17.   name: test-user-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: tenant
    22.     user: test-user
    23. spec:
    24.   slackTokenSecret: 
    25.     key: token
    26.     name: test-user-slack-secret
    28. ---
    29. apiVersion: notification.kubesphere.io/v1alpha1
    30. kind: SlackReceiver
    31. metadata:
    32.   name: test-user-slack
    33.   namespace: kubesphere-monitoring-system
    34.   labels:
    35.     app: notification-manager
    36.     type: tenant
    37.     user: test-user
    38. spec:
    39.   slackConfigSelector:
    40.     matchLabels:
    41.       type: tenant
    42.       user: test-user
    43.   channel: alert
    44. EOF

    Info

    • Slack token is the OAuth Access Token or Bot User OAuth Access Token when you create a slack app.
    • This app must have the scope chat:write.
    • The user who creates the app or bot user must be in the channel to which you want to send notifications.

    The slackConfigSelector is a selector to select SlackConfig for slack receiver, if the slackConfigSelector is not set, slack receiver will use the default slack config. You can create a default slack config like this.

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   token: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: default-slack-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: SlackConfig
    16. metadata:
    17.   name: default-slack-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: default
    22. spec:
    23.   slackTokenSecret: 
    24.     key: token
    25.     name: default-slack-secret
    26. EOF

    Slack receivers with label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global slack receiver with label type: global as below:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: notification.kubesphere.io/v1alpha1
    3. kind: SlackReceiver
    4. metadata:
    5.   name: global-slack-slack
    6.   namespace: kubesphere-monitoring-system
    7.   labels:
    8.     app: notification-manager
    9.     type: global
    10. spec:
    11. EOF

    Global slack receiver will use the default slack config.