Manage Multi-Tenant Notifications with Notification Manager

    Supported senders include:

    • Prometheus Alertmanager
    • Custom sender (Coming soon)

    Supported receivers include:

    Notification Manager uses the port and API path /api/v2/alerts to receive alerts sent from Prometheus Alertmanager of KubeSphere.

    To receive Alertmanager alerts, KubeSphere already added the Alertmanager webhook and route configurations like below (by editing the Secret alertmanager-main in the namespace kubesphere-monitoring-system):

    Send Prometheus alerts to Notification Manager:

    Send event alerts to Notification Manager:

    1.     "receivers":
    2.     - "name": "event"
    3.       "webhook_configs":
    4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
    5.         "send_resolved": false
    6.     "route":
    7.       "routes":
    8.       - "match":
    9.           "alerttype": "event"
    10.         "receiver": "event"
    11.         "group_interval": "30s"

    Send auditing alerts to Notification Manager:

    1.     "receivers":
    2.     - "name": "auditing"
    3.       "webhook_configs":
    4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
    5.         "send_resolved": false
    6.     "route":
    7.       "routes":
    8.       - "match":
    9.           "alerttype": "auditing"
    10.         "receiver": "auditing"
    11.         "group_interval": "30s"

    The above is the default configuration. If you do not want to receive a certain type of alert, you can delete the corresponding configuration.

    Configure receivers

    Notification Manager now supports three types of receivers: Email, WeChat Work and Slack. Only the administrator can configure receivers.

    Email

    If a tenant named test-user who wants to receive email notifications, create an email receiver as follows:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   password: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: test-user-email-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: EmailConfig
    16. metadata:
    17.   labels:
    18.     app: notification-manager
    19.     type: tenant
    20.     user: test-user
    21.   name: test-user-config
    22.   namespace: kubesphere-monitoring-system
    23. spec:
    24.   authPassword:
    25.     key: password
    26.     name: test-user-email-secret
    27.   authUsername: abc1
    28.   from: [email protected]
    29.   requireTLS: true
    30.   smartHost:
    31.     host: imap.xyz.com
    32.     port: "25"
    34. ---
    35. apiVersion: notification.kubesphere.io/v1alpha1
    36. kind: EmailReceiver
    37. metadata:
    38.   labels:
    39.     app: notification-manager
    40.     type: tenant
    41.     user: test-user
    42.   name: test-user-receiver
    43.   namespace: kubesphere-monitoring-system
    44. spec:
    45.   emailConfigSelector:
    46.     matchLabels:
    47.       type: tenant
    48.       user: test-user
    49.   to:
    50.   - [email protected]
    51.   - [email protected]
    52. EOF

    emailConfigSelector is a selector to select EmailConfig for the email receiver. If emailConfigSelector is not set, the receiver will use the default email configuration. You can create a default email configuration as follows:

    Email receivers with the label type: tenant only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global email receiver with the label type: global as below:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: notification.kubesphere.io/v1alpha1
    3. metadata:
    4.   labels:
    5.     app: notification-manager
    6.     type: global
    8.   namespace: kubesphere-monitoring-system
    9. spec:
    10.   to:
    11.   - [email protected]
    12. EOF

    Note

    The global email receiver will use the default email configuration.

    WeChat Work

    Notification Manager supports sending notifications to WeChat Work. If a tenant named test-user who wants to receive notifications from WeChat Work, create a WeChat receiver as follows:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   wechat: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: test-user-wechat-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: WechatConfig
    16. metadata:
    17.   name: test-user-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: tenant
    22.     user: test-user
    23. spec:
    24.   wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
    25.   wechatApiSecret:
    26.     key: wechat
    27.     name: test-user-wehat-secret
    28.   wechatApiCorpId: wwfd76b24f06513578
    29.   wechatApiAgentId: "1000002"
    31. ---
    32. apiVersion: notification.kubesphere.io/v1alpha1
    33. kind: WechatReceiver
    34. metadata:
    35.   name: test-user-wechat
    36.   namespace: kubesphere-monitoring-system
    37.   labels:
    38.     app: notification-manager
    39.     type: tenant
    40.     user: test-user
    41. spec:
    42.   wechatConfigSelector:
    43.     matchLabels:
    44.       type: tenant
    45.       user: test-user
    46.   # optional
    47.   # One of toUser, toParty, toParty should be specified.
    48.   toUser: user1 | user2
    49.   toParty: party1 | party2
    50.   toTag: tag1 | tag2
    51. EOF
    • wechatApiCorpId is the id of your WeChat Work.
    • wechatApiAgentId is the id of the app sending messages to users in your WeChat Work.
    • wechatApiSecret is the secret of this app. You can get these two parameters in App Management of your WeChat Work.
    • Any user, party or tag who wants to receive notifications must be in the allowed users list of this app.

    wechatConfigSelector is a selector to select WechatConfig for the WeChat receiver. If wechatConfigSelector is not set, the WeChat receiver will use the default WeChat configuration. You can create a default WeChat configuration as follows:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   wechat: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: default-wechat-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: WechatConfig
    16. metadata:
    17.   name: default-wechat-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: default
    22. spec:
    23.   wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
    24.     key: wechat
    25.     name: default-wechat-secret
    26.   wechatApiCorpId: wwfd76b24f06513578
    28. EOF

    WeChat receivers with the label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global WeChat receiver with the label type: global as below:

    Note

    The global WeChat receiver will use the default WeChat configuration.

    Slack

    Notification Manager supports sending notifications to Slack channels. If a tenant named test-user who wants to receive notifications from Slack, create a Slack receiver as follows:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   token: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: test-user-slack-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: SlackConfig
    16. metadata:
    17.   name: test-user-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: tenant
    22.     user: test-user
    23. spec:
    24.   slackTokenSecret: 
    25.     key: token
    26.     name: test-user-slack-secret
    28. ---
    29. apiVersion: notification.kubesphere.io/v1alpha1
    30. kind: SlackReceiver
    31. metadata:
    32.   name: test-user-slack
    33.   namespace: kubesphere-monitoring-system
    34.   labels:
    35.     app: notification-manager
    36.     type: tenant
    37.     user: test-user
    38. spec:
    39.   slackConfigSelector:
    40.     matchLabels:
    41.       type: tenant
    42.       user: test-user
    43.   channel: alert
    44. EOF

    Info

    • The Slack token is the OAuth Access Token or Bot User OAuth Access Token when you create a Slack app.
    • This app must have the scope .
    • The user who creates the app or bot user must be in the channel to which you want to send notifications.

    slackConfigSelector is a selector to select SlackConfig for the Slack receiver. If slackConfigSelector is not set, the Slack receiver will use the default Slack configuration. You can create a default Slack configuration as follows:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. data:
    4.   token: dGVzdA==
    5. kind: Secret
    6. metadata:
    7.   labels:
    8.     app: notification-manager
    9.   name: default-slack-secret
    10.   namespace: kubesphere-monitoring-system
    11. type: Opaque
    13. ---
    14. apiVersion: notification.kubesphere.io/v1alpha1
    15. kind: SlackConfig
    16. metadata:
    17.   name: default-slack-config
    18.   namespace: kubesphere-monitoring-system
    19.   labels:
    20.     app: notification-manager
    21.     type: default
    22. spec:
    23.   slackTokenSecret: 
    24.     key: token
    25.     name: default-slack-secret
    26. EOF

    Slack receivers with the label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global Slack receiver with the label type: global as below:

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: notification.kubesphere.io/v1alpha1
    3. kind: SlackReceiver
    4. metadata:
    5.   name: global-slack-slack
    6.   namespace: kubesphere-monitoring-system
    7.   labels:
    8.     app: notification-manager
    9.     type: global
    10. spec:

    The global Slack receiver will use the default Slack configuration.