AWS CloudWatch data source

    This topic explains options, variables, querying, and other options specific to this data source. Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.

    To access data source settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the AWS Cloudwatch data source.

    Authentication

    There are three different authentication methods available. AWS SDK Default performs no custom configuration at all and instead uses the default provider as specified by the AWS SDK for Go. This requires you to configure your AWS credentials separately, such as if you’ve , if you’re running on an EC2 instance, or for a Service Account in a Kubernetes cluster.

    Credentials file corresponds directly to the provider in the Go SDK. In short, it will read the AWS shared credentials file and find the given profile. While AWS SDK Default will also find the shared credentials file, this option allows you to specify which profile to use without using environment variables. It doesn’t have any implicit fallbacks to other credential providers, and will fail if using credentials from the credentials file doesn’t work.

    Access & secret key corresponds to the StaticProvider and uses the given access key ID and secret key to authenticate. This method doesn’t have any fallbacks, and will fail if the provided key pair doesn’t work.

    IAM roles

    Currently all access to CloudWatch is done server side by the Grafana backend using the official AWS SDK. Providing you have chosen the AWS SDK Default authentication method, and your Grafana server is running on AWS, you can use IAM Roles to handle authentication automically.

    See the AWS documentation on IAM Roles

    IAM policies

    Grafana needs permissions granted via IAM to be able to read CloudWatch metrics and EC2 tags/instances/regions. You can attach these permissions to IAM roles and utilize Grafana’s built-in support for assuming roles.

    Here is a minimal policy example:

    Assuming a role

    The Assume Role ARN field allows you to specify which IAM role to assume, if any. When left blank, the provided credentials are used directly and the associated role or user should have the required permissions. If this field is non-blank, on the other hand, the provided credentials are used to perform an call.

    Endpoint

    The Endpoint field allows you to specify a custom endpoint URL that overrides the default generated endpoint for the CloudWatch API. Leave this field blank if you want to use the default generated endpoint. For more information on why and how to use Service endpoints, refer to the .

    EKS IAM roles for service accounts

    The Grafana process in the container runs as user 472 (called “grafana”). When Kubernetes mounts your projected credentials, they will by default only be available to the root user. In order to allow user 472 to access the credentials (and avoid it falling back to the IAM role attached to the EC2 instance), you will need to provide a for your pod.

    1. securityContext:
    2.   fsGroup: 472
    3.   runAsUser: 472
    4.   runAsGroup: 472

    AWS credentials file

    Create a file at ~/.aws/credentials. That is the HOME path for user running grafana-server.

    Example content:

    1. [default]
    2. aws_access_key_id = asdsadasdasdasd
    3. aws_secret_access_key = dasdasdsadasdasdasdsa
    4. region = us-west-2

    Using the Query Editor

    The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor. You select which API you want to query with using the query mode switch on top of the editor.

    Using the Metric Query Editor

    To create a valid query, you need to specify the namespace, metric name and at least one statistic. If Match Exact is enabled, you also need to specify all the dimensions of the metric you’re querying, so that the matches exactly. If Match Exact is off, you can specify any number of dimensions by which you’d like to filter. Up to 100 metrics matching your filter criteria will be returned.

    Dynamic queries using dimension wildcards

    In Grafana 6.5 or higher, you’re able to monitor a dynamic list of metrics by using the asterisk (*) wildcard for one or more dimension values.

    In the example, all metrics in the namespace AWS/EC2 with a metric name of CPUUtilization and ANY value for the InstanceId dimension are queried. This can help you monitor metrics for AWS resources, like EC2 instances or containers. For example, when new instances get created as part of an auto scaling event, they will automatically appear in the graph without you having to track the new instance IDs. This capability is currently limited to retrieving up to 100 metrics. You can click on Show Query Preview to see the search expression that is automatically built to support wildcards. To learn more about search expressions, visit the CloudWatch documentation.

    By default, the search expression is defined in such a way that the queried metrics must match the defined dimension names exactly. This means that in the example only metrics with exactly one dimension with name ‘InstanceId’ will be returned.

    When defining dimension values based on multi-valued template variables, a search expression is used to query for the matching metrics. This enables the use of multiple template variables in one query and also allows you to use template variables for queries that have the Match Exact option disabled.

    Search expressions are currently limited to 1024 characters, so your query may fail if you have a long list of values. We recommend using the asterisk (*) wildcard instead of the All option if you want to query all metrics that have any value for a certain dimension name.

    The use of multi-valued template variables is only supported for dimension values. Using multi-valued template variables for Region, Namespace, or Metric Name is not supported.

    Metric math expressions

    You can create new time series metrics by operating on top of CloudWatch metrics using mathematical functions. Arithmetic operators, unary subtraction and other functions are supported and can be applied to CloudWatch metrics. More details on the available functions can be found on

    As an example, if you want to apply arithmetic operations on a metric, you can do it by giving an id (a unique string) to the raw metric as shown below. You can then use this id and apply arithmetic operations to it in the Expression field of the new metric.

    Please note that in the case you use the expression field to reference another query, like queryA * 2, it will not be possible to create an alert rule based on that query.

    Period

    A period is the length of time associated with a specific Amazon CloudWatch statistic. Periods are defined in numbers of seconds, and valid values for period are 1, 5, 10, 30, or any multiple of 60.

    If the period field is left blank or set to auto, then it calculates automatically based on the time range. The formula used is time range in seconds / 2000, and then it snaps to the next higher value in an array of predefined periods [60, 300, 900, 3600, 21600, 86400]. By clicking Show Query Preview in the query editor, you can see what period Grafana used.

    Deep linking from Grafana panels to the CloudWatch console

    CloudWatch deep linking

    Left clicking a time series in the panel shows a context menu with a link to View in CloudWatch console. Clicking that link will open a new tab that will take you to the CloudWatch console and display all the metrics for that query. If you’re not currently logged in to the CloudWatch console, the link will forward you to the login page. The provided link is valid for any account but will only display the right metrics if you’re logged in to the account that corresponds to the selected data source in Grafana.

    This feature is not available for metrics that are based on metric math expressions.

    To query CloudWatch Logs, select the region and up to 20 log groups which you want to query. Use the main input area to write your query in CloudWatch Logs Query Language

    You can also write queries returning time series data by using the . When making stats queries in Explore, you have to make sure you are in Metrics Explore mode.

    To the right of the query input field is a CloudWatch Logs Insights link that opens the CloudWatch Logs Insights console with your query. You can continue exploration there if necessary.

    Using template variables

    As with several other data sources, the CloudWatch data source supports the use of template variables in queries. See the documentation for an introduction to the templating feature and the different types of template variables.

    Deep linking from Grafana panels to the CloudWatch console

    If you’d like to view your query in the CloudWatch Logs Insights console, simply click the CloudWatch Logs Insights button next to the query editor. If you’re not currently logged in to the CloudWatch console, the link will forward you to the login page. The provided link is valid for any account but will only display the right metrics if you’re logged in to the account that corresponds to the selected data source in Grafana.

    Alerting

    Since CloudWatch Logs queries can return numeric data, for example through the use of the stats command, alerts are supported. See the documentation for more on Grafana alerts.

    Curated dashboards

    The updated CloudWatch data source ships with pre-configured dashboards for five of the most popular AWS services:

    • Amazon Elastic Compute Cloud Amazon EC2,
    • Amazon Elastic Block Store Amazon EBS,
    • AWS Lambda AWS Lambda,
    • Amazon CloudWatch Logs Amazon CloudWatch Logs, and
    • Amazon Relational Database Service Amazon RDS.

    To import the pre-configured dashboards, go to the configuration page of your CloudWatch data source and click on the Dashboards tab. Click for the dashboard you would like to use. To customize the dashboard, we recommend saving the dashboard under a different name, because otherwise the dashboard will be overwritten when a new version of the dashboard is released.

    Templated queries

    See the documentation for an introduction to the templating feature and the different types of template variables.

    Query variable

    The CloudWatch data source provides the following queries that you can specify in the Query field in the Variable edit view. They allow you to fill a variable’s options list with things like region, namespaces, metric names and dimension keys/values.

    In place of region you can specify default to use the default region configured in the data source for the query, e.g. metrics(AWS/DynamoDB, default) or dimension_values(default, ..., ..., ...).

    Read more about the available dimensions in the .

    For details about the metrics CloudWatch provides, please refer to the CloudWatch documentation.

    Examples templated queries

    Example dimension queries which will return list of resources for individual AWS Services:

    ec2_instance_attribute examples

    The ec2_instance_attribute query takes filters in JSON format. You can specify pre-defined filters of ec2:DescribeInstances. Note that the actual filtering takes place on Amazon’s servers, not in Grafana.

    Filters syntax:

    Example ec2_instance_attribute() query

    1. ec2_instance_attribute(us - east - 1, InstanceId, { 'tag:Environment': ['production'] });

    Selecting attributes

    Only 1 attribute per instance can be returned. Any flat attribute can be selected (i.e. if the attribute has a single value and isn’t an object or array). Below is a list of available flat attributes:

    • AmiLaunchIndex
    • Architecture
    • ClientToken
    • EbsOptimized
    • EnaSupport
    • Hypervisor
    • IamInstanceProfile
    • ImageId
    • InstanceId
    • InstanceLifecycle
    • InstanceType
    • KernelId
    • LaunchTime
    • Platform
    • PrivateDnsName
    • PrivateIpAddress
    • PublicDnsName
    • RamdiskId
    • RootDeviceName
    • RootDeviceType
    • SourceDestCheck
    • SpotInstanceRequestId
    • SriovNetSupport
    • SubnetId
    • VirtualizationType
    • VpcId

    Tags can be selected by prepending the tag name with Tags.

    Example ec2_instance_attribute() query

    1. ec2_instance_attribute(us - east - 1, Tags.Name, { 'tag:Team': ['sysops'] });

    Some queries accept filters in JSON format and Grafana supports the conversion of template variables to JSON.

    If env = 'production', 'staging', following query will return ARNs of EC2 instances which Environment tag is production or staging.

    Pricing

    The Amazon CloudWatch data source for Grafana uses the ListMetrics and GetMetricData CloudWatch API calls to list and retrieve metrics. Pricing for CloudWatch Logs is based on the amount of data ingested, archived, and analyzed via CloudWatch Logs Insights queries. Please see the CloudWatch pricing page for more details.

    Every time you pick a dimension in the query editor Grafana will issue a ListMetrics request. Whenever you make a change to the queries in the query editor, one new request to GetMetricData will be issued.

    Please note that for Grafana version 6.5 or higher, all API requests to GetMetricStatistics have been replaced with calls to GetMetricData. This change enables better support for CloudWatch metric math and enables the automatic generation of search expressions when using wildcards or disabling the Match Exact option. While GetMetricStatistics qualified for the CloudWatch API free tier, this is not the case for GetMetricData calls. For more information, please refer to the .

    Service quotas

    AWS defines quotas, or limits, for resources, actions, and items in your AWS account. Depending on the number of queries in your dashboard and the number of users accessing the dashboard, you may reach the usage limits for various CloudWatch and CloudWatch Logs resources. Note that quotas are defined per account and per region. If you’re using multiple regions or have set up more than one CloudWatch data source to query against multiple accounts, you need to request a quota increase for each account and each region in which you hit the limit.

    To request a quota increase, visit the .

    Please see the AWS documentation for Service Quotas and for more information.

    Configure the data source with grafana.ini

    In the Grafana file, there’s an AWS section that allows you to customize the data source.

    allowed_auth_providers

    Specify which authentication providers are allowed for the CloudWatch data source. The following providers are enabled by default in OSS Grafana: default (AWS SDK default), keys (Access and secret key), credentials (Credentials file), ec2_IAM_role (EC2 IAM role).

    assume_role_enabled

    Allows you to disable assume role (ARN) in the CloudWatch data source. By default, assume role (ARN) is enabled for OSS Grafana.

    list_metrics_page_limit

    When a custom namespace is specified in the query editor, the is used to populate the Metrics field and the Dimension fields. The API is paginated and returns up to 500 results per page. The CloudWatch data source also limits the number of pages to 500. However, you can change this limit using the list_metrics_page_limit variable in the grafana configuration file.

    It’s now possible to configure data sources using config files with Grafana’s provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page

    Using AWS SDK (default)

    1. apiVersion: 1
    2. datasources:
    3.   - name: CloudWatch
    4.     type: cloudwatch
    5.     jsonData:
    6.       authType: default
    7.       defaultRegion: eu-west-2

    Using credentials’ profile name (non-default)

    1. apiVersion: 1
    3. datasources:
    4.   - name: CloudWatch
    5.     type: cloudwatch
    6.     jsonData:
    7.       authType: credentials
    8.       defaultRegion: eu-west-2
    9.       customMetricsNamespaces: 'CWAgent,CustomNameSpace'
    10.       profile: secondary

    Using accessKey and secretKey

    1. apiVersion: 1
    2. datasources:
    3.   - name: CloudWatch
    4.     type: cloudwatch
    5.     jsonData:
    6.       authType: default
    7.       assumeRoleArn: arn:aws:iam::123456789012:root
    8.       defaultRegion: eu-west-2